8
College Readiness vs College Admissions
College Readiness vs College Admissions: Will we ever resolve chasm between K-12 and Higher Education?
Paper prepared for the Conference on Defining Enrollment in the 21st Century at the
USC Center for Enrollment Research, Policy and Practice
August 2008
Wayne J. Camara
The College Board
The author gratefully acknowledges the assistance of Krista Mattern and the research on non-cognitivenoncognitive predictors by Neal Schmidt, Fred Oswald and their colleagues.
Despite the fact that only 14% of colleges and universities are considered selective (admitting less than 50% of applicants), the media, as well as many educators, parents and students, have increasingly focused on college admissions through this narrow and highly restricted lens. Students are increasingly applying to more colleges, utilizing early admissions and early decision, and focusing on building a resume that is attractive to selective institutions. However, college matriculation and graduation rates among graduating high school students have generally been stagnant for the past three decades and remediation rates have increased (Camara, In Press). Such facts underscore the disconnect between college readiness and college admissions.
Admissions decisions are heavily weighted around high school grades and test scores. However, academic success comprises only one aspect of what colleges consider relevant in assembling a class and admitting students. and whenOften, other factors (e.g., teacher recommendations, student essays, references) are used but (e.g., teacher grades, standardized test scores, teacher recommendations, student essays, references) they are employed in a ‘holistic’ or ‘judgmental manner’ that precludes any external empirical evaluation for reliability, validity or fairness. Academic success comprises only one aspect of what colleges consider relevant in assembling a class and admitting students. HoweverThat is, other predictors used in admissions are unstandardized and generally reviewed through more subjective processes.
Diversity is also a major concern related to college admissions. Research consistently shows that African American and Hispanic students score substantially lower, about one standard deviation, than White and Asian American students on cognitive tests and about 2/3 of a standard deviation lower on high school grades. These differences appear to hold across tests, grades and schools;, however, much smaller differences are found on personality and non-cognitivenoncognitive measures which have been shown to be valid predictors of performance in work settings (Guion, 1998). As the realization of a more diverse college applicant pool materializes, the need for expanded definitions of both predictors and criteria of college success will intensify.
The College Board has been pursuing a long term research agenda designed to identify factors related to college readiness and to evaluate potential new predictors of college success. This paper will provide a brief overview of both efforts.
College Readiness and College Success
First, we must develop a definition of college readiness which is distinct from college admissions. Following a review of the literature, some empirical studies of student success in college, and consultation with educational leaders in policy and research, it is apparent that college readiness must be grounded in an objective and measureable definition of college success. The most popular definitions of college success include one or more of the components below:
· Freshmen enrollment in college level courses with no remediation required
· Full-time enrollment in college
· Graduation from a 4-yr. institution in 6 years or less
· Attaining a specific GPA or course grades
Of course, there are interim measures that could be used in place of graduation (e.g., persistence, completion of a specific number of credits associated with graduation in 6 yrs, freshmen grades). Each of the above outcomes has its associated problems.
As we noted above, metrics of college success should be objective and measureable and the data should be relatively easy to obtain from most colleges without a great deal of effort. Course level data is notoriously difficult to work with across institutions. Understanding the classification and sequencing of courses in domains such as mathematics or English language arts is complex across institutions and higher educational systems. The College Board has obtained course level grades on freshmen from over 110 institutions and documenting which courses are considered remedial, determining comparability of courses (e.g., the first course in calculus for Engineers vs social scientists; differences in science courses across majors) and determining if students have obtained sufficient credits to be considered a junior are time consuming and require extraordinary effort (and a good deal of interpolation). Comparing grades across these difference different courses is difficult while FGPA is a more comparable and reliable measure within an institution. In addition, differences between semester, trimester, and quarter systems, as well as grading systems, does present challenges to aggregation across higher education institutions.
We have tentatively, determined that graduation from a 4-year institution in six-years is the appropriate metric of college success for the College Board, and that interim metrics such as freshmen grades and credits completed during freshmen year also have efficacy for tracking college readiness overtime. Even though academic factors (such as test scores and high school grades) are not the sole determinants of Graduation and persistence and graduation, are not solely determined by academic factors (such as test scores and high school grades) yet these area moderately strong relationship exists predictors of college success[1] (Burton and Ramist, 2001).
College readiness is a metric that provides students, educators and policymakers some indication of students’ the probable success in college – ideally, early enough to address any gaps prior to college matriculation. If college success is related to the probability of graduation and college readiness can be associated with freshmen grades and persistence, then we need effective predictors of readiness.
Some organizations have defined college readiness based solely on the strong relationship between test scores and college grades. This logic can also be defended based on the relationship between SAT scores and graduation rates. For example, Adelman (1999) found that noted that only 7% of students scoring in the bottom quintile attained a bachelor’s degree compared to 67% of those with scores in the top quintile. However, his studies (1999, 20043) also showed that predictive models that combined the academic rigor of high school curriculum, to test scores and grades provided efficacy in predicting college success. Therefore, the College Board is developing models of college readiness that employ multiple measures and multiple methods to determine college readiness. Specifically, a composite measure that reflects the academic rigor of the high school curriculum, combined with high school grades and test scores, provides the best estimate of college academic success. Much of the current work is devoted to how to optimally classify course rigor across schools and how to evaluate AP, IB, dual enrollment, honors and standard college prep courses.
What About Non-Academic Factors that Influence College Success?
Too often, college success is operationally defined as a high grade point average and there are many additional attributes of success that are noncognitive in nature. Therefore, the likelihood of any substantial change to the admissions process but also any success in improving the graduation rates across higher educational institutions is not only contingent on identifying valid and reliable predictors for college admissions but also in the expansion of what constitutes successful college performance[2].
College Board Research
Borrowing from multi-dimensional theories of job performance, researchers assisting the College Board reviewed the literature and mission statements of about 100 representative universities and college via their homepage and coding coded them for student characteristics that they colleges and univeristies purport to value (Schmitt, Oswald, & Gillespie, 2005; Oswald, Schmitt, Kim, Ramsay, & Gillespie, 2004). Subject matter experts then identified twelve dimensions that are valued by at most institutions.
Dimension / DefinitionIntellectual behaviors
Knowledge, learning, and mastery of / Gaining knowledge and mastering facts, ideas, and theories and how they interrelate, and
general principles (Knowledge) / understanding the relevant contexts in which knowledge is developed and applied. Grades or grade
point average can indicate, but not guarantee, success on this dimension.
Continuous learning, and intellectual interest / Being intellectually curious and interested in continuous learning. Actively seeking new ideas and new
and curiosity (Learning) / skills, both in core areas of study and in peripheral or novel areas.
Artistic cultural appreciation and curiosity / Appreciating art and culture, either at an expert level or simply at the level of one who is interested.
(Artistic)
Interpersonal behaviors
Multicultural tolerance and appreciation / Showing openness, tolerance, and interest in a diversity of individuals (e.g., by culture, ethnicity, or
(Multicultural) / gender). Actively participating in, contributing to, and influencing a multicultural environment.
Leadership (Leadership) / Demonstrating skills in a group, such as motivating others, coordinating groups and tasks, serving as a
representative for the group, or otherwise performing a managing role in a group.
Interpersonal skills (Interpersonal) / Communicating and dealing well with others, whether in informal social situations or more formal
school-related situations. Being aware of the social dynamics of a situation and responding
appropriately.
Social responsibility, citizenship, and / Being responsible to society and the community and demonstrating good citizenship. Being actively
involvement (Citizenship) / involved in the events in one’s surrounding community, which can be at the neighborhood, town/
city, state, national, or college/university level. Activities may include volunteer work for the
community, attending city council meetings, and voting.
Intrapersonal behaviors
Physical and psychological health (Health) / Possessing the physical and psychological health required to engage actively in a scholastic
environment. This would include participating in healthy behaviors, such as eating properly,
exercising regularly, and maintaining healthy personal and academic relations with others, as well as
avoiding unhealthy behaviors, such as alcohol/drug abuse, unprotected sex, and ineffective or
counterproductive coping behaviors.
Career orientation (Career) / Having a clear sense of career one aspires to enter into, which may happen before entry into college
or at any time while in college. Establishing, prioritizing, and following a set of general and specific
career-related goals.
Adaptability and life skills (Adaptability) / Adapting to a changing environment (at school or home), dealing well with gradual or sudden and
expected or unexpected changes. Being effective in planning one’s everyday activities and dealing
with novel problems and challenges in life.
Perseverance (Perseverance) / Committing oneself to goals and priorities set, regardless of the difficulties that stand in the way.
Goals range from long-term goals (e.g., graduating from college) to short-term goals (e.g., showing
up for class every day even when the class is not interesting).
Ethics and integrity (Ethics) / Having a well-developed set of values, and behaving in ways consistent with those values. In
everyday life, this probably means being honest, not cheating (on exams or in committed
relationships), and having respect for others.
Note. / Taken directly from Schmitt Oswald et al., 2004
After the taxonomy of college success was identified, items were written to assess those dimensions. Drawing on measures commonly employed in industrial organizational psychology research and practice, a situational judgment inventory (SJI) and a biographical data measure (i.e., biodata measure) were developed. Items for the situational judgment inventory present a typical college scenario and then ask respondents what they are mostly likely to do and least likely to do. As for the biographical data measures, the items are multiple choice questions assessing past experiences and interests. See Appendix A for sample items for both measures. Items were reviewed for fairness, sensitivity, relevance, and acceptability by current and former college admissions personnel, high school counselors, and College Board staff and consultants. The measures revealed acceptable psychometric quality with adequate reliability and convergent/divergent validity with other measures (for a more detailed description of the development of the two noncognitive measures, see Camara et al., Sathy, & Mattern, 20078 and Oswald et al., 2004).
Two studies were conducted that demonstrated that these particular new measures of college success were reliable, fair, and valid. While they added validity above and beyond admissions tests and high school GPA in predicting freshmen grades (increment of .02), they had greatest utility in predicting non- academic outcomes like time to degree, absenteeism and self- rated student performance. The second study involved 2,771 freshmen at 10 colleges and universities across the country (Schmitt, Oswald, Kim, Imus, Merritt, Friede, & Shivpuri, Schmitt, 2007). Tables 2-4 present the relationship between traditional admissions tools and different criteria of success. The incremental validity of adding in the noncognitive measures into the prediction is also displayed. The addition of the noncognitive measures resulted in significant incremental validity for all outcomes, albeit to a lesser degree for first year GPA. These findings provide preliminary evidence for the utility of adding noncognitve measures to the admissions process.
Outcome / ValidityFirst Year GPA
SAT/ACT and HS GPA / 0.70
Noncognitive Measures / 0.72
Incremental Validity / 0.02*
Self Rated Performance
SAT/ACT and HS GPA / 0.09
Noncognitive Measures / 0.50
Incremental Validity / 0.41*
Class Absenteeism
SAT/ACT and HS GPA / 0.22
Noncognitive Measures / 0.36
Incremental Validity / 0.14*
Unlike cognitive measures such as the SAT/ACT and high school GPA, the results revealed that there was small to no subgroup differences on the two noncognitive measures (See Table 5) (Oswald et al., 2004). (See Table 5) The implications of these findings are significant in terms of their potential to provide incremental validity as well as a their potential to increase the diversity of an their admitted class. Table 6 demonstrates the relative efficacy of admissions models that employ two different models on the admitted class under three different selection ratios (85% admitted, 50% admitted, and 15% admitted)[3]:
AB (equal weighting of tests and grades)
AB+ (equal weighting of tests, grades, biodata and SJI)
In sum, the results of these studies are highly promising on a number of levels and there is enthusiasm from the 158 institutions involved in the next phase of the work. The limitations of the research will be addressed in the next section.
There are significant challenges for this type of research. First, the biodata and SJI are most effective in predicting college success on non-academic measures (e.g., absenteeism, ratings, possibly persistence, engagement, leadership), which are not systematically collected by institutions. Despite the extensive claims from that higher education that they define success broadly, the only systematic and objective measures widely available from most colleges are restricted to the academic realm. These new predictors will have limited utility in predicting cognitive outcomes and universities have not developed cross-institutional outcome measures in non- academic domains (with the exception of persistence and graduation). In addition, introducing new measures for college admissions is a lengthy process that involves various constituencies, political concerns, and economic issues.