Nathan Lindsay & Dan Stroud

8-1-2012

ETS Proficiency Profile Test Report

The Education Testing Service Proficiency Profile (ETS-PP)is a standardized assessment that focuses on student learning outcomes for undergraduates as they enter major field course work. The instrument isdesigned toprovide evidence regarding the institution’s effectiveness in helping students achieve the General Education learning outcomes. All UMKC undergraduates are required to take the test before they graduate. During this past academic year, there werea total of 1,448 students who took the exam from August 2011 throughJuly 2012.

Two sets of analyses were run on the UMKC data from the ETS-PP: theCustom Comparative Data Report and the Item Analysis Report. The Custom Comparative Data Report indicates how UMKC’s students' skills and knowledge compare with the skills and knowledge of students at similar institutions. The report generates descriptive statistics (over a three year period) based on a reference group of 10 or more comparable institutions or programs of interest which were selected based on similar institutional characteristics. The anonymity of each institution's performance is maintained by reporting only the aggregate performance of the selected reference group.The number of students tested and sampling procedures vary from one institution to another. Therefore, it is impossible to verify that the students tested at each institution are representative of all the institution's students in that program.

Table 1: List of Benchmarking Institutions
School Name / Number
of
Students
Ball State University, IN / 307
Indiana State University / 477
Florida International University, FL / 2,479
Lamar University, TX / 672
Missouri State University, MO / 14,994
Southeast Missouri State University, MO / 11,226
University of Akron, The, OH / 515
University of Alabama at Birmingham, AL / 1,700
University of Cincinnati, OH / 321
University of South Florida – St. Petersburg, FL / 27
Total / 33,359

Table 1 outlines the ten institutions chosen for the comparative review of peer institutionsfor the ETS-PP. Among this group are several institutions with which UMKC regularly benchmarksits data. If there are other institutions of higher learning that would be beneficial to exchange for those in this comparison, the report can easily be adapted. However, please note that all of our regular peer institutions do not participate in this testing process, and thus some information may not be available to us for review.

The numbers presented in Table 2(below) include the mean scores, first for UMKC and then for the peer institutions listed above. These sub-scores and assessment indicators are both composite scores that are derived from the scores of many items. The numbers indicate, at least from this comparative sample of similar institutions, that our students are learning and ultimately performing ahead of the curve.

Table 2 - Custom Comparative Data Report
Skill Dimension / Proficient / Marginal / Not Proficient
UMKC / Comparative Data / UMKC / Comparative Data / UMKC / Comparative Data
Reading, Level 1 / 77% / 59% / 15% / 19% / 9% / 23%
Reading, Level 2 / 53% / 34% / 16% / 16% / 31% / 50%
Critical Thinking / 11% / 6% / 21% / 16% / 68% / 77%
Writing, Level 1 / 70% / 57% / 21% / 26% / 9% / 17%
Writing, Level 2 / 25% / 18% / 38% / 33% / 37% / 49%
Writing, Level 3 / 9% / 7% / 34% / 24% / 58% / 69%
Mathematics, Level 1 / 65% / 56% / 18% / 23% / 17% / 21%
Mathematics, Level 2 / 38% / 29% / 25% / 24% / 37% / 48%
Mathematics, Level 3 / 12% / 8% / 21% / 16% / 68% / 75%

The total test consists of 108 assessable items that are split over three abbreviated tests. As outlinedbelow in the Item Information Report, the items are prioritized first by skill area, and subsequently by proficiency level to make comparisons easier for grouping.The proficiency level in Table 3 refers to the difficulty level of the test questions. Scores compared in this report are referenced to the overall national percentage for each item.Items that are lower than the national average are highlighted.

In looking over the Table 3 data below, the overall performance at the institutional level is very positive. Reading and critical thinking were especially impressive, as every question and indicator tracked above the national average. The writing and mathematics areas proved nearly as strong, with only one indicator that fell below the national average in each area.

We recommend that these data be used to stimulate dialogue across campus about the curriculum and pedagogy surrounding these topics. Faculty should reflect on whether these scores are congruent with the skills and knowledge that students demonstrate in the classroom, as well as the skills that they struggle to demonstrate. Faculty might also ask whether they are teaching the “type of content” listed in the first column of Table 3. We plan to meet with faculty from the General Education Committee, the University Assessment Committee, and the Higher Learning Commission Assessment Academy team to discuss these scores. Based on these conversations, we encourage the development of an action plan to respond to the findings in this report.

Nathan Lindsay & Dan Stroud

8-1-2012

Table 3 - Item Information Report (n = 1,448)
Type of Content / Number of Questions / Skill Area / Average Proficiency Level / Percent Correct Institution / Percent Correct National
Discern facts from a passage / 11 / Reading / I / 75.57 / 68.74
Meaning in context / 2 / Reading / I / 77.65 / 68.45
Recognize a valid inference / 5 / Reading / II / 49.76 / 44
Discern purpose of a reference / 2 / Reading / II / 70.95 / 67.65
Synthesize material / 2 / Reading / II / 66.75 / 56.65
Discern main idea / 2 / Reading / II / 68.6 / 58.5
Discern primary purpose / 3 / Reading / II / 68.07 / 58.8
Recognize flaws in argument / 2 / Critical Thinking / III / 52.3 / 47.15
Extrapolate from known facts / 1 / Critical Thinking / III / 48.5 / 42.9
Recognize an assumption / 5 / Critical Thinking / III / 69.94 / 62.02
Evaluate hypotheses / 3 / Critical Thinking / III / 61.33 / 52.73
Determine relevance of information / 10 / Critical Thinking / III / 52.01 / 46.09
Evaluate an argument / 1 / Critical Thinking / III / 23.1 / 21.2
Evaluate data for consistency / 2 / Critical Thinking / III / 50.2 / 41.25
Evaluate explanations / 1 / Critical Thinking / III / 57.3 / 47.8
Recognize salient features / 2 / Critical Thinking / III / 67 / 63.75
Recognize agreement / 6 / Writing / I, II^ / 66.6 / 64.32
Order sentences / 2 / Writing / I / 71.8 / 64.2
Recognize appropriate transitions / 1 / Writing / I / 72.3 / 74.3
Recognize incorrect punctuation / 1 / Writing / I / 91.5 / 87.2
Recognize incorrect construction / 1 / Writing / III / 47.7 / 46.2
Incorrect word choice / 1 / Writing / I / 61.5 / 54
Recast an existing sentence / 5 / Writing / II / 80.98 / 77.74
Combine simple clauses / 1 / Writing / II / 67.3 / 62.2
Recognize correct construction / 7 / Writing / III / 56.79 / 51.4
Incorporate new material in paragraph / 1 / Writing / II / 67.8 / 55.5
Recognize most effective revision / 1 / Writing / III / 51.9 / 52
Arithmetic word problem - profit / 1 / Mathematics / I / 77.9 / 77.3
Algebraic expression / 1 / Mathematics / I / 75.6 / 72.1
Word problem - algebraic expression / 1 / Mathematics / III / 11.5 / 7.6
Arithmetic word problem / 1 / Mathematics / I / 82.5 / 73.9
Arithmetic word problem - rates / 1 / Mathematics / I / 83.3 / 77.2
Arithmetic word problem - work units / 1 / Mathematics / I / 94.2 / 93.8
Geometric word problem - lengths / 1 / Mathematics / I / 76.7 / 68.4
Data Interpretation - percents / 1 / Mathematics / I / 53.1 / 47.2
Translation to algebraic expression / 2 / Mathematics / II / 69.65 / 62.85
Simplify algebraic expression / 1 / Mathematics / II / 85.5 / 81
Arithmetic word problem - embedded ratios / 1 / Mathematics / II / 59.4 / 53.8
Arithmetic word problem - complex wording / 1 / Mathematics / II / 80.8 / 76.4
Arithmetic word problem - weighted mean / 1 / Mathematics / II / 58.7 / 57.4
Draw conclusion from algebraic expression / 1 / Mathematics / II / 31 / 29.5
Solution is algebraic expression / 1 / Mathematics / III / 15 / 10.5
Percent change - ratio and proportion / 1 / Mathematics / III / 31.6 / 21.2
Data interpretation - percent increase / 1 / Mathematics / III / 51.4 / 36.5
Arithmetic word problem - ratios, approximation / 1 / Mathematics / II / 54.8 / 54.3
Exponential growth / 1 / Mathematics / III / 70.2 / 64.3
Data Interpretation - range / 1 / Mathematics / I / 70 / 66.7
Data Interpretation - compare two charts / 1 / Mathematics / I / 35 / 26.8
Data Interpretation - percent change / 1 / Mathematics / III / 42.7 / 41.4
Data Interpretation - rate of change / 1 / Mathematics / III / 78.3 / 73
Percent decrease / 1 / Mathematics / III / 64 / 50.6
Understanding of properties of integers / 1 / Mathematics / III / 60.6 / 54.9
Measurement conversion / 1 / Mathematics / III / 68.8 / 66.8
^ Questions split between levels I (4) and II (2)