COMPARATIVE COHORT ANALYSIS FOR THE STEM SCHOLARS PROGRAM

Final Report

9/18/15

SUBMITTED BY:

Dr. Keith Thompson, Associate Professor

Civil and Environmental Engineering Department

Dr. Michael Black, Assistant Professor

Math Department

Overview

This AAF project was funded to supplement summative analysis of the STEM Scholars Program within the College of EMS. The STEM Scholars Program was a five-year NSF sponsored S-STEM project which ended in Spring 2015. The program provided scholarships for 69 new freshman and transfers pursuing majors offered by the College of EMS between Fall 2010 and Spring 2015. NSF S-STEM projects also required participation in support programming to enhance the scholarship recipients’ academic and career skills and was specifically targeted to recruit underrepresented groups into STEM majors at UW-Platteville. As part of the final work for the grant, summative assessment of the program was conducted throughout the summer of 2015. Summative results of interest includeSTEM Scholars’ academic success and participation in practices known to foster retention and enhance career preparation.

Furthermore, there is strong general interest in methods of assessment that can be used to assess programs such as STEM Scholars because there are other similar programs that currently exist.Hence, a broader goal of this AAF project is to enhance the institutional capacity of the College of EMS Office of Student Success Programs for program assessment. The Director of EMS Student Success Programs, Tammy Salmon-Stephens (who is also a Co-PI of the STEM Scholars project) was involved in this study.

Assessment Methods

The key assessment method utilized in this project was a statistical comparison between the students in the STEM Scholars Program and similar cohorts of students within the College of EMS who did not participate in the STEM Scholars program. The purpose of this assessment was to determine the degree to which the STEM Scholars program enhanced academic performance of the participants in comparison to reasonably similar students who were not in the program. This has been an attempt to determine value-added by the program.

The Office of Institutional Research (now the Office of Institutional Effectiveness and Assessment) was contacted to compile data for comparison cohorts. To createcomparison cohorts, the following traits were desired:

  • Each comparison cohort would exclusively contain students enrolled in STEM majors (considered to be any major offered by the College of EMS for this study). This was also true of the STEM Scholars.
  • Each comparison cohort would contain students with entry level academic indicators matching those required to apply to the STEM Scholars program (HS Graduating Percentile  75% or ACT  22; HS GPA  3.0 could not be set as a criteria because data on HS GPA is not stored within the university’s student information system and was therefore inaccessible for this study).
  • Each comparison cohort would contain underrepresented/majority student ratio and a male/female ratio matching the ratios found in each cohort of STEM Scholars.
  • Comparison cohorts would be enrolled in EMS majors over the same time period of the STEM Scholars Program, so five cohorts were created in which the admit terms were Fall 2010 through Fall 2014 respectively (matching the five cohorts of Scholars recruited for the STEM Scholars Program).
  • Nobody in the comparison cohorts would have been a STEM Scholar at any time in their academic history.
  • Beyond the constraints listed above, each comparison cohort was to be sufficiently large for statistical analysis with as many students chosen randomly as possible from the available pool of EMS majors (the students least likely to be randomly chosen were underrepresented minorities, who were more likely to be the entire population of such students matching the criteria listed above).

Data requested for each comparison cohort included: EMPLID; New Freshman/Transfer status; gender; primary ethnicity group; first generation status; high school percentile; ACT Score; and, for each semester possible from their admit term to the most recently completed term(Spring 2015), records of each student’s academic program, units taken, units passed, term GPA, cumulative GPA, and academic standing at the end of each term. Similar data were also collected for the STEM Scholars. Because of the sensitive nature of this data and in accordance with the IRB approval for this project, no raw data or samples of analysis are provided as appendices to this report.

A secondary assessment method attempted under this AAF project was to compare participation in High Impact Practices (HIPs) between STEM Scholars and students from the comparison cohort. This data is not routinely measured at UW-Platteville, so the researchers could not simply ask the institutional researcher to gather this data. Originally, the researchers on this grant proposed to randomly survey students from each comparison cohort on their participation in internships, co-ops, undergraduate research, and other HIP’s in order to gather the data needed. However, in spring 2015 and mid-way through this project, the College of EMS implemented an exit survey on HIP participation to the all students graduating from the College of EMS in that spring. This survey will become an institutional practice within EMS each semester and it will ultimately be more efficient if these survey results become the benchmark for assessment of HIP participation rather than implementing a separate survey for each special program or other desired assessment need. Thus, the original plan to create a survey to collect comparison data was abandoned and the College of EMS survey results were used for comparison instead. This meant that the comparison data was not as good a match for statistical comparison, however, in terms of streamlining practice for institutionalization, the survey results from graduating seniors will be more cost-effective until the College or the University implements more rigorous data tracking of participation in HIPs.

Data provided by the College of EMS HIP participation survey included (but was not limited to) the number of times a graduating student had: completed a co-op; completed a summer internship; completed a semester research project; and completed a study abroad (international experience).

Analysis and Results – Academic Success

STEM Scholar and comparative cohort data was available for five admit termsand are summarized in Table 1. The values shown are the number of students in each cohort at the beginning of the admit term. The total number of STEM Scholars for which any data is available is 69. This number excludes two students who were offered scholarships, but left the university prior to receiving any money. In each successive semester, the number of students in a cohort shrinks as students voluntarily leave EMS majors for non-STEM majors; voluntarily leave UW-Platteville for other universities; are dismissed from their academic program or the university; are dismissed from the STEM Scholars program (for STEM Scholar cohorts only); or graduate from the university.

For analysis purposes, the cohort data was reorganized to reflect the student’s semester at the university (1st, 2nd, 3rd, 4th, etc.). Students admitted in Fall 2010 and Fall 2011 provided at least 8 semesters of data for analysis; students admitted in Fall 2012 provided 6 semesters of data; students admitted in Fall 2013 provided 4 semesters of data; and students admitted in Fall 2014 provided 2 semesters of data. Analysis on the basis of semesters spent in the College of EMS provided more reliable statistical analysis for the earlier semesters because it allowed all five STEM Scholar cohorts to be combined, increasing the amount of data in the statistical pool. However, the size of each semester-based data set diminishes over time due to natural sources of attrition and the limitations of the cohort data imposed by the admit term and the number of available semesters for analysis. Table 2 summarizes the number of students available for semester analysis in the College of EMS.

Using the data available for STEM Scholars and Comparison Students, the following items were compared: average credits taken each semester, average credits passed each semester, average term GPA, and average cumulative GPA. Figure 1 plots the averages for these four results over time. For the most part, particularly for earlier semesters, STEM Scholars out-perform the comparison cohorts for each measure, but these trends diminish over time. The diminishment may be influenced by the diminishing pool of data available for each semester, so this result should be reassessed at a later time when later semesters of data are available to the given cohorts.

Table 1 – Cohort Size Summary

Admit Term of Cohort / Initial Size of STEM Scholars Cohort / Initial Size of Comparison Cohort / Number of Semesters of Available Data / Semesters Contributed to in Analysis
Fall 2010 / 10 / 329 / 10 / 1st through 8th
Fall 2011 / 15 / 419 / 8 / 1st through 8th
Fall 2012 / 17 / 434 / 6 / 1st through 6th
Fall 2013 / 13 / 461 / 4 / 1st through 4th
Fall 2014 / 14 / 489 / 2 / 1st and 2nd

Table 2 – Available Data per Semester of Attendance at UW-Platteville

Semester of Comparison / Available STEM Scholar Data / Available Comparison Cohort Data
1st Semester / 69 / 2,132
2nd Semester / 68 / 1,979
3rd Semester / 65 / 1,797
4th Semester / 48 / 1,290
5th Semester / 46 / 1,219
6th Semester / 35 / 833
7th Semester / 32 / 776
8th Semester / 18 / 472
Analysis is unreliable beyond the 8thsemester due to diminishing size of the cumulative STEM Scholar group.

Figure 1 – Four academic indicators plotted over time for STEM Scholars and the comparison cohorts

The data plotted in Figure 1 were analyzed statistically. T-test analysis was performed to measure significance of the difference between the STEM Scholars and the comparison cohorts. P-Values (given as proportions) from this analysis are presented in Table 3. The minimum criteria for significance was set at the 5% level. For all values in which a significant difference was observed, the difference corresponded to a positive gap between the STEM Scholars and the comparison cohort (i.e. the STEM Scholars significantly out-performed the comparison cohort). The results from Figure 1 and Table 3 indicate the following:

  • STEM Scholars took on average 1 semester credit more than the comparison cohort students in three of their first four semesters in college.
  • STEM Scholars passed on average 1 to 1.5 more credits than comparison cohort students in all of their first four semesters in college.
  • STEM Scholars earned on average 0.3 to 0.45 more Term GPA points than comparison cohort students in all of their first four semesters of college.
  • STEM Scholars consistently sustained on average 0.3 more Cumulative GPA points than comparison cohort students in all of the eight semesters examined under this analysis.

Table 3 – Results (p-Values) of T-Tests of Difference between STEM Scholars and Comparison Cohorts

Semester / Credits Taken / Credits Passed / Term GPA / Cumulative GPA
1st / 0.0011** / 0.0001*** / 0.0000*** / 0.0000***
2nd / 0.0760 / 0.0126* / 0.0000*** / 0.0000***
3rd / 0.0038** / 0.0089* / 0.0178* / 0.0001***
4th / 0.0127* / 0.0129* / 0.0091* / 0.0001***
5th / 0.3779 / 0.1929 / 0.0648 / 0.0002***
6th / 0.4673 / 0.4527 / 0.0774 / 0.0002***
7th / 0.0692 / 0.0340* / 0.0114* / 0.0015**
8th / 0.4208 / 0.4494 / 0.4353 / 0.0026**
*
**
*** / Significant at a 5% level.
Significant at a 0.5% level.
Significant at a 0.005% level.

Another aspect that was examined was retention in EMS majors and retention in good standing at the university between the two groups. Retention in good standing indicates that students stayed at the university and maintained good academic standing (cumulative GPA  2.0 and term GPA  1.0). Within the group of students who do not retain in good standing are students who undergo academic dismissal or who voluntarily leave the university with or without good academic standing. Tables 4 and 5 summarize numerical results from these analyses and Figure 2 presents plots of the data. The data show that STEM Scholars retained in EMS majors and in academic good standing within the university at higher rates than students in the comparison cohorts for their first six semesters of college. For retention in good standing, the difference between STEM Scholars and the Comparison Cohort was 11% higher in the first three semesters and maintained at around 8-9% up through semester 7. Retention in STEM majors (those offered by the College of EMS) improved steadily until the 7th and 8th semesters. Sharp reversals in both sets of data in the 7th and 8th semesters may be a result of the limited data available for these semesters.

Table 4 – Retention in EMS Majors

Semester / STEM Scholars / Comparison Cohorts / Difference
N / n / n/N / N / n / n/N
1st / 69 / 69 / 1.000 / 2020 / 2020 / 1.000 / -
2nd / 69 / 61 / 0.884 / 2020 / 1744 / 0.863 / 0.021
3rd / 55 / 44 / 0.800 / 1533 / 1115 / 0.727 / 0.073
4th / 55 / 40 / 0.727 / 1533 / 996 / 0.650 / 0.078
5th / 42 / 28 / 0.667 / 1106 / 658 / 0.595 / 0.072
6th / 42 / 27 / 0.643 / 1106 / 619 / 0.560 / 0.083
7th / 25 / 13 / 0.520 / 699 / 369 / 0.528 / -0.008
8th / 25 / 12 / 0.480 / 699 / 353 / 0.505 / -0.025
N – Number of students of the given group listed in EMS in the 1st semester and who could have taken classes in the current semester. The number of students who could have taken courses in the current semester decreases due to the shrinking numbers of semesters for which data can be analyzed for students in cohorts for later admit years.
n – Number of students listed in the college of EMS in the current semester.
The “Difference” is the calculation of (n/N)STEM Scholars – (n/N)Comparison Cohorts.

Table 5 – Retention at UW-Platteville in Good Academic Standing

Semester / STEM Scholars / Comparison Cohorts / Difference
N / n / n/N / N / n / n/N
1st / 69 / 67 / 0.971 / 2132 / 1827 / 0.857 / 0.114
2nd / 69 / 66 / 0.957 / 2132 / 1756 / 0.824 / 0.133
3rd / 55 / 48 / 0.873 / 1643 / 1244 / 0.757 / 0.116
4th / 55 / 45 / 0.818 / 1643 / 1186 / 0.722 / 0.096
5th / 42 / 32 / 0.762 / 1182 / 794 / 0.672 / 0.090
6th / 42 / 30 / 0.714 / 1182 / 750 / 0.635 / 0.080
7th / 25 / 18 / 0.720 / 748 / 449 / 0.600 / 0.120
8th / 25 / 14 / 0.560 / 748 / 443 / 0.592 / -0.032
N – Number of students of the given group in good standing at the start of the 1st semester and who could have taken classes in the current semester. The number of students who could have taken courses in the current semester decreases due to the shrinking numbers of semesters for which data can be analyzed for students in cohorts for later admit years.
n – Number of students listed in good standing at the end of the current semester.
The “Difference” is the calculation of (n/N)STEM Scholars – (n/N)Comparison Cohorts.

Figure 2 – Retention in EMS and in good standing by semester for STEM Scholars and the comparison cohorts

Further analysis of STEM Scholar retention examined why students left the program. This analysis could be performed for the STEM Scholars due to the researchers’ direct knowledge of that pool, but could not be performed for the comparison cohort as reasons for leaving are not tracked in the student information system. Table 6 summarizes why students left the STEM Scholars Program. Scholars that withdrew from the university typically did so to transfer to another university. Scholars who left to change major, change universities, or who become dismissed did so predominantly within their first 4 semesters. Thus for the 23 Scholars who are still in the program with 4, 6, or 8 semesters completed at the university, it is likely that they will successfully graduate with a STEM major and maintain the academic requirements (Cumulative GPA  3.0) required from the program until that point.

Table 6 – Reasons STEM Scholars Left the Program

Semester at which Status Applies / Status
Graduated / Changed Major to non-STEM within Univ. / Voluntary Withdrawal from Univ. / Dismissed from STEM Scholars / Still in the STEM Scholars Program
1st / 0 / 1 / 3 / 0 / 0
2nd / 0 / 4 / 2 / 7 / 12d
3rd / 0 / 0 / 0 / 1 / 0
4th / 2 / 2 / 0 / 2 / 9c
5th / 0 / 1 / 0 / 1 / 0
6th / 2 / 0 / 0 / 0 / 10b
7th / 1 / 0 / 0 / 0 / 0
8th / 2 / 0 / 0 / 0 / 4a
9th / 1 / 0 / 0 / 0 / 0
10th / 2 / 0 / 0 / 0 / 0
All Students / 10 / 8 / 5 / 11 / 35
a –2011 Admit Term; b – 2012 Admit Term; c – 2013 Admit Term; d - 2014 Admit Term

Specific Results by Race/Ethnicity, Gender, and First Generation Status:

Value Added - Race/Ethnicity

To examine value added specifically for non-whiteSTEM Scholars, the data available from the STEM Scholars includes only 16 students (23% of the total) spread out over all five admit term cohorts. Because the pool of data is small to begin with and shrinks for each successive semester, results beyond the 1st and 2nd semester should be treated cautiously. The data that is available indicates a gap in value-added between white and non-white students. Within the first two semesters, the STEM Scholars Program correlated with improvements in credits taken, credits passed, term GPA, and cumulative GPA for both whites and non-whites; however, these improvements were less for non-white students. Data plotted in Figure 3 illustrates this. The data plotted for white STEM Scholars comes from comparison with only white comparison cohort students, while the data plotted for non-white STEM Scholars comes from comparison with only non-white comparison cohort students. Each groups of STEM Scholars shows improvement over their respective comparison group, but the improvements were less for non-whites.

Figure 3 – Gaps between white and non-white STEM Scholars in value-added by the STEM Scholars Program for four academic indicators

Value Added - Gender

To examine value added specifically for female STEM Scholars, the data available from the STEM Scholars includes 31 students (45% of the total). This is better than the data available based on race and ethnic differences, but should still be treated cautiously beyond the 4th semester (where the number of available data for female STEM Scholars begins to dip below 20 individuals). As with race and ethnicity, academic indicators for female STEM Scholars were compared only against females in the comparison cohort and data for male STEM Scholars were compared only against males. The resulting differences are plotted in Figure 4. The plots show that female STEM Scholars took on higher credit loads each semester for the first four semesters than their female peers who were not in the STEM Scholars, whereas the male STEM Scholars tended to take less than their peers. Aside from this result, the remaining indicators show inconsistent results, perhaps suggesting that there was no significant difference in the value added by the STEM Scholars Program between males and females.