Long Beach City College Promise Pathways Evaluation Matrix
(as of September 20, 2012)
Overall program evaluation
Timeframe / Comparison groupsBaselines / Three most recent years available (F09-F11 for current metrics, earlier for longitudinal metrics)
1) General student population (students non-exempt from matriculation requirements)
2) First-time direct from High School
3) First-time direct from LBUSD
Entering Fall 2012
1) General student population (students non-exempt from matriculation requirements)
2) First-time direct from High School
3) First-time direct from LBUSD
4) Near-pathway students (completed agreement and/or workshop but not in program)
5) Pathways students
6) Comparisons of subgroups of Pathways students
a. Student success course (Counseling 1, Learn 11, none)
b. Achievement coaches (Yes, no)
c. Reading pilot (Paired, Standard, none)
Measurable Objective / Measure
1. Preparedness
1.1: Number of students assessed / First time students assessed in Math, English, and Reading
1.2: Basic Skills Load / Courses required to achieve full eligibility to take transfer-level course-work for all assessed students who matriculate
1.3: Transfer-eligibility / Students assessed as fully eligible for transfer level work/judged proficient at matriculation
1.3a: Math / Students assessed as eligible for transfer level math
1.3b: English / Students assessed as eligible for transfer level English
1.3c: Reading / Students assessed as proficient in reading
1.3d Overall / Students assessed as transfer-eligible or proficient in all three disciplines
1.4: Transfer-eligibility achievement / Students who complete basic skills work and become fully eligible for transfer level work/judged proficient at matriculation
1.4a: Math / Students who complete basic skills math requirements by end of 1st year
1.4b: English[i] / Students who complete basic skills English requirements by end of 1st year
1.4c: Reading / Student who completes basic skills math requirements by end of 1st year
1.4d Overall / Students who achieve transfer-eligibility or proficiency in all three disciplines
1.5a: Semesters of HS Math completed[ii] / Number of semesters of Math taken in HS
1.5b: Level of first HS Math course taken / Frequency distribution of point at which students begin taking Math in high school
1.5c: Level of highest HS Math completed ii / Frequency distribution of highest level of high school Math successfully completed (C or better grade at end of course)
1.5d: Prepared for transfer-level Math ii / Number of students successfully completing Algebra 2/Intermediate Algebra (preparatory course for transfer-level math) or higher
1.6: Completion of student success course/intervention by end of first year / Number of full-time, first-time degree-seeking (FTFTDS) students who complete a student success course or alternative student success intervention (e.g., summer bridge, intersession acceleration, etc.) sometime in first year
1.7: Student goal identification, development, and encouragement / First time students identifying AA/AS or 4-year degree as goal (but also tracking student identification of certificate goals)
1.7a: Student educational plan / First time students completing educational plan by end of first year
Measurable Objective / Measure
2. Progress Milestones
2.1: Units attempted / Total units attempted in first and subsequent terms
2.1a: Transfer level units / Total transferable units attempted in first and subsequent terms
2.2: Full-time student status[iii] / First-time, degree-seeking students who achieve full-time status (FTFTDS students)
2.3: Successful completion rate / Percentage of courses enrolled past census in which students have achieved an A, B, C, P or CR in the course in first and subsequent terms.
2.4: GPA / Grade point average achieved in first and subsequent terms
2.5: Reduction in probation / Percentage of FTFTDS students receiving probationary status in first and subsequent terms
2.5a: Academic probation / Percentage of FTFTDS students whose cumulative GPA drops below 2.0 after attempting 12 or more units.
2.5b: Progress probation / Percentage of FTFTDS students whose completion ration drops below 50%.
2.6: Persistence to second term / FTFTDS who persist to 2nd term
2.7: Achievement of concentration (Jenkins, 2011) / FTFTDS who complete 3 or more courses in field of study within 1 year.
2.8: Persistence to second year (third term) / FTFTDS who persist for two consecutive terms
2.9: Behavioral intent to transfer / FTFTDS who earn 12 transferable units and attempt transfer-level English or Math within 2 years.
2.10: Transfer-directed / FTFTDS who complete transfer-level English and Math within 3 years.
2.11: Completion of full year of college[iv] / FTFTDS who successfully complete 30 transfer-level units within 3 years
2.12: Transfer-preparedness / FTFTDS who complete 60 transferable units within 3 years.
Measurable Objective / Measure
3. Completion of meaningful educational outcomes
3.1: Certificate (18 or more units) / FTFTDS who complete a 18-unit or greater certificate within 3 years
3.2: AA/AS degree / FTFTDS students who complete an AA/AS or AAT/AST degree within 3 years
3.3: Transfer to 4 year institution of higher education / FTFTDS students who transfer to a four-year institution within 3 years
3.3a: Transfer with degree / FTFTDS students who transfer to a four-year institution within 3 years with AA/AS degree
3.3b: Transfer w/o degree / FTFTDS students who transfer to a four-year institution within 3 years without AA/AS degree
3.4: Completion of educational outcomes / Unduplicated number of students completing educational outcomes above.
3.4: Total number of educational outcomes / Total number of educational outcomes achieved (duplicated)
Program component evaluation
Component / Specifications / Outputs / Outcomes / Data Source /Agreement to Participate in Promise Pathways (prescriptive scheduling) / Students agree to responsibilities in exchange for program benefits / · Count of students who complete contract
· % of students who complete process of those eligible
· Percentage of students who persist (are retained in courses) of those who commit to program / · Students understand responsibilities and benefits of program
· Students commit to program requirements. / · First Semester Student Survey (FSSS): mid-semester survey targeting all LBUSD students currently enrolled and LBUSD students who started the process but did not enroll.
· Signed contracts (currently hard copy)
· Student records from LBUSD
· PeopleSoft P2 student group counts for drop and opt outs
Alternative Placement in English and math / English placement based on HS grade or Accuplacer Score (whichever is highest)
Math based on (a) HS perquisite to Math 130 which is A or B in 2nd semester of Algebra or (b) / · Count of students placed at each level in English/math / Sub-cohort comparisons: English – Accuplacer vs. HS grade; Math – Accuplacer, college prerequisite, comprehensive HS evaluation
· Course success and completion rates increase and attrition rates decrease
· Percent students who are “college-ready” in one year decreases
· Increase rate of completion of transfer English, math
· Students feel appropriately placed (prepared and sufficiently challenged)
· Faculty report students more appropriated placed in courses / · LBCC MIS files; LBCC Data Warehouse
· Faculty reported preparation of students collected by English department/Matriculation
· FSSS
Student Success Course / LEARN11 enrollment (1/2 with academic coach/1/2 without academic coach) and ePortfolio, without academic coach COUNS1 (with academic coach/without academic coach) / % students who utilize ePortfolio (levels of utilization: upload course assignments; write reflective summaries; upload extra-curricular work) / · Higher retention in courses to census
· Gain in fall-to-spring and fall-to-fall persistence
· Greater course success rate and higher overall GPA
· % completion of education plan by end of first year higher than other first-time students / · LBCC MIS; Data warehouse (compare P2 students with other first-time high school graduates; compare LEARN11 and COUNS1 student persistence, completion rates
· PeopleSoft educational planning module
· Student experience survey end of 1st semester (comparison groups as indicated for academic milestones above), end of first year (EOFY) and end of second year (EOSY).
Academic Coaches / ½ P2 students enrolled in Couns1 and ½ P2 students in LEARN11 to be assigned academic coach / # meetings with coaches each semester / · % of students on probation less than general incoming high school grad first-time students
· Fall to spring and fall to fall persistence rates; time to complete education goal (degree, transfer, certificate)
· Higher levels of reported self-confidence, improved social integration skills, enhanced sense of belonging and meaningful participant in community of student learners, increased knowledge of college resources for success; reported gains in behavioral follow-
through on steps towards educational/career goal achievement.
More self-reported interactions with peers and instructors (inside and outside classroom) / · Gains in student self-reporting (pre and post) academic coaching inventory
· Student experience survey at EOFS, EOFY, and EOSY.
ePortfolio / All P2 students in LEARN11 will be issued ePortfolio account; plus students in COUNS1 with academic coaches / # of students who complete assigned uploads to ePortfolio for pilot LEARN 11 sections. / · Improved ability to connect course and program learning with personal interests, aptitude and abilities with educational goals and career aspirations. / · Qualitative review of ePortfolios licensed to pilot group enrolled in LEARN11 sections (focus on reflective narratives assigned by LEARN11 faculty (on LASSI and LMT inventory) and on short and long-term goal development
Paired Reading / Based on Accuplacer scores, students placed at 1-4 levels below proficient. Students placed 1-3 levels below (READ 882, 883 and 82) divided into 3 groups: 1/3 paired with content course, 1/3 in Reading course only, 1/3 with no Reading (but taking some elective) / · Course success and retention rates in discipline courses will increase for students with Reading support (compare to students assessed at same level)
· GPA in transfer-level courses will be higher for students with reading support
· Students will report improved comprehension, summarization and argumentative writing skills.
· Faculty will observe improved course performance that requires mastery of college text / · Student items that address reading skill development (EOFS, EOFY, EOSY).
· Focus group interviews with students in paired reading courses, in general education electives w/o reading class and in non-paired reading class.
Student Support
· Counseling
· Orientation picnic
· Enrollment
· Educational Plan counseling
· Financial Aid
· Web-based services / Counseling workshop
Picnic and other student/parent orientations; mandatory completion of educational plan by end of first year / · Participation rates at workshops (% of Pathways students)
· Participation rates at orientations / · Count and percent completion of educational plans by end of first year.
· Student resource knowledge and service satisfaction with matriculation services, financial aid, admissions and records, web-based self-service tools / · PeopleSoft Student Administration reports
· Student survey at EOFS, EOFY, and EOSY
Engagement of faculty and staff / · Communication regarding Promise Pathways (website, presentations, emails)
· Implementation of pilot innovations
· Professional development activities / Participation rates of faculty and staff / · Faculty and staff will report high level of motivation and engagement as active participants in Promise Pathways components
· Faculty and staff will report transformations in thinking and behaving on the job and with their interactions with students and colleagues. / · Survey for faculty and staff EOFY
September 20, 2012 4
[i] Because no students are placed into English 801B, this will only be able to track students starting in English 105 in the Fall semester.
[ii] This will be tracked for Promise Pathways students only (they will be the only students for whom we have this data) to determine whether efforts with LBUSD to increase the number and level of math courses taken in high school are working.
[iii] Students with official DSPS status can achieve full-time status with less than 12 units. Key milestone that predicts persistence and educational outcome completion.
[iv] This is a pivotal educational milestone – it is first one that is associated with an increase in student’s lifetime earnings.