Early College High Schools at Scale:
Probing Impacts and Generalizability with a Quasi-Experiment Benchmarked Against an RCT
Douglas Lee Lauen, UNC Chapel Hill, Department of Public Policy
Fatih Unlu, Abt Associates, Inc.
Summary of Presentation Topic
This talk will present early findings from a within-study replication analysis of an educational intervention — validating a quasi-experimental model against an experimental estimate from the same sites. We will then use one or more QE models with the best internal validity to estimate impacts on a set of sites implementing the same intervention but with no lotteries and, hence, no experiments. Our talk will be about both within-study replication (internal validity) and generalizability (how to extrapolate from one sample to another). There is no paper yet, so we provide references below on related topics.
Project Summary
Purpose: This project will provide evidence about the effectiveness of Early College High Schools (ECHS) in North Carolina (NC). ECHS are small schools of choice primarily located on campuses of two- or four-year colleges or universities. Students who attend ECHS have the opportunity to earn, at no financial cost to them, two years of transferable college credit or an associate's degree while simultaneously satisfying high school graduation requirements. This study will leverage existing data to provide information about whether, how, and for whom NC ECHS are effective. Prior IES-funded research examined impacts from19 of the 70 ECHS sites in NC. These 19 sites used a lottery for admissions to ECHS. Lotteries approximate random assignment designs and allow for rigorous assessment of efficacy. This study builds on the prior study by examining additional mediators of effects and longer term outcomes. In addition, this study will use the data and results of the prior study to build complex statistical models that can replicate the RCT and be applied, retrospectively, to the other 51 NC ECHS to estimate their effects. A secondary aim of the project is to develop and refine this new statistical technique.
Project Activities: In order to complete this retrospective efficacy study, the researchers will obtain administrative data from all NC high schools, including ECHS, and match the data with longitudinal data from other local, state and federal databases to examine secondary, postsecondary, and other life outcomes (incarceration and voting) for the students. The sample will be limited to students who applied to ECHS. Those who were offered admission are considered to be the treatment group, and those who were not serve as the control group. Outcomes for students in the two groups will be compared.
Products: The products of this project will be evidence of the efficacy of ECHS for high school students in NC, a newly developed statistical technique, and peer-reviewed publications.
Structured Abstract
Setting: Data will be obtained from NC high schools (including ECHS), community colleges, universities, and public databases which include incarceration and voter registration information in North Carolina. In addition, data on students' postsecondary completion (within and outside NC) will be obtained from the National Student Clearinghouse.
Sample: The size of analytic samples will vary by outcome. By year two of the study, more than 40,000 students will have primary outcome measures such as achievement in high school, high school graduation, and postsecondary enrollment during and after high school. The 19 sites that participated in the lottery and the 51 sites that did not participate in the lottery have similar characteristics, though the lottery sites have slightly more African Americans and slightly more students eligible for free or reduced price lunch, while the non-RCT ECHS have slightly more Hispanic students and slightly higher 8th grade algebra passing rate. The traditional high schools in NC have similar characteristics to the ECHS, with the exception that there are more students with disabilities and fewer gifted students in traditional schools. All demographic differences between school types will be taken into account in the statistical models.
Intervention: The ECHS model includes a core set of design principles: college readiness, powerful teaching and learning, personalization, redefined professionalism, leadership, and purposeful design. ECHS are limited to 400 students. ECHS is operated through the auspices of North Carolina New Schools (NCNS).
Research Design and Methods: Using statewide administrative data and student-level information about lotteries conducted by ECHS, the best performing (lowest bias) propensity score models will be chosen from a within-study replication analysis that tests models using data from the 19 lottery sites as a benchmark. A variety of covariate sets and propensity score techniques will be tested separately for each outcome of interest. A mediational analysis will examine the mechanisms of the ECHS model design principles.
Comparison Condition: The comparison students are those attending a traditional public high school in the state of North Carolina.
Key Measures: Intermediate and ultimate outcomes include high school course taking, NC standardized end of grade test scores, and graduation; postsecondary applications, enrollment, course taking, GPA, and completion; incarceration; and voter registration. Pre-treatment covariates include levels and trends of middle school NC end of grade standardized test scores, absences and suspensions; parental education, poverty status, school enrollment changes, school-based educational classifications, and socio-demographic characteristics.
Data Analytic Strategy: The data analytic strategy is propensity score matching. The researchers will explore heterogeneity across sites, across student subgroups, and mechanisms that may mediate impacts.
Related references
Cole, S. R., & Stuart, E. A. (2010). Generalizing Evidence From Randomized Clinical Trials to Target Populations: The ACTG 320 Trial. American Journal of Epidemiology, 172(1), 107–115.
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750.
Edmunds, J. A., Bernstein, L., Unlu, F., Glennie, E., Willse, J., Smith, A., & Arshavsky, N. (2012). Expanding the Start of the College Pipeline: Ninth-Grade Findings From an Experimental Study of the Impact of the Early College High School Model. Journal of Research on Educational Effectiveness, 5(2), 136–159.
Edmunds, J. A., Willse, J., Arshavsky, N., & Dallas, A. (2013). Mandated Engagement: The Impact of Early College High Schools. Teachers College Record, 115(7), 31.
Kaizar, E. E. (2011). Estimating treatment effect via simple cross design synthesis. Statistics in Medicine, 30(25), 2986–3009.
Kern, H. L., Stuart, E. A., Hill, J., & Green, D. P. (2016). Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations. Journal of Research on Educational Effectiveness, 1–25.
LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 604–620.
Pressler, T. R., & Kaizar, E. E. (2013). The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Statistics in Medicine, 32(20), 3552–3568.
Steiner, P. M., Cook, T. D., Shadish, W. R., & H, M. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250–267.
Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386.
Tipton, E. (2012). Improving Generalizations From Experiments Using Propensity Score Subclassification: Assumptions, Properties, and Contexts. Journal of Educational and Behavioral Statistics.
Tipton, E. (2013). Stratified Sampling Using Cluster Analysis: A Sample Selection Strategy for Improved Generalizations From Experiments. Evaluation Review, 37(2), 109–139.
Tipton, E. (2014). How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations. Journal of Educational and Behavioral Statistics, 39(6), 478–501.
Tipton, E., Hedges, L., Vaden-Kiernan, M., Borman, G., Sullivan, K., & Caverly, S. (2014). Sample Selection in Randomized Experiments: A New Method Using Propensity Score Stratified Sampling. Journal of Research on Educational Effectiveness, 7(1), 114–135.