Quantifying the Robustness of RD Inferences

The following is from

Saw, G., Schneider, B., Frank, K., Chen, I. C., Keesler, V., & Martineau, J. (2017). The Impact of Being Labeled as a Persistently Lowest Achieving School: Regression Discontinuity Evidence on Consequential School Labeling.American Journal of Education,123(4), 585-613.

Quantifying the Robustness of RD Inferences

To inform policy debates and theoretical interpretations of the causal effects of the PLA list, it is useful to quantify the discourse about the robustness of the inferences in this study. We quantify how much bias there must be in our RD estimates to invalidate inferences in terms of replacement data,16focusing only on the positive PLA list effects on the average of students’ scale scores in writing and the percentage of students who met proficiency level in social studies. As shown intable 5, to invalidate our causal inference of the PLA list effects on the average of students’ scale scores in writing, we would need to replace about 25% to 32% of our PLA schools with school samples for which there is no effect of being on the list. These 17 to 22 replacement schools could represent populations not directly in our sample, such as schools from outside of the selected bandwidth. Additionally, to invalidate the inference of an effect of assignment to the PLA list on social studies achievement, we would have to replace 6% to 8.6% of schools with schools in which there was no effect of being on the PLA list.

Table 5.Quantifying the Robustness of Inferences from RD Impact Estimates of the 2010 PLA List

This analysis helps us to quantify the robustness of the inference with respect to internal validity by considering the replacement schools to come from different bandwidths. The same analysis can apply to external validity by considering the replacement schools to come from a different state or time. Thus, the analysis tells us how much we would have to change our sample to change our inference. In summary, based on this quantification of possible bias to invalidate an inference, we find that our RD estimates of the PLA list on the average of students’ scale scores in writing are particularly robust, but they are less so for the percentage of students who met proficiency level in social studies.

16.As defined by Frank et al. (2013), “The proportion of bias to make inference invalid = 100% × (estimate – [SE ×tcritical,df])/estimate.”

Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013.What would it take to Change an Inference?:Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences.Education, Evaluation and Policy Analysis.Vol 35:437-460.