Differential Prediction Generalization in College Admissions Testing

Supplemental Materials

Differential Prediction Generalization in College Admissions Testing

by H. Aguinis et al., 2016, Journal of Educational Psychology

Supplemental File A

Proof that Differential Prediction Generalization Results are Unaffected by Criterion Measurement Error

Suppose that the observed criterion includes measurement error. Note that indexes individuals and let index subgroups and subscript denoting different samples is dropped to simplify notation. Classical test theory (CTT) implies that where is the true criterion score for individual in subgroup and . It is standard in CTT to assume that for all and , which implies that measurement errors are systematically unbiased and subgroups may differ in the degree of criterion measurement error.The aforementioned CTT assumptions imply that and . The primary research interest for differential prediction studies is the regression of the true score on a set of predictors for subgroup ,

/ (A1)

where is dimensional vector of predictors, is a dimensional vector of regression coefficients for subgroup , and is a prediction error with variance .

The probability density function for the distribution of given and after marginalizing over the unobserved true score is,

/ (A2)

It is customary to assume criterion measurement errors and prediction errors are conditionally normally distributed as,

/ (A3)
/ (A4)

Substituting the models for and from Equations A3 and A4 into Equation A2 implies that the conditional density for is,

/ (A5)

where the last equality was obtained by noting the integral (which equals ) corresponds to a normal distribution with mean and variance,

/ (A6)

The aforementioned result implies that if is measured with error that .

The model parameters for the latent regression in Equation A1 can be estimated via marginal maximum likelihood. Let be a vector of dependent variables for subgroup and be a matrix of predictors. The likelihood for a sample of observations is,

/ (A7)

The log-likelihood for group is,

/ (A8)

Let . The partial derivatives of with respect to and are,

/ (A9)
/ (A10)

Accordingly, the maximum likelihood estimator (MLE) for the regression coefficients is,

/ (A11)

which is the same as the least squares solution that assumes that criterion measurement error is nonexistent. Similarly, setting the partial derivative in Equation A10 to zero yields the MLE estimator for ,

/ (A12)

The asymptotic variance-covariance matrix for is related to Fisher information,

/ (A13)

The estimated MLE of the variance-covariance matrix among the regression coefficients is:

/ (A14)

which is the estimator of discussed in the text.

The aforementioned results confirm that the MLE for group in the case of criterion measurement error is identical to the typical ordinary least squares solution. These derivations imply that subgroup differences in criterion measurement error variances do not alter the subgroup estimator for the regression coefficients. In fact, the groups could be jointly modeled. Let . The joint marginal likelihood function is,

/ (A15)

The MLE for and from the joint marginal likelihood yields the same estimators as in Equations A11 and A12.

Table S1
Range Restriction Corrected Meta-analytic Regression Coefficients and Standard Errors for Differential Prediction Analysis for Female-Male Comparisons (Model 2—see Table 4)
Focal Interactions
SAT Subtest / SAT Subtest
Predictor / HSGPA / CR / M / W / Focal / HSGPA / CR / M / W
Intercept / -0.0092 / 0.0002 / 0.0011 / 0.0007 / -0.0363 / -0.0031 / 0.0004 / -0.0002 / 0.0000
0.0791 / 0.0004 / 0.0004 / 0.0004 / 0.0393 / 0.0745 / 0.0004 / 0.0004 / 0.0005
1/N / -2.0681 / -0.0368 / 0.0390 / 0.0029 / -7.2676 / 4.2461 / 0.0056 / -0.0512 / -0.0125
4.7290 / 0.0298 / 0.0276 / 0.0302 / 2.7846 / 5.2787 / 0.0377 / 0.0320 / 0.0402
p / -0.1168 / 0.0011 / -0.0017 / 0.0001 / 0.0641 / 0.0161 / 0.0005 / 0.0014 / -0.0002
0.0792 / 0.0004 / 0.0004 / 0.0004 / 0.0417 / 0.0779 / 0.0005 / 0.0004 / 0.0005
Mean Differences
HSGPA / 0.2751 / -0.0007 / 0.0004 / 0.0006 / 0.2432 / 0.0206 / 0.0007 / 0.0001 / -0.0006
0.1281 / 0.0007 / 0.0007 / 0.0007 / 0.0685 / 0.1289 / 0.0009 / 0.0008 / 0.0009
SAT-CR / -0.0014 / 0.0000 / 0.0000 / 0.0000 / -0.0020 / 0.0003 / 0.0000 / 0.0000 / 0.0000
0.0011 / 0.0000 / 0.0000 / 0.0000 / 0.0006 / 0.0011 / 0.0000 / 0.0000 / 0.0000
SAT-M / -0.0021 / 0.0000 / 0.0000 / 0.0000 / -0.0012 / -0.0007 / 0.0000 / 0.0000 / 0.0000
0.0009 / 0.0000 / 0.0000 / 0.0000 / 0.0005 / 0.0009 / 0.0000 / 0.0000 / 0.0000
SAT-W / 0.0013 / 0.0000 / 0.0000 / 0.0000 / 0.0032 / 0.0006 / 0.0000 / 0.0000 / 0.0000
0.0011 / 0.0000 / 0.0000 / 0.0000 / 0.0006 / 0.0011 / 0.0000 / 0.0000 / 0.0000
Standard Deviations
HSGPA / 0.7999 / -0.0004 / -0.0023 / -0.0008 / 0.1930 / 0.0697 / -0.0013 / 0.0029 / 0.0010
0.1289 / 0.0007 / 0.0007 / 0.0007 / 0.0677 / 0.1253 / 0.0008 / 0.0007 / 0.0009
SAT-CR / 0.0040 / 0.0000 / 0.0000 / 0.0000 / 0.0025 / 0.0003 / 0.0000 / 0.0000 / 0.0000
0.0020 / 0.0000 / 0.0000 / 0.0000 / 0.0011 / 0.0020 / 0.0000 / 0.0000 / 0.0000
SAT-M / -0.0042 / 0.0000 / 0.0000 / 0.0000 / 0.0011 / -0.0011 / 0.0000 / 0.0000 / 0.0000
0.0017 / 0.0000 / 0.0000 / 0.0000 / 0.0009 / 0.0016 / 0.0000 / 0.0000 / 0.0000
SAT-W / -0.0004 / 0.0000 / 0.0000 / 0.0000 / -0.0039 / -0.0006 / 0.0000 / 0.0000 / 0.0000
0.0025 / 0.0000 / 0.0000 / 0.0000 / 0.0013 / 0.0024 / 0.0000 / 0.0000 / 0.0000
Note. Columns correspond to the institutional regression coefficients and rows indicate institutional predictors of regression coefficients. Standard errors of estimates are reported under coefficients. Some coefficients and standard errors round to 0.0000. The Focal group included Females, Blacks, and Hispanics. Criterion: first-year college grade-point average. Predictors: Model 1 predictors (see Table 4) as well as those shown in this table. 1/N: inverse of sample size, p:proportion of test takers in reference group, HSGPA: High-school grade-point average, SAT-CR: SAT Critical Reading, SAT-M: SAT Math, SAT-W: SAT Writing. Predictor mean differences were included in the model as the focal mean minus the reference mean.
TableS2
Range Restriction Corrected Meta-analytic Regression Coefficients and Standard Errors for Differential Prediction Analysis for Black-White Comparisons (Model 2—see Table 4)
Focal Interactions
SAT Subtest / SAT Subtest
Predictor / HSGPA / CR / M / W / Focal / HSGPA / CR / M / W
Intercept / -0.0152 / 0.0007 / 0.0011 / 0.0003 / -0.4544 / 0.3966 / 0.0004 / -0.0028 / -0.0023
0.0894 / 0.0004 / 0.0004 / 0.0004 / 0.1134 / 0.1691 / 0.0013 / 0.0010 / 0.0013
1/N / -18.5100 / -0.0013 / 0.0408 / -0.0392 / -9.5271 / 33.0554 / 0.0256 / -0.2465 / 0.0697
5.5440 / 0.0303 / 0.0288 / 0.0308 / 7.5811 / 11.0574 / 0.0895 / 0.0713 / 0.0963
p / 0.0448 / -0.0004 / 0.0001 / 0.0006 / 0.0304 / 0.1637 / 0.0013 / 0.0011 / -0.0011
0.0851 / 0.0005 / 0.0005 / 0.0005 / 0.0888 / 0.1260 / 0.0009 / 0.0007 / 0.0009
Mean Differences
HSGPA / 0.0729 / -0.0006 / -0.0001 / 0.0004 / 0.2310 / -0.0178 / 0.0012 / 0.0014 / 0.0024
0.0611 / 0.0003 / 0.0003 / 0.0003 / 0.0825 / 0.1197 / 0.0009 / 0.0007 / 0.0010
SAT-CR / -0.0011 / 0.0000 / 0.0000 / 0.0000 / 0.0019 / -0.0022 / 0.0000 / 0.0000 / 0.0000
0.0007 / 0.0000 / 0.0000 / 0.0000 / 0.0009 / 0.0014 / 0.0000 / 0.0000 / 0.0000
SAT-M / -0.0007 / 0.0000 / 0.0000 / 0.0000 / -0.0016 / 0.0024 / 0.0000 / 0.0000 / 0.0000
0.0006 / 0.0000 / 0.0000 / 0.0000 / 0.0008 / 0.0011 / 0.0000 / 0.0000 / 0.0000
SAT-W / 0.0020 / 0.0000 / 0.0000 / 0.0000 / 0.0003 / 0.0011 / 0.0000 / 0.0000 / 0.0000
0.0007 / 0.0000 / 0.0000 / 0.0000 / 0.0009 / 0.0014 / 0.0000 / 0.0000 / 0.0000
Standard Deviations
HSGPA / 1.1480 / -0.0015 / -0.0016 / 0.0004 / 0.0422 / -1.0455 / -0.0001 / 0.0039 / 0.0044
0.1312 / 0.0006 / 0.0006 / 0.0006 / 0.1741 / 0.2555 / 0.0020 / 0.0015 / 0.0021
SAT-CR / 0.0068 / 0.0000 / 0.0000 / 0.0000 / 0.0090 / -0.0109 / 0.0000 / 0.0001 / 0.0001
0.0023 / 0.0000 / 0.0000 / 0.0000 / 0.0031 / 0.0046 / 0.0000 / 0.0000 / 0.0000
SAT-M / -0.0043 / 0.0000 / 0.0000 / 0.0000 / -0.0081 / 0.0094 / 0.0000 / -0.0001 / 0.0000
0.0019 / 0.0000 / 0.0000 / 0.0000 / 0.0025 / 0.0037 / 0.0000 / 0.0000 / 0.0000
SAT-W / -0.0028 / 0.0000 / 0.0000 / 0.0000 / 0.0030 / 0.0022 / 0.0001 / 0.0000 / -0.0001
0.0026 / 0.0000 / 0.0000 / 0.0000 / 0.0033 / 0.0049 / 0.0000 / 0.0000 / 0.0000
Note.Columns correspond to the institutional regression coefficients and rows indicate institutional predictors of regression coefficients. Standard errors of estimates are reported under coefficients. Some coefficients and standard errors round to 0.0000.The Focal group included Females, Blacks, and Hispanics. Criterion: first-year college grade-point average. Predictors: Model 1 predictors (see Table 4) as well as those shown in this table. 1/N: inverse of sample size, p:proportion of test takers in reference group, HSGPA: High-school grade-point average, SAT-CR: SAT Critical Reading, SAT-M: SAT Math, SAT-W: SAT Writing. Predictor mean differences were included in the model as the focal mean minus the reference mean.
Table S3
Range Restriction Corrected Meta-analytic Regression Coefficients and Standard Errors for Differential Prediction Analysis for Hispanic-White Comparisons (Model 2—see Table 4)
Focal Interactions
SAT Subtest / SAT Subtest
Predictor / HSGPA / CR / M / W / Focal / HSGPA / CR / M / W
Intercept / 0.0133 / 0.0004 / 0.0010 / 0.0001 / -0.1077 / -0.1532 / -0.0002 / 0.0013 / -0.0001
0.0850 / 0.0004 / 0.0004 / 0.0004 / 0.0883 / 0.1520 / 0.0010 / 0.0010 / 0.0012
1/N / -17.2200 / 0.0023 / 0.0138 / -0.0151 / 9.9070 / 0.2479 / 0.0716 / -0.0252 / -0.0740
4.5920 / 0.0255 / 0.0239 / 0.0253 / 4.4090 / 7.1307 / 0.0516 / 0.0487 / 0.0566
p / -0.0204 / -0.0005 / 0.0010 / 0.0009 / -0.0381 / 0.2291 / 0.0005 / -0.0005 / 0.0005
0.0743 / 0.0004 / 0.0004 / 0.0004 / 0.0692 / 0.1131 / 0.0007 / 0.0007 / 0.0008
Mean Differences
HSGPA / -0.0016 / -0.0003 / -0.0005 / 0.0000 / 0.0670 / 0.0293 / -0.0002 / -0.0005 / 0.0010
0.0663 / 0.0003 / 0.0003 / 0.0003 / 0.0736 / 0.1239 / 0.0009 / 0.0009 / 0.0010
SAT-CR / -0.0011 / 0.0000 / 0.0000 / 0.0000 / 0.0000 / 0.0012 / 0.0000 / 0.0000 / 0.0000
0.0007 / 0.0000 / 0.0000 / 0.0000 / 0.0008 / 0.0014 / 0.0000 / 0.0000 / 0.0000
SAT-M / -0.0002 / 0.0000 / 0.0000 / 0.0000 / -0.0001 / 0.0012 / 0.0000 / 0.0000 / 0.0000
0.0005 / 0.0000 / 0.0000 / 0.0000 / 0.0006 / 0.0011 / 0.0000 / 0.0000 / 0.0000
SAT-W / 0.0016 / 0.0000 / 0.0000 / 0.0000 / 0.0017 / -0.0014 / 0.0000 / 0.0000 / 0.0000
0.0007 / 0.0000 / 0.0000 / 0.0000 / 0.0008 / 0.0013 / 0.0000 / 0.0000 / 0.0000
Standard Deviations
HSGPA / 1.1210 / -0.0015 / -0.0009 / 0.0010 / -0.0758 / -0.1063 / 0.0004 / 0.0009 / -0.0015
0.1168 / 0.0005 / 0.0005 / 0.0005 / 0.1268 / 0.2129 / 0.0014 / 0.0015 / 0.0017
SAT-CR / 0.0035 / 0.0000 / 0.0000 / 0.0000 / 0.0009 / -0.0059 / 0.0001 / 0.0000 / 0.0000
0.0024 / 0.0000 / 0.0000 / 0.0000 / 0.0026 / 0.0044 / 0.0000 / 0.0000 / 0.0000
SAT-M / -0.0042 / 0.0000 / 0.0000 / 0.0000 / 0.0015 / -0.0017 / 0.0000 / 0.0000 / 0.0000
0.0019 / 0.0000 / 0.0000 / 0.0000 / 0.0021 / 0.0035 / 0.0000 / 0.0000 / 0.0000
SAT-W / 0.0001 / 0.0000 / 0.0000 / 0.0000 / -0.0010 / 0.0099 / 0.0000 / -0.0001 / 0.0000
0.0026 / 0.0000 / 0.0000 / 0.0000 / 0.0027 / 0.0046 / 0.0000 / 0.0000 / 0.0000
Note. Columns correspond to the institutional regression coefficients and rows indicate institutional predictors of regression coefficients. Standard errors of estimates are reported under coefficients. Some coefficients and standard errors round to 0.0000. The Focal group included Females, Blacks, and Hispanics. Criterion: first-year college grade-point average. Predictors: Model 1 predictors (see Table 4) as well as those shown in this table. 1/N: inverse of sample size, p:proportion of test takers in reference group, HSGPA: High-school grade-point average, SAT-CR: SAT Critical Reading, SAT-M: SAT Math, SAT-W: SAT Writing. Predictor mean differences were included in the model as the focal mean minus the reference mean.

1