1
Slide 1
Summary
Almost all range restriction in studies is indirect (Thorndike, 1949), yet the usual correction for indirect range restriction (Thorndike’s Case III) cannot be used because the needed information on the third variable is not available. Because of this, the correction for direct range restriction is usually made (Thorndike’s Case II) as a default correction, both in individual studies and in meta-analysis methods.
In this paper we present a method for correcting for indirect range restriction that does not require information on third variables. This procedure has been shown via computer simulation to be much more accurate than use of the formula intended for direct range restriction.
We apply this method to the GATB validity generalization data base and show the use of the correction for direct range restriction results in these data in undercorrections of between 20% and 35%. Similar findings were found when data from McDaniel et al’s meta-analysis (McDaniel, Whetzel, Schmidt, & Maurer, 1994) on the validity of employment interviews were re-analyzed. There is every reason to believe that this will also be true for other published meta-analysis that have corrected for range restriction using the Case II correction formula, whether these are validity generalization studies or other types of meta-analyses. That is, the implication is that meta-analytic correlations in the literature are substantial underestimates. This has important substantive implications of both a practical and a theoretical nature.
Time will not permit the full derivation of this new range restriction correction procedure at this symposium, but the resulting equation will be presented and discussed. Use of this method requires that corrections for measurement error be made first, followed by the correction for range restriction. The reasons for this will be explained and this sequence of corrections will be contrasted with the required sequence for direct range restriction. A full derivation of the procedure is presented in Hunter, Schmidt, and Le (2005), which is available from the authors on request. The computer simulation calibrating the accuracy of the new method is presented in Le (2003).
Slide 2
In many research situations such as educational and employment selection researchers have data only from a restricted population and yet must attempt to estimate parameters of the unrestricted population. For example, the validity of the Graduate Record Examination (GRE) for predicting performance in graduate school can only be estimated using samples of students admitted to the graduate program (the restricted sample). However, the goal is to estimate the validity of the GRE when used in a different population: the population of applicants to the graduate program. The population of admitted students usually differs systematically from the population of applicants. Typically, the admitted students have higher mean GRE scores and a smaller standard deviation (SD) of scores; i.e., there is range restriction on GRE scores. To estimate the validity in the applicant population from the observed validity in the “incumbent” population of admitted students, one must correct for the effects of range restriction on GRE scores. In such a situation, if applicants have been selected directly on test scores top down, we have what is called direct or explicit range restriction. On the other hand, if students have been selected on some other variable that is correlated with GRE scores (such as a composite of undergraduate grade point and letters of recommendation), then the range restriction is said to be indirect.
Slide 3
Range restriction can generally be classified into two basic types: univariate range restriction which includes direct range restriction and indirect range restriction, and multivariate range restriction.
Direct Range Restriction
Explicit selection on X results in distortion of the correlation between X and Y.
Indirect Range Restriction
Explicit selection on the third variable Z results in distortion of the correlation between X and Y (which are correlated with Z).
MultivariateRange Restriction
Explicit selection can simultaneously occur on more than one variable in a sample. There can also be multiple incidental selection variables whose inter-correlations are affected by the selections.
In practice, it can be seen that multivariate range restriction occurs only when selection is made on a composite or sum of variables. An example of the situation is where the military services administer a battery of 10 tests to potential recruits and select people based on the sum of four of these tests (direct range restriction on this composite of four tests); this process produces indirect range restriction on the other 6 tests (e.g., Held & Foley, 1994). As shown in Ree et al., (1994) and Sackett and Yang (2000), to use the correction procedure for multivariate range restriction, one must know the intercorrelations of the variables in both the restricted and unrestricted populations and the selection ratio for each explicit selection variable. This information is mostly available in the military testing context (Held & Foley, 1994) and in predictive validation studies. Most researchers outside the military testing field, however, have rarely used the multivariate range restriction correction procedure in their studies.
We do not consider multivariate range restriction in our study.
Slide 4
Thorndike’s Case II, by far the most popular one, is appropriate under the condition of direct range restriction (i.e., selection directly occurs on the predictor variable). Using Thorndike’s Case II formula to correct for the effect of range restriction is relatively simple because it requires only knowledge of (a) the degree of range restriction in the predictor, and (b) correlation between the variables in the restricted sample. Unfortunately, effects of range restriction in personnel selection and education are mostly indirect (Thorndike, 1949), which means selection often occurs on a third variable (variable Z) that is correlated with both the two variables of interest (X and Y). Thorndike’s Case III is specifically reserved for this situation. The formula for Thorndike’s Case III shows that the correlation between variables X and Y in the unrestricted population can be estimated from (a) observed inter-correlations between the three variables (X, Y, and Z) in the restricted population, and (b) the degree of range restriction on the third variable Z where direct selection occurs. In practice, however, the third variable Z often cannot be measured because the selection process may occur on variables which were not quantified (e.g., recommendation letters, unquantified subjective judgments, or self-selection - Gross & McGanney, 1987; Linn, 1968; Schmidt, 2002). Consequently, Thorndike’s Case III cannot be applied in the majority of studies.
Slide 5
Correction for Range Restriction in Current Meta-Analysis Methods
The issue of indirect range restriction correction poses a serious problem to meta-analysis. While indirect range restriction is prevalent, meta-analysts often have little control over their data, so it is practically impossible for them to apply the Thorndike’s Case III formula. The popular practice has been to apply the correction method for direct range restriction (i.e., using the formula for Thorndike’s Case II). This practice implicitly assumes that effects of direct range restriction and indirect range restriction are similar. This assumption has been proven wrong; using direct range restriction formula to correct for indirect range restriction can lead to substantial underestimating the correlations of interest (Linn, Harnisch, & Dunbar, 1981). This problem has been noted long ago (Schmidt & Hunter, 1977), but its implications have generally been discounted until recently (Hunter et al., 2002). The fact that all simulation studies have consistently shown that the current meta-analysis approaches yield accurate estimates of and SDρ may have led researchers think that the inappropriate use of range restriction correction is inconsequential. These studies, however, are misleading because they simulated datasets under direct range restriction condition, which automatically ensures the accuracy of approaches using the Thorndike’s Case II correction formula (Schmidt and Hunter, 2001). When these approaches are applied to real data which are likely affected by indirect range restriction, they can yield inaccurate results that may affect substantive research conclusions.
Slide 6
The New Model of Indirect Range Restriction (Hunter & Schmidt, 2004, Hunter, Schmidt, & Le, 2005)
Hunter et al. (2004) suggested a model that elaborates on the combined effects of range restriction and measurement error on the observed correlation in the restricted population. This model is illustrated in the slide. T and P are the true scores (constructs) underlying the variables X (predictor) and Y (criterion). Selection on the third variable Z (often unknown/unmeasurable) creates incidental range restriction on T, which in turn leads to range restriction on P. Here it can be seen that the difference between direct and indirect restriction is that unlike indirect selection, direct selection operates on test scores. Thus errors of measurement are part of the selection process in direct selection. That is, when people are selected on test scores, they are selected partly on their true scores and partly on their measurement errors. For indirect selection, scores on the test of interest are never considered and so errors of measurement in those test scores have no effect on the selection process. The impact of indirect selection is on the predictor true score T; there is no effect on the errors of measurement in the observed scores.
Slide 7
Assumption. The model for indirect range restriction suggested by Hunter et al. (2005) relies on an assumption about the structural relationship between Z, T, and P. Specifically, it is assumed that the effect of the third variable where explicit selection occurs (Z) on the dependent variable P (criterion construct) is fully mediated by the predictor construct T. Arguably, the assumption can be met in personnel selection situations where a new selection procedure (X as a measure of T) is comprehensive in a sense that it captures the constructs that determine the criterion-related validity (i.e. correlation with P) of the third variable on which direct selection has occurred earlier because of either employers’ selection practices or applicants’ self-selection. For example, a structured interview is likely to capture the constructs of cognitive ability, job knowledge, and personality (Schmidt & Rader, 1999; Moscoso, 2000).
Correction Procedure. As is evident in the Figure, range restriction on T is caused directly by range restriction on Z. Although this value (range restriction on Z) is often not known, Hunter et al. (2005) showed that correction for indirect range restriction only require knowledge of range restriction on T. They derived the formula enabling estimating range restriction on T from range restriction on X, which is often observable. The Thorndike’s case II formula is then used with uT (instead of uX) to correct for the effect of indirect range restriction. (See Handout 1 for details of the formulas involved and correction procedure).
Slide 8
Correction procedure (cont.) As evident from the procedure discussed above, the major difference between the new meta-analytic approach and existing approaches is that the former uses range restriction in the true score (uT) instead of range restriction in the observed score (uX) to correct for the effect of range restriction. Since uT is always smaller than uX (equality occurs only when the predictor measure is perfectly reliable), the new approach always yields larger estimates for than those provided by the existing approaches. Based on this finding, Hunter et al. (2004) pointed out that the existing meta-analysis methods almost always underestimate the mean true correlation.
Estimating the standard deviation of the true correlation (SDρ). When information about the statistical artifacts is available in all primary studies, the standard deviation of the true correlation (SDρ) can be estimated following the procedure for the individual correction for statistical artifacts method detailed in Hunter and Schmidt (1990; 2004). When such information is unavailable, a more common situation, direct estimation of the standard deviation is difficult. Hunter et al. (2005) suggested two alternative approaches based on equation (5) above: (a) using the modified nonlinear interactive approach (cf. Law, Schmidt, & Hunter, 1994); and (b) using Taylor series approximation approach (cf. Raju, Burke, Normand, & Langlois, 1991). Details of the methods are available in Hunter and Schmidt (2004) and Hunter et al. (2005).
Slide 9
Simulation Study
Purposes. The new meta-analysis method is evidently more complicated than all the existing methods. The basic attenuation formula of the method is complex and nonlinear, which renders estimating the standard deviation of true correlation (SDρ) difficult. Another complicating important factor is that Ux and Rxx needed to estimate UT are not independent of one another. Since both the nonlinear interactive and Taylor’s series approximation approaches suggested to estimate SDρ assume these elements to be independent, their accuracy may be affected when the assumption is violated. In this study, we simulated data to examine the accuracy of the new meta-analysis method and compared the results with those provided from an existing method based on direct range restriction correction.
Slide 10
Simulation parameters.
Range restriction: The uT distribution underlying the well-established generalized uX distribution used in previous meta-analyses (e.g. Gaugler, Rosenthal, Thornton, & Bentson, 1987; Pearlman, Schmidt, & Hunter, 1980; Vinchur, Schippmann, Switzer, & Roth, 1998 ) and simulations (Callender, Osburn, Greener, and Ashworth 1982; Raju & Burke, 1983; Burke, Raju, & Pearlman, 1986; Law, Schmidt, & Hunter, 1994).
Reliabilities: the Rxx and Ryy distributions used in previous meta-analyses and simulations.
Sample: Primary study sample size varies randomly from 75 to 150 ; Number of studies randomly varies from 30 to 100.
True-score distributions: Total 9 conditions: 3 under fixed-effect model(SD= .00 with M= .30; M=.50; M=.70); 3 under random- effect model with moderate variations (SD= .055 with M= .30; M=.50; M=.70); and 3 under random-effect model with larger variations (SD= .110, SD= .148; SD=.184 with M=.50) (cf. Law et al., 1994).
500 data sets were simulated for each condition.
Analysis . For each data set: (1) Revised interactive approach, (2) Taylor-series based approach, and (3) Interactive approach based on direct range restriction correction (Law et al., 1994)
Slide 11
Results. Handout 2 presents the results. Under the condition of fixed effect (i.e., SD = .00, conditions 1, 2, 3), all the approaches overestimated the standard deviation SD. This result is expected because when the true standard deviation is actually zero, the estimated standard deviation will be negative 50% of the times due to sampling error, but all the approaches set the estimated variance (and consequently the standard deviation) zero under such situation. Thus, when averaged across simulations, the results become positively biased. Accordingly, the accuracy of the approaches under this condition can only be examined by the magnitudes of their estimated standard deviations. As can be seen in Handout 2, in average, the two new approaches overestimated true score standard deviation SD from .048 to .087 (Var =.002- .008).
To have a concrete idea of the extent of the overestimation, it is helpful to compare the current results to those in the Law et al’s study (1994) which showed the performance of the existing meta-analysis methods based on direct range restriction when their assumption of direct range restrictionis met (Law et al., 1994). In that study, the existing methods overestimated the variance of true score correlation from .003 to .010 (mean SD = .057 - .100) when it was actually zero (Law et al., 1994; Table 4, p. 983). Thus, it appears that the new approaches provide relatively accurate results.
When the true score correlation varies slightly (SD=.055, conditions 4,5, and 6), both new approaches overestimated the true score standard deviation. This pattern of overestimation is also expected because of the same problem discussed earlier. That is, the approaches automatically set the estimated variances to be zero when the calculated values are negative due to sampling error. While the proportions of overestimation appear to be very high under certain conditions, the absolute magnitudes are not.
Results of the current analyses can again be compared to those in Law et al. (1994) to provide more concrete evaluations of the performance of the new meta-analysis approaches vis-à-vis those of the existing meta-analysis methods when their respective assumptions are met. In the Law et al. (1994) study, the interactive nonlinear mean r method, the best overall method, overestimated the true variance Var by .002 (Law et al, 1994; Table 6, p. 984), which is equivalent to the overestimation of .015 in SD. As can be seen in Handout 2 the average overestimations of the interactive and the Taylor’s series approaches are .023 and .003, respectively. Thus, it is evident that the new approaches can be compared favorably with the currently best meta-analysis method (when their respective assumptions of range restriction are met).
Above comparison ignores the fact that range restriction is mostly indirect in practice. Under this realistic condition, the currently best meta-analysis method (i.e., interactive non-linear meta-analysis method based on direct range restriction correction) appears to seriously overestimate the standard deviation SD: the overestimations are averaged .070 across simulation conditions. Coupled with the underestimations of mean true score correlation discussed earlier, this finding shows that using existing meta-analyses based on direct range restriction would result in serious underestimations of the generalizability of relationships between variables of interest.
When variations of the true score correlations are greater (conditions 7,8, and 9), both new approaches generally underestimated the standard deviations of true score correlation SD. The underestimations, however, are generally small in absolute values. The method based on direct range restriction provides inconsistent estimates of SD across simulation conditions, ranging from overestimation (.033 when true SD=.110) to underestimation (-.012 when true SD=.184).
Slide 12