1
Adolescent Connectedness Online Supplement
Running head: A multicultural assessment of adolescent CONNECTEDNESS
A Multicultural Assessment of Adolescent Connectedness:
Testing MeasurementInvariance across Gender and Ethnicity
Online supplement
Statistical Analysis Procedures
Reliability Analyses
Reliability analyses were conducted for all students and each subgroup separately. Internal consistency coefficients revealedgood to excellent (i.e., α’s ranging from .70 to .90) internal consistency using the entire sample, with gender-specific and ethnic subgroup reliabilities varying slightly between the groups. For the subscale analyses by each subgroup, internal consistency coefficients ranged from .61 to .94, with a mean of .79 (SD = .07). In general, the lowest internal consistency coefficients were on the Peers subscale for those groups labeled as Other (), Latina/o (), and African American (). The only other internal consistency coefficients below .70 were the Self-in-the-Present () and Self-in-the-Future () subscales for the African American samples. All other internal consistency coefficients were greater than .70 across the other subscales and subsamples. A table of internal consistency coefficients for each subsample may be obtained from the author.
One reason for the smaller internal consistency coefficients for certain groups was the reduction in total score variance (for more details see Feidt, Woodruff, & Sailh, 1987). In addition, these subscales also displayed lower reliability due to the lack of measurement invariance within these groups. As indicated in the invariance resultsbelow, several items produced different factor loadings and intercepts across, with both these components directly linked to the subscale’s reliability.
Tests of Latent Mean Differences across Gender and Ethnicity
In this supplementwe provide the gender mean differences across the ten factors, along with the associated effect size (Cohen’s d) for both the full and partial invariance models. Effect sizes are based on standards suggested by Cohen (1988), which are as follows: small (|d| = 0.20), medium (|d| = 0.50), and large (|d| = 0.80). Absolute z-values[1] greater than 3.30 and p < .001were considered statistically significant for these analyses using a Bonferroni adjustment (05/40 = .001). Recall that the mean difference (MDiff) always favors the reference group (Byrne, 1989). Therefore, the latent mean difference of .16 between boys and girls on the connectedness to Neighborhood factor indicates the mean score was .16 units higher for boys than for girls.
Whether the full or partial invariance model was used for latent mean comparisons (see Table 7) often did not influence the results. More specifically, the Cohen’s d did not change by more than .10 (i.e., |Δd| < .10) between the groups. The exceptions, discussed below, were between boys and girls on Connectedness to Friends and Self-in-the-Present and between Caucasians and African Americans on Connectedness to Friends and Self-in-the-Future.
Of the ten comparisons, only three latent mean differences were not statistically different between boys and girls using the full invariance model: Connectedness to Parents, Self-in-the-Present, and Self-in-the-Future. On all but the Connectedness to Neighborhood factor, girls scored significantly higher than boys. Most differences were for practical purposes relatively small based on Cohen’s (1988) standards. Only the difference on the Connectedness to Friends and Reading subscales, favoring girls, reflected a medium effect size.
When using the partial invariance model, however, the effect size between boys and girls on the Connectedness to Friends factor dropped noticeably (dfull = -.53, dpartial = -.26), although still statistically significant,MDiff–partial = -0.28, z = -7.70, p < .001. This finding might be expected given that three items were not invariant and the two largest Δχ2 values (see Table 4) were present within this factor. Effect size differences also emerged on the Self-in-the-Present factor between boys and girls. The effect size increased from -0.04 to -0.18 in the partial invariance model (Δd = .14), with girls scoring significant higher than boys,MDiff-partial = -0.19, z = -5.46, p < .001. Although there were three non-invariant intercepts (v33, v43, & v53), only the difference on v53 was large from a practical standpoint (Diff = -.57, see Table 4). On this item, girls had a larger intercept than boys. Thus, in this way, the partial invariance model, a mean difference emerged with boys lower on Connectedness to Self-in-the-Present.
Analyses revealed that Caucasians and African Americans differed on 6 of 10 factor means, and Caucasians and Latino/as differed on 5 of 10 factor means (see Table 7), both using the full invariance model. Caucasians scored higher than African Americans on latent means for Connectedness to Neighborhood, Friends, and Teachers, while African Americans scored higher than Caucasians on the Self-in-the-Present, Self-in-the-Future, and Sibling connectedness factor. However, the differences between Caucasians and African Americans on the Friends and Self-in-the-Future factors were not found with the partial invariance models (z = -0.38 p > .05; z = -1.22, p > .05, respectively). This makes sense given the larger number of non-invariant Connectedness to Friends (ni = 4) and Self-in-the-Future (ni = 5) items. Curiously, Connectedness to Peers had 4 non-invariant item intercepts but full and partial invariance model results were identical.
Caucasians were higher than Latino/as on Connectedness to Neighborhood, Friends, Self-in-the-Present, Self-in-the-Future, and Reading factors, but Latino/as were higher on the Sibling connectedness. African Americans and Latino/as differed on the Self-in-the-Present, Self-in-the-Future, and Reading factors, with African Americans scoring higher on all three factors.
Practically speaking (based on Cohen’s d), differences between the three ethnic groups were relatively small. The largest effect sizes were between African Americans and Latino/as on the Self-in-the-Present and Self-in-the-Future factors with effect sizes of .39 and .44, respectively. It is worth noting that the lower number of statistically significant group differences between African Americans and Latino/as is partially due to the smaller sample sizes (i.e., n = 2932 & 2950 vs. 764) in the minority group comparisons. For example, the effect size of .09 and mean difference of -.14 on Teacher connectedness between Caucasians and African Americans was statistically significant due to the large sample size (n = 2932), but this same mean difference was not statistically significant for the comparison between African Americans and Latino/as, nor was the same effect size on the Sibling connectedness (n = 764) significant. It should be noted that these differences are not likely associated with school characteristics, as none of the intraclass correlations (ICCs) for any of the outcomes exceeded .04.
In summary, then, latent mean differences did not differ greatly whether the full or partial invariance model was employed. While there were only a few subscales demonstrating evidence of partial rather than invariance, the latent mean difference results from these invariance models render the use of the Connectedness to Friends and the Connectedness to Self subscales suspect. In most cases, relaxing the constraints on non-invariant items to create a partial invariance factor model resulted in smaller or non-significant mean differences. For example, the effect size magnitude between boys and girls on the Connectedness to Friends factor decreased by about half when the partial invariance model was used. Similarly, significant mean differences between Caucasians and African Americans on Connectedness to Friends and Self-in-the-Future factors were rendered non-significant when non-invariant items were unconstrained. This suggests that when an assumption of scale invariance is made, but in fact scale invariance is not present, it is possible that group differences will be reported that may not truly exist or vice versa. With the possible erroneous interpretation of such results, a mischaracterization of group differences may be promulgated by the field when in fact the differences are considerably smaller or non-existent. Of course, the opposite could also occur. Based on these results, the implication of using non-invariant measures in cross-cultural studies is obviously problematic.
Yet there also was evidence that group differences may not appear unless the items on which they actually differ are allowed to be freely estimated through factor analytic score estimation. This means that, for example, when raw score subscale means on the Connectedness to Self-in-the-Present are computed as item averages, a partial invariance model is not being employed, and true differences between groups may not be revealed. Most unfortunate for the field is that where non-invariance occurs, this serves as a harbinger for researchers, signaling them to further explore the meaning of a given construct and the reasons for between-group variability. But, where invariance has not been tested, no such signal will be heard.
Given that an assumption of partial invariance, rather than full invariance, is theoretically more appropriate, it is important for scale users to consider early on how to deal with items found to be invariant for the groups under investigation. Research (Cheung & Rensvold, 1999; Millsap & Kwok, 2004) indicates that several procedures can be utilized if the factor model is not invariant: 1) delete the non-invariant items, 2) use all the items assuming that differences are small in the population and will not adversely influence the mean differences, 3) avoid using the scale all together or use them but interpret the latent variable scores independently (avoiding group comparisons), and/or 4) use the partial invariance model.
Sibling vs. Singleton Comparison
As indicated above, singletons were omitted from the above analyses due to missing data on the Connectedness to Siblings factor. To test whether the factor structure differs between subjects with and without siblings on the nine other connectedness factors, invariance tests were conducted between these two groups with the omission of the connectedness to Sibling factor. (The sample size precluded tests of gender and ethnic/racial differences across sibling statuses.)
Results revealed a good model fit for the sibling and singleton samples, which together provided a good fitting configural model. Unlike the invariance models comparing gender and race/ethnicity, the Δχ2 was not statistically significant when testing for factor loading and intercept invariance. The regression equations for computing the latent variable scores were nearly identical between the groups and latent mean comparisons are appropriate. A slightly larger Δχ2 (p < .001) occurred when testing for item residual invariance, although the ∆RMSEA, ∆SRMR, and ∆CFI remained in the acceptable range. Testing for variance/covariance invariance also revealed a statistically significant Δχ2 (p = .01), but again the ∆RMSEA, ∆SRMR, and ∆CFI were very small. Analysis of the latent variable means did not reveal any statistically significant differences between the groups at α = .05, with all the effect sizes being very small (|d| < .05).
1
Adolescent Connectedness
Online Supplement: Table 1Latent Variable Differences between Gender and Ethnic/Racial Groups for only Subjects with Siblings.
Neighborhood / Friends / Self-in-
the-Present / Parents / Siblings / School / Peers / Teachers / Self-in-the-Future / Reading
Boys vs. Girls
/ 0.16 / -0.46 / -0.04 / 0.05 / -0.13 / -0.27 / -0.14 / -0.32 / -0.05 / -0.57
/ 3.93 / -16.01 / -1.25 / 1.53 / -3.44 / -10.00 / -5.28 / -9.34 / -1.45 / -12.81
dfull / 0.13* / -0.53** / -0.04 / 0.05 / -0.11** / -0.33** / -0.18** / -0.31** / -0.05 / -0.43**
dpartial / 0.12* / -0.26** / -0.18* / 0.07 / -0.11** / -0.29** / -0.22** / -0.29** / 0.01 / -0.43**
Caucasian vs. African American
/ 0.30 / 0.24 / -0.24 / -0.07 / -0.34 / 0.00 / 0.03 / 0.14* / -0.35 / -0.02
/ 4.34 / 4.77 / -5.01 / -1.28 / -5.79 / 0.04 / 0.63 / 2.48 / -6.23 / -0.26
dfull / 0.16** / 0.18** / -0.18** / -0.05 / -0.21** / 0.00 / 0.02 / 0.09* / -0.23** / -0.01
dpartial / 0.13** / -0.01 / -0.12* / -0.06 / -0.23** / -0.04 / -0.05 / 0.07* / -0.05 / -0.01
Caucasian vs. Latina/o
/ 0.45** / 0.23 / 0.10 / -0.08 / -0.25 / 0.04 / -0.06 / 0.02 / 0.11 / 0.24
/ 7.05 / 4.41 / 2.15 / -1.81 / -4.54 / 1.00 / -1.40 / 0.34 / 2.07 / 3.69
dfull / 0.26 / 0.16** / 0.08* / -0.07 / -0.17** / 0.04 / -0.05 / 0.01 / 0.08* / 0.14**
dpartial / 0.27 / 0.18** / 0.13** / -0.06 / -0.17** / -0.04 / -0.07 / 0.02 / 0.08* / 0.11*
African American vs. Latina/o
/ 0.15 / -0.05 / 0.33 / -0.01 / 0.10 / 0.07 / -0.11 / -0.14 / 0.46 / 0.27
/ 1.80 / -0.74 / 5.36 / -0.10 / 1.25 / 1.23 / -1.74 / -1.81 / 6.11 / 2.96
dfull / 0.13 / -0.05 / 0.39** / -0.01 / 0.09 / 0.09 / -0.13 / -0.13 / 0.44** / 0.21*
dpartial / 0.10 / -0.07 / 0.36** / -0.01 / 0.09 / 0.07 / -0.09 / -0.14 / 0.38** / 0.18*
Note. Effect sizes marked with an * and ** were statistically significant at α = .05 and α = .001, respectively, with bolded effect sizes having Δd> |.10|. Mean differences () and z-statistics correspond to the full invariance model, with groups marked with an “R” acting as the reference group.
1
Online Supplement: Table 2
Connectedness Observed Score subscale Means and Standard Deviations By For Each Gender and Ethnic Group
Neighborhood / Friends / Self-in-the-Present / Parents / Siblings / School / Peers / Teachers / Self-in-the-Future / ReadingGirls vs. Boys
Girls / 3.20(1.02) / 3.80(0.65) / 3.95(0.76) / 3.91(.77) / 3.61(1.03) / 3.60(.75) / 3.55(.71) / 3.85(.84) / 3.93(.76) / 3.22(1.24)
Boys / 3.32(1.00) / 3.40(0.71) / 3.92(0.76) / 3.95(.74) / 3.50(1.03) / 3.34(.80) / 3.43(.70) / 3.57(.88) / 3.88(.81) / 2.68(1.18)
African American
All / 3.07(1.00) / 3.44(.78) / 4.09(.70) / 3.93(.76) / 3.78(0.95) / 3.47(.76) / 3.42(.73) / 3.56(.87) / 4.10(.74) / 2.96(1.14)
Girls / 2.85(1.00) / 3.53(.75) / 4.05(.72) / 3.91(.74) / 3.75(0.99) / 3.60(.72) / 3.41(.77) / 3.66(.87) / 4.08(.72) / 3.19(1.20)
Boys / 3.26(0.96) / 3.36(.80) / 4.12(.68) / 3.95(.78) / 3.78(0.91) / 3.36(.78) / 3.42(.70) / 3.47(.86) / 4.13(.76) / 2.77(1.04)
Caucasian
All / 3.34(1.01) / 3.63(.69) / 3.93(.77) / 3.93(.77) / 3.50(1.04) / 3.47(.80) / 3.49(.71) / 3.73(.89) / 3.90(.79) / 2.97(1.28)
Girls / 3.31(1.01) / 3.87(.60) / 3.96(.76) / 3.91(.78) / 3.58(1.03) / 3.62(.75) / 3.58(.70) / 3.87(.85) / 3.94(.76) / 3.26(1.26)
Boys / 3.36(1.01) / 3.41(.69) / 3.89(.78) / 3.93(.75) / 3.43(1.04) / 3.34(.82) / 3.41(.71) / 3.60(.91) / 3.86(.82) / 2.70(1.23)
Latina/o
All / 2.95(.96) / 3.48(.77) / 3.82(.70) / 3.96(.64) / 3.73(.91) / 3.43(.71) / 3.52(.63) / 3.70(.77) / 3.79(.75) / 2.74(1.06)
Girls / 2.80(.95) / 3.60(.77) / 3.75(.74) / 3.88(.70) / 3.79(.98) / 3.50(.72) / 3.48(.69) / 3.88(.76) / 3.75(.74) / 3.04(1.08)
Boys / 3.08(.95) / 3.36(.76) / 3.89(.66) / 4.03(.58) / 3.68(.85) / 3.36(.69) / 3.56(.58) / 3.53(.75) / 3.82(.75) / 2.48(.98)
Note. These are the observed means, not the latent means compared in Table 4. These are provided to give information about the variability of observed subscale scores.
Online Supplement: Table 3
Model Fit Statistics across Sibling Status
/ / / / RMSEA / ∆RMSE / SRMR / ∆SRMR / CFI / ∆CFIFit for Siblings / 11321.38 / 1188 / 0.053 / 0.050 / 0.961
Fit for Singleton / 2526.55 / 1188 / 0.064 / 0.072 / 0.942
Configural / 13847.93 / 2376 / 0.054 / 0.072 / 0.959
Factor Loadings / 13895.58 / 2418 / 47.65 / 42 / 0.053 / 0.000 / 0.073 / 0.000 / 0.959 / 0.000
Item Intercepts / 13944.98 / 2469 / 49.39 / 51 / 0.053 / -0.001 / 0.073 / 0.000 / 0.959 / 0.000
Item Residuals / 14100.60 / 2520 / 155.62 / 51 / 0.052 / -0.001 / 0.071 / -0.001 / 0.959 / 0.000
Variance/Covariance / 14170.36 / 2565 / 69.76 / 45 / 0.052 / 0.000 / 0.079 / 0.008 / 0.959 / 0.000
Note. Sample sizes for siblings, no siblings, and combined were 3633, 294, and 3927, respectively. All values were statistically significant at α = .001. for the item residuals and variance/covariance matrix were statistically significant at α = .001 and α = .01, respectively.
[1] This study compared latent variable means following the procedure of Byrne (1998, pp. 303-325). Therefore, the limitation of using the Δχ2 to statistically compare latent means with complex models was not encountered (see Fan & Sivo, 2009).