Factor Structure, Discriminant and Convergent Validity of TIMSS Math and Science Motivation

TIMSS MOTIVATION MEASURES

Factor Structure, Discriminant and Convergent Validity of TIMSS Math and Science Motivation Measures: A Comparison of Arab and Anglo-Saxon Countries.Supplemental Material for External Website

<Note: The external appendices are intended to appear only on a website associated with the journal that is hot-linked to the article, not the printed article.>

External Appendix 1

Table 1

Structure of Residual Variance/Covariance Matrix Representing Parallel Items (PI) and Negative Items (NI)

1 MVAL1 UNQ

2 MVAL2 0 UNQ

3 MVAL3 0 0 UNQ

4 MVAL4 0 0 0 UNQ

5 MSCP1 0 0 0 0 UNQ

6 MSCP2 0 0 0 0 0 UNQ

7 MSCN1 0 0 0 0 0 0 UNQ

8 MSCN2 0 0 0 0 0 0 NI1 UNQ

9 MAFFP1 0 0 0 0 0 0 0 0 UNQ

10 MAFFP2 0 0 0 0 0 0 0 0 0 UNQ

11 MAFFN1 0 0 0 0 0 0 NI NI 0 0 UNQ

12 SVAL1 PI 0 0 0 0 0 0 0 0 0 0 UNQ

13 SVAL2 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

14 SVAL3 0 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

15 SVAL4 0 0 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

16 SSCP1 0 0 0 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

17 SSCP2 0 0 0 0 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

18 SSCN1 0 0 0 0 0 0 PI/NI NI 0 0 NI 0 0 0 0 0 0 UNQ

19 SSCN2 0 0 0 0 0 0 NI PI/NI 0 0 NI 0 0 0 0 0 0 NI UNQ

20 SAFFP1 0 0 0 0 0 0 0 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

21 SAFFP2 0 0 0 0 0 0 0 0 0 PI 0 0 0 0 0 0 0 0 0 0 UNQ

22 SAFFN1 0 0 0 0 0 0 NI NI 0 0 PI/NI 0 0 0 0 0 0 NI NI 0 0 UNQ

23 MMORE 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0a

24 SMORE 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a

Note. UNQ=uniqueness, the residual variance term for each item. PI = correlated uniqueness associated with the same item used to assess parallel math and verbal construct. NI = correlated uniqueness associated with negatively-worded items (a negative item method effect)

aThese two items are used to define single-item factors and so the uniqueness term is fixed to zero and not estimated.

TIMSS MOTIVATION MEASURES

External Appendix 2

MPlus Syntax for Final Model 17 (see published article)

TITLE: Model 17 (Extended MIMIC Model);

DATA:FILE='D:\timss2007\data7jun\GRD8\TIMSS26May12Grd8imputation.txt'; TYPE=Imputation;

! Brings in the 5 multiple imputation files;

VARIABLE:

NAMES ARE

IDCNTRY IDSCHOOL IDCLASS IDGRADE IDSTUD ITSEX HmLang ITLANG

MALG MDAT MNUM MGEO SCHE SEAR SBIO SPHY MACHtotSACHtot MACH SACH

mscp1 mscn1 mscn2 mscp2 MaffP1 Maffn1 MaffP2

SSCp1 SSCn1 SSCn2 SSCp2 SaffP1 Saffn1 SaffP2

mval1 mval2 mval3 mval4 mmore sval1 sval2 sval3 sval4 smore

HOUWGT L2MACH L2SACH L2MACHmi L2SACHmi L2MACHmx L2SACHmx

FHIEdMHIEdEdAsphied AGE L2AGE TchAgeTchSEXTchYrs BSDMPATM;

! Variables in the files;

cluster is IDCLASS; !complex design cluster by class

IDVARIABLE IS IDSTUD;

USEVARIABLES ARE

MVal1 MVal2 MVal3 MVal4 mscp1 mscp2 mscn1 mscn2 Maffp1 Maffp2 Maffn1

SVal1 SVal2 SVal3 SVal4 SSCp1 SSCp2 SSCn1 SSCn2 Saffp1 SafFp2 Saffn1

MMoreSMore MACH SACH EdAspITSEX ;

! variables actually used in the analysis

CLUSTER is IDCLASS; ! Level 2 variable in Multilevel model;

weight is HOUWGT; !TIMSS weighting factor.

useobservations

IDCNTRY eq 840 or IDCNTRY eq 926 or IDCNTRY eq 36 or IDCNTRY eq 927 or

IDCNTRY eq 682 or IDCNTRY eq 400 or IDCNTRY eq 512 or IDCNTRY eq 818;

!select 8 countries used in the analysis

grouping is IDCNTRY

(840=usa 926=ENG 36=Aus 927=SCO 682=SAUD 400=JORD 512=Oman 818=EGYP);

!defines the grouping variable for multiple group analysis to be the 8 countries

define: edasp = (edasp - 3.77)/1.382;

define: ITSEX = (ITSEX - 1.500)/.500;

!standardize variables to facilitate interpretations ofunstandardized estimates

ANALYSIS: type=complex; ESTIMATOR=MLR;

!uses complex design option and robust ML estimator

model:

!This is the general model specification that applies to all multiple groups but can be qualified in relationto country specific models that follow

mval by MVal1-MVal4*(1-4);msc by mscp1-mscn2*(5-8);MAFF by MAFFp1-MAffn1*(9-11);

! The values in parentheses (1-4) refer to parameter estimates; Because these are the samefor the math and science factor loadings, they are estimated subject to the constraint that they are invariant over the math and science domains;

sval by SVal1-SVal4*(1-4);ssc by sscp1-sscn2*(5-8);SAFF by SAFFp1-SAffn1*(9-11);

!metric defined by fixing variances

mval-saff@1; MASP by MMore@1; MMore@0;SASP by SMore@1; SMore@0;

!Intercepts Invariant over Domain

!CUs for matching items

MVal1-MAffn1 pwith SVal1-SAffn1;

!CUs for negative item effect

mscn1 mscn2 MAffn1 sscn1 sscn2 SAffn1 with mscn1 mscn2 MAffn1 sscn1 sscn2 SAffn1;

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX

!estimate relations with new variables added in the MIMIC Model

modelusa :

!Country specific model for USA

! Factor Loadings: The values in parentheses [e.g. (A1-A4)] refer to parameter estimates; Because these are the same for the math and science factor loadings, they are estimated subject to the constraint that they are invariant over domain (within-group invariance). Because they are the same for each country, factor loading are estimated subject to the constraint that they are invariant over country (between-group invariance). Six of the motivation factors are based on multiple items, but two are single-item factors;

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-maff@1; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

! Item intercepts: Values in brackets are intercepts. The values in parentheses e.g. (A101-A108) refer to parameter estimates of these intercepts; Because these are the same for the math and science factor loadings, they are estimated subject to the constraint that they are invariant over domain (within-group invariance). Because they are the same for each country, factor loading are estimated subject to the constraint that they are invariant over country (between-group invariance). Six of the motivation factors are based on multiple items, but two are single-item factors;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (C109-C111); [sscn1,SSCN2,sval3] (C109-C111);[sval1,MVAL1*];

! In this model of partial invariance of intercepts, the constraint of invariance over country is relaxed for scn1, SCN2 and val3 and val1 items in both domains;

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

Modeleng :

!Country specific model for England (see discussion of country-specific parameters in the detailed description of USA Model

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (B109-B111); [sscn1,SSCN2,sval3] (B109-B111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

Modelsco :

! Country-specific model for Scotland

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (C109-C111); [sscn1,SSCN2,sval3] (C109-C111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

Modelaus :

! Country-specific model for Australia

mval by MVal1-MVal4*(A1-A4);msc by MScp1-MScn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (C109-C111); [sscn1,SSCN2,sval3] (C109-C111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

Modelsaud:

! Country-specific model for Saudi-Arabia

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-maff@0];[masp@0];[sval-saff*];[sasp*];masp-sasp*;MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (E109-E111); [sscn1,SSCN2,sval3] (E109-E111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

Modeljord:

! Country-specific model for Jordan

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (F109-F111); [sscn1,SSCN2,sval3] (F109-F111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

Modeloman:

! Country-specific model for Oman

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (G109-G111); [sscn1,SSCN2,sval3] (G109-G111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX

Modelegyp:

! Country-specific model for Eqypt

mval by MVal1-MVal4*(A1-A4);msc by mscp1-mscn2*(A5-A8);maff by maffp1-Maffn1*(A9-A11);

sval by SVal1-SVal4*(A1-A4);ssc by sscp1-sscn2*(A5-A8);

Saff by Saffp1-Saffn1*(A9-A11);

mval-saff* ; Masp by MMore@1; MMore@0;Sasp by SMore@1; SMore@0;

[MVal2,Mval4-mscp2,Maffp1-Maffn1,mmore] (A101-A108);

[sVal2,sval4-sscp2,Saffp1-Saffn1,smore] (A101-A108);

mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1 with mscn1 mscn2 Maffn1 sscn1 sscn2 Saffn1;

[mval-saff*]; [masp-sasp*]; masp-sasp*; MVal1-MAffn1 pwith SVal1-SAffn1;

[mscn1, MSCN2 ,mval3] (H109-H111); [sscn1,SSCN2,sval3] (H109-H111);[sval1,MVAL1*];

mval-sasp,Mach-ITSEX with mval-sasp,Mach-ITSEX;

OUTPUT: TECH1 TECH4 STDYX sampstatsvaluesMODINDICES(ALL) ;

! Note: Modification indices are not available with multiple imputation (and are simply ignored) but areavailable for tests of individual imputation data sets

TIMSS MOTIVATION MEASURES