1
Course Experience Questionnaire
Supplemental Materials
(to appear on the official APA website that is hot-linked to the published version of the article
Appendix 1
Preliminary Analyses: Factor Structure of CEQ Responses.
In the present investigation, we evaluate the ability of the global satisfaction rating and specific CEQ factors to differentiate between universities and courses within universities. Although the five a priori CEQ factors have been previously identified with exploratory factor analyses (EFAs) and confirmatory factor analyses (CFAs; GCCA, 2002), no previous research has explicitly compared the goodness of fit for parallel EFAs and CFAs based on the same data. However, Marsh et al. (2009; in press) suggests that the traditional CFA requirement that all indicators load on one and only one factor is often overly restrictive, resulting in a poor fit to the data and biased parameter estimates such as inflated factor correlations that detract from the discriminant validity of the results. They argued for the use of exploratory structural equation modelling (ESEM) that combines many of the advantages of EFA and CFA. They argued the ESEM solutions should be routinely compared with those based on CFA in terms of goodness of fit and parameter estimates – particularly factor correlations.
In preliminary analyses, we evaluated two different structures, each with five factors (based on the 23 specific CEQ items; see Appendix 1). Analyses were conducted with Mplus (version 5.2, MuthénMuthén, 2010). In each case we used the robust maximum likelihood estimator (that is robust in relation to non-normality and the use of Likert scales) and the Mplus “complex design” option that controls for the fact that responses by students are nested within universities and courses within universities. The analysis first was based on a CFA, but the fit of this model was only moderate (e.g., CFI = .910, TLI = .886). We then evaluated an ESEM solution based on the same data. Here the goodness of fit was substantially better (e.g., CFI = .965, TLI = .940). A comparison of the two solutions (Supplemental Table 1) shows that both analyses identified the five a priori factors. However, results of the ESEM showed that there are several moderate cross-loadings that were constrained to be zero in the CFA, thus accounting for the marginal fit of the CFA solution.
Importantly, the factor correlations based on the CFA solution are systematically larger than the corresponding factor correlations based on the ESEM solution (CFA: .053 to .708, Md r = .387; ESEM = -.040 to .425, Md r = .213). Thus, for example the largest correlation in both models is between the Good Teaching factor and Goals factor, but this correlation is .708 in the CFA and .425 in the ESEM. The sizes of factor correlations is potentially important in the present investigation as we seek to evaluate the discriminant validity of the five CEQ factors – and profiles of these factors – in relation to universities and courses within universities.
Based on these preliminary analyses, we concluded that the ESEM solution was better than the CFA solution – both in terms of its ability to fit the data and the sizes of the factor correlations. Consistent with this conclusion, for purposes of the present investigation, the five CEQ specific factors are represented as ESEM factor scores (produced by the Mplus statistical package). We also note that our results are similar to CFA results based on previous CEQ research (e.g., the CFA in the GCCA 2002 report), but previous research did not explicitly compare ESEM and CFA solutions in terms of goodness of fit or size of factor correlations. Although summaries of CEQ factors are typically based on simple scale scores that are not entirely consistent with factor scores based on either CFA or ESEM results, ESEM factor scores better reflect these preliminary factor analysis results.
We also conducted further supplemental analyses to evaluate the higher-order structure of CEQ responses. We found that with or without the inclusion of the overall rating item, there was one dominant factor. When the overall rating item was included, it had the highest factor loading (.87), followed by the Good Teaching factor (.71). These results are consistent with the primary focus of CEQ reports on the overall rating item, supplemented, perhaps, by the Teaching scale. These results also justify our focus on the overall rating item in the present investigation and explain why the results based on the overall rating are so similar to those based on the specific CEQ factors.
1
Course Experience Questionnaire
Supplemental Table 1
Confirmatory and Exploratory Factor Solutions for 23 CEQ Items
Confirmatory Factor Solution Exploratory Factor Solution Item Wording (paraprased)
TCH GOL WRK ASS SKL TCH GOL WRK ASS SKL
Factor Loadings
1 (15) .738 .479 .235 -.039 .086 .194 effort to understand my difficulties
2 (17 ) .723 .645 .136 -.009 .092 .003 helpful feedback on how I was going
3 (18 ) .706 .649 .076 .081 .072 .034 good at explaining things
4 (20) .768 .702 .100 .027 .090 .037 make their subjects interesting
5 (3 ) .719 .524 .192 .059 .063 .112 motivated me to do my best work
6 (7 ) .673 .497 .125 .027 .100 .147 time into commenting on my work
7 (1 ) .681 -.013 .681 .075 -.015 .059 know the standard of work expected
8 (13r) .734 .041 .646 .055 .004 .137 to discover what was expected of me in this course.
9 (24 ) .627 -.033 .585 .134 .243 -.070 clear what they expected from students.
10(6 ) .664 .247 .516 -.009 .004 .024 clear idea of where I was going and what was expected
11(14 ) .632 -.036 .018 .634 .060 .017 enough time to understand the things I had to learn.
12(21r) .515 .225 .123 .436 -.001 .123 pressure on me to do well in this course.
13(23r) .596 -.076 .020 .656 -.006 -.091 volume of meant it couldn’t all be comprehended.
14(4r ) .694 .030 .030 .623 .119 -.027 workload was too heavy.
15(12r) .606 -.048 -.071 .014 .648 .072 testing what I had memorised than what I had understood.
16(19r) .784 .081 .033 .024 .736 -.030 Too many questions just about facts.
17(8r ) .614 -.030 .045 .119 .578 .023 all you really needed was a good memory.
18(10 ) .711 -.024 .132 -.061 .084 .659 tackling unfamiliar problems.
19(11 ) .702 -.013 .098 -.069 .122 .647 improved my skills in written communication.
20(2 ) .410 .127 .042 -.012 -.108 .375 developed my problem-solving skills.
21(22 ) .687 .079 .069 .033 .014 .625 develop the ability to plan my own work.
22(5 ) .553 .122 .027 .074 .093 .458 sharpened my analytic skills.
23(9 ) .607 .076 .079 -.055 .109 .503 develop my ability to work as a team member.
Factor Correlations
TCH 1.000 1.000
GOL .708 1.000 .425 1.000
WRK .269 .397 1.000 .096 .213 1.000
ASS .406 .365 .376 1.000 .213 .220 .213 1.000
SKL .580 .550 .053 .341 1.000 .301 .328 -.040 .201 1.000
An exploratory and confirmatory factor analyses were conducted with Mplus based on responses to the 23 CEQ items designed to measure five a priori factors. Both analyses clearly identify the five factors, but the goodness of fit for the ESEM was better than for the CFA (CFI = .965 vs. .910; TLI = .940 vs. .896; RMSEA = .035 vs. .046; SRMR = .017 vs. 0.046)
1
Course Experience Questionnaire
Appendix 2
Profile Analysis. In the final set of analyses we evaluate whether there are unique profiles of CEQ factors that are able to differentiate between universities or departments. Adapting a traditional profile analysis model (Tabachnik & Fidell, 2001) like that used in SET research (e.g., Marsh & Bailey, 1993), we separate this differentiation into level and shape components. Level differences refer to the extent to which some universities and departments receive consistently high or consistently low ratings across the five CEQ factors. However, based on the separate analyses of each of the five CEQ factors and of the global satisfaction rating, we already know the answer to this question – there are only very small level effects.
Of more interest are the shape differences reflecting unique profiles of CEQ factors (e.g., high in teaching and low in workload). Whilst this sort of analysis has not been a focus of previous research with CEQ ratings (or DUE ratings more generally), we suggest that this information – if reliable and valid – might be useful information in benchmarking universities and departments, as well as being useful to prospective students. Results in Table 2 are based on two somewhat different approaches to this issue – repeated measures ANOVA and a parallel MANOVA. However, the key findings are essentially the same for both approaches.
Level effects (Supplemental Table 2) are consistent with previous analyses, showing that variance components (η2) are statistically significant but very small. Averaged across the five CEQ factors, there is little differentiation between universities, disciplines or departments. Although results based on such different statistical procedures need to be compared with caution, variance components based on this analysis are even smaller than those based on the multilevel analyses for the overall CEQ rating and each of the five CEQ factors considered separately.
Shape effects (Supplemental Table 2) evaluate the extent to which profiles of specific CEQ factors vary as a function of university, disciplines, or departments (i.e., the interaction between universities and disciplines). Although these effects are statistically significant, the variance components associated with each of these tests are again very small.
In summary, these profile analyses indicate that there are little effects of profile level or profile shape. Neither universities nor departments are meaningfully differentiated in relation to either the average of the five CEQ factors (a level effect) or the unique profiles of CEQ factors (a shape effect).
Supplemental Table 2.
Profile Analysis: Level and Shape Effects Associated with University the Course.
Source of Variation SS DF MS F-ratio η2 Wilks MultES
Level Effects
University (Univ) 305.29 40 7.63 4.82** .004
Discipline (Dis) 35.93 9 3.99 2.67** .001
Course (Univ x Dis) 1439.67 253 5.69 3.57** .019
Error Between 70639.42 44629 1.58 .975
Total between 72420.31 44931
Shape Effects
CEQ Factors (Fact) 2.14 4 .53 1.03 .000 .999 .000
Univ x Fact 237.98 160 1.49 2.88** .003 .990** .003
Dis x Fact 50.28 36 1.40 2.70** .000 .998** .001
Course x Fact 1686.99 1012 1.67 3.23** .018 .934** .017
Error Within 92207.11 178516 .52 .979
Total Within 94184.50 179728
Grand Total 166604.81 224659
Note. Results are presented for a repeated measures analysis of variance [represented by the Sums of Squares (SS), degrees-of-freedom (DF), means squares (MS), F-ratio, and proportion of variance explained ( η2)] and a multivariate analysis of variance [represented by the Wilks Lambda (Wilks) and the Multivariate effect size (MultES)]. Level effects represent the extent to which there are effects of university, course, and their interaction generalize across the five CEQ factors. Shape effects are the extent to which Level effects vary as a function of the five CEQ factors. As noted earlier, the Univ x Discipline interaction in this analysis corresponds to the Course variance component in the multilevel analyses – the extent to which differences between disciplines vary from university to university—whilst the main effect of discipline in both approaches is the extent to which there are consistent discipline differences that generalize across universities.