G Theory SeminarShavelson & Haertel
Education 353C
Problems in Measurement: Generalizability Theory
Richard J. Shavelson & Edward H. Haertel
Tuesdays 9:00AM – 11:50AM
Cubberley 207
Generalizability Theory (G theory)provides a powerful class of psychometric models for behavioral and social measurement data. G theory was developed beginning in the 1960s by Lee J. Cronbach here at StanfordUniversity with his collaborators Goldine Gleser, Harinder Nanda, and Nageswari Rajaratnam (Cronbach, Rajaratnam, & Gleser, 1963; Cronbach, Gleser, Nanda, & Rajaratnam, 1972). It encompasses classical test theory (CTT) as a special case, but goes far beyond CTT in clarifying conceptual confusions and providing more powerful statistical tools. Most importantly, G theory systematizes distinctions among the many reliability coefficients of CTT, and enables the calculation of reliability coefficients and/or standard errors of measurement tailored to specific measurement applications and/or interpretations. G theory also provides a powerful extension of the Spearman-Brown "prophecy" formula, enabling the design of efficient data collection plans for specific measurement applications.
In this course, we will provide a systematic introduction to the concepts and models of G theory, as well as special-purpose software useful in carrying out the requisite calculations. The goals of the course are to provide a working knowledge of common G-theory models and applications, to enable students to read critically journal articles employing G theory, and to introduce some unsolved problems and current research topicsrelated to G-theory.
The class will typically be organized as follows. The instructors might begin with a mini-lecture on the topic of the day with discussion. Then, a problem with data will be introduced and students will work in teams to solve the problem within the G theory framework. Toward the end of the quarter, the problems will be replaced by student-led presentation of self-chosen paper that uses G theory.
Occasional homework will consist of students working on abrief problem with data. There will be a final project in which either a student will apply G theory to a somewhat novel set of data and report her/his findings to the class or will read up on a topic not covered in class (e.g., newly developed standard errors and confidence intervals for variance component estimates)and teach the class about the topic. There will be no final examination.
The primary text for the course is Generalizability Theory: A Primer by Richard J. Shavelson and Noreen M. Webb. In addition, Generalizability Theory by Robert L. Brennan is recommended. Additional readings will be provided during the quarter. Note that the Primer is just that—an introductory text on G theory. Brennan’s is the currently authoritative text. So, when you complete a chapter in the Primer and you want more statistical background, turn to Brennan. Brennan covers roughly the same materials in his Chapters 1-4 as the Primer does—but in a somewhat different structure.
You can purchase both textbooks online so they are not in the bookstore. For convenience, we have copied the first two chapters of the Primer and will email them to you.
Some References
Brennan, R. L. (1997). A perspective on the history of generalizability theory. Educational Measurement: Issues and Practice, 16(4), 14-20.
Brennan, R. L. (2000a). (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice, 19(1), 5-10.
Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.
Cardinet, J., Tourneur, Y., & Allal, L. (1976). The symmetry of generalizability theory: Application to educational measurement. Journal of Educational Measurement,13, 119-135
Cardinet, J., Tourneur, Y., & Allal, L. (1981). Extension of generalizability theory and its applications in educational measurement. Journal of Educational Measurement, 18, 183-204.
Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391-418
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
Cronbach, L. J., Linn, R. L., Brennan, R. L, & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.
Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: American Council on Education/Praeger.
Shavelson, R. J. (2004). Editor’s preface to Lee J. Cronbach’s “My Current Thoughts on Coefficient Alpha and Successor Procedures.” Educational and Psychological Measurement, 64(3), 389-391.
Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W. (1999). Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36, 61-71.
Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133-166.
Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage.
Shavelson, R.J., Webb, N.M., & Rowley, G. (1989). Generalizability theory. AmericanPsychologist, 44(6), 922-932.
Webb, N.M., Shavelson, R.J., & Maddahian, E. (1983). Multivariate generalizability theory. In L.J. Fyans (Ed.), Generalizability theory: New directions for testing and measurement (pp. 67-82). San Francisco: Jossey-Bass.
Webb, N. M., Shavelson, R. J., & Haertel, E.H. (2007). Reliability coefficients and generalizability theory. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics: Psychometrics(Vol.26, pp. 81-124).Amsterdam: Elsevier B. V.
Wiley, E. W. (2000). Bootstrap strategies for variance component estimation: Theoretical and empirical results. Unpublished doctoral dissertation, StanfordUniversity.
Spring 20071
G Theory SeminarShavelson & Haertel
Tentative Course Schedule[1]
DateTopicReadingsaAssignmentb
4-3Introduction to G Theory & StatisticalSW Chapter 1
Demo Software (GENVA & EduG)CGNR Preface &
SW Tables 1.1 and 1.3Pp 1-14
4-10AERA Week—No class meetingSee 4-3 ReadingsHomework #1
SW Chapter 2A. Handout P x I G study Brennan Appx. F B. Run Exercise 3 in SW Ch. 3
EduG Users Guide with GENOVA & EduG
4/17Discuss Homework #1 SW Chapters 2 & 3Homework #2—Brief proposal for finalproject (bring handouts
Statistical Model for G Theoryto class 4/24)
G Studies with crossed facetsRead Cardinet’s Why EduG
One-facet G Study simulation
4/24Discuss final project ideasSW Chapter 4Homework #3—Read prepare to
Discuss Why EduGlead discussion:
In Class Exercise #1Group 1: Cardinet et al 1981
G studies with nested facetsGroup 2: CLBH
Group 3: SRW
5-1Group presentations of papersSW Chapter 5Homework #4 Prepare final Random & Fixed Facets project description; hand in at class on 5-8
5-8Decision StudiesSW Chapter 6Homework #5 Cardinet Exercise#2_Cardinet #1 Exercise #2 & Cardinet
Collect project descriptionsComplex Exercise
5-15Discuss Homework #5SW Chapter 7 & 8Work on projects
G & D Studies for same and
different Designs
5/22Multivariate G TheoryWS
mGenova ExerciseBrennan 267-285
(Optional)
5-29Presentations of student projects
6-5Presentations of student projects
______
aReadings to be done before class on date given
CGNR refers to Cronbach, Gleser, Nanda, & Rajaratnam (1972).
CLBH refers to Cronbach, Linn, Brennan, & Haertel (1997).
SW refers to Shavelson & Webb (1991).
SRW refers to Shavelson, Ruiz-Primo, and Wiley (1999).
WS refers to Webb & Shavelson (1983).
b Assignments due the week following their assignment
Spring 20071
[1] This schedule is tentative and may change depending on coverage in class or other unexpected events.