HS 410a: Applied Research Seminar - Quantitative
Spring 2016
Grant Ritter (781)736-3872
Room 268, Heller-Brown Building
Recommended Texts:
Quantitative Research Methods in the Social Sciences - Paul S. Maxim, Oxford Press, 1999
Seminar Description:
The purpose of this seminar is to help Heller PhD students with the quantitative research needs of a dissertation. This is the 4th course in a quantitative sequence for Heller PhD students. Prerequisites are to know the material from the prior three quantitative courses in the set. The seminar will focus on real world applications of statistical and quantitative methods. In particular, we will address the issues which arise in course of analyzing large administrative or research datasets, including complex sample design, quasi-experimental design, sophisticated model techniques, construction of scales and indices, dealing with threats to validity, and imputation for missing data.
At the beginning of the course all students are required to choose a dataset. It should be of sufficient quality, detail, and scope that it could be used for a PhD dissertation. If the student doesn’t already have such a dataset in mind, they are encouraged to identify one as soon as possible. The dataset does not have to be the exact one which a student would use for a dissertation, but the more closely related the better. To complete the course, students must identify research question(s) based on their dataset, formulate hypotheses, determine appropriate analytic methods to test these hypotheses, draw conclusions from these tests, interpret how these conclusions fit the theory they have been formulating, and discuss policy implications.
Actual statistical programming will represent only a small component of the course, but student may benefit greatly from using statistical software and the analytic methods they have learned in past courses to become familiar with their data. Real success in this course requires that you engage with the data (the same may almost certainly be said with respect to success with a dissertation).
I will try to make each class meeting a discussion of methodological theory, data issues, statistical techniques, and statistical programming advice. There are a number of writing assignments throughout the semester, which should be e-mailed to the instructor. Each writing assignment deals with topic(s) discussed in the seminar meeting, but requires a student to further explore the topic with respect to his/her own research project. As a final writing assignment for the course, student are required to combine all their previous writing assignments into one document, then edit it to read like the methodology chapter of a PhD dissertation.
A second requirement for all students in the seminar is to give a presentation on their research project. Beginning the 7th week of the semester (March 9/ 2016), two students per week will be chosen to give a presentation highlighting the quantitative and methodological features of their research and receive advice and feedback from other seminar members.
Weekly Topics and Homework Assignments
Lecture 1 – January 13, 2016
Topics: Scientific Inquiry, Theory, Hypothesis testing, Causality, Relationship between empirical phenomena and scientific truth, role of probability in statistical inference, types of errors associated with hypothesis testing, significance, power and the trade-off between them.
Reading Assignment:
Maxim Chapters 3 and 4 (Causality and Statistical Inference)
G. Dallal Passages (at http://www.tufts.edu/~gdallal/LHSP.HTM):
a. Cause and Effect
b. Study Design
c. creating Data files
d. Look at the Data!
Project Assignment – Choose a dataset of sufficient complexity to support a dissertation
Lecture 2 – January 27, 2016
Topics: Random Samples, Sampling Design, Strata and Clusters,
Reading Assignment:
Maxim Chapter 5 (Sampling: Basic Statistics)
G. Dallal Passages (at http://www.tufts.edu/~gdallal/LHSP.HTM):
a. Intention to Treat
b. Random samples /Randomization
c. Units of Analysis
Writing Assignment – Write a page describing the research project associated with your dataset. List the set of research questions that you hope to address.
Lecture 3 – February 3, 2016
Topics: Complex Sample Designs, Weighting
Reading Assignment:
Maxim Chapter 6 and 7 (Sampling: Design and Special Problems)
Writing Assignment – Write a section describing the sampling design of the dataset you are planning to use.
Lecture 4 – February 10, 2016
Reading Assignment:
Topics: Experimental and Quasi-Experimental Design
Maxim Chapter 8 (Experimental Design)
Thomas E. Dawson – A Primer on Experimental and Quasi-Experimental Design
http://www.tele.sunyit.edu/expdes.htm
Yale stat course definitions: http://www.stat.yale.edu/Courses/1997-98/101/expdes.htm
Writing Assignment – List the hypotheses you wish to test as part of your research project. Describe your conceptual framework and relate how it can be used do your hypothesis testing.
Lecture 5 – February 24, 2016
Topics: Types of Quasi-Experimental Designs, Cross-sectional and longitudinal data
Reading Assignment:
G. Dallal Passages (at http://www.tufts.edu/~gdallal/LHSP.HTM):
a. The Most Important Lesson you’ll ever Learn about Multiple Linear Regression Analysis
William M.K. Trochim - Research Methods Knowledgebase . This is a web-based primer on research methods. Begin at www.socialresearchmethods.net/kb/ and navigate by way of the index on the left side. You’re welcome to read any part of this but I recommend starting with Design, and in particular read:
Menu item web page
Introduction to Research Design desintro.htm
Type of Design destypes.htm
Experimental Design desexper.htm
Quasi-experimental Design quasiexp.htm
Non-Equivalent Groups Design quasnegd.htm
Relationships Among Prepost desrel.htm
Designing Designs for Research desdes.htm
Internal Validity intval.htm
External Validity external.htm
Writing Assignment – Describe the (quasi-) experimental design you will use in your project.
Lecture 6 – March 2, 2016
Topics: Statistical Models – Logistic, Linear Regression, Hierarchical
Reading Assignment:
More Trochim sections on Experiment and Quasi-Experimental Design
Menu item web page
Dummy Variables dummyvar.htm
Randomized Block Design statblck.htm
Analysis of Covariance statcov.htm
Non-Equivalent Groups statnegd.htm
Factorial Design Analysis statfact.htm
Writing Assignment – Construct a table identifying the variables you’ll use in your analyses. Put them in categories as treatment variables, covariates (other independent variables used as adjustors) and dependent variables.
Lecture 7 – March 9, 2016
Topics: ANOVA and Multiple Comparisons, Fixed and Random Effects
Reading Assignment:
SEMATECH sections on Analysis of Variance, Variance Components, Fixed and Random Effects, and Multiple Comparisons. Go to http://www.itl.nist.gov/div898/handbook/prc/section4/ and click on individual pages:
Menu item web page
Are the means equal prc43.htm
The 1-way ANOVA prc432.htm
The 2-way ANOVA prc437.htm
What are Variance Components prc44.htm
How can we compare results prc45.htm
Multiple Comparisons prc47.htm
Writing Assignment –Begin writing your analysis plan. Formulate appropriate model(s) based on your chosen (quasi-) experimental design.
Lecture 8 – March 16, 2016
Topics: Measurement, Reliability, Validity
Reading Assignment:
Maxim Chapter 9 (Measurement Theory)
Trochim web pages (at http://www.socialresearchmethods.net/kb/) regarding Measurement/reliability.
Chung Ho Yu web page giving various views on reliability and validity http://seamonkey.ed.asu.edu/~alex/teaching/assessment/reliability.html
Menu item web page
Theory of Reliability reliablt.htm
Types of Reliability reltypes.htm
Writing Assignment – Continue writing your analysis plan by describing the statistical method you will use to analyze your model(s).
Lecture 9 – March 23, 2016
Topics: Cohen’s Kappa, Scales and Indices, Cronbach’s Alpha, ICC coefficients
Reading Assignment:
Maxim Chapter 10 (Classical Test Theory)
Lecture notes at http://www.shsu.edu/~icc_cmf/cj_787/research10.doc
David Howell’s http://www.uvm.edu/~dhowell/StatPages/More_Stuff/icc/icc.html
McGraw, K. and Wong S.P. “Formulating Inference about some Intraclass Correlation Coefficients” Psychological Methods (1996) Vol 1. 1 30-46
Cohen: A coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement XX, 1, 1960)
Computer Assignment:
1) Selection a number of items in your data that might combine to form an index.
2) Use Cronbach’s alpha to check the reliability of items if they were combined unweighted.
Lecture 10 – March 30, 2016
Topics: Factor Analysis, realities of modern data collection: IRBs, DUAs, and HIPAA
Reading Assignment:
Maxim Chapter 11 (on Confirmatory Factor Analysis)
Government description of HIPAA regulations at:
http://www.hhs.gov/ocr/part3.html and http://www.hhs.gov/ocr/regtext.html
Computer Assignment:
1) Use Principle Component Factor Analysis on the same variables used in last assignment to check the reliability of the items if they were combined in a weighted fashion (test for number of factors within)
Writing Assignment: Write a description of your index construction process suitable for -inclusion in a Methods chapter of a dissertation.
Lecture 11 – April 6, 2106
Topics: Missing Data and Imputation
Reading Assignment:
Maxim Chapter 13 (Missing Data)
Writing Assignment: Write a section on the limitations of your analyses. Point on threats to validity and describe the steps you have taken to limit the impact these threats. Selection and its interaction with other threats (e.g., selection x treatment interaction) is a particular concern with quasi-experimental design, so but sure to discuss how you hope to deal with it.
Lecture 12– April 13, 2106
Topics: Dealing with Selection – matching, propensity score matching,
Reading Assignment:
D’Agostino article on “Propensity Score Methods for Bias Reduction”
Statistics in Medicine 17, 2265-2281 (1998)
Newhouse, J. & McClellan, M. article on Instrumental variables: “Econometrics in Outcomes Research” Annual Review of Public Health (1998) Vol 19, 17-34
Writing assignment: The dataset you are going to use for your project is virtually guaranteed to have missing data on some essential variables. Write a section identifying and describing how you intend to impute such variables. Identifying which variables are appropriate for such action will require some computer work to determine the amount of missingness.
Lecture 13– April 20, 2016
Topics: Dealing with Selection – Using instrumentals
Final Writing Assignment: Combine all previous writing assignments into one document supplementing with additional bridging material to make it read right. When done, you should have something that looks very much like the Methods chapter of a dissertation (earlier material regarding research questions and conceptual model may move to earlier chapter when such chapter are written).