Study Questions for Chapter 2 of Shadish et al.
“Statistical Conclusion Validity and Internal Validity”
p. 33 Shadish et al. introduce the concept of validity by telling the story of a famous horse. Briefly tell this story.
p. 34 Validity is a property of ______
p. 35 ______theory says that a claim is true if it corresponds to the world.
p. 35 ______theory says that a claim is true if it belongs to a coherent set of claims.
p. 35. ______is a theory of truth that says that a claim is true if it is useful to believe that claim.
pp. 35-36 Shadish et al. identify three dominant theories of truth.
Name and explain each of these theories. Which of these three theories do Shadish et al. endorse as the one that is most likely correct? Which of these three theories do Shadish et al. consider to be "compromised"?
pp. 37-39 What are the four validity types identified by Shadish et al that correspond to four questions that researchers face when interpreting causal studies? For each type of validity, state the corresponding question that researchers face when interpreting causal studies (see page 39).
p. 39 ______are specific reasons why we can be partly or completely wrong when we make an inference about covariance, about causation, about constructs, or about whether the causal relationship holds over variations in persons, settings, treatments, and outcomes.
p. 40 What is the main method that Shadish et al. recommend for ruling out potential threats to validity?
p. 40 If threats to validity cannot be ruled out by design controls, Shadish et al. recommend that researchers identify and explore the role and influence of the threats in the study. In doing this, three questions are critical. What are these three questions?
pp. 40-41 Shadish et al. give an example about how their "three critical questions" could be used to identify and explore "history" as a threat to the validity of the WIC program. Discuss this example in detail, explaining how each of the three questions is addressed by Shadish et al.
p. 41 Shadish et al. state that because all researchers have difficulty criticizing their own work, what is the most common source of identified threats to the validity of studies?
p. 41 Why do Shadish et al. prefer design control over statistical control to rule out threats to validity?
p. 41 Without a theory that identifies possible threats to validity, most researchers rely on ______to identify such threats.
p. 42 Statistical conclusion validity concerns two related statistical inferences. What are these two inferences?
p. 43 Most criticisms of NHST reduce to two. What are these two?
p. 43. In the debate about NHST, how do most parties prefer that results be reported?
pp. 45-48 Suppose that of friend of yours has designed an experiment and estimated its power, which is .30.
(a) What is the probability that this study will fail to find a significant effect, assuming that an effect really exists?
(b) What is the power that is generally recommended for experiments in psychology?
(c) What practical advice should you give to your friend?
pp. 46-47 Be able to name, explain, and give concrete examples of the methods to increase power listed in Table 2.3 on pages 46 and 47 of Shadish et al.
pp. 46-47 Name, explain, and give concrete examples of THREE methods (other than increasing the level of alpha) to increase power, that are listed in Table 2.3 of Shadish et al.
p. 49 What are the two reasons that Shadish et al. believe that a complete reporting of results in research reports is unlikely:
p. 49 What is the likely outcome of the practice of incomplete reporting of research results?
p. 49 Unreliability always ______univariate relationships.
p. 49 What is the effect of unreliability if three or more variables are involved?
p. 49 Name any TWO remedies suggested by Shadish et al. for unreliability of measurement.
pp. 49-50 Floor and ceiling effects are examples of what threat to statistical conclusion validity?
pp. 49-51 How does restriction of range affect power and bivariate relationships?
p. 50 What are floor and ceiling effects? Give an example of each.
p. 50 Why is it undesirable to dichotomize or trichotomize continuous variables?
p. 50 If you are thinking about using a particular measure, but are afraid that it may have floor or ceiling effects in your subjects, how could you deal with this?
p. 51 Suppose you are conducting a field experiment in which features of the experimental setting may fluctuate and adversely affect the treatment. If you cannot control these factors, but suspect that they exist, what do Shadish et al. suggest that you do?
p. 51 Suppose that your subjects are very heterogeneous. What effect will this have on your ability to detect an effect? How do Shadish et al. suggest that you might deal with this problem?
pp. 52-53 Explain the maxim that "we cannot prove the null hypothesis". What might you do if you need to know if two treatments are equivalent?
p. 53 What form of validity refers to inferences about whether observed covariation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured?
p. 54 To support inferences that A causes B, what three things must the researcher show?
p. 54 Campbell (1986) proposed re-labeling internal validity as...
p. 54 Explain the implications of each of the first three words in the term "local molar causal validity"
p. 54 What is a shorter term for "local molar causal validity?
p. 54 Mackie (1974) stated "Typically we infer from an effect to a cause (inus condition) by eliminating other possible causes." Shadish et al. say that another term for these "other possible causes is"
p. 55 What is a bidirectional or reciprocal causal relationship? Give a concrete example.
pp. 56-57 Define and give an example of each of the following threats to internal validity: Selection History Maturation
p. 56 What is the most effective (although not always the most practical) design strategy for eliminating selection bias?
p. 56 In a field study without random assignment, how might you reduce the plausibility of history as a threat to internal validity?
p. 57 In a field study without random assignment, how can you reduce maturation as a threat to internal validity?
p. 57 If research participants are selected or select themselves because they had scores that were higher or lower than average, what threat to internal validity (besides selection) becomes more plausible?
p.58 If the participants in a study are selected precisely because they have extreme scores on a variable, what is the best solution?
p. 59 What procedures involving measurement at multiple time points can help identify and reduce regression effects?
p. 59 A teacher has received a list of students who received a D average during the last quarter. Her job is to select half of these students and tutor them. If their grade goes up to C during the next quarter, she will receive a bonus. If the teacher wants to maximize her chances of receiving a bonus, how can she use the principle of "regression to the mean" to select the students she will tutor?
p. 59 What is another, shorter name for "experimental mortality"?
p. 60 Explain the difference between the validity threats of (1) testing and (2) instrumentation. Give a concrete example of each.
p. 61 Which treats to internal validity are eliminated by randomization?
Which additional threats are eliminated by randomizing subjects AND measuring groups equivalently?
p. 62 Given random assignment, inferential problems about causation arise only in TWO situations. What are they? What are the ways to manage these two problems?
p. 62 What is "response burden"? Give a concrete example.