Study Questions for Chapter 14 of Shadish et al.
“A Critical Assessment of Our Assumptions”
Dear Students,
Yes, there will be a quiz on this material on Tuesday.
Jim Wood (Saturday, November 03, 2001)
p. 457 Describe the view of causality represented by “pretzels.”
pp. 457-458 In the United States, objections to experimentation and to simple causal arrows are most prevalent in what group of researchers? What inconsistency do Shadish et al describe in this group?
p. 458 Give a concrete example of (a) a causal relationship that changes magnitude under some circumstances, and (b) a causal relationship that changes sign under some circumstances.
p. 458 Explain why in education the discovery of simple causal relationships may often be of more practical value than the discovery of complex “contingent” causal relationships. Give an example to illustrate this point.
p. 459 Shadish et al suggest that some critics (a) rightfully reject the idea that quantification and logic are the only basis for knowledge, (b) but then go on to commit a logical error. What is that error?
p. 459 According to critics who focus on the incommensurability of theories, when disconfirming data seem to contradict a theory, what can scientists always do?
p. 460 Discuss the following statement “The strength of science is its ability to construct definitive tests of proposed hypotheses.”
p. 461 What kinds of replications are particularly useful to keep “observations” from being “overly impregnated” with one theory?
pp. 460-461 Describe how a repetitive cycle of criticism and new analyses has led to a consensus regarding the effectiveness of psychotherapy?
p. 462 Name any three causal relationships that, according to Shadish et al are (a) important but (b) not amenable to experimentation.
p. 463 Cronbach (1982) criticized the concept of “internal validity” because it was based on a “pointless” and “trivial” idea of cause. What did he think was “trivial” about the idea of “cause” with which internal validity is concerned?
p. 463 “The history of science is replete with examples of famous series of experiments in which a causal relationship was established early, but it took years for the cause (or effect) to be named.” Shadish et al are arguing that which two types of validity warrant separate attention.
p. 465 Explain how the legal process of proving that “Jones’ action cause the death of Smith” address the issues raised in John Stuart Mill’s criteria for inferring causality.
p. 466 Shadish et al. assert that if we “wiped the slate clean” and started to reconstruct our knowledge anew, we would probably reinvent the notion of descriptive causation all over again? Why do they think this is so?
p. 470 Shadish et al state “Thus, we come full circle. We began with multiple operational representations of the same cause or effect when testing a single causal relationship; then the data forced us to invoke more than one relationship; and finally the pattern of the outcomes and their relationship to the existing literature can help improve the labeling of the new relationships achieved.” Give a concrete example of a program of research that follows this “circle”.
p. 475 How do Shadish et al define validity? What is the chief difference between Messick’s (1989) understanding of validity and the understanding of Shadish et al?
p. 475 Briefly describe the historical events that led to the publication of Cronbach and Meehl’s (1955) classic article on construct validity.
p. 476 If validity is concerned with the truth of knowledge claims, then why is it nonsensical to speak of the validation of actions?
p. 477 What is the difference between validation and evaluation?
p. 480 Give a counterexample that undermines each of the following definitions of truth: (1) A statement is true if it is meaningful and coherent; (2) A statement is true if it is credible to those who hear or read it.
p. 481 What is wrong about claims that the idea of validity “historically arose in the context of experimental research”?
p. 484 What does it mean that “fuzzy plausibility” is a problem for quasi-experiments?
p. 484 Explain the rationale that guided Shadish et al. when they chose the order of the chapters on quasi-experiments and randomized experiments.
pp. 487-488 According to Shadish et al, what has been the recent attitude of relevant governmental agencies and programs toward randomized experiments?
p. 488 Some critics argue that although experiments are superior in theory, they are not superior in practice. Why do they take this position? How do Shadish et al respond to these critics?
p. 491 What argument was made against an “empirical approach” to school reform in a letter to New York Times? What are your own thoughts on the issue of “waiting until the studies have told us what works”? What are the pluses and minuses of waiting?
p. 493 If randomized experiments are not used to evaluate educational reforms, why can this limit the inferences about “what works”? Give a concrete example mentioned by Shadish et al.
pp. 494 Some critics have argued that experiments really aren’t worth doing because they have a limited or ambiguous effect on the real decisions of policy makers. Describe any two of the points that these critics have made. Describe any two points that Shadish et al. make in response.
p. 495 How does the experience of Tennessee and California with smaller classrooms illustrate that a treatment that is effective on a small scale will not necessarily work as well if it is implemented on a large scale?
p. 496 Some critics have argued that locally developed solutions work better than federally imposed solutions that are based on experiments. According to Shadish et al, what is the ideological orientation of these critics? What is “ironic” about their objections? How strong is the evidence that locally developed solutions work better?
pp. 499-502 Shadish et al discuss four major approaches (other than experimentation) that have been used to answer questions about causal hypotheses. What are these four approaches? (Hint: Look at the headings in this section
pp. 500-501 According to Shadish et al., what are the three reasons that case study methods generate enthusiasm?
pp. 501-502 Give an example of how a researcher might design a theory-based evaluation.
pp. 504-505 Succinctly characterize the attitude of Shadish et al. toward the use of statistical adjustment to achieve control in quasi-experiments.