Pelham, B. W., & Blanton, H. (2003). Conducting research in psychology: Measuring the weight of smoke (2nd Ed.). Toronto: Thompson Wadsworth

Chapter 4: How do we misinterpret? Common threats to validity

The chapter focuses on four general categories of threat to the validity of a set of research findings (each divided into three more specific sub-categories).

  • People are different
  • People change
  • The process of studying people changes people
  • Variables that systematically accompany a treatment (confounds) change people

People are different (p. 86)

  • Individual differences
  • Pseudo-experiments are prone to this threat (absence of some kind of comparison group or control group) … individual differences are a very good alternate explanation for the findings observed in pseudo-experiments
  • Individual differences are a threat to internal validity
  • Experimental method is relatively immune to threat to internal validity, including those based on individual differences
  • Selection bias
  • Refers to sampling from an unrepresentative bias
  • Threat to external validity
  • Nonresponse bias
  • Respondents themselves are the source of bias because people who choose to answer surveys are systematically different from people who choose not to do so
  • Surveys that have low response rates may yield information that is misleading
  • Relying on convenience sample of college students may represent a serious shortcoming of much psychological research
  • It is important to realize that the efficacy or utility of a sample technique depends as much upon the group of people about whom we would like to generalize as much as it does on the group of people we have sampled

People change (p. 89)

  • People change over time and across situation
  • If people change for reasons that have nothing to do with the treatment, then these changes can lead to some inappropriate conclusions about the treatment
  • History
  • Refers to changes that are occurring more or less across the board in a very large group of people such as a nation or culture
  • Common threat to internal validity
  • Maturation
  • Refers to the specific developmental or experiential changes that are occurring in a particular person, or particular age cohort, over time
  • Common threat to internal validity
  • Regression toward the mean
  • Refers to the tendency for people who receive high or low scores on a particular measure to score closer to the mean on a subsequent testing
  • Changes that are mistaken for experimental treatments occur when research participants do not experience any true changes (such as maturation) but score differently than they did originally when they take a retest of a particular measure of personality or performance
  • People fail to appreciate the potency and ubiquity of regression toward the mean
  • True scores are more stable than observed scores (influenced by the true score and error) … regression toward the mean exists because performance is always the joint product of skill and luck
  • Threat to internal validity
  • Pre-post studies that do not include a control group are most susceptible

The process of studying people changes people (p. 94)

  • Hawthorne effect
  • Studying people can dramatically change the way that people behave
  • Testing effects
  • Refer to the tendency for most participants to perform better on a test or personality measure the second time they take it
  • Probably reflect learning on the part of the test taker
  • In the case of many psychological tests, test effects may represent a form of attitude polarization (allowing people to give a little thought to their attitudes often leads them to become more extreme in these attitudes)
  • There are several ways to correct the problem of testing effects
  • Conduct a true experiment in which you estimate the pretest (requires some faith in random assignment)
  • Wait as long as possible between pre and post test
  • Use an alternate version of the test
  • Experimental mortality (attrition)
  • Refers to the failure of some participants in an experiment to complete the study
  • Can represent a threat to both internal and external validity
  • Homogeneous attrition: equal levels of attrition across all experimental conditions (threat to external but not internal validity)
  • Heterogeneous attrition: the attrition rates in two or more conditions of an experiment are noticeably different (threat to internal and external validity) … basically erases the normal benefits of random assignment
  • Ways to reduce attrition include: Provide realistic expectations, incentives, emphasize the scientific importance of attrition
  • Participant reaction bias
  • Occurs when people realize they are being studies and behave in ways that they normally wouldn’t
  • Three basic types of reaction
  • Participant expectancies: People may try to do what they think the researcher expects
  • Participant reactance: People may try to do the opposite of what they think the researcher expects
  • Evaluation apprehension: Concerns about being judged favorably or unfavorably by another person. In an experiment, people may try to do whatever will make them look good
  • These represent threats to internal validity
  • Demand characteristics: characteristics of an experiment that subtly suggest how people are expected to behave
  • Ways to reduce participants reaction bias include:
  • guarantee anonymity
  • use a cover story (a false and often elaborate story about the purpose and nature of the study to create expectancies unrelated to the real predictions of the experiment) … deception (ethics?)
  • unobtrusive observations (participants do not realize they are being observed, or if they do, they don’t realize what is being observed)
  • use a bogus pipeline (convince participants that researchers can read people’s minds)
  • use indirect measures of people’s attitudes and opinions (e.g., ask about reactions to another’s behaviour) … pencil-and-paper version of an unobtrusive observation
  • measure attitudes people don’t know they posses (e.g., IAT)

Variables that accompany a treatment change people (p. 105)

  • researchers sometimes unintentionally allow things to vary along with their independent variables
  • can be hard to detect, but represent threats to internal validity
  • Experimenter bias
  • Refers the influence of the experimenter’s expectations about their studies on their experimental observations
  • Sometimes see what we expect to see
  • Experimenters may treat their participants differently according to their expectations about how their participants should perform (double-blind procedure can overcome this problem)
  • Confounds
  • A broad term for any design problem in which some additional variable varies systematically along with the independent variable …sometimes referred to as a third variable problem
  • Confounds can influence correlational and experimental studies
  • Relative to other designs, true experiments are less likely to contain confounds
  • Represent a serious threat to internal validity
  • Ways to avoid confounds include:
  • Be careful,
  • read the existing literature
  • measure any variables that are likely to be confounded with the variable(s) in which you are interested