Chapter 9 – Experimental Design Research Methods (pp. 187-232)

Overall teaching objective: To introduce undergraduate criminal justice research methods students to the various experimental design research methods and to demonstrate their applications.

Note to instructors: This chapter is presented in two sections. The first section provides an overview of the research method. The second section uses a research report to demonstrate how previous researchers applied this method to a project relevant to criminal justice practice. In both sections the material is organized by the generic research process that was presented in Chapter 2.

  • Experimental design research methods are considered one of the ‘purest’ forms of social science inquiry.
  • When done correctly, results from an experimental design provide very good insight into the actual causes of social phenomena.
  • Because of their ability to isolate and measure the effect of a single independent variable on a single dependent variable, experimental design models are especially useful in explanatory research.

Making Research Real 9.1 – Would a Speed Trap Reduce Traffic Crash Fatalities? (p. 187)

  • A police department conducts an experiment to determine if a speed trap would reduce traffic fatalities on an interstate highway that traverses the community.
  • The experiment contains all of the features of an experimental design – pretest, posttest, treatment, experimental group, control group

Experimental Design Basics (p. 189)

  • An experiment is a research method that measures the effect of an independent variable on a dependent variable.
  • All experimental design models feature an;
  • experimental group (the group that actually gets exposed to a treatment),
  • a treatment (the independent variable that is alleged to cause change to the dependent variable), and
  • a posttest (a measurement of the dependent variable after the treatment.
  • More sophisticated experimental design models also include;
  • a pretest, (a measure of the dependent variable before the treatment), and
  • a control group (the group, equivalent to the experimental group in terms of the dependent and other variables, that is not exposed to the treatment.

Types of Experimental Designs (p. 191)

  • There are several types or variations of the experimental designs model.
  • The five most commonly used experimental design models.
  • The one group no pretest experimental design model
  • The one group pretest/posttest experimental design model
  • The two group no pretest experimental design model
  • The two group pretest/posttest experimental model
  • The Solomon Four Group experimental model
  • The one group, no pretest experimental design only includes the basic elements of an experimental design model – the experimental group, a treatment and a posttest. This design does not include a pretest or a control group. Because of this the effect of the treatment cannot be accurately measured and the influence of other factors on the dependent variable cannot be identified.

Figure 9.1 – The one group, no pretest experimental design model. (p. 192)

PRETEST / TREATMENT / POSTTEST
EXPERIMENTAL GROUP / NO / YES / YES
  • The one group pretest/posttest experimental design model includes the basic elements of an experimental design model – an experimental group, a treatment and a posttest. In addition, this model includes a pretest that allows the researcher to measure the actual effect of the treatment (independent variable) on the dependent variable. This design does not include a control group so there is really no way for the researcher to know whether something other than the treatment caused a change to the dependent variable.

Figure 9.3 – The one group pretest/posttest experimental design model.(p. 193)

PRETEST / TREATMENT / POSTTEST
EXPERIMENTAL GROUP / YES / YES / YES
  • The two group no pretest experimental design model includes the basic elements of the experimental design model – an experimental group, a treatment and a posttest. This design includes a control group so the researcher would be able to determine that the independent variable, by itself, had some effect on the dependent variable. However, because there is no pretest the researcher cannot measure how much effect the independent variable had on the dependent variable.

Figure 9.5 – The two groupno pretest experimental design model. (p. 194)

PRETEST / TREATMENT / POSTTEST
EXPERIMENTAL GROUP / NO / YES / YES
CONTROL GROUP / NO / NO / YES
  • The two group pretest/posttest experimental design model includes the basic elements of the experimental design model – an experimental group, a treatment and a posttest. In addition this model has both a pretest and a control group. These two features enable the researcher to measure the actual effect of the treatment on the dependent variable and to determine whether or not other factors might have caused a change in the dependent variable.

Figure 9.7 – The two group pretest/posttest experimental design model. (p. 195)

PRETEST / TREATMENT / POSTTEST
EXPERIMENTAL GROUP / YES / YES / YES
CONTROL GROUP / YES / NO / YES
  • The Solomon Four Group experimental design model includes the basic elements of the experimental design model – an experimental group, a treatment and a posttest. This model also contains a pretest and a control group. More significantly, this model contains an extra experimental group and an extra control group. These additional features enable the researcher to determine how much, if any, the research subjects’ exposure to the pretest affected their performance on the posttest.

Figure 9.9 – The Solomon Four Group experimental design model.

PRETEST / TREATMENT / POSTTEST
PRETEST/POSTTEST
EXPERIMENTAL GROUP / YES / YES / YES
PRETEST/POSTTEST
CONTROL GROUP / YES / NO / YES
POSTTEST ONLY
EXPERIMENTAL GROUP / NO / YES / YES
POSTTEST ONLY
CONTROL GROUP / NO / NO / YES

Threats to Internal Validity in Experiments(p. 199)

  • You may recall that validity is the extent to which a measure actually measures the concept it purports to measure. For example, feet and inches would be valid measures for height, but not for weight.
  • In experimental designs, the term validity is used in a slightly different sense. There are two types of validity in experimental design research – internal and external
  • Internal validity refers to the ability of an experimental design to document the causal relationship between an independent variable and a dependent variable.
  • There are seven common threats to the internal validity of an experiment: history, maturation, mortality, testing, instrumentation, selection bias and regression.

Table 9.1- Threats to internal validity in an experiment. (p. 201)

Threat to internal validity / Description / Example
History / Major events happen during an experiment that affect the research subjects and thus the dependent variable. / During an experiment on the effect of an education program intended to reduce underage drinking, a popular rap star dies from alcohol poisoning. Empathetic research subjects may decide to drink less as a result. This will affect the researcher’s ability to measure the independent effect of the educational program.
Maturation / Natural developmental changes in the research subjects affect the outcome of the experiment. / A researcher is testing the effects of a new drug use prevention program on high school students. Because young people change within a very short time span, some of the research subjects may mature and become less likely to abuse drugs regardless of whether they participated in the program.
Mortality / A loss of research subjects can occur over the course of an experiment and affect the outcome of the experiment. / A criminologist studies juvenile delinquents from their fifteenth through thirtieth birthday to determine the effect of life course changes on criminal behavior. Some of these research subjects move, drop out of the experiment or die over the 15-year study. As a result, the researcher does not have enough research subjects to ensure her results are significant.
Testing / Exposing research subjects to a pretest prior to the treatment can change the outcome of the posttest. / To evaluate the effect of a prison-based rehabilitation program, a researcher administers a pretest to members of the experimental and control groups. Following the six week training program, in which only the subjects from the experimental group participate, the researcher administers the same test to both groups. The members of the experimental group report higher scores suggesting that the training program has its intended effect. But so does the control group. It is possible that the members of the control group ‘improved’ because they had taken the pretest beforehand.
Instrumentation / Differences between the pretest and posttest instruments cause a change in the dependent variable. / In order to avoid a testing effect, the researcher in the example above decides to make the pre- and posttest slightly different. The experiment indicate that the rehabilitation program had an enormous effect, so big that a peer reviewer asks to review the pre- and posttests (i.e., the instruments). The reviewer argues that the questions were so different between the two tests that the results are not comparable.
Selection bias / Differences between the members of the experimental and control groups result in different effects of the treatment. / In this case the researcher either systematically or inadvertently assigned research subjects to the groups so that the groups were not equivalent with respect to the dependent or other variables that are affected by the treatment
Regression / Although there may be an initial treatment effect, the effect diminishes over time, indicating that the independent variable has no long-term effect. / An educator conducts an experiment to evaluate a program similar to Head Start, whereby impoverished children are allowed to enter school at an earlier age. His results suggest that the program does improve reading and math scores in elementary school. But by the time the children enter junior high school, their reading and math scores are more or less equal to the children in the control group. In short, the independent variable has no long-term effect.

Making Research Real 9.2 – So Much for Community Relations (p. 199)

  • A media consultant is helping a large police department improve its public image
  • She is evaluating the effect of directed public safety announcements (the treatment) on the public perception of the police department (the dependent variable)
  • Prior to the posttest (a survey to measure the effects of the public safety announcement) the local news broadcasts a video of police officers beating a subject following a high speed chase.
  • This is an example of how history affects the internal validity of an experiment’s results.

Making Research Real 9.3 – Measuring the Effect of Pornography (p. 200)

  • A psychologists is conducting an experiment on the effect of pornography (the independent variable) on the frequency of risky sexual behaviors (the dependent variable).
  • He tests and retests a sample of adolescents and learns that most of them experience an increase in risky sexual behaviors as they approach 18 years of age, regardless of whether they were exposed to pornography.
  • Another psychologists reminds the researcher that humans naturally increase sexual desire, and therefore sexual behaviors, during late adolescence.
  • This is an example of maturation.

Making Research Real 9.4 – Measuring the Effect of Video Games (p. 200)

  • A criminologist conducts an experiment on the effect of video games (the independent variable) on aggressive behavior (the dependent variable).
  • Toward the end of the experiment she learns that several members of the control group (who were prohibited from playing video games during the experiment) ‘forgot’ and played them anyway.
  • She removed these errant members from the experiment. Fortunately, there were not enough of them to affect the outcome of the experiment.
  • This is an example of mortality.

Threats to External Validity in Experiments(p. 202)

  • External validity refers to the generalizability of an experiment’s results to other settings and situations. There are two common threats to external validity: reactivity and selection bias.
  • Reactivity occurs when research subjects change their behavior when they become aware that they are being watched or measured.
  • An interaction between selection bias and the experimental variable occurs when;
  • The experimental and control groups are not equivalent with respect to the dependent and other variables that could affect the outcome of the experiment, and
  • One of these groups includes members with characteristics that cause them to be more or less susceptible to change caused by the treatment (independent variable)

Table 9.2 - Threats to external validity in an experiment. (p. 204)

Threat to
external validity / Description / Example
Reactivity / An awareness that they are being measured causes a change in the behavior of research subjects. / A psychologist conducting an experiment on the effect of a new exercise program on obese juvenile delinquents requires his research participants to report their weight on a weekly basis. Overweight participants ashamed of their weight may report lower weights, thwarting the researcher’s ability to measure the effect of the exercise program.
Interaction between selection bias and the experimental variable / There is a failure to ensure that the subjects assigned to the experimental and control groups are more or less equivalent with respect to the variables that might influence the dependent variable.
One of these groups either more or less susceptible to change caused by the treatment (independent variable) / A researcher is conducting an experiment to determine the effect of frequent shift changes on police officers’ cardiovascular health. She inadvertently assigns a larger proportion of older officers to the experimental group. Because age is an influential factor in cardiovascular health, age and not frequent shift changes may affect a change in the dependent variable.

Making Research Real 9.5 – Watching Out for Bullies (p. 203)

  • A school resource officer conducts a study on bullying.
  • This experiment involves observing students on a playground.
  • The officer realizes, almost too late, that his presence, in uniform on the campus would be reactive because potential bullies would be deterred
  • He decides instead to use non-reactive security cameras.

Making Research Real 9.6 – Self Esteem Among Child Abuse Victims (p. 203)

  • A researcher inadvertently assigns a higher percentage of girls to the experimental group.
  • Gender plays a role in the effect of self-esteem
  • This selection bias may affect the outcome of the research.

The Benefits and Limitations of Experimental Research (p. 204)

Table 9.3 - The benefits of experimental research.(p. 205)

  • Ability to isolatethe effect of an independent variable on a dependent variable
  • Ability to measurehow much of an effect a treatmenthas on an outcome
  • Ability to demonstrate causality, or cause and effect

Table 9.4 - The limitations of experimental design. (p. 206)

  • Requirement of much time, money and control
  • Potential for serious ethical concerns
  • Possible lack of feasibility

Note to instructors: In this section the research method is discussed within the context of a relevant research project. The material is organized by the generic research process that was presented in Chapter 2. The story (i.e. the research project that provides the context) is presented in a series of set out boxes called Developing the Method. These are repeated here to allow instructors to use the research to teach the concepts, tools and techniques related to this research method.

The Experimental Research Process (p. 206)

  • Reading about how a research method should be conducted is important.
  • However, learning a new research method is easier when you can see how another person applied the method in an actual research situation.
  • In this section, we will take a look at how a researcher might implement an experimental design following the basic steps of the research process outlined in Chapter 2.
  • At the end of each research step, you will notice a box called “Developing the Method”.
  • Within these boxes, you will read about an actual research experiment in the field of criminology. We begin with a general introduction of this study.

Developing the Method 9.1 - A Case Study in Experimental Research (p. 206)

One of the most difficult questions in American policing is how to effectively respond to domestic violence incidents. Calls for service to locations experiencing domestic violence are among the most dangerous of policing activities. Strong emotions, drug and alcohol abuse, and/or severe economic problems are present in nearly every domestic violence incident. These are very personal events and the police are often viewed, even by the victims, as interlopers into a private matter. How to respond to these calls and how to handle cases of alleged domestic violence, therefore, are important to the police.

In 1984, Lawrence W. Sherman and Richard A. Berk published an article called “The Specific Deterrent Effects of Arrest for Domestic Assault” in the American Sociological Review (1984a). The research considered whether the threat of an arrest had a deterrent effect on perpetrators of domestic violence. The research was conducted from 1981 to 1982 in Minneapolis, Minnesota with the cooperation of the Minneapolis Police Department and the Police Foundation. The project was funded through a National Institute of Justice grant.

The results of this research had a profound effect on the policing procedures relating to domestic assault and violence (Buzawa and Buzawa, 1990). Specifically, policing leaders throughout the nation reconsidered their long standing policies and procedures for dealing with domestic abuse. In short, this research was a ‘game changer’ in criminal justice practice.

Ask a Research Question (p. 207)

  • Because of their ability to isolate the effect of a single variable on an outcome, experimental designs are most often used in explanatory research. Experiments are appropriate for both pure and applied research purposes.

Developing the Method 9.2 - Asking a Research Question in Experimental Research(p. 208)

Before the 1980s, the standard response to domestic violence in most police departments was to not get involved. Police officers were routinely trained that domestic assault was a private matter not warranting an official response, such as an arrest. In fact, in only the most egregious cases were police officers even allowed to arrest an individual they suspected of being guilty of domestic assault. In addition to being emotionally charged, domestic assault cases often involve individuals who pose a real safety risk to the police officers who respond to the call. Moreover, the victims of domestic violence, usually women, are often hesitant to testify against their domestic partners in court. In many cases, the alleged abuser is the principal wage earner in the family and without their income, the family might be destitute. Regardless of the reasons, a victim’s unwillingness to testify makes it very difficult for a police officer to convince a prosecutor to file charges, especially since the officer does not witness the abuse in most cases.