In-class Problems: Study Design
For each problem below, you are given a method for collecting data and a corresponding conclusion. If the conclusions are justifiable from the method of data collection, explain why you think the method of data collection is effective. If the conclusions are not justifiable from the method of data collection, say why you think the method of data collection is ineffective. Don't assume that all the study designs below are ineffective; some may be perfectly reasonable.
For the surveys, assume there is no ambiguity in question wording. For the causal studies, assume the definition of any treatments and response is specific. In other words, don't worry about the method of stating the question, just focus on the data collection. Assume sample sizes are sufficiently large.
Write at most three sentences for your answer (graders only will read up to the first three sentences).
1. A survey is carried out by the finance department to determine the distribution of household size in a certain city. They draw a simple random sample of 1,000 households. After several visits, the interviewers find people at home in only 653 of the sample households. Rather than face such a high nonresponse rate, the department draws a new batch of households, and uses the first 347 completed interviews in the second batch to bring the sample up to its planned strength of 1,000 households. The department counts 3,087 people in these 1,000 households, and estimates the average household size to be about 3.1 persons. Is this estimate likely to be too low, just about right, or too high? Explain concisely.
2. On October 20, 1993, the San Francisco Chronicle reported on a survey of top high school students in the U.S. According to the survey, “Cheating is pervasive. Nearly 80 percent admitted dishonesty, such as copying someone’s homework or cheating on an exam. The survey was sent last spring to 5,000 of the nearly 700,000 high achievers included in the 1993 edition of Who’s Who Among American High School Students. The results were based on the 1,957 completed surveys that were returned.”
a) Do you think that the survey provides evidence that roughly 80% of high school students are cheating? Explain why or why not.
b) Do you think that the survey provides evidence that roughly 80% of the students in Who’s Who Among American High School Students are cheating? Explain why or why not.
3. In the early 1990s, California evaluated a program to rehabilitate prisoners before their release. The object of the program was to reduce the recidivism rate, which is the percentage who will be back in prison within two years of release. The program involved several months of “boot camp”, military-style basic training with very strict discipline. Admission to the program is voluntary. According to a prison spokesman, “Those who complete boot camp are less likely to return than other inmates” (San Francisco Chronicle, October 7, 1993).
a) What are the two treatments in this causal study? (Identically, what are the treatment and control groups?)
b) Is the prison spokesman’s comparison based on an observational study or a randomized experiment? Justify your choice.
c) True or false: the data are evidence that the boot camp worked. Justify your choice.
4. People who get lots of vitamins by eating five or more servings of fresh fruit and vegetables each day (especially cruciferous vegetables like broccoli) have much lower death rates from colon cancer and lung cancer, according to many observational studies. These studies were so encouraging that two randomized controlled experiments were done: treatment groups were given large doses of vitamin supplements, while people in the control groups just ate their usual diet. One experiment looked at colon cancer, the other looked at lung cancer.
The first experiment found no difference in the death rate from colon cancer between the treated and control group (Greenberg, et al., 1994). The second experiment found that beta carotene (as a diet supplement) increased the death rate from lung cancer (Heinonen, et al., 1994).
a) True or false, and justify your choice: The experiments confirmed the results of the observational studies.
b) True or false, and justify your choice: The observational studies could have easily reached the wrong conclusions due to confounding. People who eat lots of fruit and vegetables have lifestyles that are different in many other ways, too.
c) True or false, and justify your choice: The experiments could have easily reached the wrong conclusions due to confounding. People who eat lots of fruit and vegetables have lifestyles that are different in many other ways, too.
5) A headline in USA Today in 1998 announced that “Prayer can lower blood pressure”. The opening sentence of the news story stated, “Attending religious services lowers blood pressure more than tuning in to religious TV or radio, a new study says.” The news story was based on a journal article (Davis, 1998) describing a study done by the National Institute of Health. The study followed 2391 people aged 65 or older for six years. One of the principal findings in the article was that, “People who attended a religious service once a week and prayed or studied the Bible once a day were 40% less likely to have high blood pressure than those who don’t go to church every week and prayed and studied the Bible less” (Davis, 1998). Comment on the whether the news paper is correct; that is, comment on whether this study proves that prayer reduces blood pressure.
6) A researcher conducted an experiment to see if her meditation technique helps people relax. First, she interviewed every person in the study to record their background characteristics. Then, she randomly assigned the subjects to two groups. The researcher taught one group her meditation technique for relaxation, and simply told the other group to relax the way they usually relax. After each person used his or her assigned relaxation technique, the researcher re-interviewed the person and rated his or her relaxation level. Based on the ratings, the group that used meditation was significantly more relaxed than the group that did not use meditation. The researcher attributes the difference to the effect of her meditation technique.
7) Here is a story from The Washington Post (1993).
”Challenging the common assumption that guns protect owners, a multi-state study of hundreds of homicides has found that keeping a gun at home nearly triples the likelihood that someone in the household will be slain there. The study, published in New England Journal of Medicine, studied the records of three populous counties surrounding Seattle, Washington, Cleveland, Ohio, and Memphis, Tennessee. The counties offered a sample representative of the entire nation because of the mix of urban, suburban, and rural communities. Although 1860 homicides occurred during the study period, the team looked only at those that occurred in the homes of the victims—about 400 deaths. The researchers found that members of households with guns were 2.7 times more likely to experience a homicide than those in households without guns.”
From this study, can you conclude that owning a gun causes people to have higher likelihoods of experiencing a homicide than not owning a gun? Defend your answer. Be specific in your defense, referring to strengths or deficiencies of the study design.
8) According to a poll of scientists reported in Science (Mervis, 1998), 82% of scientists “strongly or somewhat agree” with the assertion, “The U.S. public is gullible and believes in miracle solutions.” Another 82% agreed with the assertion, “The media do not understand statistics well enough to explain new findings.” What further information about the data collection would you like to have to decide whether or not to trust the results of the survey? List at least two, and no more than three, pieces of information (answers with at least two correct pieces of information will receive full credit). Assume the wording of the questions is not a problem. The survey was mailed to 1400 scientists.