Political polls do a remarkably good job of predicting the winner of national elections. However, we occasionally hear of a case where a poll has gone badly wrong. A recent example was the 1992 election in Colorado, where a measure was on the ballot to prohibit the state legislature and cities from passing anti-discrimination laws concerning homosexuals. Polls showed that the Colorado measure would be defeated, but the measure passed (See the New York Times, November 8, 1992). What may have accounted for the difference between the result in the polls and the result in the election? In this activity you will examine how the design of a survey can affect the answers that are given.

PART ONE

Within your group (2-3 people), pick a topic of interest to you and write a questionnaire that contains about 10 items about this topic; at least 3 should be based upon national statistics you have discovered in research. For example, I have gathered the following from a website on teen statistics:

“More than 40% of teens who admitted drinking said they drink when they are upset; 31% said they drink alone; 25% said they drink when they are bored.”(Make sure to cite your statistical sources).

Use your researchednational statistics to write the following survey:

  1. Do you drink?
  2. If so, do you:

a. Drink while upsetb. Drink alonec. Drink when bored

You will use this data to calculate at least three confidence intervals (in this example, a result might be: “Through the data we gathered, we are 95% confident that the true population proportion of teens who drink when they are upset is between 25 and 32%...”).

Decide upon a sample size that will be large enough to establish the statistical significance of any difference you feel is important. Randomly select your classes to survey (as an example, use the Master Schedule online to randomly select your classes, then contact the teacher during lunch or non-instructional time to ask permission to survey).

PART TWO

Now, you will introduce a bias into your survey; you will be attempting to sway the results by changing your original survey somehow. For example, I might introduce the following paragraph (verbal or written) before my second survey to try and influence the results:

“Research indicates that the human brain continues to develop into a person’s early twenties and that exposure of the developing brain to alcohol may have long-lasting effects on intellectual capabilities. Exposing the brain to alcohol during this period (i.e, before age 21) may interrupt key processes of brain development; alcohol can do long-term and irreversible damage to critical neurological development that is ongoing during the teen-age years. Further, research shows the human brain doesn't stop growing until about age 21 or 22, and that alcohol consumption can alter or retard that growth, including memory and test-taking ability.”

In this example, I would give anunbiased questionnaire to my first set of classes and then give an biased questionnaire to my second set of classes.

Here are some ideas for introducing bias into your survey:

  1. Does the order in which 2 candidates appear on a ballot make a difference in the percentages of votes they receive?
  2. Is it possible to word a question in 2 different ways that are logically equivalent but that have a different percentage of students agree with them?
  3. Does the order in which 2 statements appear in a survey make a difference in the percentage of students who agree with them?
  4. Can the percentage who agree with a statement be changed by having respondents read some introductory material?
  5. If a statement is rewritten to be logically equivalent but to have a more complicated sentence structure and bigger words, will it affect the percentage of students who agree with it?
  6. Does the appearance of the interviewer make a difference in how students will respond to a question? For ex, do students tend to respond the same way about a controversial issue when the interviewer is female as when the interviewer is male?
  7. If the interviewer does not know how a student responds (as on a secret ballot), does it make a difference in the percentage of students who agree with a controversial statement?
  8. If a student knows absolutely nothing about an issue, will he or she give an opinion anyway? Will students admit if they don’t know the answer to a question?
  9. Do students report events (such as how many days last week had rain or the description of a person who just walked by) as accurately as they think they do?
  10. If you let students volunteer to be in your poll, do you get a different result than if you approach students?

Here’s an example:

Unbiased (to be given to Classroom #1):

  1. Do you think Asians are smarter than Whites?
  2. Yes b. No
  3. Do you think wealth has an affect on GPA?
  4. Yes b. No
  5. Do you think brunettes are smarter than blondes?
  6. Yes b. No
  7. Do you believe Asians are better at drawing anime than Whites?
  8. Yes b. No
  9. Do you think health has an affect on GPA?
  10. Yes b. No
  11. Who do you think is smarter: males or females?
  12. Malesb. Females
  13. Do you think drugs have an affect on SAT scores?
  14. Yes b. No
  15. Does using big words in everyday conversation affect success in school?
  16. Yes b. No
  17. Who has a higher GPA: artist or jock?
  18. Artistb. Jock
  19. Do you think young teen Mexicans are more prone to pregnancy?
  20. Yes b. No

Biased (to be given to Classroom #2):

  1. For the 2007-2008 school year, the mean average of all seniors who took the math SAT that were Asian was 581 and for Whites was 537. Do you personally think Asians are smarter than Whites?
  2. Yes b. No
  3. Sarah and Mary both go to the same school. Mary always wears designer clothes and is extremely organized, while Sara is very messy and wears ripped or dirty clothes. Who do you think has a higher GPA?
  4. Maryb. Sara
  5. Statistics say that, in general, brunettes have a higher IQ score than blondes by nearly 30 points. Do you think brunettes are smart than blondes?
  6. Yes b. No
  7. 63% of anime is drawn and created by Japanese. Who do you think draws anime better, a Japanese or a White person?
  8. Japaneseb. White
  9. Jenny is overweight and doesn’t exercise. Susan is skinny and eats healthy. Who do you think has better grades in school?
  10. Jennyb. Susan
  11. The average math SAT score for males is 534 and 499 for females. Do you think males are smarter than females?
  12. Yes b. No
  13. Billy took the SAT for the first time and received 2300, while Bobby received 1320 on his first time. One of these boys uses drugs. Which one do you think it is?
  14. Billyb. Bobby
  15. Larry uses high vocabulary when he talks, making it hard for people to understand. Ronnie talks on a second grade reading level, saying like every other word. Who does better in school?
  16. Larryb. Ronnie
  17. Joe is a jock. He plays football, soccer, and lacrosse. Julia is an artist, and plays no sports at all. Who has a higher GPA?
  18. Joe b. Julia
  19. One in every five young teenage Mexicans is pregnant. Do you believe Mexican teenagers get pregnant more than Whites?
  20. Yes b. No

Wrap Up: Your group should prepare a PowerPoint about what you have learned and present it to the rest of the class. In your presentation, consider the following:

  1. Explain your randomization design (stratified, cluster, etc.) and be sure to reference it (email it to me: 5 pts!); that is, explain how you picked your random sample (less points awarded for convenience sampling or a voluntary response sample).
  2. Analyze and summarize the results in a meaningful way (for example, compare and contrast your “biased” results vs. your “unbiased” results); include graphs. Consider the following:

 How did changing the survey change your results?

Was there any bias you noticed that you did not yourself introduce?

Did you discover any problems with the bias you introduced?

How did it affect the outcome?

  1. Useresults from your unbiased survey to do inference. Use Chapter 10 concepts to calculate 3+ confidence intervals based upon your results (make sure to use PATC!).
  2. Did your study suffer from other types of unintendedbias (for example, undercoverage or nonresponse)?
  3. Did you discover anything unexpected in your study?
  4. If you were to repeat this survey, how would you change your procedure(s)?
  5. Does your data/confidence data match/come close to your national statistics? (Discuss your research here).

Special Notes:

  1. When presenting your data, make sure you reference the question. For example, don’t write “Q2 of biased survey saw 90% agreed with the statement whereas Q2 of unbiased survey saw 45% agreed with the statement). It’s hard to make your data meaningful if you don’t reference it!
  2. Include a clear introduction, analysis, and conclusion.

1