Writing Good Questions: Part 1

Slide 1

The goal of this lecture is to provide some basic guidelines and rules of thumb for designing good questions.

Slide 2

A questionnaire is only as good as the questions it asks. Questionnaire designers must be careful that their questionnaires are as well designed as possible and that their questions are as precise and as easy to answer as possible.

Slide 3

This Calvin and Hobbes cartoon illustrates why we should care about the quality of questions we ask. We could design questions in such a way that any set of responses is possible. As this cartoon shows, Calvin agrees with many statements about graphic violence until the question is asked a certain way. Suddenly, it becomes much more difficult for him to answer affirmatively. The same is true of questionnaires that we would write. Fielding a questionnaire is meant to reduce decision-related uncertainty. The point is to query respondents and to assess their attitudes and behaviors accurately; hence, questions should be asked in a way that yields the most accurate responses. What managers do with those responses is up to them.

Slide 4

This cartoon illustrates why we should care about the preciseness of questions we ask. The advertising slogan indicates this television is preferred 2-to-1 by service people. Is that good? Do service people prefer this television because it breaks down repeatedly and thus gives them multiple opportunities to earn money fixing them, or do they prefer this television for their own use because they’re relatively reliable? In other words, the statement is ambiguous. We should strive to avoid this type of ambiguity with our questions.

Slide 5

The first thing we must do to ensure we ask good questions is to plan what we’ll measure. We must know the required information and the targeted respondents for our questions. Different respondents will have different abilities to answer different types of questions. Once we understand the target, then we’ll have a better sense for how to position our questions. Finally, we must know the data collection method. We can ask far more complex questions with many more choices in a self-administered mail questionnaire than we can in a telephone-administered questionnaire, in which the responses choices must be few and straightforward.

Slide 6

Think about open-ended questions being equivalent to essay or short answer questions and close-ended questions being equivalent to multiple-choice questions.

Slide 7

The advantages of close-ended questions are as follows:

  • The communication skills required to answer such questions are lower.
  • Respondents can answer rapidly.
  • Such questions are easy to answer.
  • Responses to these questions are easy to code and enter into a computer.
  • It’s easier to analyze responses to close-ended questions.
  • Either little or no interviewing skill is needed to administer close-ended questions, which is why close-ended questions often appear on mail surveys, internet surveys, or other self-administered surveys.

The disadvantages of close-ended questions are as follows:

  • It’s impossible to obtain in-depth answers. Respondents merely are looking at several options and picking the one that most represents their opinions and/or behaviors.
  • Close-ended questions are poor for providing insights. Such questions assume we already know the likely responses and we’re merely asking respondents to pick one. Any response beyond what we’ve provided respondents is beyond the scope of the study and it’s unlikely that respondents will volunteer insights into unlisted reply options.
  • It’s harder to write good close-ended questions because they must anticipate all possible responses.
  • The provided answers may not fully reflect respondents’ attitudes. We ask respondents to indicate the choice that’s most reflective, but that most reflective response could be very close or not very close.
  • Finally, by providing the possible responses, the interviewer is hinting at the right answers. Perhaps respondents would’ve answered differently; yet, once they see the answers, they sense what comprises an appropriate answer and respond accordingly.

Slide 8 (No Audio)

Slide 9

Here’s an example of the problems introduced by hinting at the correct answers. Suppose we want to ask undergrads how many hours a day they study and watch television. We can provide two different choice sets for answering these questions. We could provide the choices under version ‘A’, which range from less than ½ hour daily to more than 2½ hours daily. Alternatively, we could present the choices under version ‘B’, in which the first five choices from version ‘A’ are lumped into a single 2½ hours or less daily choice and the last choice from version ‘A’ is split into five choices, which range from 2½ to 3 hours daily to more than 4½ hours daily. The percentages in the table indicate that choice format strongly influences students’ answers about their daily studying and television viewing time. If we ask the question as 2½ hours or more versus less than 2½ hours for studying, as per version ‘A’, then far fewer students report studying more than 2½ hours daily; compare the 23% for mail surveys, the first number in the upper-left-hand corner, to the 69%, the third number in the same row. Similarly, self-reported television viewing varies by choice format. If asked as per version ‘A’, with more of the fewer or shorter periods, versus version ‘B’, with more of the longer periods, students indicate they watch fewer hours of television daily.

Slide 10

The relative advantages and disadvantages of open-ended questions are basically the reverse of the advantages and disadvantages of close-ended questions. The advantages of open-ended questions are that respondents have an opportunity to provide a wide range of responses. Such questions encourage people to respond, and because these responses may be unexpected, they encourage additional probing. In general, open-ended questions encourage additional probing about respondents’ answers.

However, the disadvantages of such questions are numerous.

  • To some extent, responses to open-ended questions are weighted unintentionally by respondent articulateness, as more articulate respondents will say more. Given the way responses are entered into a computer, more words will count more than fewer words.
  • Another disadvantage is interviewer bias. Open-ended questions do not lend themselves to administered questionnaires; they’re best used with a live interviewer, either in person or via telephone, and interviewers are not all alike. Therefore, inter-interviewer differences introduce additional response bias.
  • It’s harder to code and record responses to open-ended questions. Imagine asking people questions and then writing what they said. Although audio or video recording seems a great alternative, many people are reluctant to allow themselves to be audio or video recorded. As a result, the only record of their answers is whatever interviewers can scribble as quickly as they can scribble it. Often, those scribbles are incomplete and erroneous.
  • As for coding inconsistencies and difficulties, any post-data-collection numerical analysis would require examining every response to every question, developing basic categories that represent all possible responses, revisiting all the answers, and then assigning a numeric code to each one. Such an effort requires much additional time and effort. This is one reason why I urge you, if you’re conducting a survey project for this course, to use close-ended questions. They may take longer to write, but they’ll make your life easier for subsequent data analysis.
  • Open-ended questions are more difficult to use for cross-study comparisons, where choices and contexts change over time.
  • Finally, open-ended questions are more costly because their best use requires live—and expensive—interviewers and additional costly data handling (associated with recording, coding, entry, and tabulation).

Slide 11

Here are three basic guidelines for writing good questions that relate to research proposals. A proposal requires you to develop a research problem and related set of research questions. The three guidelines are as follows: (1) Create specific questions only after you have thoroughly thought through your research questions, which you’ve formalized in writing. (2) When you’re working on a questionnaire, constantly refer to your research questions. (3) For each question you write, understand how responses to that question will help you to answer your research questions. If you’ve written a question that’s unrelated to your research questions, then that question should be deleted from your questionnaire.

Slide 12

Now I’ll discuss several examples of why language is critical. This cartoon introduces that idea. Here, Junkyard Dog refers to a dog that works in the security division of an automobile wreckage site. That particular definition denotes something vastly different from the historical use of Junkyard Dog.

Slide 13

One rule for writing good questions is to be certain they are as clear and precise as possible. Consider these two examples; neither one is great but version ‘A’ is better than version ‘B’. Version ‘B’ asks how frequently do you drink coffee—extremely often to never? Although seemingly straightforward, the question is ambiguous because different people may mean very different things by drinking coffee very often. One person may believe that drinking coffee very often means drinking a cup every few hours, yet somebody else may believe that drinking coffee very often means drinking a few cups per week. Obviously, it’s a problem when different people provide the same answer but refer to vastly different behaviors or attitudes. Version ‘A’ of the question is a bit better because it’s a bit more specific: how many cups of coffee do you drink in a typical work day? Nonetheless, the question is ambiguous is several ways. What’s a typical work day? What’s a cup of coffee; an 8 oz. cup or a 32 oz. mug? Version ‘A’ is better than Version ‘B’, but Version ‘A’ could be made far more precise. In addition, asking people to model their own behavior is problematic.

Slide 14

This slide hints how version ‘B’ from the previous slide about coffee consumption might be improved. The quantifiers for the question “How often did you attend religious services during the past year—never, rarely, occasionally, or regularly?” could be more precise. We’ve replaced those with not at all, a few times, about once a month, two to three times a month, once a week, and more than once a week. This form doesn’t ask someone to remember behaviors back more than roughly one month, and as most people only attend religious services once a week and such services are relatively memorable, the response set is reasonable.

Slide 15

These next two slides reinforce the point about people differing markedly in the way they interpret vague quantifiers. The two questions address frequency of movie attendance and beer consumption. The response categories are quite vague.

Slide 16

This slide shows the extensive range of responses to vague quantifiers. In terms of annual movie visits, very infrequently was interpreted as 0 to 12 times, infrequently was interpreted as 2 to 52 times, frequently was interpreted as 4 to 104 times, and very frequently was interpreted as 12 to 265 times. In fact, respondents could have said very infrequently and meant the same thing, if they were thinking 12 times annually, as respondents who answered very frequently. So, two vastly different answers correspond to the identical set of frequency behaviors. The category quantifiers for beer are flawed similarly. Hardly any, in terms of 6 packs monthly, is reasonably consistent with the 0-1 range, but quite a lot is from 1 to 22 6-packs monthly and very much is from 1 to 40 6-packs monthly. Once again, one person may answer hardly any and mean one 6-pack monthly, yet someone else could answer very much and also mean one 6-pack monthly. Again, it’s a problem when people who behave differently answer similarly about their behavior.

Slide 17

Good questions provide respondents with categories that are either mutually exclusive or exhaustive. Exhaustive means all possible answers are present and mutually exclusive means that one answer precludes all other answers. The choices in the sample question here are not mutually exclusive, although they are exhaustive because all incomes from 0 to infinity are represented. The choices aren’t mutually exclusive because there’s no clear answer if the respondent’s income is $15,000 or $25,000. If it’s $15,000, is the correct answer #2 or #3? If it’s $25,000, is the correct answer #3 or #4? Again, there is needless ambiguity.

Slide 18

The first of three questions on this slide is an example I hope never to see in any questionnaire written by a student in this class. The choices are muddled; radio, television, someone at work, while at home, and while traveling to work, are different kinds of sources and places. Instead of muddling sources and places, it’s better to ask two separate questions. In the revised version, question #1 asks “From which one of these sources did you first hear about the tornado?” and question #2 asks “Where did you first hear: work, home, or traveling somewhere else?”

Slide 19

It’s important that our questions use language that is natural and familiar to respondents. Here’s an example of ensuring that language is familiar. Remember, a questionnaire is meant to assess respondent’s attitudes and behaviors; it’s not intended as a vocabulary test. The first version of the question about intoxication reads: “In the past year, how often did you become intoxicated while drinking any kind of alcoholic beverage?” Respondents answer relative to a given response set. Although the choices may be problematic, the major problem with the question is the word intoxicated. People tend to mean vastly different things by that word. Many people mean the legal limit, which in New Mexico is 0.08 blood-alcohol level. Other people may mean being fall down drunk or being totally out of control. There’s a vast difference between that latter alcohol level and being ever so slightly in excess of the 0.08 level. Rather than assume (or hope) that all respondents will interpret intoxicated similarly, the solution suggested at the bottom of the slide is reasonable. Instead, respondents could be told the following: “Sometimes people drink a little too much beer, wine, or whiskey, so that they act different from usual. What word do you think we should use to describe people when they get that way so that you will know what we mean and feel comfortable talking about it?” By describing intoxication in terms of a modification in behavior, people are more likely to think about the same consumption level when responding to the question about consumption frequency.

Slide 20

Here’s another example about natural and familiar language. There are regional differences in the terms that people use to describe the identical thing. For example, here are five different terms for the type of sandwich you can buy at Subway. In some areas of the country, those sandwiches are called grinders; in other areas, hoagies. In Chicago, they’re called hero sandwiches; in New York, they’re called submarine sandwiches; and in Texas and Louisiana, they’re called poor boys. Although it’s the same sandwich, respondents should receive a version of the questionnaire with the regionally accepted term for it. Alternatively, the questionnaire could include a description of the sandwich’s physical characteristics. Regardless, sensitivity to regional differences and terminology is important.

Slide 21

Another rule to remember is to avoid questions that lead to a certain conclusion. For example, consider the question “What did you dislike about the product you just tried?” If the respondent didn’t dislike anything, he or she is forced to think about something that may have been less than totally satisfactory or to answer the question in the way it was asked. This is a biased question. Instead, the question should be divided into two parts: (1) Did you dislike any aspect of the product you just tried, yes or no? and (2) If yes, then what aspect and why is it important? Another example (although somewhat dated) relates to Johnson & Johnson poisoning episode in Chicago back in the 1970s. Asking people “Do you think Johnson & Johnson did everything possible in its handling of the Tylenol poisoning situation?” is not a good way to assess people’s attitudes towards the firm’s actions. Did they do everything possible? Obviously, the firm could have done something. Nonetheless, business ethics case studies written about this episode tend to reinforce the firm’s highly ethical behavior. Asking the question “Did they do everything possible?” suggests Johnson & Johnson’s behaviors were problematic when they weren’t.

Slide 22

Here’s another example of a leading question and a way to revise it to remove the bias. In the leading version, respondents are asked “Do you believe private citizens have the right to own firearms to defend themselves, their families, and property from violent criminal attack?” This is a rather emotional issue when posed this way. Most people would answer yes to the question “Do I have the right to protect my family?” As a result, it’s a leading question because it’s emotionally laden. In the improved version, the question is “Do you believe a ban on private ownership of firearms would significantly reduce the number of murders and robberies in your community?” This revised version no longer raises family defense issue, so its emotional overtone is markedly reduced. It’s less likely that someone would automatically answer a question of the second type than of the first type.