Writing Effective Survey Questions
Training I for the Great Lakes Social Science Network on December9, 2011
Prepared by Chris Ellis, Coastal Services Center and Caitie McCoy, Illinois-Indiana Sea Grant
For our first training, we will review the Survey on Public Knowledge of Coldwater Lake Shoreline: Fish and Wildlife Habitat Issues. We have modified the survey to contain mistakes; we will discuss what we like about the survey and how it could be improved with changes or additions.
Included is a list of 25 guidelines, which will enhance your ability to critique and improve survey questions (adapted from Survey Research and Analysis by Jerry J. Vaske).
As we go through the survey, find which guidelines are represented in the survey questions. Write notes on the worksheet and survey throughout the activity. Note: all guidelines below may not apply to the survey exercise.
25 Guidelines for Writing Effective Survey Questions
Adapted from Jerry J. Vaske’sSurvey Research and Analysis
Guideline 1: Identify exactly what kind of information you want respondents to provide.
Guideline 2: Use fixed-scale or close-ended questions rather than open-ended questions wherever possible.
Guideline 3: Choose simple words and phrases over more specialized alternatives.
Guideline 4: Use as few words as possible to ask a question.
Guideline 5: Use complete sentences; avoid cryptic phrases when asking questions.
Guideline 6: Avoid vague quantifiers if precise estimates can be obtained.
Guideline 7: Use an equal number of positive and negative responses for scale questions.
Guideline 8: Distinguish “neither” from “no opinion” in response options.
Guideline 9: Avoid double-negatives or asking people to say “yes” in order to mean “no.”
Guideline 10: Avoid double-barreled questions.
Guideline 11: Make every question count.
Guideline 12: Use an appropriate timeframe for questions and responses.
Guideline 13: Reduce impact of sensitive or objectionable questions.
Guideline 14: Use terminology that makes sense to respondents and define unclear terms.
Guideline 15: Define abbreviations and acronyms that respondents may not understand.
Guideline 16: Avoid bias from unequal comparisons.
Guideline 17: Avoid using leading or loaded introductions and questions.
Guideline 18: State both sides of scales in question stems or introductions.
Guideline 19: Minimize number of “check all that apply” questions.
Guideline 20: Make response categories mutually exclusive.
Guideline 21: Make sure that each question or statement is accurate.
Guideline 22: Select questions and responses that permit comparisons with existing information and data.
Guideline 23: Provide information instead of assuming too much respondent knowledge.
Guideline 24: Avoid questions that are too difficult for respondents.
Guideline 25: Use multiple questions to measure complex concepts.
Writing effectivesurvey questions:
Guidelines that may not have been covered in training
Adapted from Jerry J. Vaske’sSurvey Research and Analysis
Guideline 9: Avoid double-negatives or asking people to say “yes” in order to say “no.”
Notes: Double negatives are confusing and make intended responses difficult to achieve.
Example: “Do you oppose not being allowed to fish for Lake Trout?”
□Yes
□No
□Unsure
A better way to ask this question would be “do you support or oppose current Lake Trout restrictions?”
Guideline 13: Reduce the impact of sensitive or objectionable questions.
Notes: Questions that are perceived as sensitive or objectionable in nature may lead to nonresponse. Open-ended questions can be especially challenging in regard to this guideline.
Example: “What was your total annual household income in 2010?”
$______
A better approach to this question would be to provide income categories/choices from which to select the range of one’s household income level.
Guideline 21: Make sure that each question or statement is accurate.
Notes: Basically, avoid errors in your survey. Have the survey team proofread multiple times, and have a couple external people review it. Errors can range from spelling and grammar to inaccurate information.
Example: “The bald eagle is an “endangered” species within the Great Lakes Basin Ecosystem, meaning that it is at a high risk of becoming extinct in the imediate future. Should the U.S. federal government provide more funding for research on this species?
□Yes
□No
□Unsure
There are two major problems. First, the bald eagle is a federally-listed “threatened” species. Second, “imediate” is misspelled; it should be spelled “immediate.”
Guideline 22: Select questions and responses that permit comparisons with existing information and data.
Notes: One of the great things about quantitative survey research is the ability to compare results to other studies. To compare data, researchers must ask for information in the same way as earlier studies; this is measurement consistency. When possible, keep question wording and response options the same. Certain academic fields are even partial to certain question phrasing or response scales.
Example: One great example is demographic information, which when phrased the same as the Census, can then easily be compared to Census data.
Example II: Recreation “crowding” literature is partial to a 9-point scale that ranges from “Not at all crowded” to “Extremely crowded” when asking the extent to which recreators feel crowded. Using a 9-point scale in future work to measure this concept would allow for easy comparisons across studies. Using a 5-point or 7-point scale would make comparisons difficult. Consult the literature to see if your field has a certain way of doing things.
Guideline 24: Avoid questions that are too difficult for respondents.
Notes: Don’t ask respondents to make unnecessary calculations, like reporting percentages. It increases the respondent burden, which leads to bad data.
Example: What percent of days that you fished in the last three months were spent at Foster Lake?
_____ % of days fishing in last three months spent at Foster Lake
This question would be difficult for avid anglers who may have fished multiple days at multiple locations. Instead, separate it into two questions, and have the computer report the percentage for you:
Approximately how many days did you go fishing in the last three months?
I fished approximately _____ days in the last three months.
Now, approximately how many of these days did you go fishing at Foster Lake?
I fished approximately ____ days at Foster Lake in the last three months.
Guideline 25: Use multiple questions to measure complex concepts.
Notes: Respondents will answer questions the same about topics that are more factual like demographics or a past behavior whether you ask them today or three weeks from now. Questions about difficult or complex concepts, like beliefs, norms, attitudes, and intentions are prone to measurement error because they may change from day to day or may require a lot of thought from the respondent.
Example: To what extent do you disagree with the statement: “A moose hunt in Anchorage would benefit the future moose population in the area?”
□Strongly disagree
□Somewhat disagree
□Neither agree nor disagree
□Somewhat agree
□Strongly agree
No single question is a perfect measure of a complex concept like an attitude. Researchers typically use scales asking multiple questions to measure single complex concept. Statistics like reliability analysis can help analyze patterns, which then justifies combining answers into a single estimate (ex. average) of the concept. For example (modified from Whittaker et al. 2001)…
Please tell us whether you think each of the following outcomes of the proposed moose hunt would be good or bad.
Very bad / Slightly bad / Neither / Slightly good / Very goodReducing the number of accidents involving moose is… / 1 / 2 / 3 / 4 / 5
Providing more hunting opportunities for moose hunters is… / 1 / 2 / 3 / 4 / 5
Being able to see moose as often as I like is… / 1 / 2 / 3 / 4 / 5
A hunt that prevents nonhunters from using the area is… / 1 / 2 / 3 / 4 / 5
Holding a hunt that might injure someone else is… / 1 / 2 / 3 / 4 / 5
Keeping moose from becoming overpopulated is… / 1 / 2 / 3 / 4 / 5