Draft: Do Not Quote Or Cite Without Permission

DRAFT: DO NOT QUOTE OR CITE WITHOUT PERMISSION.

EVIDENCE OF SOCIAL DESIRABILITY BIAS IN CONTINGENT VALUATION SURVEYS ADMINISTERED THROUGH IN-PERSON INTERVIEWS*

Naomi S. Kleckner,a Christopher G. Leggett,b† Kevin J. Boyle,c John Duffieldd, and Robert Cameron Mitchelle

aIndustrial Economics, Incorporated

2067 Massachusetts Avenue

Cambridge, MA 02140

(617) 354-0074

bIndustrial Economics, Incorporated

2067 Massachusetts Avenue

Cambridge, MA 02140

(617) 354-0074

fax: 617-354-0463

cDepartment of Resource Economics and Policy

University of Maine

Orono, ME 04469-5782

dDepartment of Economics

University of Montana

Missoula, MT 59812

eDepartment of Geography

Clark University

Worcester, MA 01610

April 12, 2002

† Corresponding author

* We thank Mark Griswold, Marla Markowski, Bruce Peacock, and Robert Unsworth for helpful comments. All errors are the responsibility of the authors. Funding for this research was provided by the National Park Service.
ABSTRACT

The National Oceanic and Atmospheric Administration’s Blue Ribbon Panel (Arrow et al., 1993) suggests that social desirability bias associated with in-person contingent valuation surveys may be diminished by using a “ballot box” to collect responses to contingent valuation questions. This paper presents empirical evidence that social desirability bias may affect in-person surveys even when a ballot box is used. We conducted an on-site, split-sample contingent valuation survey of visitors to Fort Sumter National Monument, South Carolina. All respondents were told that the survey was being conducted on behalf of the National Park Service. We find that WTP for a fort visit is approximately 23 percent higher when the survey is administered through face-to-face interviews with a ballot box rather than being self-administered by the respondent.

JEL Categories: Q26, H40

I. Introduction

The National Oceanic and Atmospheric Administration’s (NOAA) Blue Ribbon Panel (Arrow et al., 1993) and Mitchell and Carson’s (1989) widely cited contingent valuation (CV) text both recommend in-person interviews over phone interviews or mail surveys for CV research. Mitchell and Carson write that “the in-person survey, where the interviewer conducts the interview in the respondent’s dwelling place, is the method of choice for most CV studies” (Mitchell and Carson, 1989, p. 110). The NOAA Panel concurs: “it [is] unlikely that reliable estimates of values could be elicited with mail surveys. Face-to-face interviews are usually preferable, although telephone interviews have some advantages in terms of cost and centralized supervision” (Arrow et al., 1993, p. 4611). Other researchers have been less forceful in recommending a particular survey mode. Dillman, for example, writes that “until the attributes of each method are considered in relation to the survey topic, the population to be surveyed, and the precise survey objectives, the question of which is best cannot be answered” (Dillman, 1978, p. 39).

The advantages of in-person interviews include high response rates and the ability to communicate rather complex scenarios to survey respondents. However, the survey research literature indicates that social desirability bias can occur in responses to in-person surveys (e.g., Wiseman, 1972). The NOAA Panel suggests that this bias may be diminished by using a “ballot box” to collect responses to CV questions (p. 4611). The effectiveness of the ballot box technique has not yet been documented in the literature.

This paper presents empirical evidence consistent with social desirability bias in CV surveys administered through in-person interviews, despite the use of a ballot box technique. We conduct an on-site, split-sample survey of visitors to Fort Sumter National Monument designed to estimate WTP for a fort visit. We find that WTP estimates from an in-person survey are significantly higher than WTP estimates from a self-administered survey. Although several previous studies have investigated social desirability bias in responses to CV questions, existing research either focuses exclusively on telephone versus mail comparisons (Loomis and King, 1994; Whittaker et al., 1998; Ethier et al., 2000) or compares surveys with low response rates and different sample frames (Mannesto and Loomis, 1991; Maguire et al., 2002). Thus, the existing literature fails to provide conclusive evidence of social desirability bias in CV surveys administered through in-person interviews.

II. Literature Review

Social desirability bias can be identified by comparing responses to identical survey questions administered through two different survey modes, one which makes use of interviewers and one which does not. The challenge in making such comparisons, however, is to hold constant all other mode-related factors that could potentially affect responses.

The survey research literature identifies two reasons why the data collected through surveys may vary by mode. First, the characteristics of the sample may differ across modes as a result of coverage bias or non-responsebias. Coverage bias arises when a segment of the population of interest is omitted from the sample frame (Groves, 1987; Edwards and Anderson, 1987; Dillman, 1991). Non-response bias arises when some members of the sample choose not to respond to the survey (Delecki, Whitehead, and Blomquist, 1993).

Second, even if identical samples are obtained, question responses may differ due to different survey contexts. Context effects involve the effect of survey mode on the way in which individuals interpret and answer questions (Schwarz and Sudman, 1992). Dillman et al. (1996) identify three different context-related factors that may cause differences in responses across modes. First, survey questions and supplemental materials are presented through either aural or visual communication. Second, the pace of the survey is either controlled by the respondent or by the interviewer. Third, completing the survey is either a private activity with minimal interaction with an interviewer or it is a social activity where an interviewer asks each question and records all responses.

The impact of this third context effect has been the focus of a great deal of investigation in the survey research literature. It has been shown that “people sometimes distort their answers in surveys in order to present themselves as having more socially desirable or respectable characteristics” (Krosnick 1999) and that the presence of an interviewer may lead to social desirability bias, whereby respondents provide responses that they think will please the interviewer or be consistent with societal norms (Fisher, 1993, Krosnick, 1999). For example, Atkin and Chaffee (1972-1973) find that respondents express more positive attitudes toward firefighters when they suspect that the interviewer is a firefighter. Furthermore, several studies have found that African-Americans express more favorable attitudes towards whites when their interviewer is white (Anderson, Silver, and Abramson, 1988; Schuman and Converse, 1971) and vice versa (Cambell, 1981; Cotter, Cohen, and Coulter, 1982). Social desirability bias may vary by measurement method, with self-reporting instruments and those that provide complete anonymity showing lower levels of socially desirable responses. Wiseman (1972), for example, finds that Catholics are more likely to report favoring legalized abortion and birth control in self-administered surveys than in telephone or in-person surveys.

Five recent CV studies examine the impact of survey mode on responses to CV questions (Mannesto and Loomis, 1991; Loomis and King, 1994; Whittaker et al., 1998; Ethier et al., 2000; Maguire et al., 2002). Three of the five studies compare mail surveys to telephone surveys and find mixed evidence for social desirability bias in telephone surveys.

Loomis and King (1994) investigate willingness to pay (WTP) for improvements in wildlife habitat in California and find that a mail survey yields higher WTP than a telephone survey. It would be difficult to attribute the observed difference in WTP to any type of context effect. The sample frames were slightly different and response rates were quite low and differed across modes (35% for the mail survey and 55% for the telephone survey).

In contrast, recent studies by Whittaker et al. (1998) and Ethier et al. (2000) employ identical sample frames in telephone-mail comparisons and achieve high response rates, providing an ideal opportunity to isolate the impact of social desirability bias in telephone surveys. Whittaker et al. investigate WTP for increased state park user fees in Colorado, and they find that a telephone survey yields significantly higher WTP than a mail survey. They attribute this difference to a combination of social desirability bias in the telephone survey and potential strategic bias in the mail survey.[1] On the other hand, Ethier et al. (2000) find no significant difference in WTP between mail and telephone surveys in a study of consumers’ WTP for “green” electricity. They do, however, find evidence of social desirability bias in telephone survey responses to three non-WTP questions. Telephone respondents are more likely than mail respondents to claim to have contributed to environmental causes, provide high ratings for the service provided by their power company, and express an interest in programs that involve planting trees on public lands.

Mannesto and Loomis (1991) compare mail and in-person surveys in an investigation of boaters’ WTP for increased wetland habitat in California, and they find that the in-person survey yields significantly higher WTP than the mail survey. The in-person survey was conducted by intercepting anglers and boaters at public boat ramps, while the mail survey was conducted by placing questionnaires on the windshields of vehicles with boat trailers parked at these boat ramps. With slightly different sample frames and dramatically different response rates (97% for the in-person survey and 24% for the mail survey), Mannesto and Loomis cannot disentangle the influences of non-response bias, coverage bias, and social desirability bias.

Finally, Maguire et al. (2002) compare CV responses to mail, in-person, and telephone surveys. They find no significant difference in WTP between the mail and telephone surveys, but they find that WTP in surveys administered through in-person interviews is significantly lower than in the two other modes. The use of different sample frames limit the generalizability of these results. The sample frame for the mail and telephone surveys (supplemented telephone listings) was quite different from the sample frame for the in-person survey (members of specific community organizations in the Atlanta metropolitan area).

Carson et al. (1994, p. 93) in a national CV survey using the Exxon Valdez oil spill survey instrument find no difference in responses between in-person surveys administered with and without a ballot box. However, as only one survey mode was employed, it is not possible to conclude whether social desirability bias was absent or the ballot box technique was not effective in eliminating this bias.

Thus, there is only limited evidence for social desirability bias within the existing CV literature. Further, we are not aware of any studies that successfully examine the impact of social desirability bias in in-person surveys by comparing an in-person survey with a survey conducted using an alternative mode, where the two surveys have identical sample frames and high response rates. Yet the concern about social desirability bias within the survey research literature indicates that this issue deserves further attention.

III. Study Design

The data are from a survey of visitors to Fort Sumter National Monument conducted between November 21, 2000 and November 29, 2000. Fort Sumter was the site of opening hostilities during the Civil War and is located on a small island near the city of Charleston, South Carolina (Figure 1). The fort is maintained by the National Park Service. Although the Park Service does not charge an entry fee, nearly all visitors access the fort via a concessionaire ferry. The ferry ride takes approximately 35-45 minutes one way, and the round-trip fee at the time of the survey was $11 for individuals 12 and older and $6 for children between 6 and 12 years old. Visitors are allowed to tour the fort for approximately one hour before returning to Charleston on the ferry.

The survey was administered to a split sample of adult visitors during the return trip to Charleston on the ferry. At the time of the survey, two separate ferries were operated by the private firm, departing from and returning to two different locations within the city of Charleston, City Marina and Patriots Point (Figure 1). We sampled visitors on both ferries using sampling rates that were proportional to the number of visitors. Only visitors residing outside of the Charleston area were sampled, as local visitors to the fort were addressed in a separate survey effort.[2] During each ferry trip from the fort, both the in-person and the self-administered surveys were administered to sub-samples of visitors chosen through a predetermined seating pattern. Interviewers approached a total of 951 visitors and 847 agreed to participate in the survey, providing an overall response rate of 89 percent.[3] The survey firm did not collect information that would allow us to calculate response rate by survey mode.

Half of the respondents were approached by a professional interviewer and asked to fill out a survey by hand. The respondents were informed that the survey was being conducted for the National Park Service and that it would take approximately ten minutes to complete. Each interviewer wore a pin that indicated his or her name as well as the name of the local marketing research firm conducting the survey. Interviewers returned to respondents periodically throughout the trip in order to collect completed surveys.

The other half of the respondents were approached by professional interviewers and asked to complete an in-person survey. The interviewers who administered the in-person survey were the same interviewers who handed out and collected the self-administered survey. Again, respondents were informed that the survey was being conducted for the National Park Service and that it would take approximately ten minutes. The interviewer sat next to the respondent and administered a survey that was identical to the self-administered survey with the exception of minor changes in question wording introduced to accommodate questions being asked by an interviewer. Hand cards (listing potential responses to a question) were used for some questions in order to minimize dependence upon aural communication and reduce the memory burden for respondents.

The survey instrument was pre-tested on 140 respondents (63 in-person surveys and 77 self-administered surveys). In the pre-test we used a dichotomous-choice CV question (with bid amounts of 1, 3, 5, 10, 15, and 20 dollars) for both modes, and the in-person version did not use a ballot box. The response data to the CV question are shown in Table 1.[4] An examination of the data in Table 1 indicates that WTP may be affected by survey mode when a ballot box is not used: for every bid amount greater than $3, the percentage of “Yes” responses from in-person surveys is greater than or equal to the percentage of “Yes” responses from self-administered surveys. In order to test the hypothesis that survey mode affects WTP, we estimate the following simple model using the methodology described by Cameron (1988):

Here, INPERSON is a dummy variable representing the in-person survey mode. The estimated value of is 0.403 and the asymptotic t-statistic is 1.37, which is significant at the 10% level using a one-sided test (= 2.81, t = 11.2). These results, although based on a small sample from a pre-test, indicate that a social desirability bias may be present in WTP responses when a ballot box is not used.

Based on these findings, we chose to implement a ballot box for all in-person interviews during the main survey in order to provide the respondent with anonymity in responding to the CV question. In addition, we chose to use a payment card rather than a dichotomous choice question to elicit WTP. Aside from the efficiency gains provided by payment card data, we felt that allowing respondents to see all bid levels might reduce anchoring and “yea-saying” in the CV responses. At the $15 bid, 75 percent of respondents in both samples answered yes, and at the highest bid ($20), 31 percent of the in-person sample answered yes. Finally, we added two higher bid amounts ($22 and $25) and three intermediate bid amounts ($8, $12, and $18) for the final survey.

The text of the CV question used in the main survey, which is designed to estimate the value of a visit to Fort Sumter, is reproduced in Figure 2. For the self-administered survey, the respondent simply read the question text and circled the appropriate value. For the in-person survey, the interviewer read the question text aloud and handed each respondent a payment card “ballot.” The respondent was instructed to circle the appropriate value, place the ballot in an envelope, and return it to the interviewer, who placed it in a “ballot box,” a manila envelope that contained ballots from other respondents. Similarity between the surveys was maintained to minimize the potential for survey context effects affecting CV responses.

IV. Econometric Model

The statistical analysis of payment card data is relatively straightforward. We follow the approach outlined in Cameron and Huppert (1988), which involves directly estimating the parameters of a willingness-to-pay function using maximum likelihood techniques. The intuition underlying the Cameron and Huppert approach is that an individual circling $5 on the payment card reveals that $5 WTPi < $8. Thus, this individual’s contribution to the overall likelihood function is the probability that WTPi lies between $5 and $8, conditional on a vector of explanatory variables, Xi, and a set of unobservable factors that are captured by the error term, i.

More formally, let the vector represent the values on the payment card (listed in increasing order) and let tirepresent the value selected by the ith respondent. Assume that the willingness-to-pay function can be written as where ~ N(0,2). Then the probability that the respondent will select ti is given by