Principal Investigator/Program Director (Last, First, Middle): Kapteyn, Arie

A. Specific Aims

The overarching goal of the current joint project between RAND and the University of Michigan, “Internet Interviewing and the HRS,” has been to inform the Health and Retirement Study (HRS) of the potential of Internet interviewing of its respondents. We have conducted Internet interviews of a subsample of HRS respondents and have set up a separate Internet panel (the “American Life Panel”, ALP) of approximately 1,500 respondents over 40. The ALP has been used to compare telephone interviewing and Internet interviewing; recently, we provided non-Internet users with Web TVs, which allow them to access the Internet by TV and a telephone line. Both the HRS Internet interviews and the ALP have been used to conduct a large number of pilot experiments dealing with mode effects, selectivity, new ways of measuring beliefs and preferences (including probabilities of several events and preferences for different Medicare Part D plans), measurement of health and health histories, visual presentation of tasks, and measurement of objective quantities such as wealth and consumption.

Based on these results, we propose a competing renewal to further integrate Internet interviewing into the tool box of the HRS and conduct a substantial number of new substantive experiments that will not only be specifically helpful in designing the content of the HRS interviews but also have potential applications in other surveys. These experiments will be aimed at measurement of preferences, expectations, cognition, health and well-being, as well as at supplementing current interview modes (e.g., CATI, Mail, CAPI) by Internet interviewing, high-frequency interviewing, and event-related interviewing.

We expect that experiments will lead to knowledge about how the increased use of high-quality and cost-effective Internet surveys can enhance and supplement HRS data collection.

Specifically our aims are as follows:

  1. To contribute to the long-run improvement in the HRS instrument by designing and fielding new experiments in questionnaire and question design and measurement methods, exploiting the potential of Internet interviewing, These activities will include
  2. Experimental measurement of variables such as health, consumption, well-being, expectations, and preferences, among others, with special attention to cognitive functioning.
  3. High frequency interviewing to measure expectations with respect to stock markets, house prices, or health for example.
  4. Use of mixed-mode designs combining Internet and mail surveys to determine optimal approaches to maximizing response rates and minimizing costs.
  5. Hypothetical choice experiments related to important policy issues that are important for middle-aged and older Americans, such as Medicare Part D, retirement decisions, and savings and consumption.
  6. Choice experiments with real pay-offs to increase understanding of decision-making processes, expectations formation, and preferences.
  7. To develop an Internet version of the HRS that could be administered as substitute for core HRS. This would involve
  8. Developing HRS-core modules that could be administered in successive sessions
  9. Testing these modules and piloting new content with the ALP
  10. Identifying mode and selection effects and developing ways to eliminate or correct for them.
  11. To develop Internet surveying to supplement HRS, such as follow-up interviewing and targeted interviewing surrounding retirement, health events, and so forth.
  12. To expand the use of Web TVs and explore other ways of giving Internet access to respondents lacking access, and to further work on selectivity and propensity scoring, with the aim of producing datasets with sample weights that make them representative of the general population.
  13. To make the data generated by these activities available to the research community.

B. Background and Significance

B.1. General Context

When we started the current project, several methodological issues needed addressing. In addition, we aimed to conduct experiments exploiting the advantages of the Internet as a data collection environment. We will discuss these in our progress report (Section C.1). The proposed project aims to further integrate Internet interviewing into the toolbox of HRS and to take advantage of the unique possibilities of Internet to carry out new studies that help us better understand health and retirement issues.

We discussed major aspects of Internet data collection in the Background and Significance section of the proposal that led to the current project. Here, we mainly update issues pertaining to data collection over the Internet, mostly based on recent studies. Background and significance specifically relevant to each of the proposed new experiments is mainly given in Section D.

B.2. Collection of data over the Internet

Overall, the web survey literature describes a substantial body of research on dimensions like data quality - in particular about response rates, timeliness, cost, selection, and mode effects. The literature about access gaps in the context of web surveys is sparse.

B.2.1 Internet Access and response rates

Internet access in the U.S. is increasing at a steady pace. In April 2006, 73% of the US population usedInternet ( up from 66% in January 2005. About 42% of Americans have now broadband connections at home. By contrast, telephone coverage is about 94% (Fricker et al, 2005). Americans without Internet access are more likely to be poor, poorly educated, elderly, black or Hispanic than people with Internet access. See for instance Couper et al. (2006), Robinson et al. (2003), and Bimber (2000).

Recruiting web survey participants. People without Internet access cannot participate in web surveys, leading to coverage error. Two approaches have emerged in the quest to represent the full population and not only those with Internet access. One solution is to draw a probability sample in a traditional way (e.g. using random digit dialing (RDD)) and provide Internet access to those who do not have it. Several panels (Knowledge Networks and the American Life Panel in the U.S., CentERpanel in the Netherlands) have given respondents without Internet access devices that allow them to complete surveys using their TV screen as a monitor and to hook up to the Internet using a telephone line. Because of the costs of supplying hardware, this approach is most effective in the case of a panel (in contrast to a cross-section).

The second approach does not rely on a probability sample. Instead, a volunteer panel is used and adjusted for selectivity afterwards by reweighting. Harris Interactive has a volunteer panel of 6 million people. Participants are recruited from Harris Interactive’s web site, from banner ads and other advertising. Harris Interactive conducts monthly RDD phone surveys containing a set of so called webographic questions, which are used to calibrate the web survey with the phone survey. See Schonlau et al. (2004) for more details. Other companies that have taken this approach, but without or with a limited adjustment for selectivity, include the NPD Group with 3 Million panelists globally and Carbonview research with 25,000 panelists in the US. Unusually for a web survey company, Carbonview recruits web survey respondents by approaching shoppers at malls (Enright, 2006). Other well-known panels include NFO research ( and Greenfield online (

Development of a standard contact methodology. When the Web survey is offered as part of a mixed mode strategy, an effective means for raising response rates is to first offer one mode and to offer non-respondents to take the interview using a different mode. (e.g. Dillman et al., 2001).When a single response mode is offered, respondents can be confused if pre-notification and reminder contacts are in a different mode. In a randomized experiment Schaefer and Dillman (1998) studied various combinations of mail and e-mail pre-notification, survey implementation and follow-up contact. Their survey was attached as an e-mail attachment. They achieve similar response rates when respondents are only contacted by e-mail as when they are only contacted by mail. The response rates were lower for e-mail surveys that had a mail pre-notification or a mail follow-up contact.

Unlike in Dillman’s “Total Survey Design strategy” for mail and phone surveys personalization of contact e-mails may not improve response rates, presumably because respondents are already inundated with personalized junk mail. Explicitly telling the respondent that he/she is part of a small group chosen for the survey may help the response, especially when combined with a deadline (Porter and Whitcomb, 2003).

Timeliness. Response speed is generally higher than in mail surveys (e.g. Tuten et al, 2002). Harris Interactive, for example, often keeps surveys open for only a few days.Carbonview Research receives 80% of the eventual responses within 3 days and a “full” response at around 12 days (Enright, 2006). However, respondents who respond after 3 days may be important because they use the Internet less often and may be more representative of the offline population.

B.2.2. Data quality

Response rates (Unit Nonresponse).The literature on response rates is inconclusive, mostly because response rates are often not directly comparable. Response rates are affected by the contact mode. For example, if respondents are contacted by mail and given the option to reply either by mail or on the Web they will tend to favor the mail option even when that option is only offered to them later (Schonlau, 2003).

Kapplowitz et al (2004) in an experimental study in a university setting find that a Web survey with a postcard pre-notification achieves almost the same response rate as a mail survey with a postcard pre-notification. However, in both cases the response rate is only about 30%.

In a survey of university and community educators Kiernan et al (2005) find a higher response rate over the Web (95%) than by mail (79%). In this case, all respondents were initially contacted by e-mail. Web survey participants provided longer and more substantive responses to quantitative questions.

Parks et al (2006) randomly assign college women to a telephone or a Web survey for collecting data on alcohol use and alcohol-related victimization. The response rate of Web survey respondents was higher (60%) than for the telephone survey (45.7%).

Fricker et al (2005) achieve a much higher response rate using phone interviews (97.5%) than using Web surveys (51.6%). In this study all respondents were initially contacted by phone and passed a screener.

Item non response. Web surveys may have somewhat lower item nonresponse rates than other modes. There is strong evidence that they tend to generate longer answers in open ended questions. Tuten et al (2002) review several papers on data quality issues. In a study of alcohol use,Link and Mokdad (2005) report very low item non-response among telephone and Web survey respondents compared to mail survey respondents. Fricker et al (2005) find less item non-response in a Web survey than in a telephone survey. Schaefer and Dillman (1998) find that their e-mail questionnaire had a higher completion rate (69%) than their identical mail survey (57%). Open-ended questions from the e-mail version contained an average of 40 words compared to 10 words for the paper version.

Mode effects. The survey mode (web survey, phone survey, mail survey) may affect the responses. In auditory modes (phone) there may be a tendency to choose the last response option for questions for which the respondent does not have a pre-formed opinion (recency effect). In visual modes (web survey, mail survey) the respondent may have a tendency to choose the first response option (primacy effect). The respondent may be satisfied with the first response option without evaluating later options. Knauper (1999) finds that recency effects are exacerbated by older age. For example, in a question about whether divorce should be easier or more difficult to obtain she finds that the reordering of response options unveils a recency effect of only 5% among 18-54 year old respondents but 36% among more than 70 year old respondents. In a randomized study of 15-year-old schools children on smoking Denscombe (2006) finds that there is little evidence of a mode effect between the web-based and paper-based questionnaires.

Time to complete the survey. Parks et al (2006) found that Web survey participants took slightly longer than respondents who did the same survey by phone(medians of 19.5 minutes versus 18.8 minutes). Fricker et al (2005) also reported that Web survey respondents took longer than phone survey respondents (21.6 minutes versus 20.8 minutes). The longer completion time is consistent with longer answers in open ended questions and less item nonresponse. Chang and Krosnick (2003b) compared both modes in a laboratory setting where subjects were randomly assigned to either self-administered interviews with a computer or to oral interviewing over an intercom system. They find completion time in the computer mode to be significantly less than in the intercom mode (17.3 minutes versus 26.6 minutes).

Social Desirability. Like mail surveys, Web surveys do not require the presence of an interviewer. For sensitive questions the presence of an interview might prompt a respondent to give responses that are socially more acceptable, thus distorting the answers towards socially desirable responses. SeeParkset al (2006), Rietz and Wahl (1999), Chang and Krosnick (2002a) .

B.2.3. Web Survey Design for Elderly Respondents

Rogers and Badre (2002) review web site design for older adults. Design recommendations for older adults include the use of sans serif fonts (e.g. Arial), a font size of 14 points for body text and 18-24 points for headers, avoiding italics and avoiding the use of all capital letters.

Web sites for older adults should not require users to distinguish between colors of similar hue (especially in the violet, blue, and green range). The text and background on websites should be of high contrast (e.g. black on white background) rather than low (e.g. black on blue background). Informational cues like color or highlighting are especially helpful for older adults.

All in all, neither the literature nor our own project provides evidence that the quality of the answers in Internet surveys is lower than with other interview modes (mail, phone, or personal interviewing). On the contrary, there is some evidence that Internet interviewing does better in certain respects, e.g. avoiding social desirability and triggering longer and more complete open-ended answers. On the other hand, selection remains a concern in the near future, even though Internet penetration has been increasing steadily in the past decade. Particularly for the elderly respondents, Internet access can be expected to be far from complete, also in the next ten years. Providing non-Internet users access to the Internet using a web-TV or some other device is a promising development that needs to be explored further, and this is part of the proposed work. Still, it seems reasonable to expect that non-response among some groups may remain substantial, and research on reweighting, matching, and other ways to correct for unit non-response will remain necessary.

C. Preliminary Studies

Since this is a renewal application, we limit the overview of preliminary studies to a progress report of the (roughly) first four years of the current project.

C.1. Progress Report

This progress report is organized as follows. After describing the project organization,we discuss the main findings. In doing so, we distinguish two broad (and somewhat overlapping) domains. First, we consider studies that shed light on the properties of Internet interviewing vis-à-vis other modes under the heading “measurement and design.” This includes selectivity, reweighting, and mode effects. Next, we pay attention to a number of substantive topics for which the Internet is a particularly effective survey mode under the heading “content.”

C.1.1. Surveys and Organization

The original proposal for the project anticipated two Internet interviews with a subset of HRS respondents, eight interviews with respondents in our own special-purpose Internet panel, and four interviews with a telephone control group (the so-called CATI sample). The non-HRS sample is obtained from the University of Michigan’s Survey Research Center (SRC) respondents to the Monthly Survey (MS).

Anticipated sample sizes were 500 for the telephone interviews (which we will denote as CATI1, CATI2, etc.) and 1000 for the Internet interviews (which are denoted as MS1, MS2, etc.). For HRS, we planned a gross sample of 2,800 for the first Internet interview and 1,400 for the second one. Currently (end of October 2006), the following interviews have been conducted.

  1. HRS Internet 1. This one was completed in 2003. Total number of observations: 2,180, response rate 80.2%.
  2. HRS Internet 2. Because we wanted to monitor the progress of the Medicare Part D drug plan, we divided HRS Internet 2 into two phases. Phase 1 went in the field in February 2006 and measures respondents’ experience with the initial insurance offerings. We obtained 1338 observations (70.0% response rate). We are planning a phase 2 to be administered in early 2007 to take advantage of the period in which Medicare Part D beneficiaries can change plans.
  3. CATI1. Started in 2003. 616 observations at this moment. This amounts to a response rate of 84.4 percent if we exclude as ineligible potential respondents who are deceased, ill, or for whom English as interview language poses a problem.
  4. CATI2: Started in 2005. We have not contacted all respondents yet who have completed CATI1. At the moment, there are 403 respondents. This amounts to a response rate of 87.4 percent if we apply the same exclusions as before for respondents who are ill or deceased. If we exclude no contacts from the base (some of these may still be contacted) the response rate rises to 93.7 percent. If we also exclude respondents for whom an appointment for an interview is pending, the response rate rises to 98 percent.

As noted above, the original plan was to have four CATI waves. It turned out that most of the mode effect issues we wanted to study could be covered by the first two telephone interviews. In view of the limited use for further CATI interviews and the cost and operational advantages of having everyone on the Internet, that plan has been adjusted by offering CATI households without Internet access a Web TV, using the funds from other grants to create an American Life Panel (ALP[1]) ( ). This set-up follows similar set-ups of CentERdata ( ) and of Knowledge Networks ( In addition we have simplified the funding of the panel, by costing it at a fixed price per interviewee minute[2].