Effects of Context 1

Running head: CONTRAST EFFECTS

The Effects of Context on Context Effects: How Modality, Time Limits, and Distraction Impact Contrast Effects

Michael J. Walk

University of Baltimore

The Effects of Context on Context Effects: How Modality, Time Limits, and Distraction Impact Contrast Effects

There are many possible sources of context effects in survey measurement (see Sudman, Bradburn, & Schwarz, 1996, for a detailed discussion). And although prominent survey researchers once denied the quantitative importance of context effects, those researchers are recognizing that context effects are an important phenomenon that can affect respondents’ answers in substantial ways (e.g., Schuman, 1992).

Context effects at the question-level come in two main forms: (1) when question order affects the mean response to a question and (2) when question order affects the correlation between two questions (Tourangeau, Rips, & Rasinski, 2000). Moreover, correlational context effects have been observed to fall into two additional categories: (1) assimilation (i.e., the correlation between two items is increased due to their dual reliance on similar information) and (2) contrast (i.e., the correlatoins between two items is reduced due to the respondent “subtracting” out the the shared information the two items brought to mind (Sudman, Bradburn, & Schwarz, 1996).

A classic example of the phenomena of contrast and assimilation can be found in research by Schwarz, Strack, and Mai (1991). They asked respondents about their satisfaction with life in general as well as satisfaction with relationships, work, and leisure. They were interested in examining what happens to the correlation between relationship and general satisfaction when they varied the presentation of the items. They manipulated both the order of the items (i.e., general-specific or specific-general) and the number of specific items (i.e., all three or just the relationship question). They also varied the preamble to the questions such that respondents were either told nothing about the satisfaction questions, given a joint lead-in introducing the questions (e.g., “We would first like to ask you to report on two aspects of your life…(a) relationship satisfaction; (b) satisfaction with life-as-a-whole.”, told explicitly to think of the questions jointly, or told explicitly to exclude the questions from one another.

Schwarz, Strack, and Mai (1991) found that by asking the relationship question first and then the general question second, they produced an part-whole assimilation effect (r = .67). This affect occurs when a specific question (“How satisfied are you with your relationships?”) concerning a particular domain (e.g., life satisfaction) is followed by a general question (“How satisfied are you with life in general?”). A similar correlation (r = .61) was found in the condition when respondents were explictly told to think of the questions jointly. When respondents were given a joint lead-in, the researchers created a contrast effect (r = .18). A similar result (r = .20) was found when the respondents were explicitly told to exclude the questions from influencing one another.

A Model of Context Effects

A major question in the context effects literature is when should researchers expect contrast effects, and when should researchers expect assimilation effects? Schwarz and Bless (1992, as cited in Sudman, Bradburn, & Schwarz, 1996) created a model to predict the direction of context effects called the inclusion/exclusion model. Their model is based on the assumption that individual questions bring information “to mind.” The effects of this temporarily accessible information depend on the characteristics of the temporary information, knowledge about the source of the information, the characteristics of chronically available information, and adherence to conversational norms (Sudman, Bradburn, & Schwarz, 1996).

One important untested prediction of the model is that certain survey conditions should produce particular context effects. This is because the model assumes that contrast effects require more cognitive effort than assimilation effects, and research has found this to be the case (Martin, Seta, & Crelia, 1990). Therefore, survey situations in which respondents have more opportunity and more motivation to use more cognitive resources, we should expect to see a greatly likelihood of contrast effects.

The present research is interested in testing this prediction. In particular, which survey modes will predict a higher degree of contrast effects and which will predict a higher degree of assimilation effects. If Schwarz and Bless’s (1992) inclusion/exclusion model is correct, then, limiting the amount of available cognitive resources or limiting the time respondents have to engage their resources should result in a smaller contrast effect.

Hypotheses

Based on the previous research, this study will test the following hypotheses:

(1) Limiting respondent’s time to answer individual questions should decrease a contrast effect.

(2) Limiting the amount of available cognitive resources should decrease a contrast effect.

(3) The above two hypotheses should be most pronounced in situations where question order is controlled by the experimenter (i.e., a paper-and-pencil survey should show a lesser degree of support of Hypotheses 1 and 2 than a web-based survey).

Method

Participants and Design

Participants will be 480 college students recruited from several different schools in the University of Maryland system. They will receive extra credit in exchange for their participation.

The experiment will follow a 2 x 2 x 4 factorial design. Factor A will be mode of administration (web or paper), Factor B will be distraction (present or absent), and Factor C will be amount of time available per question (5 seconds, 10 seconds, 15 seconds, or no limit). In total, the experiment contains 16 conditions. Participants will be randomly assigned to one of the conditions by means of a randomly generated number received upon signing up for the study; there will be 30 participants in each condition.

Procedure

Participants will come to a designated on-campus location in groups of five to ten—the location will not vary across participant conditions. All participants during a given session will be in the same condition. Participants will receive directions corresponding to their assigned condition.

Participants in all conditions will be told they are participating in a study to test a series of survey questions for their usefulness and reliability. Participants in the Distraction-Present Condition will additionally be told that the researchers are interested in the effects of distractions during surveys. Participants a timed condition will additionally be told that the researchers are interested in the effects of time limits during surveys.

Participants will then be given a brief questionnaire consisting of eight filler questions and two target questions. Four of the filler questions will precede the target questions and will ask about radio and TV preferences. The other filler questions will come after the target questions and will ask about opinions on political issues and gather simple demographic information (i.e., gender and age).

The target questions will be those used by Schwarz, Strack, and Mai (1991), and will be presented in the following order: satisfaction with work and satisfaction with life-as-a-whole. The questions will be preceded by a joint lead in similar to Schwarz, et al., which, in their study, produced a pronounced contrast effect. Therefore, all conditions in this experiment should be expected to produce contrast effects—the degree to which a contrast effect emerges should depend on the experimental condition.

Manipulations

Mode. Participants in the Paper Condition will receive a paper-and-pencil version of the above described questionnaire. All questions will be on a single page.

Participants in a Web Condition will be provided with a URL to take the survey. Each question will be presented individually and participants will not be allowed to go back and edit responses. Computers will be present in the administration room.

Distraction. In order to create distraction during the survey administration, Martin, Seta, and Crelia’s (1990) procedure will be used. Participants will hear a random series of digits and letters spoken by a male voice, which was recorded on an audiocassette. They will be told to pay attention to the number of digits spoken by the voice and to also to pay attention to answering the survey questions. As a manipulation check, participants will be asked how many digits were spoken by the voice. Participants who are incorrect by more than 50% will have their data removed from analyses. Participants that are in the Distraction-Absent Condition will be told to focus completely on the survey and to not talk to one another.

Timing. In conditions with time limits, participants in the Web Condition will have to answer each question within five, ten, or fifteen seconds (depending on the time-limit condition) following question presentation. Answers marked after the time limit will not be included in analyses. In the Paper Condition, participants will have either 50, 100, or 150 seconds to complete the survey (this gives participants an average of five, ten, or fifteen seconds per question, respectively). In order to closely resemble the Web Condition without restricting the order in which participants answer the paper-based questions, the experimenter will sound a small beeper when each interval of five, ten, or fifteen seconds has passed in order to remind participants of the pace they should be keeping.

Data Analysis

In order to test the hypotheses, correlations between the target questions will be calculated for each experimental condition, and these conditions will be examined for significant differences using a 2 x 2 x 4 between-subjects analysis of variance.

Expected Outcomes

Given the theory and research upon which the current experiment was built, I expect that the most pronounced contrast effect (i.e., the lowest target question correlation) will be found in the Web Condition in which there is no time limit and no distraction. In this condition, the order of the questions cannot be altered by the participant (keeping the order effect intact). Also, there is no distraction to limit the amount of cognitive resources available and there is no time limit to force participants to answer quickly (therefore more temporarily accessible information should come into play and conversational norms should be allowed to affect responses).

I expect that the least pronounced contrast effect (i.e., the highest target question correlation) to be found in the paper-based, five second, distraction-present condition. Participants can switch the order of the questions (decreasing the probability of a contrast effect), have very little time to process related information (decreasing the opportunity to use temporarily available information), and have reduced cognitive resources available to process that information (due to the distraction of counting digits).

Table 1 presents an expected rank-ordering of the target question correlations. The ranking of one represents an absence of a contrast effect (i.e., the highest correlation), the ranking of 16 represents the strongest contrast effect (i.e., the lowest correlation).

Table 1

Expected Outcomes: Rank-Ordering of Expected Target Correlations by Experimental Condition

Administration Mode
Paper / Web
Distraction / Distraction
Time Limit (per question) / Present / Absent / Present / Absent
5 sec. / 1 / 2 / 3 / 4
10 sec. / 5 / 6 / 7 / 8
15 sec. / 9 / 10 / 11 / 12
None / 13 / 14 / 15 / 16

References

Martin, L. L., Seta, J. J., & Crelia, R. A. (1990). Assimilation and contrast as a function of people’s willingness and ability to expend effort in forming an impression. Journal of Personality and Social Psychology, 59, 27-37.

Schuman, H. (1992). Context effects: State of the art/state of the past. In Schwarz, N., & Sudman, S. (Eds.), Context Effects in Social and Psychological Research. New York: Springer-Verlag, pp. 5-20.

Schwarz, N., Strack, F., & Mai, H.-P. (1991). Assimilation and contrast effects in part-whole question sequences: A conversational logic analysis. Public Opinion Quarterly, 55, 3-23.

Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco: Jossey-Bass.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The Psychology of Survey Response. New York: Cambridge University Press.