ESRA 5Th Conference of the European Survey Research Association

ESRA 5th Conference of the European Survey Research Association

Ljubljana, Slovenia, July, 15–19, 2013

Authors

Prof. Silvia Biffignandi - University of Bergamo, Department of Management, Economics and Quantitative Methods, Director of Centre for Statistical Analyses and Survey Interviewing

Dr. Roberto Artaz - University of Bergamo, collaborator at Centre for Statistical Analyses and Survey Interviewing

Title

Saturation effect on over-surveyed people versus willingness to participate in online panels: experimental results.

Keywords

Elapsing time, incentives, inner motivations, response rate, saturation effect

Abstract

Response rate in online panels is affected by several factors. During 6th ISM Workshop 2012 in Ljubljana Caroline Jarrett’s contribution (Effortmark Ltd) explored with an empirical approach and from a respondent perspective the reasons of response and of the declining trend. Saturation effect on over-surveyed people was emphasized as a determinant leading to lower willingness to respond. In order to shed more light on this phenomenon a special experiment was carried out by the CASI Research Center at the University of Bergamo (Italy) within the PAADEL project (PAnel Agro-food and DEmographic in Lombardy). We analyzed response rate of a panelists sub-population previously saturated by the quick administration of 3 surveys waves in few weeks. Findings show that elapsing time could reduce the negative saturation effect on response rate more in control group, where an incentive wasn’t provided, than in the experimental group benefited by an incentive. Results seem to point out that, in a context of over-surveyed people, inner motivations are more effective in prevent saturation effect on response rate than external inducements given in exchange for participation. Nevertheless our research with the reported online experiment is not without limitations. In fact we are aware that having recourse to all the members of a sub-population and not to a probability sample of the overall population may weaken the chance of generalize the findings. Moreover using different amount of questionnaires and diverse intervals between waves and reminders could affect the results, so outcomes seek further assessment in forthcoming experiments.

1 Introduction

Response rate in on-line panels and stand-alone surveys can be affected by several factors. From experience its trend seems to be mainly declining despite the fact that Web-tools for data collection from people are widespread and methodology essentially standardized. Moreover and predictably a large body of literature is available on the topic.

This paper explores the matter from a pragmatic point of view and interprets the reasons from the respondent’s standpoint in the wake of an empirical contribution on this theme presented at the 6th ISM Workshop 2012 in Ljubljana (Jarrett, 2012).

The study started from the hypothesis that people are becoming over-surveyed by the too many invitations received and so turn less interested in response. This problem is very serious for academic researchers, professionals and firms dealing with inquiries on the Internet. Therefore at Effortmark Ltd a pilot research was developed by the manager in order to explore the aspects of the question.

Initially the phenomenon was investigates by collecting every survey invitation received during two months in 2011 and then was carried on by recruiting four people in the UK from a variety of backgrounds to collect every survey invitation they received during August 2012. These subjects had the task of looking out for every invitation but they were free to respond or not depending on the spur of the moment.

The purpose was to report on the variety and on the reasons for own response rate as well as to recount the experiences of the people recruited and their response rate.

From our point of view the contribution presented at the 6th ISM Workshop 2012 in Ljubljana was an opportunity to meditate on the condition of over-surveyed people and to design an experiment in order to examine in depth the effects. The research was carried out within the PAADEL project (Panel on Agriculture food And Demographic Evaluating Lombardy) supervised by Professor Silvia Biffignandi Director of the Centre for Statistical Analyses and Survey Interviewing at the University of Bergamo (Italy).

2 Foreword to the experiment

It’s generally admitted that offering incentives promote response and retention even if the dimension of their actual effectiveness have to be assessed every time: incentives motivate people to start a Web survey and once they have been recruited in a survey they are more likely to be retained if an incentive is offered (Göritz, 2006). So paying an incentive appears to be effective in increasing response rates in online and offline surveys, but assuring a wide participation is especially important in research based on panels, since early participation sets a ceiling for the response and the retention rate over the whole life of the panel.

Actually in offline studies the non collaboration on a subsequent wave is generally predicted by prior non collaborative attitude even when non-respondents to first wave are re-contacted in subsequent waves (Presser, 1989). And also in online panels completing a given wave seems to be an indicator for responding to next wave (Göritz and Wolff, 2007; Göritz, Wolff and Goldstein, 2008) while incentives increase participation at the beginning but the positive effect falls throughout the waves (Göritz, 2008) unless they are proffered in the subsequent surveys (Bandilla and Haas, 2009).

Therefore researchers appear to face similar problems in using traditional mail techniques or internet-based methods to carry out surveys and panels.

But the question is: does early participation really set a ceiling for the response in the next waves or this ceiling is due to other reasons as the saturation effect?

3 Hypothesis formulated and tested

In the light of quoted experience it was questionable that response rate of people have to be inevitably declining if they aren’t over-surveyed: harassing solicitations are really a problem. A person could have good inner motivations to respond but inquirer’s stubbornness and lack of time can throw out respondents.

Of course, for the researcher it’s very difficult to influence respondent’s time availability but it’s possible to develop a strategy in order to mitigate saturation effect on respondents.

Clearly we can’t screen and remove all the surveys coming from the Internet, but we can administrate a lot of questionnaires to a group in a short time so as to saturate the targets and then stop the flood for a moment. During the waves some respondents will drop out so response rate will be declining but after a rest people with inner motivations perhaps could be recovered.

So we opted for sending after a 12-months break a new questionnaire to the members of a panel where we wreaked a bit of saturation effect accidentally due to the fact that we administered 3 questionnaires from the 14th of September to the 12th of October 2011.

The topic of the three-waves first experiment conducted on-line was the consumption of special foodstuffs like the biological, those directly sold by producers, and those bought through purchasing groups.

E-mail invitations were sent to a sample of 502 individuals who are members of local governments in the province of Bergamo. The choice of the target population was made due to the fact that in Italy publishing online e-mail address and curriculum vitae of every official is required by law. Such circumstances made easier the task of collecting a respectable database with updated information on all employees that are well-distributed all over a given territory.

Of course this option can appear to have a contraindication emphasized by a part of web-survey literature, namely the remark that “people who connect for business purposes do not seem to be willing to participate in the survey” (Biffignandi et al., 2002, p. 12).

In this case, however, it is mitigated by the fact that the target population doesn’t connect for mere business purposes, but also in order to strengthen the openness of government and bureaucracy to citizenry. So actually it seems instead to occur a different phenomenon also reported by the researchers: “well-educated users tend to respond faster than the others” (Biffignandi et al., 2002, p. 12).

Anyway, the mean age of the individuals that were requested to take part in the surveys was 50 years, and 43,02% (n = 216) were female.

The members of the sample were randomly assigned to one of two conditions. In the control condition (n = 250), individuals did not receive any information in exchange for the information that was demanded. In the reciprocity condition (n = 252), individuals received unusual recipes for appreciating the special foodstuffs mentioned in the questionnaires and diverse information about the characteristics, the properties and the availability of products by redirecting to specialist websites. This assignment was kept constant over the three waves.

Note that the study was not announced as a longitudinal research at the outset. Instead, in the second wave (W2) and without prior notice, individuals were told that this was a follow-up to a study in which they were invited about 2 weeks before. In the third wave (W3), they were told that this was the final part of a study in which they had been invited before.

In substance an informal panel was organized where the participation wasn’t made official by a request to take part in it: the option was aimed at preventing a potential distortion in the response rate due to the early expectation of several waves of questionnaires.

Moreover, not only people who responded in a given wave were invited to the following wave, but also those who didn’t respond. In fact besides the link to the questionnaire specific to every wave, those who didn’t respond received again in the e-mail a reminder of the previous questionnaires with a special request to participate and in the invitation it was specified that they could take part in the prior waves by completing online surveys.

Invitations and e-mail reminders were sent weekly to target population since periodicity and number of soliciting messages were based on the empirical evidence of a pilot survey where reaction to e-mail was prompt. Furthermore, this evidence is consistent with literature which reports a reaction in about two working days from the sending (Biffignandi and Pratesi, 2000; 2002) so that even “a week should have been enough” between the messages (Biffignandi and Pratesi, 2002, p. 24).

On the 3rd of November 2012 we sent the fourth wave, a questionnaire about changes in consumer behavior and needs in agro-food sector.

Also in this wave the members of the sample were randomly assigned to one of two conditions. In the control condition (n = 249), individuals received nothing in exchange for the information that was demanded . In the reciprocity condition (n = 251), individuals received the invitation to the conference where results would be presented.

4 Results

Table 1 shows a summary about the trend of response rate in the two experimental conditions over the four waves.

TREND OF THE RESPONSE RATE
TABLE 1 / Units in mailing list / Units in experimental group / Units in control group / Total amount of respondents / Respondents in experimental group / Respondents in control group / Response rate aggregate / Response rate in experimental group / Response rate in control group
Wave 1
14.09.2011
Reminder
21.09.2011 / 502
=286M
+216F / 252
=145M
+107F / 250
=141M
+109F / 154
=95M
+59F / 70
=41M
+29F / 84
=54M
+30F / 30,67%
=33,21%M
+27,31%F / 27,72%
=28,27%M
+27,10%F / 33,60%
=38,29%M
+27,52%F
Wave 2
28.09.2011
Reminder
01.10.2011 / 501*
=286M
+215F / 251
=145M
+106F / 250
=141M
+109F / 95
=48M
+47F / 46
=21M
+25F / 49
=27M
+22F / 18,96%
=16,78%M
+21,86%F / 18,32%
=14,48%M
+23,58%F / 19,60%
=19,14%M
20,18%F
Wave 3
12.10.2011
Reminder
19.10.2011 / 500**
=285M
+215F / 251
=145M
+106F / 249
=140M
+109F / 84
=42M
+42F / 39
=20M
+19F / 45
=22M
+23F / 16,80%
=14,73%M
+19,53%F / 15,53%
=13,79%M
+17,92%F / 18,07%
=15,71%M
+21,10%F
Wave 4
03.11.2012
Reminder
10.11.2012 / 500
=285M
+215F / 251
=145M
+106F / 249
=140M
+109F / 97
=54M
+43F / 43
=24M
+19F / 54
=30M
+24F / 19,40%
=18,94%
+20% / 17,13%
=16,55%
+17,92% / 21,68%
=21,42%
+22,01%
* One subject (F) refused to take part in the wave 2 expressly and asked to be deleted from the mailing list.
** One subject (M) refused to take part in the wave 3 expressly and asked to be deleted from the mailing list after he received the reminder of the wave 2. Another subject (M) asked to be deleted from the mailing list after he received the invitation to wave 3 but didn’t refuse to take part in it expressly.
Label: M = male F = female

5 Discussion

We have presented a longitudinal experiment on the effect of elapsing time on people previously over-surveyed: the saturation effect seems to be reduced by the long rest and more in control group, where an incentive wasn’t provided, than in the experimental group benefited by an incentive. In fact the fourth wave shows in the former group a response rate of 21,68% (even better than W2) and in the latter group a response rate of 17,13% (better than W3). So the findings also point out that in a context of over-surveyed people left in peace for a long moment, inner motivations are more effective in boosting response rate than external inducements given in exchange for participation.