SPONSORSHIP ON COMMUNICATION
Luxembourg, 19 May 2010
Item 4.2 of the agenda
Project:“User satisfaction survey”
By Aurélien Daubaire / Petra Kuncová / Adolfo Galves Moraleda
1/5
Project of harmonised user satisfaction and confidence survey
DRAFT PROJECT TO BE DISCUSSED BY THE SPONSORSHIP GROUP ON COMMUNICATION
Aurélien DAUBAIRE / Petra KUNCOVÁ / Adolfo GALVES MORALEDA
May 17, 2010
1.Background
The sponsorship group on communication approved two projects which were presented on February 4 and 5 in Madrid :
A project to measure confidence in statistics.
A project to harmonise user satisfaction surveys.
It appeared that the project should have “strong focus with a clear target in order to be improved”.
The question of user satisfaction surveys has already been discussed by the working group on quality in statistics in 2008 and 2009. An overview of the different experiences in the Member States has been discussed in May 29, 2009.
This paper proposes a framework for harmonised opinion and user satisfaction surveys. It should obviously be discussed by the sponsorship group on communication, and possibly by the working groups on dissemination and/or quality in statistics.
- Preliminary statement about the difference between user satisfaction surveys and confidence surveys
Confidence surveys aim at measuring the credibility of statistics and of the ESS among general public or specific targets (opinion leaders, etc.). Their relate to the image of statistics and the ESS among a large public. Examples : do people trust the rate of unemployment such as disseminated by their NSI ? Do they believe that the professional independence of statisticians is effective ? Etc.
User satisfaction surveys aim at measuring the quality of statistics as perceived by users, ie by people who need data for their own use, and improving the relevance of statistics (statistics as a product) and their dissemination (dissemination as a service). Examples : is the unemployment rate of European Member States easily available on the internet ? Is the employment rate enough detailed by age categories ?
The issue can be seen with reference to the principles of the European Statistics Code of Practice :
Confidence surveys mainly refer to the institutional environment (professional independence, impartiality and objectivity, etc.)
User satisfaction survey mainly refer to statistical results themselves and the way users can access the results (relevance, accessibility and clarity, timeliness and punctuality, etc.).
The next table summarizes the main differences between user satisfaction surveys and confidence surveys.
User satisfaction surveys / Confidence surveysWhat for / To check the relevance of statistical results and their dissemination
To improve the quality of statistics and their dissemination (with reference to users’ needs) / To assess the credibility of statistics and statisticians among a large public (users and others)
To define priorities of communication in order to correct the image of statistics and statisticians.
What / To ask users what they think about statistical products and their dissemination.
Examples : attractiveness of a website, ease of use, search and display functions, products and services... / To ask people if they trust statistics and statisticians.
Who the respondents are / Generally a specific (although large) public.
Examples : website users, contacts, researchers, medias... / The general public (although sometimes focused on a specific public such as opinion leaders).
How / Often an online survey on the website / Often a gallup (ex : Eurobarometer...)
- How far to harmonise user satisfaction surveys and confidence surveys ?
Many NSIs and/or Eurostat already implement user satisfaction surveys. Several countries also implement confidence surveys and the Eurobarometer published in 2008 tackled the question at a European level.
The costs and benefits of harmonized surveys must be assessed before initiating the project.
Limitations and drawbacks of harmonized surveys :
A new questionnaire would break time series for NSIs who have already implemented surveys ;
Fully harmonised surveys would not allow to tackle country-specific questions (ex : the accessibility of a specific national web page, the efficiency of a specific call center, etc.) ;
Results for different countries are difficult to compare, due to cultural or sociological differences ;
Impossibility of financing two similar surveys (a country specific survey and a harmonised survey).
Advantages of harmonized surveys :
A support for countries who initiate new surveys through the collection of existing “best practices” ;
A common assessment for common actions and actions of common interest (European data, project of a “common set of tables”, etc.).
Two principles could be defined about harmonisation of user satisfaction and confidence surveys :
Common surveys can be made on common subjects or common actions ;
Guidelines should be provided to the NSIs, on a voluntary basis, for user satisfaction surveys and confidence surveys.
- Proposals about user satisfaction surveys
Project 1 : guidelines for general user satisfaction surveys on websites
The objective is to provide guidelines and “good practices” for surveys about websites. Countries would optionally choose to follow these guidelines or not. The guidelines would be useful to countries who set up new user satisfaction surveys.
Description:
The sponsorship group would define a common set of basic questions which are usually asked in user satisfaction surveys. Each NSI could implement additional questions specific to its needs ;
The common set of questions would focus on websites (rather than on services as the call centers, or the detail of database, which are more country-specific) ;
The common set of questions would consider satisfaction referring to the common principles of the European Statistics Code of Pratice, mainly “accessibility and clarity”;
The guidelines would also include examples of how to communicate the results of such surveys and examples of concrete actions derived from that.
Process :
The questionnaire could be implemented easily on the website of each participating NSI.
Project 2 : common surveys about common subjects (European data, common set of tables, etc.)
The objective is to make a common assessment of common actions of the ESS. The European data, as well as the project of a “common set of tables”, would be examples of such cooperation.
Description :
A common questionnaire would be accessible on the “common set of tables” web page, on each web site.
And/or
A common survey would be carried out by the European Statistical Data Support among his users.
Process :
Website questionnaires.
- Proposal about confidence surveys
Project 3 : guidelines for confidence surveys
The objective is to provide guidelines for confidence surveys. Some countries who already implement confidence surveys could adapt their questionnaire, on a voluntary basis, in order to compare oneself to a some other countries. Countries who do not implement confidence surveys would thus get a help on the subject.
Description :
A limited set of questions which would be common to all voluntary countries. Each country could obviously ask additional questions ;
The common questions of the confidence survey would focus on some principles pf the European Statistics Code of Practice : professional independence, impartiality and objectivity, relevance ;
As far as possible, the guidelines would include examples of communications and actions derived from such surveys.
Process :
Two ways of collecting the data could be explored :
Participating country could adapt their questionnaire and share the results on a voluntary basis ;
The implementation of a short common set of questions in the Eurobarometer could be a great help.
1/5