Essnet “Preparation of standardisation”
Work package 1
Statistical methodological standards
- Review of current international and methodological handbooks and guidelines -
Draft report
Version 12 August 2011
1. Definition of a statistical standard and of guidance 3
1.1. Introduction 3
1.2. Current use of the words “standard” (noun or adjective) and standardise/ standardisation in statistics 3
1.2.1.1. Method used 3
1.2.2. Some statistical references to standards as “sets of rules to be applied” and the corresponding clues 5
1.2.3. The current use of the words standard and standardisation is not fully consistent 10
1.3. Taking stock of standardisation in other domains 11
1.3.1. Definition of a standard 11
1.3.2. How should this proposal be understood in the ESSnet STAND-PREP ? 12
2. Review and analysis of six methodological manuals 16
2.1. A framework for the analysis 16
2.1.1. Conceptual scheme of the repository for the standards 17
2.1.2. The IT tool for the management of the repository 23
2.1.3. Procedure followed for the definition and refinement of repositories for single handbooks and for the construction of the unified repository 26
2.2. Analysis of the six methodological handbooks 27
2.2.1. Eurostat sampling references guidelines 28
2.2.2. Recommended practices for editing and imputation in cross-sectional business surveys (EDIMBUS) 32
2.2.3. Guidelines for statistical metadata on the Internet 41
2.2.4. ESS hHandbook for qQuality rReports 45
2.2.5. Recommended practices for questionnaire design and testing 52
2.2.6. Survey mMethods and pPractices 57
2.3. Description of the unified repository for methodological standards 65
2.3.1. General description of the repository 65
2.3.2. Examples of compared analyses of standards derived from different handbooks for the same GSBPM sub-processes 72
3. Review of support measures 8279
3.1. Introduction 8279
3.2. Support measures related to standards 8279
3.3. Overview of support measures 8380
3.3.1. Motivation for implementation 8380
3.3.2. Making the implementation easy 8481
3.3.3. Supporting right implementation 8683
3.4. Summary 8784
4. Quality of statistical standards and criteria for the assessment 8885
4.1. Introduction 8885
4.2. The concept of quality of standards 8885
4.2.1. Applicability 8986
4.2.2. Effectiveness 9087
4.2.3. Efficiency 9087
4.3. Relations between the proposed quality criteria on methodological standards 9188
4.4. Main problems connected with standards that may have quality character 9188
5. Conformity to standards 9390
5.1. Definition 9390
5.2. Conformity assessment is considered during the development of standards 9390
5.3. Standards in the EU and EU conformity assessment 9592
5.3.1. Framework for EU standardisation 9592
5.3.2. Modules for conformity assessment 9693
5.4. How to assess conformity to methodological standards in statistics? 10097
5.4.1. Problem statement in the ESS 10097
5.4.2. Proposal for conformity assessment 10299
Annex 1 - Extracts referring to standards with another meaning 105102
Annex 2 – Review of literature on standards 107104
1. Definition of a statistical standard and of guidance
1.1. Introduction
The definition of a standard in the ESSnet STAND PREP is a concern that is felt to be important by all participants. This issue has been tackled several times (even when preparing the answer to the Eurostat “call for interest”!). This is an issue that deserves serious consideration both for this ESSnet but very probably as well for further discussions in the sponsorship and in the ESSC and PG that will discuss the reports of that sponsorship.
This kind of work is not exceptional in the ESS: for instance, the statistical definition of “enterprise” was a lasting concern for several years in Business statistics. The definition of “quality” required some work as well. For these two examples, there were some pre-existing references in other domains or in other statistical offices that helped settle the debate relatively quickly. For the definition of “enterprise”, the existing definitions in national accounts and accounting standards helped a lot. As for the definition of quality in statistics, papers from the US Bureau of the Census, Statistics Canada and Statistics Sweden provided very useful references
In the case of standards, the ESSnet has taken stock of the literature on the subject. Whereas there are many references on standards in domains other than statistics, we did not find references in statistical organisations that could help to solve the issue quickly.
In this document, we refer to the practice in existing statistical texts using the word of “standard” in order to define the underlying agreed concept for statisticians. This method is used below.
However, since the previous method does not bring useful results, we have then used the ISO vocabulary to propose a definition for the concept of “standard” and the corresponding definition of standardisation.
1.2. Current use of the words “standard” (noun or adjective) and standardise/ standardisation in statistics
12.21.1. Method used
“Standards” are often referred to by statisticians, with different meanings. This is largely due to the fact that “standard” has several meanings in English. We should therefore select for our purpose only texts in which the meaning of “standard” used by the author refers to standards that are the result of the standardisation activity.
In short, standardisation consists in developing standards.
The definition of standardize given by the Merriam-Webster dictionary is:
1: to reduce to or compare with a standard <standardize a solution>
2: to bring into conformity with a standard
3: to arrange or order the component items of a test (as of intelligence or personality) so that the probability of their eliciting a designated class of response varies with some quantifiable psychological or behavioral attribute, function, or characteristic
The last meaning is corresponds to some technical work in statistics, and is not relevant here.
The first two meanings of standardise refer to the word standard with the same underlying meaning of this noun. As a first approximation, such standards would be “a set of rules to be applied”.
We have to take into account the various meanings of “standard” in English, in order to keep for our analysis only the texts that refer to the meaning of standard we are interested in.
Whereas it might be easier to understand for native English speakers, it can be tricky for non native ones.
If we refer to the Cambridge dictionary Online[1], the definition of standard is:
1 - meaning “usual”:
o a pattern or model that is generally accepted. ex:
This program is an industry standard for computers.
o a song or other piece of music which has been popular and often played over a long period of time
2- in the context of quality:
o a level of quality - ex:
This essay is not of an acceptable standard - do it again.
This piece of work is below standard/is not up to standard.
We have very high safety standards in this laboratory.
Not everyone judges success by the same standards - some people think happiness is more important than money.
Her technique became a standard against which all future methods were compared.
o a moral rule which should be obeyed - ex:
Most people agree that there are standards (of behaviour) which need to be upheld, but agreeing on them is rather more difficult.
The definition given by the Merriam-Webster dictionary [2] (“definition for English learners”) is:
1 a : a level of quality, achievement, etc., that is considered acceptable or desirable [count]
▪ high/low standards of quality ▪ His work is not up to our standards. [=is not as good as we require it to be] ▪ By modern/today's standards, the house is just too small. ▪ industry standards ▪ Their standards are slipping. [noncount] ▪ His work is not up to standard. = His work is below standard. [=his work is not good enough] — see also double standard, standard of living
b standards [plural] : ideas about morally correct and acceptable behavior
▪ She has high moral standards. [=principles]
2 [count] : something that is very good and that is used to make judgments about the quality of other things
▪ This book is the standard by which all others must be judged.
3 [count] : a fixed official unit of measurement
▪ a standard of weight — see also gold standard
4 [count] : a flag that is used in official ceremonies
5 [count] : a song that has been sung by many different artists
▪ an old standard
The meaning we are interested in is the first meaning listed by the Cambridge dictionary and in the Webster dictionary (reference to “industry standards”).
After taking into account these possible meanings of the word “standard”, we have rejected some references in the statistical literature using the word “standard” (see Annex) and kept the ones below. For each of these, we derive “clues” for the definition of the concept of “standard” that is implicit in the references below. These clues are shaded in light grey.
1.2.2. Some statistical references to standards as “sets of rules to be applied” and the corresponding clues
1.2.2.1) In the report of the “Canadian Advisory panel on Telephone public opinion survey quality” (February 2007):
Standards are defined as “Practices that should be requirements for all telephone studies conducted by the Government of Canada ».
Clue 1:
We deduce from this definition that:
o there are several statistical standards for just the statistical activity of “telephone opinion survey”. The paper lists a series of such standards. There are more or less as many standards as steps in the GSBPM. These sentences are made of a few sentences (generally 5 to 6).
Example: for data collection dealing with call-backs, the standard is defined as:
“Unless circumstances strongly indicate a different number of call-back attempts and scheduling of call-backs is required, there will be a minimum of eight call-backs made, varying the day and time of each call.”
o these standards are requirements, i.e. compulsory rules that have to be applied.
1.2.2.2) In the “ESS code of practice” (CoP), the word “standard” is used several times.
1.2.2.2.1: It is stated that: “The head of the statistical authority and, where appropriate, the heads of its statistical bodies have the sole responsibility for deciding on statistical methods, standards and procedures”.
Clue 2:
This would suggest that standards differ from statistical methods (which refer to academic knowledge) and from procedures (which refer to ad hoc organisations of the production process).
1.2.2.2.2: It is also stated that “European and other international standards, guidelines and good practices must be fully observed in the processes used by the statistical authorities to organise, collect, process and disseminate official statistics.”
Clue 3: This would suggest that standards are of a more binding nature than guidelines and good practices. However, since the CoP requires the guidelines and good practice “fully observed” leads to some confusion: what is then the difference between a standard and a guideline?
Standards may be relevant for the following activities in NSIs: organisation, data collection, processing, dissemination.
1.2.2.2.3: The overall methodological framework of the statistical authority follows European and other international standards, guidelines, and good practices.
Clue 4: Standards may define the methodological framework. There are international standards, international guidelines and international good practices that refer to three different types of entities.
1.2.2.2.4: Procedures are in place to ensure that standard concepts, definitions and classifications are consistently applied throughout the statistical authority.
Clue 5: there is a conformity assessment of standards inside the NSI.
1.2.2.2.5: Revisions follow standard, well-established and transparent procedures.
[…]
A standard daily time is set for the release of European Statistics.
Clue 6: standards are defined for procedures.
1.2.2.2.6: Statistics are compiled on the basis of common standards with respect to scope, definitions, units and classifications in the different surveys and sources.
Clue 7: standards include some aspects of the definition of statistics like scope, definition (of characteristics to be collected?), statistical units. Standards are defined as well for classifications…These standards have to be consistent (“common”) for the various statistics and classifications.
1.2.2.2.7: Metadata are documented according to standardised metadata systems.
Clue 8: the products than can be standardised include whole systems, for instance metadata system.
1.2.2.3) In the draft regulation on ESA 2010, it is written that:
The objective of the Regulation is to revise ESA both concerning the methodology on common standards, definitions, classifications and accounting rules, and the programme for transmitting for Union purposes accounts and tables compiled according to the methodology.
Clue 9: The ESA includes a methodology (= a set of methods) dealing with standards. These standards have to be distinguished from definitions, classifications, accounting rules.
Article 1 states that:
ESA 2010 provides for:
(a) a methodology (Annex A) on common standards, definitions, classifications and accounting rules that shall be used for compiling accounts and tables on comparable bases for the purposes of the Union, together with results as required by Article 3;
Articles 3 states that “Member States shall transmit to the Commission (Eurostat) the data and metadata required by this Regulation in accordance with an interchange standard specified by the Commission (Eurostat).”
Clue 10: Standards in statistics will be available for the activity “interchange of data and metadata”.
Paragraph 1.21 states that: The concepts are internationally compatible because:
a) The concepts in the ESA are consistent with those in the worldwide guidelines on national accounting, i.e. the SNA;
b) For the EU Member States, the ESA is the standard for submitting national accounts data to all international organisations;
Clue 11: The whole ESA is considered as one single standard.
It is stated as well that :
“The main concepts in the ESA are well-established and fixed over a long period, because:
a) They have been approved as the international standard for many years;
Clue 13: The set of all concepts of the ESA (but not the other rules/ statements) is in itself one single standard.
1.2.2.4) The November 2010 conclusion of the Ecofin Council states that:
The Council CONSIDERS that it is desirable - for administrative purposes – to refer to the agreed set of national accounts standards in the EU when the ESA is revised, in order to reinforce quality, enhance clarity and transparency of communication vis-à-vis the authorities (EFC, Finance Ministers) and other users, and increase efficiency, while reducing burden.
Clue 14: The Esa is not one single standard, but a set of standards.
1.2.2.5) The regulation 223/2009 uses as well the term “standards”:
For the purposes of this Regulation, the following definitions shall apply:
1- statistics means […]
2- ‘development’ means the activities aiming at setting up, strengthening and improving the statistical methods, standards and procedures used for the production and dissemination of statistics as well as at designing new statistics and indicators;