Doc. Eurostat/ITDG/October 2011/4.4
IT Directors' Group
18 and 19 October 2011
BECH Building, 5, rue Alphonse Weicker, Luxembourg-Kirchberg
Room QUETELET
Vision Infrastructure Project
"Enhanced Dissemination Chain"
Item 4.4 of the agenda
Vision Infrastructure Project
"Enhanced Dissemination Chain"
1. Purpose of the document
The ITDG is asked
§ to take note of this Vision Infrastructure Project (VIP) and to comment on the different actions to be carried out within this project;
§ to particularly comment on the actions where Member states will be involved and be concerned.
The project will start on 16 October 2011, and will run for 2 years. This document includes the initial version of the project management plan, which will be adapted after the start-up of the project. At the working level, NSIs will be involved via the Dissemination Working Group and the Metadata Working Group, together with ad hoc task forces as needed.
2. Background
The VIP "Enhanced Dissemination Chain" aims to improve the existing Eurostat Dissemination Chain in order to answer better to a number of main requirements being in the focus of the ESS vision: enhancement and better integration of the Eurostat/ESS dissemination; progress towards developing an ESS data warehouse and better horizontal integration of ESS indicators. The work described in the following is carried out at Eurostat, but requires – for some of the actions - also an involvement of the ESS Member states.
This VIP is part of a series of VIPs aiming at implementing the ESS strategy of business process integration, determined by the joint ESS strategy paper related to the Commission Communication on the production method of EU statistics, COM (2009) 404 final.
3. Objectives and deliverables
o Scope
The project covers the enhancement and further exploitation of the New Dissemination Chain used at Eurostat, covering the reference database Eurobase, Comext dissemination and the Eurostat website, including its data access and visualisation tools. This New Dissemination Chain stores most of the statistical data and metadata and disseminates them to users and to the internal data/metadata producers. The New Dissemination Chain covers statistics, indicators and metadata using a SDMX compliant IT architecture and IT applications.
This VIP contributes to the strategic objectives of the ESS vision: it supports the production and dissemination of data and metadata in IT systems which are not tied to domain-specific stovepipes, but which facilitate cross-domain data and metadata sharing within Eurostat and ultimately within the ESS. The project also aims to open a new indicator production line disseminating more integrated (cross-domain) ESS indicators.
The project covers all statistical domains. It aims to strengthen Eurostat and the ESS capacity to react quickly to requests for new EU policy indicators (like the Europe 2020 indicators), without having to launch new projects in each case.
o Aims and Objectives
The New Dissemination Chain at Eurostat has been put in place in 2010. It is the basis for the online dissemination of European data and metadata. Although the New Dissemination Chain is already operational, it has become evident in 2010-2011 that the increased visibility of European statistics and the ever-growing use of European statistics and indicators in policy decisions requires rapid enhancements of this chain, which cannot be achieved in a short timescale (2011-2013) with the existing staff resources. These improvements are the precondition for achieving more vertical and horizontal integration of the European Statistical System. The improvements also enable the more integrated dissemination of European data and indicators.
The short term objectives are
· Improvement, simplification and better integration of the New Dissemination Chain;
· Improvement of the quality of the data disseminated;
· Introduction of a new production line for EU policy indicators;
· Involvement of the ESS Member States in the New Dissemination Chain.
The medium-term objectives are
· Further development of Eurobase towards a more integrated data warehouse for Eurostat and the ESS;
· Better integration of the Eurostat and national dissemination.
These objectives will be achieved through the following actions, for each of which the tasks, deliverables and milestones are listed below.
Action 1: The further development of Eurobase towards a Eurostat/ESS data warehouse; strategic development of the New Dissemination Chain in opening it towards an ESS data warehouse (also using the SDMX technical and statistical standards); additional Eurobase functionalities will improve the quality of the European data which is disseminated, in particular through the use of checking algorithms on all published data. This action will be implemented via Tasks 1a and 1b.
Action 2: The simplification and better integration of the New Dissemination Chain in order to rationalize the chain, to diminish the risk of disruption of the daily data updates and the online data access, and for increasing the performance of the New Dissemination Chain. This action is implemented via Tasks 2a and 2b.
Action 3: Improving functionality for producing and disseminating policy-oriented indicators. It includes the development of (a) a new production line for indicator tables, to facilitate flexible combination of data from several domains and (b) new ways of presenting indicators and dashboards on the Eurostat website, which would be used for the EU policy indicators, like Europe 2020 indicators, the Sustainable Development Indicators, PEEIs, the macroeconomic imbalance scorecard, etc, as well as new sets of policy indicators. The aim of this action is to move towards a more user-oriented, integrated and cross-domain dissemination of our data and metadata. This action is implemented via Tasks 3a and 3b
Action 4: The enhancement of the Eurostat dissemination of data and metadata in terms of better integration within the dissemination of Member States. This covers work to facilitate NSIs putting European data and metadata on the websites, as NSIs are committed to do for the action on common ESS information which was developed under the Sponsorship on Communication and agreed by the ESSC in February 2011. This action is implemented via Tasks 4a and 4b.
The following Eurostat units are involved in the following actions: unit B 3: Information systems for statistical production; B 4: Management of IT infrastructure; B 6: Reference databases and metadata; D 4: Dissemination; D 5: Key indicators for European policies. The project is initially planned for two years.
o Deliverables and Key Milestones
Task 1a
Deliverable: Strategy on the opening the New Dissemination Chain towards an ESS data warehouse
In also involving Member states this task should define a more detailed strategy of opening the New Dissemination Chain towards an ESS data warehouse. This strategy will be based on an ESS business architecture which is currently discussed within Eurostat and the ESS (Lead: the Eurostat units B3, B6 and D4).
Milestones:
Q3 2011: Input on the strategy on the ESS business architecture
Q1 2012: Assessment of the impact of this strategy on the new Dissemination Chain
Q2 2012: Communication on the impact of these actions to Member states
Q3 2013: Involvement of Member states in the further development and implementation of this strategy.
Task 1b
Deliverable: checking algorithms used for all data published in Eurobase.
The ESS data are currently sent by the Eurostat production units to Eurobase (being part of the New Dissemination Chain). Eurobase currently only checks the incoming files structures and not the statistical contents (data) contained in the files. As in many production processes, we aim to install a final control for the ESS data released within the New Dissemination Chain. This should strongly improve the quality of the data disseminated by us, by avoiding some rather obvious errors in datasets published by Eurostat, such as negative values in datasets where only positive values are valid (Lead: unit B6, associated: units D4, B3). Results of similar work (e.g. on the Editing Building Block) should be taken into consideration.
Milestones:
Q3 2011: results of preparatory study available
Q4 2011: launch contract for implementation
Q2 2012: test of checking algorithms for pilot domains in Eurobase
Q3 2012: analysis and organisation of the workflow
Q4 2012: checking algorithms in production for selected domains in Eurobase
Q3 2013: checking algorithms in production for all domains in Eurobase
Task 2a
Deliverable: Service Impact Manager (SIM).
The Service Impact Manager is a simple IT application (to be developed) aiming to monitor the functioning and performance of the Eurostat New Dissemination Chain. This dashboard will give an overview and browse the impacts of the all IT components of the New Dissemination Chain on their corresponding business processes and sub-processes. It includes the monitoring of all applications, databases, file systems etc, linked to all business processes for dissemination at Eurostat (including all stages from the transmission of data from the local Eurostat production databases up to the Eurostat website). The information would be displayed on an interface with different availability metrics. As the New Dissemination Chain includes multiple IT applications managed by 3 different Eurostat units (D4, B6, B3), with high traffic loads, this tool is needed to ensure continuous availability and adequate performance of the whole chain (Lead: D4; associated units: B6, B3, B4).
Milestones:
Q3 2011: results of feasibility study available
Q4 2011: launch contract for implementation
Q2 2012: functional specifications and technical specifications
Q4 2012: development
Q3 2013: Production
Task2b
Deliverable: Feasibility and possible implementation of a direct connection between the Data Explorer and Eurobase.
This task aims to simplify the Eurostat dissemination chain. Preparatory work has already started, with an internal workshop organised in May 2011, to identify the initial technical requirements for this task. Further key milestones of this work – together with an implementation phase which needs to be decided upon - are described in the following (Lead: unit D4; associated units: B6, B3).
Milestones:
Q3 2011: feasibility study
Q4 2011: functional specifications
Q2 2012: technical specifications in close cooperation with DIGIT
Q4 2012: potential development
Q3 2013: Potential roll out and production
Task 3a
Deliverable: new production line for indicator tables, with a system enabling users in Eurostat and possibly in other Commission DGs to set up derived tables combining data from several domains
The new production line should use existing ESS data and indicators and combine them in a more user oriented and ad-hoc manner. This new production line should create much more horizontal integration of the ESS dissemination compared to the rather producer oriented dissemination currently existing (example: all data and metadata disseminated on nuclear energy responding to high ad-hoc user needs). (Lead: unit B6; associated units: D4, D5, B3)
Milestones:
Q4 2011: feasibility study
Q2 2012: technical specifications for a potential implementation
Q4 2012: potential development
Q3 2013: Potential roll out and production
Task 3b
Deliverable: flexible dashboards for policy indicators
These Eurostat dashboards would be developed using the latest web technologies and visualisation tools, building on the experiences acquired with the PEEIs and HICP dashboards. It would be possible to implement quickly different types of presentations for policy indicators (Lead: unit D4; Associated unit: D5)
Milestones:
Q3 2011: results of feasibility study available
Q4 2011: functional specifications and technical specifications
Q2 2012 development
Q4 2012: Implementation of a sound release management of the dashboards as well as monitoring reports
Q3 2013: Production
Task 4a
Deliverable: a simple visualisation tool to be embedded in the NSIs websites in their own language
This IT tool would enable NSIs to display the Eurostat data and metadata in their websites with minimal effort. It could follow the approach of IT tools such as that developed by the Guardian newspaper (Europe by numbers: the complete interactive guide), and the "widgets" developed by the statistical institute of Catalunya (http://www.idescat.cat/api/indicadors/giny/?lang=en). The development of such an IT tool will be a cooperative exercise with NSIs, which can be used by NSIs for exploiting the ESS data and metadata which are accessible through the New Dissemination Chain (Lead: D4; associated unit: D5)
Milestones:
Q3 2011: analysis
Q4 2011: functional specifications and technical specifications
Q4 2012: development and roll out, in cooperation with NSIs
Task4b
Deliverable: providing NSIs with access to Eurostat data and metadata via SDMX-compliant web services
This task builds on the first version of SDMX-compliant web services, which were initially developed under IDABC/X-DIS and are due for release in Q3 2011, after a long delay due to DIGIT policy on web services. A second version of these web services should be developed and implemented (Lead: D4; associated units: B6, B3, B5, DIGIT).
Milestones:
Q3 2011: analysis and feasibility
Q4 2011: functional specifications and technical specifications
Q2 2012: development and roll out
Q4 2012: Production
o Progress indicators
Progress indicators can be derived from the various tasks where concrete milestones are determined. A number of tasks are spreading over the whole project period of two years; some other ones are finalised already earlier.
4. Work approach
Supported by the project team, one Eurostat IT and one statistical project leaders will put in place the work programme described above. External contractors will be involved in the carrying out of a number of the tasks described above. The two Eurostat project leaders will invest major effort in working with NSIs and the Eurostat production units, to ensure awareness and active involvement in the project.
Regular communication and missions to NSIs are envisaged as well as workshops for NSIs, for Eurostat units and for staff in other DGs. The ITDG, the DIME, the Metadata Working Group and the Dissemination Working Group will be kept informed regularly.
The different actions described above will improve data quality and data accessibility to European Statistics. It will also enhance the Eurostat New Dissemination Chain and move it more towards an ESS data warehouse. A new production line linked to ESS statistics will enhance further the use of European statistics. The use of dashboards will ease the access to the information and will allow the public to see it interactively on the web. These dashboards should in line with the overall IT components of Eurostat. The usage of web services is growing and several international data providers are following this trend. The experience acquired over the last year (for example with Google) shows that visible errors in output data is an important issue, and therefore final data checks in the published data in Eurobase seem to be necessary.