Directorate B: Quality, methodology
and information systems / Directorate General Statistics

3rd Meeting

SA tools User testing group

Rome, 14 September 2009

Minutes

Participants: Mark Boxall (ECB); Anna Ciammola (ISTAT); Marco Marini (ISTAT), Kevin Moore (ONS); Dominique Ladiray (INSEE); Roxana Ilie (NSO Romania); Diana Hodor (NSO Romania); Sylwia Grudkowska (Bank of Poland); Andreas Lorenz (Bundesbank); Joerg Meier (Bundesbank); Marcus Scheiblecker (WIFO); Jean Palate (NBB); Calizzani Cristina (ESTAT); Pascal Jacques (ESTAT);

Chair: Dominique Ladiray (DL)

1_ Approval of the agenda

The agenda was approved.

2_ Updated work plan (Eurostat)

The timetable proposed by Eurostat was adopted. The majority of the group and Eurostatare confident on the feasibility to have the beta version of Demetra+ released in June 2010. Given the delay in signing the grant with the NBB, Prototype III will be available in October 2009 (not before 20th). Eurostat will adapt (minor changes) the work plan to take in account the delay in prototype III delivery.

The group was nevertheless concerned about the possibility to have native X12-X13 products in the first release of Demetra+. Users of the X family products need to start testing X13. As the integration into the tool of the native X13 depends on the delivery of the X11 and RegArima dll by the USCB, possible alternatives were considered. In the case the X13 dll should not be delivered in time to be plugged into prototype IV (end of 2009),JP proposed to integrate in Demetra+ what is already available in the current Demetra[1]. JP considers this alternative as a feasible one (less expensive and requiring less time) respect to other alternatives envisaged in the SASG in order to release Demetra+ in June 2010.

JP made a list of the news (IT and new features) in Prototype III:

- Factoring of the code to be able to migrate to an OS environment;

- All the external parts more integrated (more developed than in the current version)

- New version of C# (visual studio 2008) meaning a faster compiled version of the prototype. No other change in the language is foreseen till the release

- Way for Java prepared (to make integration easier)

- New version Tramo-Seats integrated (more stable, 2 times faster of the current version)

- Integration/fixing of all the comments provided by testers on prototype II

- Possibility to re-use current files (excel files only) of Demetra

- Possibility to integrate user defined regressors

- Possibility to define/integrate national calendars

- Possibility to save parameters and results

- Some wizards to facilitate the input parametrisation and a separate window to show what is going on respect to the SA process

The possibility to re-use DEM files of old Demetra was not considered as a priority. Should users require such functionality, Eurostat could consider developing it through an external contract. As DEM files are using Windows internal tools for their construction, the reuse of DEM files will only be limited to the Windows version of Demetra+ and is not therefore recognised as a priority.

The possibility for previous users of Demetra to re-use their calendars was requested. It was nevertheless noticed that Demetra allows exporting user defined regressors but the functionality for exporting calendars has never worked properly.

It was established again that at the current state of progress of the development of the tool the aim of the testing is not checking results (from different versions of the engines) but checking how Demetra+ works, how to use functionalities (for example how to do forecasts), if it is user-friendly, etc.

3 _ Work on OSOR Repository System (Eurostat)

P. Jacques (PJ) presented the OSOR website as the central repository system for discussion and for the software. He explained that at the moment Eurostat is moving to the 'OSOR.eu FORGE' pages of Osor in order to integrate Demetra+ as anopen source software (

A project called Demetra+ has been created and the entire testing group has been given the access as a developer. This system consents to track the history of developments, assuring more transparency. Activities can be monitored by the "tracker" facility which consents to monitor fixed bugs. The system contains a section for the documentation which now has 2 subsections: users' documentation (how to use a specific features for example) and developers' documentation (how to add modules for examples). Another subsection for the testing documentation will be added. The documentation can be modified-updated by each member of the group.

All the Demetra+ community will be moved in Forge. Any change will be always notified by e-mail automatically.

4 _ Testing prototype II – progress and additional remarks

No additional remarks were done by the group concerning prototype II. Previous comments and remarks will be integrated in the prototype III. It was clarified that the foreseen changes in C# won't preclude any changes in the interface (the main windows will be the same).

5 _ Organisation of the next testing activities

Testing protocol

A document including the protocol for testing was presented previously to the group and to the SASG. It will be further developed by Eurostat.

In order to complete the testing protocol "use cases" need to be further developed in order to facilitate the testing activity.

It was noticed that in the second use case (proposed by Eurostat) a missing element is the comparison between TS and X12 results. Specific use cases should also been prepared for forecasting, time series analysis, explanatory analysis and training.

MS and DL will make a draft proposal of those use cases on which the TF will be asked to react. Dario Buono will be asked to propose the training use case.

The document with the testing protocol and the template for communication of bugs will be made available in OSOR in two weeks time. Another document on the governance of Demetra+ (whose previous version was presented to the SASG in July 2009) which contains some tricky issues on OSS will also be available in OSOR

User testing group was requested to react/comment on the document containing the test protocol as soon as it is available in OSOR.

Test data base

DL made an extraction of series (at least 5 years) which will be made available to the user testing Group:

- 8000 quarterly series

- 3000 monthly series

For those series, also the official SA series will be made available as well as the seasonal adjustment run with default parameters with both X12 and Tramo-Seats (no user defined regressors).

Other datasets will be soon made available:

  • A dataset with only short series;
  • A dataset with only problematic series (it could be interesting to see how the new tool deals with them, for example with very flat series);
  • A dataset with indicators from non European countries (extracted from the MEI database);
  • A dataset with a very limited number of economic indicators from several European countries.
  • A dataset with time series with calendar effects and time series of user-defined regressors.[2]
  • A dataset with time series with extremely high values or with many decimals[3].

The series will be provided in csv format.

Action: DL to send the various DB to Eurostat which will put it in OSOR

The idea behind the testing with this database is to test the stability of the tool, not the results (if it works, how long it takes…)

PJ proposed to load the series with different format also to test ODBC facilities

A proposal to generate time series with increasing length was done (AC) to see how the tool reacts.

Wizard for parametrisation

A draft proposal to trigger the discussion was presented by Eurostat.

The definition of the time span and also of the horizon for revisions (which should be in the output specification) were considered too tricky for "beginners".

In general the idea of different paths for beginners/intermediate/experts was not agreed on. It was explained that the difference was for the input only (not output) as it could be risky to give access to all parameters to every users (they could not understand the meaning).

It was agreed that the set of wizards of the old Demetra was good and it could be streamlined.

Default values of wizards have to be the same ones proposed in the guidelines.

A request was done in order to have a separate window explaining to the users what is going on, and the step of the SA workflow.

It was agreed to simplify the wizards (at the most 5 without the difference beginners/experts). In each one of them there has to be the possibility either to go further (following the wizards) or to change the parameters soon (full parameterisation).

JP specified some work has to be done as some features which will be always the same (for example the calendars); these features have to be defined and managed apart and integrated in the process when necessary.

JP proposed to put a wizard in the left presenting all the parameters leaving the possibility to go more in specific.

An additional comment was made in order to add a "Demetra+ Companion"in place of or in addition to the wizards.

Action: JP to propose some wizards in the next prototype and a separate window showing the step of the SA process. Users are expected to comment on them when testing prototype III.

Eurostat to investigate the idea of the "Demetra+ Companion"

Additional modules

PJ asked for the definition of priorities for modules to be implemented in the next years (independently from the previous features of Demetra).

ISTAT proposed to have a facility to impose a matrix of weights for indirect adjustment (as the weight of each component series could not be the same for all the period).

It was nevertheless suggested to focus on the SA process in order to have a product which could mimic Demetra in the short time possible.

The list (very detailed) of possible additional modules is provided in the final report of the users requirements TF.

Documentation

It was agreed on the need to improve the documentation. Nevertheless at the present stage of development it was considered too early to define documentation for something which is still under development. 3 different kinds of documentation are necessary: users, developers, and testing. The XML schema developed by the NBB is also part of the documentation.

6_ Open source:

The group was informed that the SASG discussed the feasibility to put all the Demetra+ development in OS. The Board of the NBB will take a decision on putting as OS all the code developed by the NBB by the next SASG meeting (November 2009). The idea is to open the new tool as much as possible including any other additional module developed under the umbrella of the SASG. This also applies for the external modules (developed through the IT framework contract) including the connection to Fame, the guidelines for testing, the documentation.

Official version of the software will be always the ones posted in OSOR as it is the one supported by Eurostat and the ECB.

Every possible additional development outside OSOR will be of property of the developers.

7_ Organisation of the work, deadlines and milestones

JP: To provide prototype III in October. It will also present some simple wizards and a separate window presenting a summary of the SA process

DL, MS: prepare first draft of two new use cases (forecast and explanatory analysis)

DL to provide the databases to Eurostat (by the delivery of next prototype)

Eurostat to provide the document with the protocol for testing in 2 weeks; to post in Osor the document on the governance issues of Demetra+

User Testing Group: to react on the document with the testing protocol; to reflect on additional modules before next prototype

Next meeting

The group required to organise the next meeting in January 2010 instead than December 2009. The meeting will be organised in Luxembourg.

1

[1] The Bundesbank highlighted that according to the mandate it has to be tested whether the current version of Demetra fulfils the user requirements or not (all options of X-12 and same results as the original software).

[2]Comment coming by email after the meeting

[3] Comment coming by email after the meeting