Extensibilityof Demetra+.
Extensible features
Demetra+ is designed to allow external IT teams to add their own implementation for several features. Those features are listed below:
- Time series providers and browsers
- Repository for the definitions of SA processing
- Storage (or further processing) of the results
- Diagnostics on the SA estimations
- Summary (reporting) of a complete SA processing
- Data formatting (drag/drop and copy/paste)
External modules can be plugged into the main module of Demetra without changing the current assemblies.Their implementation must follow some constraints:
- Implementation of a predefined programming interface, provided by the framework
- Suitable modification of the configuration file of Demetra+ (xml file)
- Definition of a set of “properties” that can be displayed and modified using the standard “PropertyGrid”control [optional]
The concepts that the users’ defined assemblies have to deal with are defined by the framework:
- Time series and related topics
- TramoSeats / X13 specifications and results
Except for the repository of SA processing definitions, which must be unique (at least for the moment), the framework accepts several implementations of each feature. They will be called sequentially when needed.
The next table provides, for each feature, a short description of what it means and indicates the interface that it must implement. It also mention the default implementations provided by the framework
Features / Description / Interface / ImplementationsTime series provider / Provides the time series for a given source. A source can be a database, a set of files, a WEB service… / TSToolkit.ITSProvider / TSToolkit.TimeSeries.Xml..XmlProvider (specialized xml files)
TSToolkit.TimeSeries.TxtFormatter (txt files)
ExcelProvider4Demetra.ExcelProvider
(Excel workbooks)
ODBCProvider.ODBCProvider
(generic ODBC solution)
[Gtab.GtabProvider (internal NBB implementation for a DB2 database]
Time series browser / Provides a graphical interface for displaying the series available for a given provider / TSToolkit.TimeSeries.ITSBrowser / TSToolkit.TimeSeries.WorkspaceTree (browser for xml files)
TSImplUI.TxtBrowser (txt files)
ExcelBrowser4Demetra.ExcelBrowser
(Excel workbooks)
ODBCBrowser4Demetra.ODBCBrowser
(generic ODBC solution)
[Gtab.GtabBrowser (internal NBB implementation for a DB2 database]
Storage of SA processing / Storage of the definition of batch processing (series identifiers + specifications + last data + some additional information) / TSToolkit.Demetra.ISARepository / TSToolkit.Demetra.XmlSARepository
(xml file; can use compressed files)
Storage of a SA estimation / Storage of the results of a SA processing
(Could be a file, a database…) / TSToolkit.Demetra.ISAOutput / TSToolkit.Demetra.XmlSASerializer
(xml files, for TramoSeats and X13)
See Appendix 1 for an example
Diagnostics on a SA estimation / TSToolkit.Demetra.ISAQualityDiagnostics / TSToolkit.Demetra.ResidualsDiagnostics
(basic diagnostics on residuals: normality, independence, spectral analysis)
Summary / Synthesis of the processing of several series / TSToolkit.Demetra.ISAReport / TSToolkit.Demetra.TramoSeatsReport (summary specific to TramoSeats)
TSToolkit.Demetra.X13Report
(summary specific to X13)
Data transfer formatting / Formats rendered through drag/drop or copy/paste / TSToolkit.TimeSeries.ITSCollectionFormatter / TSToolkit.TimeSeries.TSCollectionExcelFormatter (xml spreadsheet format for Excel)
TSToolkit.TimeSeries.TSCollectionTXTFormatter (txt format)
Technical solution
From a technical point of view, the solution used to load dynamically users’ defined implementations is simple. It is summarized below.
- When it is launched, the program read an xml file, which contains information on the required users’ defined modules (file called [program].dconfig); it contains, for each extensible feature, the names of the assemblies and of the classes that need to be loaded.
- When one of those assemblies has been loaded, an instance of the given class is created. Such a class must either provide a default constructor or be conform to the singleton pattern, as it is defined in TSToolkit.Design.Singleton[1] (user’s defined attribute); see documentation of TSDefs.dll. That instance, which is supposed to implement the suitable interface, is then added to the chain of objects that provide the considered service. The application always interacts through that chain.
- The users’ defined assemblies might also use xml configuration files to define persistent properties. The template TSToolkit.Design.Folders<T> provides some facilities for a coherent location of configuration files.
We present in the appendix a simple example for the creation of a new TSProvider (random series generated by an Airline model), and the way it can be integrated in “sacruncher.exe” (batch processing)
Appendix
We present below the different steps for the creation of a new time series provider, using MS visual Studio (C#). The making of implementations of other features should be similar.
The time series provider is able to create monthly random series which follow an airline model.
Series are identified by:
- theta = regular ma parameter
- btheta = seasonal ma parameter
- seed = starting parameter for the random generator
- n = length of the series
The provider has some properties that should be serialized:
- (for example) the starting period of each time series
- The kind of random numbers generator
Step 1.
Create a new class library (RndAirline) and add the necessary references: TSDefs.dll and TSCore.dll and NbbTools.dll (for random numbers only); they should be located in “C:\program files\Eurostat\Demetra+.
Step 2.
Create the class for the provider. It will implements the TSToolkit.ITSProvider interface, will derive from TSToolkit.Design.Folders and will have the Singleton attribute.
The first condition is mandatory, the second one slightly simplify the serialization of the properties of the provider and the second one is just a “good practice”.
[TSToolkit.Design.Singleton]
public classProvider : TSToolkit.Design.FoldersProvider>, TSToolkit.ITSProvider
{
privatestaticProvider g_instance;
publicstaticProvider Instance
{
get
{
if (g_instance == null)
g_instance = newProvider();
return g_instance;
}
}
private Provider()
{
}
}
Step 3.
Define the way time series will be identified. For example: theta$btheta$seed$n.
Provide methods to create and to read the identifier.
public structSeriesID
{
publicstaticchar Delimiter = '$';
// Parameters of the series
publicdouble theta, btheta;
publicint seed, n;
// Create an identifier
publicoverridestring ToString()
{
StringBuilder builder = newStringBuilder();
builder.Append(theta).Append(Delimiter).Append(btheta)
.Append(Delimiter).Append(seed).Append(Delimiter).Append(n);
return builder.ToString();
}
// Read an identifier
publicbool Read(string s)
{
TSToolkit.Utilities.Tokenizer tokenizer = new TSToolkit.Utilities.Tokenizer(s, Delimiter);
if (!tokenizer.HasNextToken())
returnfalse;
if (! Double.TryParse(tokenizer.NextToken(), out theta))
returnfalse;
if (!tokenizer.HasNextToken())
returnfalse;
if (! Double.TryParse(tokenizer.NextToken(), out btheta))
returnfalse;
if (!tokenizer.HasNextToken())
returnfalse;
if (! Int32.TryParse(tokenizer.NextToken(), out seed))
returnfalse;
if (! Int32.TryParse(tokenizer.NextToken(), out n))
returnfalse;
returntrue;
}
}
Step 4.
Implement the interface “ITSProvider”
#region ITSProvider Members
// Synchronous provider
public TSToolkit.TSAsyncMode AsyncMode
{
get { return TSToolkit.TSAsyncMode.None; }
}
///<summary>
/// The most important method of the interface.
/// Delegates the actual processing to an internal methods
///</summary>
///<param name="identifier">Identifier of the series</param>
///<returns>The time series data</returns>
public TSToolkit.ITSData GetData(string identifier)
{
SeriesID id = newSeriesID();
if (! id.Read(identifier))
returnnull;
return Create(id);
}
///<summary>
/// Create a new time series, corresonding to a given identifier
///</summary>
///<param name="identifier">The identifier</param>
///<param name="loaddata">Indicates if the time series must contain the data</param>
///<returns>The new time series</returns>
public TSToolkit.ITS GetTS(string identifier, bool loaddata)
{
TSToolkit.TSS.TS ts = new TSToolkit.TSS.TS();
ts.Identifier.Identifier = identifier;
ts.Identifier.Source = G_SOURCE;
ts.Identifier.Name = "Random airline series: " + identifier; // for example
if (loaddata)
{
SeriesID id=newSeriesID();
if (id.Read(identifier))
ts.TSData = Create(id);
}
return ts;
}
// Not supported
public TSToolkit.ITSCollection GetTSCollection(string identifier, bool loaddata)
{
thrownewException("The method or operation is not implemented.");
}
// Only useful for connection to external data sources (DB, Internet...)
publicbool IsAvailable
{
get { returntrue; }
}
// Not supported (synchronous provider)
publicvoid QueryData(string identifier, bool action)
{
}
// Not supported (synchronous provider)
publicvoid QueryTSCollection(string identifier, bool action)
{
}
// Identifier of the provider
publicstring Source
{
get { return G_SOURCE; }
}
publicevent TSToolkit.TSDataHandler TSDataHandler;
publicevent TSToolkit.TSDataSetHandler TSDataSetHandler;
#endregion
The previous code could be reused with very few changes in other synchronous time series providers. The actual creation of the data for this example is presented below, without further explanations
///<summary>
/// Creates the actual time series
///</summary>
///<param name="id">The parameters of the series</param>
///<returns>The data of the time series</returns>
private TSToolkit.TSS.TSData Create(SeriesID id)
{
TSToolkit.TSS.TSData data = new TSToolkit.TSS.TSData();
data.Frequency = 12;
data.FirstYear = m_firstyear;
data.FirstPeriod = m_firstperiod;
// creates the generator
Nbb.Random.RandomEngine engine;
if (m_rnd == "xorshift")
engine = new Nbb.Random.XorshiftRNG(id.seed);
else
engine = new Nbb.Random.MersenneTwister(id.seed);
Nbb.Random.StochasticRandomizer rnd = new Nbb.Random.StochasticRandomizer(engine);
// generates the data. No special treatment for the starting values !!
double[] yprev = newdouble[13];
double[] err = newdouble[13];
double[] rslt = newdouble[id.n];
for (int i = 0; i < id.n; ++i)
{
double e = rnd.Normal();
double y = e;
// ma
y -= id.theta * err[0];
y -= id.btheta * err[11];
y += id.theta * id.btheta * err[12];
// ar
y += yprev[0];
y += yprev[11];
y -= yprev[12];
rslt[i] = y;
// update buffers...
for (int j = 12; j > 0; --j)
err[j] = err[j - 1];
err[0] = e;
for (int j = 12; j > 0; --j)
yprev[j] = yprev[j - 1];
yprev[0] = y;
}
data.Data = rslt;
return data;
}
privateint m_firstyear = 1980, m_firstperiod = 1;
privatestring m_rnd = "xorshift";
publicconststring G_SOURCE = "RND_AIRLINE";
Step 5. Persistence of the properties of the provider.
The persistence of the properties of the provider should be achieved as follows:
- Define an xml class with the properties
- Load the configuration file in the constructor of the provider
- Save the current properties in the IDisposable interface.
5.1 Define an xml serialization class
[XmlRoot("rndairlineconfig")]
publicclassConfiguration
{
[XmlAttribute("generator")]
publicstring Generator="xorshift";
[XmlAttribute("firstyear")]
publicint FirstYear=1980;
[XmlAttribute("firstperiod")]
publicint FirstPeriod=1;
}
5.2 Load the configuration file in the constructor of the provider, using information provided in the TSToolkit.Design.Folders base class.
// Name of the config file
publicconststring ConfigFile = "rndairline.dconfig";
private Provider()
{
try
{
// ConfigName generates the full name of the files, using information from the base class
string file = ConfigName(ConfigFile);
using (System.IO.StreamReader reader = new System.IO.StreamReader(file))
{
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(Configuration));
Configuration config = (Configuration)serializer.Deserialize(reader);
m_firstperiod = config.FirstPeriod;
m_firstyear = config.FirstYear;
m_rnd = config.Generator;
}
}
catch { }
}
5.3 Save the current properties in the IDisposable interface
publicvoid Dispose()
{
string file = ConfigName(ConfigFile);
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(file))
{
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(Configuration));
Configuration config = newConfiguration();
config.FirstYear = m_firstyear;
config.FirstPeriod = m_firstperiod;
config.Generator = m_rnd;
serializer.Serialize(writer, config);
writer.Close();
}
}
It should be possible to change the current properties of the generator through some public properties.
Step 6.
Using the provider with sacruncher.exe
Sacruncher.exe is a light console application which is able to process time series in batch. It uses the same design than Demetra+.
Loading the new random airline generator in the application is straightforward; the following line has to be added in the tsprovider section:
<tsproviders>
…
<tsprovider module="RndAirline" class="RndAirline.Provider"/>
</tsproviders>
The application is now able to deal with time series corresponding to the RndAirline provider.
The code below generates SAProcessing containing RndAirline series. It can be used with SACruncher (or with Demetra+, provided that its configuration file has been adapted).
TSToolkit.Demetra.SAProcessingDescriptor descriptor = new TSToolkit.Demetra.SAProcessingDescriptor();
TSToolkit.Demetra.SAProcessingDefinition def = new TSToolkit.Demetra.SAProcessingDefinition();
TSToolkit.TSS.X13.Specification spec = new TSToolkit.TSS.X13.Specification();
spec.X11 = new TSToolkit.TSS.X13.X11Spec();
def.Default = spec;
descriptor.Definition = def;
uint seed = 0;
for (double th = 0; th < 1; th += 0.1)
for (double bth = 0; bth < 1; bth += 0.1)
for (uint n = 60; n <= 240; n += 12, ++seed)
{
RndAirline.SeriesID id = new RndAirline.SeriesID();
id.theta = th;
id.btheta = bth;
id.seed = seed;
id.n = n;
TSToolkit.ITS s = RndAirline.Provider.Instance.GetTS(id.ToString(), false);
descriptor.Add(new TSToolkit.Demetra.SAItemDescriptor(s, false));
}
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(".\\rndprocessing.xml"))
{
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(TSToolkit.Demetra.SAProcessingDescriptor));
serializer.Serialize(writer, descriptor);
}
[1]A class could fulfil several features. In that case, the use of a singleton should be the preferred solution.