Data Logging with National Instruments Citadel 5

Data Logging with National Instruments Citadel 5

February 2006

Executive Summary

National Instruments Citadel is a proprietary, real-time logging database designed primarily to store single-point logging data. In contrast to relational databases, Citadel stores data in terms data traces which are compressed and optimized for real-time logging and historical data retrieval.

Citadel is an integral component of many National Instruments software products such as the

·  LabVIEW Datalogging and Supervisory Control Module (DSC),

·  Lookout,

·  DIAdem, and

·  VI Logger

As a common data storage mechanism, Citadel allows these software products to easily exchange test and measurement data. For example, a researcher could use DIAdem to analyze data generated by the DSC Module or Lookout without having to modify, convert, or reprocess the original data.

Citadel offers several major benefits that will improve productivity while saving time and money. These benefits are:

·  Citadel is optimized for real-time logging and historical data retrieval. This results in increased application performance while saving valuable system resources, especially in large applications.

·  Citadel includes advanced data visualization and management components. You won’t have to develop custom data visualization or management tools for your application.

·  Citadel doesn’t require any programming or prior knowledge of database systems to use. This saves time and money when developing an application.

·  Citadel is inherently network-aware allowing you to share data seamlessly among team members, or between data terminals.

·  Citadel is ready to use out-of-the box which greatly reduces start-up and system familiarization time.

You do not need to be familiar with the contents of this white paper to be successful with Citadel. However, if you are interested in learning about the technology behind Citadel or are planning a large project using Citadel, you may want to take a few minutes to review this paper.

Table of Contents

Executive Summary 1

Citadel Database Structure 2

Supported Data Types 3

Compression of Numeric Data 3

Traces and Subtraces 4

VI Logger Data 6

Database Files 7

Database Size Considerations 8

Historical Alarm and Event Data 8

Citadel Operations 9

Installing Citadel 9

Creating, Attaching, or Detaching a Database 9

Archiving Data 10

Backing Up and Restoring a Database 10

Retrieving Data from a Database 11

Inserting Data into a Database 12

Networking and Security Considerations 12

Networking 12

Security 12

Citadel Database Structure

Citadel organizes data into a three tiered hierarchy which contains the originating computer, process name, and trace name. See figure 1 for a graphical overview of a typical Citadel database.

Figure 1: Citadel Database Layout

Notice that the Historical Data Viewer also gives you access to multiple databases on the same computer and provides access to remote databases.

Besides traces views, Citadel also includes Waveform hierarchies which contain data logged from VI Logger, and dataset hierarchies which are created using the DataSet Marking I/O server in the DSC Module.

Supported Data Types

Though optimized for numerical data logging, Citadel supports a wide range of data types including numeric, discrete, bit array, waveform, string, and variant.

Once logged, data is available to other NI software such as DIAdem and can be retrieved programmatically using the Citadel API available in the DSC Module, through the Historical Data Viewer (HDV) component in Measurement and Automation Explorer (MAX), or through the Citadel ODBC interface. There is no C, C++, or VisualBasic API for retrieving Citadel data.

In summary, you can store the following types of data in Citadel:

§  Analog

·  Analog values are compressed based on the specified logging resolution

§  Integer &Bit-array

·  Discrete are not compressed when logged and are returned as an I32 typed values

§  String

·  Citadel supports three string types; LabVIEW style string, null terminated strings, and Unicode strings

§  Variant

·  Citadel stores raw variant data, but optimizes storage by only storing the metadata associated with a variant once instead of with each logged value.

§  Waveform

·  VI Logger and DIAdem are the only two NI software products that can interact with waveform data

Compression of Numeric Data

This section contains an overview of how Citadel compresses numerical data. Understanding this section is not required to use Citadel.

The purpose of this section is to present a simplified view of how Citadel compresses numeric data. This section does not describe the full algorithm Citadel uses for compression.

When queried for historical data, Citadel will always return a value equal to measured value ± the logging resolution at the time the data point was logged. The actual resolution of the returned data point may be higher than the specified logging resolution, but will always be within the logging resolution.

Data is typically acquired a data source such as a data acquisition board, analog input device, or PLC. In most cases measured values are passed through a series of deadbands before being sent to Citadel. Once in Citadel, the values are converted to deltas and compressed using the specified logging resolution. The example shown in table 1 uses a logging resolution of 0.1 which means all values retrieved from the database will be within ±0.1 of the actual recorded value.

Citadel uses a complex compression algorithm to store data, but this compression can be approximated using the following equation.

Equation 1: Simplified Citadel Delta Calculation

where y is the set of recorded values,
k is the set of logged values, and .

The following table provides sample delta calculations, using equation 1.

Table 1: Example Citadel delta calculation

The logging resolution is a key parameter in how Citadel compresses numeric data. The logging resolution can be any real number but should be chosen in concert with the logging deadband. For instance, if you specified a logging resolution of 10 units and a logging deadband of 1 unit, Citadel would log any change of at least 1 unit, but due to compression within the Citadel database any value change under 10 could not be discerned from data retrieved from the database. To prevent this from happening you should always specify a logging resolution less than or equal to the logging deadband. You should also specify the logging deadband as a power of ten unless you have an application-specific reason to do otherwise. Specifying a logging resolution of zero (0) will always log data with full resolution.

For reasons explained later in this white paper, the number of values in a run will always be less than 100, i.e. n will always be less than or equal to 100.

Traces and Subtraces

Citadel databases are structured around traces. Traces are uniquely named and point to historical data associated with that trace. When working with a Citadel database you interact with database based on trace names. Internally to Citadel, each trace is in turn made up of subtraces which contain a historical record of values for a single run of data. Subtraces are composed of data runs with constant resolution and logging parameters.

A variety of properties are associated with each trace, all of which you can view in the HDV. Right-click a trace in the HDV and choose Properties to display trace properties. The trace properties dialog box will look like the one shown in figure 2.

Figure 2: Trace properties dialog box

Data type indicates the most generic type associated with a trace. The most generic type indicates what sort of reader should be used to retrieve the majority of the trace’s contents.

Types from most to least generic are:

§  Binary (LabVIEW String)

§  Variant

§  Unicode String

§  ANSI String

§  BitArray

§  Logical

The startTime and endTime properties of the trace indicate the actual start and end times of the trace in Universal Coordinated Time (UTC) format. UTC avoids all conflicts associated with daylight savings time or varying time zones. Although Citadel stores timestamps in UTC, the times displayed in the Hyperbrowser are shown in the local time zone. If you expand the value column you will see that the current time zone name is appended to the end of the time string.

The lifespan property indicates how long trace data will remain in the Citadel database before Citadel reuses storage space taken up by old trace data. For instance, if a trace were configured with a lifespan of 10 days, only the 10 most recent days worth of data would remain in the database. After 10 days the oldest data would be overwritten. The lifespan of a trace is defined by the application that writes data to the database and can only be configured on a per-process basis, not a per-trace basis. A lifespan of 0 indicates an infinite lifespan.

The data lifespan is not a contract. Setting a lifespan indicates signals Citadel that data older than the lifespan can be deleted in order to save space, but the database does not guarantee that expired data will be deleted on any schedule. Lifespannig only occurs when the process that originally logged the data is actively logging new data to the database. For instance, if you copied data to a CD and, 15 years later, reattached your database to retrieved data, the data would not suddenly disappear due to the lifespanner.

The pages property indicates how many 4kb pages the traces is using and provides a rough estimate of how much disk space is used by the trace.

The subtraces property indicates how many subtraces a trace is composed of.

A subtrace is an arbitrary data stream containing a single type of data, plus associated meta-data. Subtraces are indexed by time, making it possible to seek to a time position within a subtrace without reading through the entire stream. Because data points are indexed by time, all data in a subtrace data stream must have nondecreasing timestamps. If a back-in-time event occurs, or the type of the data changes, Citadel must create a new subtrace. Changing the type of the data means changing the base data type (double, bool, etc.), or changing the format with which data is serialized to disk (e.g. changing the compression algorithm by changing the value or time resolution, changing the type of Variable being logged to a Variant subtrace, or even changing the attributes that are set on the Variant. Setting attributes on Variants logged to Citadel is not recommended.)

Though Citadel provides no direct interface to work with individual subtraces it’s important to understand subtraces because they can have a significant impact on database performance. In particular, if a trace contains many subtraces it will perform less well than a trace with fewer subtraces. In typical use cases Citadel will not need to create multiple subtraces.

If any local Citadel client, reader, writer, or the Citadel service fails to shut down cleanly while accessing a database, the database will be re-indexed on restart. When a database is re-indexed, we do not know if a subtrace may have been somehow damaged by the failure that triggered the re-indexing, so all existing subtraces are closed permanently, and new data will be logged into a fresh subtrace.

The following events can trigger the creation of a new subtrace:

·  The trace data type changed.

·  You log a “back-in-time” value.

·  The logging properties, such as logging resolution changed.

·  The Citadel service terminated abnormally (power loss, system crash, etc.).

Subtraces are made of data runs of n = 100 points. This applies mainly to analog traces where data is compressed based on delta values and the logging resolution. After every 100 points an absolute measurement value is logged which resets the compression algorithm. This prevents accumulation of error and reinforces data integrity within the database.

VI Logger Data

VI Logger stores waveform data to Citadel in groups of runs, which belong to VI Logger tasks. Each run corresponds to a single acquisition of data. A VI Logger subtrace consist of between one and three subtraces, a waveform subtrace containing data from hardware, a waveform subtrace containing calculated data generated by the VI Logger application, and a subtrace containing VI Logger event data. VI Logger subtraces contain raw waveform data, which are essentially arrays of uncompressed values. Waveform subtraces do not contain serialized timestamps because the waveform timestamp can be calculated from its start time and deltaT attributes. VI Logger uses uncompressed subtraces to maximize throughput to disk, at the cost of greatly increased disk and network usage. The VI Logger event subtrace uses the same value/timestamp format as single-point traces to log arbitrary-length boolean arrays without compression.

Database Files

Citadel databases are stored in a special set of files on your hard drive. A Citadel database typically resides in a folder unique that that database. Though not required, you should avoid placing non-database related files a folder that contains a Citadel database. You can only have one Citadel database in a given folder. A typical database is comprised of a set of files similar to those presented in figure 2.

Figure 3: Typical Citadel database files

The number of CDPG and CDIB files will vary depending on the amount of data in a database. The nodetree.*, pid.cdih, and stridm.cdin files contain important information about the structure of the database. The mssql.* files contain historical alarm information and are automatically created the first time alarm or event data is written to the database.

Important: You should never modify, move, or delete a database file while the database is attached. Doing so will result in a database corruption. If you modify or delete a database file while the database is detached you may not be able to reattach the database and could loose some or all of the data in the database. If you move or copy a detached database, be sure to move/copy all database files.

The CDPG files contain trace data. Citadel stores data in a compressed format therefore it is not possible to read and extract data from these files directly. You must use the Citadel API in LabVIEW DSC or the HDV to access trace data. See the Citadel Operations section for more information about retrieving data from a Citadel database.

Each of the 1,024 Kb CDPG files contains a set of 4 Kb pages. Each page contains data for a single subtrace. Citadel will attempt to use all 4 Kb of space in a page before opening a new page. If a subtrace is terminated or a new subtrace is started before the end of a page the remainder of the space in the page will go unused. Citadel will only reclaim the remaining unused space if all trace data in that page is removed or deleted.