Report on WMO Guide to WWW Data Management WMO 788

For the Meeting at Shinfield, England 13th-16th May

Of the Expert Team on Integrated Data Management

Summary using the Contents list:

WORLD METEOROLOGICAL ORGANIZATION

GUIDE ON WORLD WEATHER WATCH
DATA MANAGEMENT

KB in 2 parts)

Geneva, April 1992

TABLE OF CONTENTS

CHAPTER 1 - The WWW Data Management Concept

1.1 Purpose and Scope of WWWDM
1.2 Principal Long-term Objectives of the WWWDM
1.3 Programme Organization
1.4 Major Influences
1.5 Programme Components and Plans
1.6 Status of the WWWDM Activities

Update Status: This needs entirely rewriting, with a modern focus on data management, and reviewed objectives, programs, plans and activity status.

CHAPTER 2 -Requirements for the Development of Specific Data Management Functions

2.1 Introduction
2.2 Data Representation
2.3 Data Exchange
2.4 Data Monitoring

Update Status: Obviously this needs a complete review, but the requirements stated in 1992 are interesting in their own right, and are commented on in detail later. We are still fleshing out the practical implementation of many of the 1992 requirements, while others are still over the horizon.

CHAPTER 3- Meteorological Data Representation: Overview of Codes

3.1 Introduction
3.2 The Manual on Codes
3.3 The Numbering System of Code Forms and Binary Representations
3.4 Code Issues

Update Status: This can still stand with a little updating, but it must be extended to reference new Internet standards from organisations such as ISO W3C openGIS and IETF.

CHAPTER 4 - Meteorological Data Representation: Alphanumeric (character) Codes

4.1 Introduction
4.2 Code Forms
4.3 Coding Procedures
4.4 Aeronautical Meteorological Codes
4.5 Observing Station Identification
4.6 Dates and Changes in Meteorological Codes

Update Status: The content can stand after details are brought up-to-date.

CHAPTER 5 - Meteorological Data Representation: Binary Representation Form

5.1 Introduction
5.2 BUFR (FM 94)
5.3 GRIB (FM 92)
5.4 Issues Raised by the use of Binary Representation Codes

Update Status: The content can stand after details are brought up-to-date.

CHAPTER 6 - The Use of Computer Graphics for Meteorological Data Representation

6.1 Introduction
6.2 Current Graphics Systems and Techniques
6.3 The User Interface
6.4 Standards for Meteorological Graphics
6.5 Representation of Image Data
6.6 Standard Environments

Update Status: This is considerably out of date, both in detail and in philosophy. The move from bespoke-programmed to package created graphics and global standard formats should be addressed. However the status of SVG (Scalable Vector Graphics) while looking to be the new de-facto standard, is still under development.

CHAPTER 7 - Data Bases in a Meteorological Environment

7.1 Introduction
7.2 Database Technology
7.3 Query Languages
7.4 Data Base Technologies Employed by Meteorological Services
7.5 Storage of Quality Control Information in Meteorological Data Bases

Update Status: None of this is really out of date, it just has a dated feel and approach. It has to be updated with reference to practical experience of RDBMS. Sections on XML databases and Xquery would be necessary.

CHAPTER 8 - The Distributed Data Bases Concept

8.1 Introduction
8.2 The Relationship of the DDBs to the Existing WWW Infra-structure
8.3 Implementation of the DDBs
8.4 A Possible Implementation Strategy
8.5 The Role of the DDBs in Data Collection

Update Status: This has still not come about in practice, but it will have to be updated with reference to FWIS, Unidart, cataloguing etc.

CHAPTER 9 - Data Monitoring

9.1 Introduction
9.2 The Identification of Errors and Deficiencies within the WWW
9.3 The Role of the GOS
9.4 The Role of the GTS
9.5 The Role of the GDPS
9.6 The Role of Data Management

Update Status: This is still relevant, but needs updating with reference to current data monitoring practice.

CHAPTER 10 - Computer Software Exchange

10.1 Introduction
10.2 Software Exchange
10.3 Software Standards

Update Status: Well out of date, both in practicality and outlook. Open Software and Commercial Off-The-Shelf functionality and availability have overtaken the concept.

CHAPTER 11 - Endnote

LIST OF ACRONYMS

Update Status: X stands for X-Windows!!, XML and any of the W3C standards wasn't even on the horizon.

Overall Summary. With so much out of date, this cannot be updated as it stands. It would require a wholesale rethink. The 1992 document was written by and for WMO with a typically introverted community approach (however - with a worthy attempt to encompass outside influences). In the intervening period, the wider world of Information Technology has taken enormous strides and dominates all but the special detail of WMO business processes. Any rewrite is in danger of becoming obsolete in some sections before it is completed.

Discussion about the Requirements section of Chapter 2.

Requirements listed:

2.2 Data Representation

Existing Requirements:

  1. Maintain the alpha-numeric forms for the representation of meteorological observations;
  2. Maintain and enhance the binary universal form for the representation of meteorological observations (BUFR);
  3. Maintain the alpha-numeric forms for the representation of gridded meteorological fields;
  4. Maintain and enhance the binary form for the representation of gridded meteorological fields (GRIB);
  5. Centres continue to meet their requirements to exchange basic observational data in a format which ensures maintenance of existing services. BTAB (BUFR Tabular Format) should be re-examined with a view to increasing its efficiency, both generally, and particularly with regard to reducing the length of a BTAB message in order to use this form for new data only available in binary representation form. During the transition period between character and binary representation, the requirement for products may need to be met with some duplication of products in each of these formats.

Comment:
These requirements still apply and are being serviced. Requirement e) reflects the limited bandwidth of fixed GTS links.

New Requirements:

  1. Establish the policy that all meteorological data should be exchanged in the appropriate binary form, and such forms be self-defining, whenever possible;
  2. Expand BUFR to permit encoding of observational metadata;
  3. Establish a mechanism to permit the encoding of information necessary to classify characteristics of the observing platforms;
  4. Expand the adoption of standards for the representation of vector graphic and image data beyond facsimile group 3T4;
  5. Establish a means for the identification of the data contents (as opposed to the addressing) of messages.

Comment:
Many of these have been achieved or are being addressed even by this ET (e). However all these requirements should recognise the affect of the development of new international standards.

2.3 Data Exchange

Existing Requirements for the GTS:

  1. The GTS should continue to deliver observational data as quickly and reliably as possible;
  2. The GTS should continue to deliver routinely distributed processed data as quickly and reliably as possible;
  3. The GTS should prevent unauthorized access to the communication facilities and to the databases;
  4. The same information should not be sent in more than one code form on the same communication link, unless it cost-effectively suits centers' requirements under bilateral or multilateral arrangements.

Comment:
Unobjectionable common sense. However d) reflects the limited bandwidth of fixed GTS links..

New Requirements for the GTS:

  1. The GTS should establish and maintain a comprehensive catalogue of meteorological messages for observational data and processed information, (an improved WMO-Publication No. 9, Vol. C) updated in near real-time;
  2. The GTS should develop a comprehensive mechanism to route ad hoc requests to the appropriate center (GTS or GDPS) for response. Although it is the responsibility of the GTS to develop the request/reply mechanism, it is the responsibility of the WGDM to develop the specific request/reply message format in co-ordination with the WG/GTS;
  3. Standard interfaces for queries to and replies from databases should be adopted with due consideration of international standards;
  4. A mechanism for the transmission of information on the status of a GDPS or other appropriate center's processing and database should be established;
  5. A mechanism for the transmission of information on the contents of a GDPS or other appropriate center's database should be established;
  6. Transmission of larger volumes of information and larger message sizes should be supported. The GTS should consider alternative types and levels of service that would be required for several ranges of traffic volume in the future;
  7. The increased traffic volume must not be allowed to adversely impact the current rapid distribution of observational data. The GTS should consider the implementation of a flow-control mechanism to ensure the timely delivery of observational data and the non-interference by the transmission of products and replies to ad hoc requests.

Comment:
While these requirements reflect the capability of the GTS to extend itself, the Inter Programme Task Team on Future WMO Information Systems recognises that many of the requirements lead to the need to replace the message switch basis of the current GTS with an Internet based GTS which can support continued bandwidth increases using cheap commercial services and protocols. Many of the requirements are not capable of being handled by the message switch paradigm. Requirement a) is being addressed by this ET (and by other projects such as UNIDART). The specific example of WMO Pub No 9 was suggested at the Geneva meeting by Alexander Besprozvannykh "Applying the Resource Description Framework(RDF) to hydrometeorological metadata"

However, these requirements are not sufficient. Before the request/reply capability can be used to satisfy the needs expressed by the GDPS, both the GOS and also the GDPS must satisfy certain requirements as well:

New Requirements for the GOS:

  1. The GOS should liaise with Data Management concerning the data representation forms to be adopted by new observing systems;
  2. The GOS should establish and maintain a comprehensive catalogue of meteorological observation stations and schedules (an improved WMO-Publication No. 9, Vol. A) updated in near real time;
  3. The GOS should establish and maintain the capability to recognize and respond to notifications sent from the GDPS regarding erroneous meteorological observations;
  4. The GOS should establish and maintain the capability to recognize and respond to requests for retransmission of meteorological observations.

New Requirements for the GDPS or Other Appropriate Centers:

  1. The GDPS or other appropriate center should establish and maintain database catalogues;
  2. The GTS nodes have limited storage capacity to support their store-and-forward capability, typically equivalent to a volume of data ranging from one hour to a few hours. The GDPS or other appropriate center should therefore establish and maintain the capability to recognize ad hoc requests and issue replies for information that does not exist at the GTS nodes;
  3. The GDPS or other appropriate centers responsible for maintaining databases should be able to respond to requests for large portions of their databases in the form of large files for efficient exchange between GTS centres.

Comment:
Many of the above requirements across chapter 2 do not reflect the emergence and dominance achieved by the development of international IT standards or the adoption of de-facto standards by the wider world IT community. Many of these have bypassed the WMO developed standards and made us a more isolated community within IT than ever we used to be.

The WMO over its lifetime has had to develop or adopt new practices in the fledgling IT industry when we were the only users. We had to develop all our codes, define and protect our data formats, data compression and communication protocols and suffered the expense of doing so. From the development of Internet software, new visualisation techniques, compression mechanisms, new communications and recently document definition standards, the outside world now has cheaper, better, faster software, communications and processes. WMO can no longer afford to develop new mechanisms itself, but must carefully pick and choose from these offerings, recognising that while we may be able to modify or to influence them, we can no longer expect to control them.