Nebraska Dept of Roads (DOR) ATIS Messages, Report Summary

(Draft Rev 01 Working draft, )

A summary of the XSD schemas and XML messages provided by the service end points found at http://aztechrads.org:8080/c2cxml/services/listServices (‘the deployment”) during the period from April 26th to May 18th 2007. During this time approximately 250 representative XML files were recovered and examined.

Summary: The deployment, with a few minor exceptions noted below, meets the formats outlined in the schema set that it uses. These locally developed schema are very substantially correct and meet the ATIS standards, the only deviations found were attributable to using an earlier version of the standard (revision #64) then the now current draft #79. The deployment can be viewed as having implemented the ATIS standard correctly, although several minor improvements could still be made.

This report is intended to establish the level of conformance of the above services and messages with the requirements outlined in the SAE ATIS Standard (SAE J2354) and its associated documents. It focuses on the structural expressions and formats found in the messages, and that they meet defined details found in the subject standards. It does not deal with any fitness of use of the data itself.

The XSD schema set found on the NebDOR documentation page (above link) consists of nine schema files, most taken from the ATIS national standards effort revision #64 and its associated supporting files.[1] A local copy of these files was made in the normal fashion and examined further as noted below.

As a preliminary step to validating the message and studying the schema set, all these files were downloaded to a local file system for further use. In this case the ITS schema repository was used and an entry created for the Neb-DOR work and labeled with revision “00-00-00” This can be found in the repository at http://serv1.itsware.net/itsschemas/NebDOR/NebDor-00-00-00/

The above link presents the schema as found (unchanged). As changes were needed to make the schema compile and comply, revisions were sequentially numbered in the normal way. A ReadMe.txt file in each directory notes the changes that were made to each file set. The changes required to use the schema in actual XML document comparisons will now be outlined. The fact that these changes were needed is typically indicative that the final schema, as presented, is NOT being used to validate the final message being produced in the production system. As a best practice, this should always be possible to do, even if it is not routinely done in operation.

The first change required was that of the defective ITIS codes[2] and phrases (found in the ITIS schema file at http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-00/itis.xsd which were present in the issued standard at that time. The use of “ID fields” in that standard required a non-numerical first character (a rule of XML which was not widely known until after balloting) a problem which has since been resolved by inserting a “_” in front of each number. This change[3] was made, and revision 01 was created.

The other changes needed to use the schema files were very minor. The complete notes can be found at http://serv1.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/ReadMe.txt but in essence were the following:

· The event count limit facet was increased from 100 (in the std) to be 1000. This allows sending every roadway segment of the entire state in one message.[4]

· Two occasions of duplicate items were created as a side-effect of adding the “_” (UVLevel and CompassDirection) This was resolved by moving them to the ATIS namespace (as was done in the latter ATIS drafts).

· Two occasions in which new items were added to more current data sets were noted as well (TransitLocations and TravelDirection) This was resolved by adding them to the schema, although the two data concept do not appear to be ever used in the actual XML.

With these changes the XSD schema file set completely validates against multiple tools. HTML documentation from that validation can be made, and an example of this can be found at: http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/docs/

Basic Checks and Assertions:

The WSDL does not link to the proper XSD schema set here.
There is no WSDL present in this system, and this deployment pre-dates the establishment of WSDL standards for ATIS (or in general for ITS at all).
Normally there would be an analysis inserted at this point in the report to compare the form of the deployment WSDL to that specified in the various ATIS and C2C rules.
The normal ATIS style for this is a specified request message which causes a response message, both of which follow the formats outlined in the standard for top level messages. The response, (which the current system sends in response for a simple HTTP Get solicitation) is of the correct form.
The complete schema set is well formed and does validate with older tools such as XMLSpy Rev 5 sp2. It also does validate with current tools such as XMLSpy Enterprise 2007 sp2.[5] A detailed breakdown follows:

Well-formed Validates-5.2 Validates-2007

a. ATIS.xsd Ö Ö Ö

b. DSRC.xsd Ö Ö Ö

c. IM.xsd Ö Ö Ö

d. ITIS.xsd Ö Ö Ö

e. Local.xsd Ö Ö Ö

f. LRMS.xsd Ö Ö Ö

g. NTCIP.xsd Ö Ö Ö

h. TCIP.xsd Ö Ö Ö

i. TMDD.xsd Ö Ö Ö

j. test.xsd Unused but present

The schema set consist of nine files as follows. The links provided below come from the repository, not the deployment web site.

a. ATIS.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/atis.xsd

b. DSRC.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/dsrc.xsd

c. IM.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/IM.xsd

d. ITIS.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/itis.xsd

e. Local.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/local.xsd

f. LRMS.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/lrms.xsd

g. NTCIP.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/ntcip.xsd

h. TCIP.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/tcip.xsd

i. TMDD.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/TMDD.xsd

j. test.xsd http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/test.xsd

The schema set does follow the ITS requirement to use a character encoding of UTF-8.
The schema set does follow the FADD naming styles for content organization (i.e. ATIS content is found in a namespace called ATIS etc).
The schema set does follow the older style of file naming each FADD[6] without providing a release string in the name.
This was the norm when this schema set was created.
The schema set does follow an older naming style and convention that provided a single local.xsd file for all the local extension content.
This was the norm when this schema set was created.
Current best practice is to have a FADD_local_xx-xx-xx.xsd style file for each FADD used because this allows better management of local changes and is more easily managed when multiple projects have a need to exchange data with others who are using multiple and different release versions of one or more standards.
It does follow ITS conventions for naming the imported files with relative path names (rather than absolute path names).
It does allow the automatic generation of XML documentation sets from the schema with common tools such as XML Spy without major change or effort. An example of this can be found at http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/docs/
Any message details mentioned in the rest of this document will have links to this documentation.
The sequence of top level messages defined in the event and other operations does seem to be correct for an ATIS exchange.
No subscription styles are implemented, and as such there is a “one message with everything in it” format.
The actual XSD schema files are available on the ITS repository with the minor changes as previously noted.

Validating XML Messages

The next section of this report deals with the recovery of actual XML messages that the deployment server produces, and compares these with the schema it claims to conform to and with the schema of the national ATIS message set. Before proceeding, it may be helpful to recall that XML is used both with and without a schema in operational systems.

It is important to bear in mind that the mere presence of a schema declaration element in the XML does not slow processing down and does not require any party to validate each message against that stated schema. Nor does it say where others can obtain the subject schema. What it does imply is that the issuer of the message asserts that there is (somewhere) a schema by that precise name to which the current XML message will validate. It may also show where this schema can be obtained. It may be that the schema which is represented is NOT the schema that the message issuer used, or that the receiver will use. It is fairly common to have centers use their own “extra tight” schema for in-house use and express the XML with a “looser” one.

As a general rule, when you see any XML messages that do not have a schema assertion line present in them, be very suspicious regarding their alleged adherence to any schema.

The current deployment DOES purport to comply with the schema outlined above. It DOES NOT declare this schema in the XML instances that it creates. We can (and do) add that line in our internal processing to aid in the comparison of each message to the schema.

Obtaining the Test Data

The normal client code base was used. This is a simple MS dot.Net application tool developed by SCSC which can request well formed WSDL and XSD files from a deployment’s documentation and build a client node to request and gather information from the deployment. Once set up, it is typically run for a number of days to extract the needed data into files for comparison and analysis. Data files are stored in a folder with a file name containing the run time, and dated for later use. The time period between runs was set for every 5 minutes in this case. The source code for this program is readily available from SCSC and is often used to create example client code for deployments. A screen shot is shown below.

Validating the Selected XML messages against the Deployment

Here is the basic rule for testing a deployment’s messages (or any message set claiming conformance) against the national ATIS standards:

A objective and testable aspect of any well done local deployment that claims to conform to one or more of the national standards can be stated as follows: Any valid XML message instance which validates to the local deployment schema set should also completely validate to the national standard schema set. The only exception to this rule would be the presence of locally defined element content. If the local extension portion of the national standard schema set was set to contain the “any” construct,[7] then any valid XML instance which validates to the local schema set should also completely validate to the national standard schema set. This is a necessary (but not sufficient) criterion to ascertain conformance with the standard.

The deployment DOES produce XML messages that in fact do validate against its own schema. Of the recovered and tested messages, NONE was found which would not validate to the schema.

These two simple sentences represents a huge amount of work on the part of the deployment. Most other conformance reports have 10~20 pages at this point going over why the deployment’s messages failed to meet its own documentation and why what was produced was incorrect and not in keeping with the spirit, or the letter, of the standard. This deployment has none of these errors. At the time this was written (mid 2007), this deployment is the most correct we have ever examined.

Validating the Selected XML messages against ATIS

The current messages DO NOT validate against the current ATIS standard due to evolutionary changes which have occurred since the deployment was created. None of these seems major or difficult to overcome, should the deployment seek to comply with the most current standard.

For those seeking to play with the current deployment validated against the current ATIS drafts, here is a link to begin the process: http://www.itsware.net/itsschemas/NebDOR/NebDor-00-00-01/example/AltSet.xml

A few of these changes are noted in passing

· The <header> structure in the current standard now demands a message count value to fulfill a requirement to detect lost messages under some subscription conditions. A deployment like this (a fire hose feed) would set this value to zero, but it needs to be present to conform.

· The values of time and date in the deployment are built using short strings. The current standard has evolved beyond this to using the native XML time and date formats. Every entry with timeStamp will need to be revised to use XML time.

· The use of outer “plural tags” has been removed in most of the XML of the current standard. For example, an entry such as responseGroups to enclose a sequence of one or more responseGroup items in the standard has been discontinued. Therefore the use of this type of tag in the deployment should be removed.

· There is a number of string values in the deployment XML which, while valid, could be placed in better suited elements. As one example of this, the string “NDOR HCRS” is placed in the tags lastName which is a part of the person tag inside of sender. This would be better placed in the <agencyName> tag of <sender> to reflect the semantic meaning of the sender.

· The tag lateralOffsetRef is now required in the element linearReferencePoint, although it can be set to be a value of zero (to indicate that the event is on the linear reference line itself). This tag is also now required when linearReference is used.

· The tag for geographic coordinate is no longer called geographicCoordinate and is now called geoCoor in use.

· The tag for ITIS phrases is now in lower case, as in <itis>.

Summary Thoughts

This report is the result of being asked to review the Nebraska DOR ATIS deployment XML with respect to the ATIS standards. Achieving true conformance with this or any other ITS standard is a difficult and thankless job of countless small details. It should be kept in the mind that the deployment, as an early adopter, had less guidance regarding best practices than now exists to help others. In that light, it is the purpose of documents like this to be overly negative so that the rigorous goal of better deployments can be achieved.