Managing Transactions in Support of Distributed Database Requests

Managing Transactions in Support of Distributed Database Requests

Proposal 29 addresses a serious functional problem for both client and server vendors—replication by the client of server maintained data. The goal is to ease the workload for both but the initial design suffers a variety of flaws. Servers would need to add significant tracking and retrieval capabilities to their maintenance processes to determine the nature and timing of changes and expose themselves to large demands for resources on single transactions by abolishing limits. Clients would still not know whether their replica database is accurate for their particular purposes despite the introduction of METADATA-FOREIGNKEYS.

Statement of Problem

RETS was designed to support a variety of resources on servers and search requests by clients in support of a wide range of anticipated functionality. The design does not require nor prohibit any particular functionality, leaving owners and users of MLS systems to negotiate the functionality to be delivered by servers and clients. An emerging class of functionality is database replication or downloads under control of a client application. The tools supplied in RETS in support of such functionality (Modification Time Stamps, Limit, Offset, METADATA-FOREIGN_KEYS, Key Fields, etc.) appear to be inadequate for the problem without private agreements as to interpretation by clients and suspension of limits by servers.

Solution Requirements

Clients need methods for discovering site-specific implementation variations that affect replication. Servers need non-discriminatory, non-proprietary methods to restrict performance-destroying demands on resources while permitting clients to reliably maintain copies of selected resources. This requires greater detail in the Metadata on the use of Modification Timestamps, fields critical to replication queries, and the applicability of limits. It requires a reliable method for clients using servers that do not suspend limits to step through a results set. Finally, it requires a method for clients to determine the relevant current settings for a resource/class.

Changes to the RETS Spec to Support the Solution Requirements

Collection of Complete Result Sets Without Disposing of Limits

RETS currently provides for a LIMIT to the number of records retrieved in a Search transaction by either the client or the server. It also provides clients the ability to specify a starting record OFFSET in that search. Client MAY request a suspension of the server imposed but the server need not honor the request. The implementation options for these parameters do not provide a predictable, reliable method for Clients to retrieve result sets that exceed a server’s limits. Servers do not have a non-discriminatory, non-proprietary method to selectively relax limits.

The Search transaction will add a new Optional Request Argument:

Key= ".EMPTY." | NextKeyValue
If this entry is not present the Server will not use or provide KEY information in generating the response.

If it is set to ".EMPTY." or a NextKeyValue and the request meets the server’s requirements for KEY support as defined in the TABLE-METADATA for the Resource and Class, the server MUST provide a <NEXTKEY> tag with a NextKeyValue if it sends a <MAXROWS> tag in the response. For Resources deemed replicable, servers SHOULD support either KeyLimit=”NONE in settings or have the resource’s Keyfield as KeySelect.

If it is set to NextKeyValue and all other arguments are identical to the request that generated the NextKeyValue, the server MUST respond with a result set that starts at an offset 1 record past the previous request and guarantees the request chain that started with an .EMPTY. value will not miss any records that still meet the query parameters. The results set MAY contain records that have been additionally modified since the beginning of a request chain that started with a Key=.EMPTY. argument. A NextKeyValue is opaque to the client, only valid once and only in the context of a single request chain. The request chain ends when the Server does not include a <MAXROWS> tag in the response. At the boundary of its limits, a server MAY send <MAXROWS> and <NEXTKEY> in a response and not deliver any records to the subsequent request (ReplyCode 20201), likewise ending the request chain.

The Search Response Body Format will be modified to support the NEXTKEY tag;

[ max-row-tag ]

[next-key-tag ]

[<RETS-REPLY [1*SP "ReplyCode=" quoted-end-reply-code 1*SP "ReplyText=" quoted-string *SP]”/>”]

next-key-tag= "<NEXTKEY>NextKeyValue</NEXTKEY>" CRLF
A tag that contains an opaque string to be used in the next transaction in this session as an offset to step through the results set of the query.

The Search Transaction will have two additional response codes:

Reply Code / Meaning
202xx / Invalid Key Request
The transaction does not meet the server’s requirements for use of the Key option
202xx / Invalid Key
The transaction uses a key that does not conform to the specification for use of the Key option.

Additional Metadata to Support Replication

The METADATA-TABLE will contain a new column:

Field Name / Content Type / Description
Implementation / 1024*TEXT / A comma separated list of keyword tokens with optional values defining implementation specific information about the server’s usage of the field. None of the tokens are required but use of the tokens indicates support for their meaning. Tokens starting with X are as yet unapproved extensions. The currently defined tokens are:
ClassTimeStamp :: the field is the modification timestamp for the resource/class
ModTimeStamp :: changes to this field update the ClassTimeStamp
ObjectTimeStamp=”ObjectType” :: the field is a modification timestamp for the objects of this type (Note that a field MAY timestamp more than one type of object such as Photo and Thumbnail resulting in multiple tokens)
ObjectCount=”ObjectType” :: the field is a count of the number of objects of this type (Note that a field MAY count more than one type of object such as Photo and Thumbnail resulting in multiple tokens)
ForeignKey=”ForeignKeyID” :: the field is populated via the specified foreign key
ForeignField=”SystemName” :: the field’s SystemName in the Child Resource/Class
KeySelect :: this field may be used in the Select argument of a Search Transaction that utilizes the Key argument
KeyQuery :: this field may be used in the Query argument of a Search Transaction that utilizes the Key argument
Deleted=”Value” :: the field is a flag indicating the record is deleted if it contains the Value

Client Access to Site/Resource Specific Settings

Introduce a new transaction:

GetSettings?Resource=ResourceID&Class=ClassID

This transaction returns the settings for a Class using CR/LF separated list of keyword tokens with optional values defining site implementation specific information about the Resource/Class. None of the tokens are required but use of the tokens indicates support for their meaning. Servers Tokens starting with X are as yet unapproved extensions. The currently defined tokens are:

CurrentTimeStamp=”DateTime” :: The current timestamp (ISO 8601 UCT) that would be used to set the modification timestamp of a record in this Resource/Class. Servers supporting this setting MUST provide a timestamp that will be less than or equal to that used for the ClassTimeStamp of any record subsequently modified.

LastTimeStamp=”DateTime” :: The latest modification timestamp (ISO 8601 UCT) of any record in this Class.

MinimumLimit=”Limit” :: NONE or the minimum limit for the number of result records for any search against the Class

KeyLimit=”Limit” :: NONE or the maximum limit for the number of result records for any search against the Class that meet the criteria for Key Selection

ReplicationSupport=”Flag” :: Y , N , or K. Y and K indicate the following configuration support:

1. All fields are marked as either ModTimeStamp or ForeignKey/ForeignField
2. LIMIT=NONE is supported or, if K, the KeyField is marked as KeySelect
3. An empty query is permitted to retrieve all records the current user is permitted to access; if K only KeySelect fields may be retrieved

The GetSettings Transaction will have the following response codes:

Reply Code / Meaning
0 / Success
20601 / Not Supported
The transaction is not supported for the resource/class.
20602 / Miscellaneous Error
The ReplyText provides additional information about the nature of the error.

Developer Notes

Use of Modification Timestamps

Because servers employ a variety of methods and business rules for collecting result sets, utilizing modification time stamp data from the returned results set MAY cause a modified record to be missed. Collecting the CurrentTimeStamp in the GetSettings transaction prior to collecting a results set is the only guaranteed reliable method for saving a timestamp for a subsequent request to only retrieve records modified since the previous request. A record modified so that parameters in the request or business rules disqualify it from the result set cannot be retrieved using a modified since request.

Detecting Deletion

Although servers MAY maintain a deletion flag field, detecting records to be removed that previously qualified for a result set can only be guaranteed by retrieving the Keyfield for all records that still remain in the result set. Servers MAY permit collecting other relevant fields like modification timestamps and object counts in the same query to allow clients a single step to retrieve the fields necessary to determine the records and objects still in the result set that need to be synchronized.

Foreign Key Fields

Fields from Foreign Keys (such as Agent and Office details for Property) may be synchronized from the Resource and Class in which they appear.