ORFEUS-ECN
Detailed Functional Specifications
Version / V07.00Author / M5
Date / 05/03/2012
Target
Mr Blau / RAILDATAMr Preval / LUSIS
Mrs Manea / LUSIS
Mr Vega / LUSIS
References:
Reference / Title of documentSD-ORFEUSV4-01-V01.16.doc / ORFEUSV4 specifications
History of changes:
date / version / author / comment02/03/2009 / 01.00 / M5 / Creation
19/03/2009 / 02.00 / M5 / Update
28/05/2009 / 03.00 / M5 / Update ECN Schema 1.3
25/10/2010 / 04.00 / M5 / Integrate GRM message and ECN flow exceptions
03/02/2011 / 05.00 / M5 / Update Acceptance rules (ChRq 146)
05/12/2011 / 06.00 / M5 / INTERVENTION 103
05/03/2012 / 07.00 / M5 / INTERVENTION 117 (PCN & CANCEL)
1ORFEUS-ECN system functions
2ORFEUS-ECN system architecture
3ORFEUS-ECN system modules
3.1INCOMING ECN MESSAGE PROCESSING
3.1.1Storing INCTINLOG, INCTINDATA
3.1.2Message manager
4ECN APPLICATION CONTROL
4.1ECN Message content
4.1.1Message types
4.1.2Dictionary
4.1.3ORFEUS Members
4.2Message acceptance
4.3Message storing
4.3.1MESSAGE_ORFEUS
4.3.2MESSAGE_ORFEUS_DATA
4.3.3DOSSIER Data Model
4.3.3.1DOSSIER_WAITING
4.3.3.2DOSSIER_CARRIERS
4.3.3.3ROUTING_DESTINATION
4.4Update ORFEUS :
4.5Message processing and routing
4.6Error handling
4.6.1List of error codes for the application control
4.6.2Log Messages
5Outgoing message processing
5.1.1Changes made in the outgoing message
5.1.2ECN to CTD/UTD
5.1.3ECN to TD (ISR)
5.1.4ECN to GRM
1ORFEUS-ECN system functions
The RAILDATA ECN Working Group it was decided to create a central system processing the message flow for the paperless international freight traffic (ECN = Electronic Consignment Note).
This proposal concerns the design and realisation of a central message broker for ECN.
The ECN application is a platform for acquisition, routing and storing ECN FRET messages, hosted by RAILDATA platform and integrated by ORFEUS system.
The ECN central message broker functions are:
-ECN message acquisition
-ECN message control
-ECN message routing management
-ECN message conversion, enrichment or filtering for outgoing generation.
-ECN message storing
The ECN processing is:
- The ECN message is sent by a partner by ftpClient application and stored in the partner’s incoming message box.
- The ECN message is read and checked against the xsd schema : log, reporting and audit trail are generated
- The ECN message is read and checked against the consistency rules: schema : log, reporting and audit trail are generated
- The message is validated against the acceptance rules : message accepted depending on TYPE and the participating RW role in the transport
- The message is validated against the dossier number, version number and consignment number: these values are unique ; the control is done over all CTD and ECN messages stored (1 month)
- The message is stored (audit trail)
- The central message broker calculates the destination partners (routing rules) depending on the TYPE of message, on the participating RW role in the transport
- The central message manages and controls the message flow process (hand-over) and the broadcast of the messages depending on the participating carriers role
- For each destination partner
-Generate the outgoing messages depending on TYPE
-Set valued for outgoing message depending on the partner role in transport (TYPE, timestamp, sender, receiver, etc.)
-Convert to CTD/UTD (if necessary) if destination partner is ORFEUS members participating in transport but not ECN member
-Convert to TD for ISR
-Manage the flow sending (flow control rules)
Transversal functions:
-Log : describes the sequence of the application processing
-Reporting : reports are generated (and distributed to the declared contacts) for processing result (applied rules result)
-Audit trail : used to track incoming and outgoing files/messages processing
-HBM communication management : controls the status of the communication between the central system and the partners
-In/Out Commit/Rollback message management for information reliability : is a standard Tango module, which guarantees the delivery of an asynchronous message
2ORFEUS-ECN system architecture
The ORFEUS system integrates the ECN central message broker and become the new ORFEUS system having the same architecture.
-Application Server = the central message broker application = TANGO application
-Database Server = ECN storage on MySQL Database Server
-System Supervision by NAGIOS
The ORFEUS-ECN system interface with ISR system is realised by Transport Description (TD) file.
The common services are:
-Tango technical layers (read, write, report, log, audit trail)
-Storing Database
-NAGIOS Supervision
From functional point of view the ECN and ORFEUS application share the dossier number/consignment number control module: a number/consignment number is unique for both ECN and ORFEUS flows.
The communication between the partners and the ECN central system will be assured by “orfeusFtpClient” application (“intelligent” Java ftp application).
This documents details only new functionalities introduced by ECN messages processing and modification of the previous ORFEUS modules generated by the integration.
3ORFEUS-ECN system modules
The system integrates
new processes for incoming/outgoing ECN flows.
In the first step the ECN flow are used by ORFEUS members wishing to manage ECN and non ECN transports.
new dossier checking procedure for ORFEUS flow which shares a common DOSSIER base with ECN flow
new application control for ECN flow
new routing mechanism based on the hand over rules
ECN-ORFEUS interface for non ECN members
ECN-ISR interface for transport matching and ETA calculation optimisation
3.1INCOMING ECN MESSAGE PROCESSING
The ECN message is fully XML and is checked against the xsd schema.
A process is defined for each partner and data flow.
A directory is set-up for each partner and each data-flow (or a filename rule within the CTD flow directory)
The incoming process contains 5 services (as described in the ORFEUS Detailed Specification document):
File reader
Session manager
Interchange manager
Message manager
Commit/Rollback manager
Thee incoming status message process collaborates with the
Integration report manager: generates file integration reports and sends reports by e-mail to NIS partner
3.1.1Storing INCTINLOG, INCTINDATA
The INCTINLOG table is used to store received files. The content is used by the Audit Trail function.
Column / Type / ContentINTCID / INT / Primary key : MySQL sequence
FILENAME / VARCHAR (255) / Input file name
SENDER_RAIWAYCODE / CHAR(4) / FileCreator
SENDER_PARTNERCODE / CHAR(2) / SendingPARTNERCODE (config)
TIMESTAMP / DATETIME / Timestamp
STATUS / INT / to be processed, processed, rejected, etc.
NUMBER_OF_MSG / INT / Number of messages in interchange
PROTOCOL / VARCHAR (12) / ECN
FORMAT / VARCHAR(10) / XML
FILENAME / INDEX / `FILENAME`,`SENDER_RAILWAYCODE`,`STATUS
TIME / INDEX / TIMESTAMP
INCTINDATA
Column / Type / ContentINTCID / INT / Primary key: MySQL sequence
DATA / MEDIUMBLOB / File Content
3.1.2Message manager
The message is checked
Against the xsd schema to verify the syntax of the message; in case of error
- anerror report is generated (by the Integration report manager)
- a log entry is generated
- an audit trail entry is generated
Against a list of integrity rules to verify the data sent is coherent ; in case of error
- an error report is generated (by the . Integration report manager)
- a log entry is generated
- an audit trail entry is generated
The integrity rules described by : are defined by an external xsl file.
4ECN APPLICATION CONTROL
The functions are:
-to decide if a message is accepted or not
-to store message information needed for flow control and hand-over
-to decide the actions to be performed on the message
-to activate the routing function for a list of destination for a message
-to filter information depending on the destination role
-to send the message to the list of destinations
-to keep an audit trail of the processing
4.1ECN Message content
4.1.1Message types
The messages handled by the system are:
Message Type / DescriptionPRN / Prior Notification
ECN / Electronic consignment note. (Legally binding document)
PCN / PCN: ECN was printed. Therefore this is a ‘printed consignment note’. Legally binding is now the printout.
INFE / This message has only informative character and can be used besides the ECN transmission.
INFP / This message has only informative character and can be used besides the PCN transmission.
CANCEL / This message announce the cancelation of the transport
DEL / This message announce the suppression of a carrier from a transport
HNDO / This message has only informative character and can be used besides the handover transmission
AOD / AOD: Return message after delivery. This message is used to inform the contractual carrier about the delivery of goods to the consignee (advice of delivery).
ACK / Acknowledgement of hand-over
GRM / The GRM will be used to notify the sender that the next carrier in the transport is not able to process ECN messages
The GRM message has a different structure ; see Annex for GRM message creation.
4.1.2Dictionary
Element / OriginFileCreator / CIMECNMessages.InterchangeHeader.SenderIdentification
FileReceiver / CIMECNMessages.InterchangeHeader.ReceiverIdentification
SendingCarrier / ECNs.ECNHeader.SendingCarrier
ReceivingCarrier / ECNs.ECNHeader.ReceivingCarrier
ShipmentType / ECNs.ECNHeader.ShipmentType
MessageType / ECNs.ECNHeader.MessageType
DossierNumber / ECNs.ECNHeader.DossierNumber
VersionNumber / ECNs.ECNHeader.ECNVersionNumber
ShippingCountry / ECNs.ECN.AcceptancePoint.Station.Country
ShippingStation / ECNs.ECN.AcceptancePoint.Station.Code
AcceptanceDate / ECNs.ECN.AcceptancePoint.AcceptanceDate
ShippingCarrier / ECNs.ECN.AcceptancePoint.CarrierCode
ConsignmentNumber / ECNs.ECN.AcceptancePoint.ConsignmentNumber
DestinationCountry / ECNs.ECN.DeliveryPoint.Station.Country
DestinationStation / ECNs.ECN.DeliveryPoint.Station.Code
Carriers / ECNs/ECN.Carriers.Carrier[i].CarrierCode
ContractualCarrier / ECNs/ECN.Carriers.Carrier[i].CarrierCode where ECNs/ECN.Carriers.Carrier[i].Status=0
DossierCreator / DOSSIER_NR p1-4
Involved in the transport ≡ECNs/ECN.Carriers.Carrier[i]
4.1.3ORFEUS Members
The table ORFEUS_MEMBERS describe the railways (Carriers) relation with the “application groups”: ORFEUS (O), ISR(I), ECN (E).
Column / Type / ContentRW_CODE / CHAR(4) / Carrier code
MEMBER / CHAR(10) / O = ORFEUS
I = ISR
E = ECN
The table is loaded by insert script.
4.2Message acceptance
The backbone of the control is the DOSSIER: all the controls are articulated around the dossier sent.
One received message is checked and accepted depending on
-the TYPE of the message and its version number
-the carriers involved in the transport = ECNs/ECN.Carriers.Carrier[i]
-on the cinematic of the hand-over
Rules are defined for the sequence of messages on TYPE, sender and time : after each message the system waits:
for type of messages (list)
from sender (list)
during a configurable period of time
Each message set these parameters (DOSSIER_WAITING).
An arriving message is checked against these parameters.
If no parameter is set for a DOSSIER, then any message is expected.
Message Received / Rule to check / If rule cheeking failsDOSSIER(DossierNumber).STATUS !=”Closed” / DOSSIER CLOSED
First message sent for a DossierNumberhas VersionNumber =0 / WRONG VERSION NUMBER
Any received message for a DossierNumbe has VersionNumber> current received version (Last VersionNumberstored in DOSSIER Table) / WRONG VERSION NUMBER
Incoming message = ECN,PCN,ACK,NACK the VersionNumbercan be = current received version / WRONG VERSION NUMBER
(ConsignmentNumber+ ShippingStation + AcceptanceDate +ShippingCarrier), DossierNumber is a 1 to1 relation / CONSIGNMENT,DOSSIER ALREADY EXISTS
The DossierCreator is theContractualCarrier or the ShippingCarrier / WRONG DOSSIER CREATOR
The SendingCarrieris theContractualCarrier or in carriers List / WRONG SENDING CARRIER
DOSSIER_WAITING (TYPE,SENDER,TIMER) / MESAGE NOT EXPECTED
PRN / VersionNumber = 0 / WRONG VERSION NUMBER
SendingCarrier=ContractualCarrier = OR ShippingCarrier / SENDING CARRIER IS NOT THE SHIPPING CARRIER
ECN / First ECN for the DossierNumber
SendingCarrier=ContractualCarrier OR ShippingCarrier / WRONG SENDING CARRIER
WRONG SENDING CARRIER
If PCN received for the same DossierNumber / PCN RECEIVED
PCN / If VersionNumber == 0 Then SendingCarrier=ContractualCarrier = OR ShippingCarrier / WRONG SENDING CARRIER
If SendingCarrier>= DOSSIER.CUSTODY_RW (sequenceID in Carriers List) / SENDING CARRIER HAS NOT THE CUSTODY OF THE GOODS
ACK,NACK / If PCN received for the same DossierNumber / PCN RECEIVED
If ECN not received for the same DossierNumber / ECN NOT RECEIVED
CANCEL / SendingCarrier= DossierCreator / SENDER IS NOT DOSSIER CREATOR
VersionNumber =1 / WRONG VERSION NUMBER
ECN was received for the DossierNumber / CANCEL NOT POSSIBLE
ACK,PCN received for the DossierNumber / CANCEL NOT POSSIBLE
AOD / If RequestAOD Then message accepted / AOD NOT REQUESTED
4.3Message storing
4.3.1MESSAGE_ORFEUS
Column / Type / ContentresultCode / INT / Treatment (technical tango result)
appCode / INT / Treatment (application result)
0 = complete process without errors
Error code : see list of error coder for application control
dateTimeLog / DATETIME / System
RECIPIENT_ID / VARCHAR(35) / FileReceiver
INCT_REF_NR / VARCHAR(14) / CIMECNMessages.InterchangeHeader.InterchangeId
TRANSPORT_TYPE / VARCHAR(14) / Not available in ECN
MSG_REF_NR / VARCHAR(18) / /CIMECNMessage/ECNs/ECNHeader/MessageReferenceNumber
DOSSIER_NR / VARCHAR(19) / DossierNumber
DOSSIER_VERSION / INT / VersionNumber
DOSSIER_DATE / DATETIME / ECNs.ECNHeader.ECNPreparationDatetime
SHIPP_COUNTRY
Shipping Station Country Code / CHAR(2) / ShippingCountry
SHIPP_CODE
Shipping Station Code / CHAR(6) / ShippingStation
SHIPP_RW
Shipping RW / CHAR(4) / ShippingCarrier
SHIPP_CSMT
Consignment number / CHAR(6) / ConsignmentNumber
SHIPP_CODE_SC / CHAR(5) / ShippingStationwithout CkD
SHIPP_CSMT_SC / CHAR(5) / ConsignmentNumberwithout CkD
SHIPP_TYPE / CHAR(3) / ShipmentTypeTo be setby ORFEUS also
FUNCTION / CHAR(4) / MessageType (ORFEUS : CTD9, UTD5, UTD1)
INCTID / INT / Unique session (file) identification : Primary key
MSG_ID / INT / Unique message identification : Primary key
4.3.2MESSAGE_ORFEUS_DATA
Column / Type / ContentINCTID / INT / Unique session (file) identification : Primary key
MSG_ID / INT / Unique message identification : Primary key
MSG / Mediumtext / Message content (blob)
4.3.3DOSSIER Data Model
The DOSSIER data model is represented by
DOSSIER : describe the main characteristics needed to identify a dossier : number, version, message type, consignment, responsibility
DOSSIER_CARRIERS : describe the list of carriers involved in the transport ; may change during the transport
DOSSIER_WAITING : describe the messages expected by the system after the current processing
ROUTING_DESTINATION : describe the destination of the outgoing messages generated by the system and their status
The link is realised by the DOSSIER_SEQ, unique technical key generating by the system.
A DOSSIER is identified by (DOSSIER_NR, TYPE, VERSION). For each consignment note exists only one dossier.
The system keeps the dossier history.
Each new version of the dossier is added in the table (DOSSIER) together with the associated carriers (DOSSIER_CARRIERS), waiting dossier parameters (DOSSIER_WAITING) and outgoing message destination (ROUTING_DESTINATION).
Only the message with the highest version number received has to be accepted for the further processing.
In case of communication problems during the hand over procedure an already sent message can be repeated(this will not create a new version number). These messages will generate a replacement of the DOSSIER information (DOSSIER, DOSSIER_CARRIERS, DOSSIER_WAITING, ROUTING_DESTINATION) : previous records concerning this DOSSIER,TYPE,VERSION are deleted and the message is reprocessed.
Column / Type / Origin / CommentsDOSSIER_SEQ / INT, Primary Key / System / Add in ORFEUS
DOSSIER_NR / VARCHAR[19] / DossierNumber
CSMT_NR / VARCHAR[16] / ShippingCountry(2)+ ShippingStation(5)+ ShippingCarrier(4)+ ConsignmentNumber(5) / SHIPP_COUNTRY+ SHIPP_CODE(5)+ SHIPP_RW(4) + SHIPP_CSMT(5)
DATE_CREATED / DATETIME / ECNs.ECNHeader.ECNPreparationDatetime / datetime
TYPE / CHAR[4] / MessageType / ECN : Message Type / ORFEUS : CTD9, UTD5,UTD1
VERSION / INT / VersionNumber
CUSTODY_RW / CHAR[4] / ECN:
If VersionNumber =0 or MessageType = ACK,PCN Then
ContractualCarrier Else previous custody_rw / ORFEUS: ContractualCarrier
STATUS / INT / System / 0 = open = VersionNumber=0 received
1 = updated = INF* received
2 = blocked =waiting for hand over
3 = unblocked = hand over done
4 = closed = CANCEL,AOD received,
5 = PCN received
SENDING_RW / CHAR[4] / SendingCarrier
Status “Closed” set by :, Timer (?), ACK from Destination Carrier?
(ConsignmentNumber+ ShippingStation + AcceptanceDate +ShippingCarrier), DossierNumber is a 1 to1 relation
This rule has also to be checked for CTD flow and for ECTD flow.
Remark : a similar rule will be integrated in ISR for “NoteCreated” flow. (see ISR specifications document)
4.3.3.1DOSSIER_WAITING
This table is used to check in the received message is indeed the message expected and if it is received in time.
Column / Type / Origin / CommentsDOSSIER_SEQ / INT, Primary Key / System / Link to DOSSIER
SENDING_RW / CHAR[4] / APP CONTROL
MESSAGE_TYPE / CHAR[4] / APP CONTROL
LIMIT_TIME / DATETIME / APP CONTROL
4.3.3.2DOSSIER_CARRIERS
For each transport the system built up the sequence of all carriers with position and role in transport.
This sequence has the following structure:
Item / Type / Origin / CommentsDOSSIER_SEQ / INT, Primary Key / System / Link to DOSSIER
CARRIER_CODE / CHAR[4] / ECNs/ECN.Carriers.Carrier[i].CarrierCode
CARRIER_STATUS / INT / ECNs/ECN.Carriers.Carrier[i].Status / “0” = Contractual/Lead carrier
“1” = Successive carrier
“2” =: Substitute carrier
CARRIER_POS / INT / Carrier position / ECNs/ECN.Carriers.Carrier[i].sequenceID
CARRIER_ACC / INT / Carrier access rights to data (consulting) / 1 - yes
the corresponding railway will be allowed to consult information linked to that consignment
4.3.3.3ROUTING_DESTINATION
Describes, for each message of the DOSSIER, the destination of the outgoing messages and the status of the processing
Column / Type / ContentDOSSIER_SEQ / INT / System
DESTINATION / CHAR(4) / System :desti
TYPE / CHAR(4) / Message type of the outgoing message to destination
TIMESTAMP / DATETIME / System
STATUS / INT / Sent/Not sent/Etc.
4.4Update ORFEUS :
-Dossier Management to take into account new table DOSSIER : check the relation with the consignment number
-Migrate table DOSSIER to the new structure -: script to generate values for existing ORFEUS information from MESSAGE_ORFEUS
-Migrate table ROUTING_DESTINATION to the new structure : script to generate values for existing ORFEUS information
4.5Message processing and routing
There are 1 to n Carriers involved in the transport. The responsibility of the transport is handed over from Carrier 1 to Carrier 2, to… Carrier i, ….to Carrier n, in the exact sequence described in the message (Carrier.sequenceID).
The message version is incremented independently from the message type.
Different from the procedure in ORFEUS today the ECN railways agree that the CDS forwards the information about data and status changes only forward in the sense of transportation.
Each message received and accepted generates
A set of management actions (storing)
A set of outgoing messages
Waiting for a new message
** When the CDS receives a PRN, INFE, ECN message and the next carrier is a CTD-carrier (and not capable of processing the ECN flow), the sending carrier will receive an GRM message (created by the CDS) and no messages have to be sent to the other carriers.