Section: LIS Downtime and System Maintenance / Subject Title: Downtime Procedures Department of Microbiology
Prepared by: LIS Committee / Original Date: October 22, 2008
Issued by: Laboratory Information System / Revision Date: June 15, 2010
Approved by: Laboratory Director / Annual Review Date: May 31, 2013
Microbiology Downtime Manual
1.0Purpose......
2.0 Introduction......
3.0Downtime Planning Documents......
3.1 User Downtime Procedures......
3.2 Communications/Escalation Plan......
3.3 HP/Informatics Recovery Procedures......
3.4 Evaluation of the Downtime Procedure......
4.0Definitions......
4.1 Scheduled Downtime......
4.2 Unscheduled Downtime......
4.3 HP
4.4 Customer Care Centre......
4.5 System Components......
5.0 Downtime Procedures For HIS Systems (Cerner, Meditech or Mysis-EPR)......
5.1 Purpose......
5.2 Downtime Policy......
5.3 System Descriptions......
5.4 Downtime Procedures Grid......
5.5 Description of Downtime Procedures......
5.5.1 AA Mysis-EPR order that has been accessioned by SMC. Cerner or Meditech order that has been received.
5.5.2 AB Mysis-EPR order that has not been accessioned by SMC. Meditech order that was not transmitted. A stat Cerner order that must be ordered and collected manually.
5.5.3 AC Results Reporting......
5.6 Recovery Process......
5.6.1 AA Mysis-EPR order accessioned by SMC, Cerner or Meditech order that has bee received....
5.6.2 AB Mysis-EPR order that has not been accessioned by SMC. Meditech order that was not transmitted and a stat Cerner order that must be ordered and collected manually.
5.6.3 AC Results Reporting......
6.0 Downtime Procedures For Healthlink Interface Engine(s)......
6.1 Purpose......
6.2 Downtime Policy......
6.3 System Descriptions......
6.4 Downtime Procedures Grid......
6.5 Description of Downtime Procedures......
6.5.1 BA HIS orders......
6.5.2 BB Results Reporting......
6.6 Recovery Process......
6.6.1 BA HIS orders......
6.6.2 BB Results Reporting......
7.0 Downtime Procedures For Order entry Interface......
7.1 Purpose......
7.2 Downtime Policy......
7.3 System Descriptions......
7.4 Downtime Procedures Grid......
7.5 Description of Downtime Procedures......
7.5.1 CA HIS orders......
7.6 Recovery Process......
7.6.1 CA HIS orders......
8.0 Downtime Procedures For HIS ADT interface......
8.1 Purpose......
8.2 Downtime Policy......
8.3 System Descriptions......
8.4 Downtime Procedures Grid......
8.5 Description of Downtime Procedures......
8.5.1 DA ADT transactions......
8.6 Recovery Process......
8.6.1 DA HIS orders......
9.0 Downtime Procedures For HIS Results Reporting Interface......
9.1 Purpose......
9.2 Downtime Policy......
9.3 System Descriptions......
9.4 Downtime Procedures Grid......
9.5 Description of Downtime Procedures......
9.5.1 EA Results Reporting......
9.6 Recovery Procedures......
9.6.1 EA1 and EA2......
10.0 Downtime Procedures For Laboratory Information System......
10.1 Purpose......
10.2 Downtime Policy......
10.3 System Descriptions......
10.4 Downtime Procedures Grid......
10.5 Description of Downtime Procedures......
10.5.1 FA Order Entry......
10.5.2 FB Results reporting......
10.6 Recovery Process......
10.6.1 FA Order entry......
10.6.2 FB Results Reporting......
Department of Microbiology
Instructions for Downtime
1.0Purpose
The purpose of the downtime policies and procedures manual is to ensure that during all types of downtimes, the Microbiology Department remains functional, no data is lost and the recovery procedures have minimal impact on the organization.
2.0 Introduction
A downtime is any period of time when a computer information system is not available. This includes scheduled downtimes (planned downtimes for system maintenance and upgrades) and unscheduled downtimes (due to hardware/software failures).
The information systems at Mount SinaiHospital are highly integrated. Interfaces exist between the Hospital Information Systems (HIS) which are Cerner and Mysis-EPR, the Laboratory Information System, SCC and the MDI Solutions and Courtyard Group Ltd. interface engines. Downtimes can occur within Cerner, Mysis-EPR, one of the departmental systems, or the interface between them. Downtimes in any of these areas likely to affect other areas and the impact must be considered. Each area should have sufficient workarounds to ensure continuity of services during a downtime.
An accurate record of all information related to a patient’s care must be maintained at all times. In the event that a system is not available, information will be recorded manually on designated forms. When the system becomes available, information captured manually will be entered into the system in the recovery phase to ensure the retention of complete and accurate information.
3.0Downtime Planning Documents
The Downtime Plan consists of three main documents:
- User Downtime and Recovery Procedures
- Communication/Escalation Plan
- HP/Informatics Recovery Procedures
- Evaluation of Downtime Plan
3.1 User Downtime Procedures
Upon notification that a system is down, or when users find they cannot access a system, they must implement the User Downtime Procedures. These procedures describe what actions to take to continue operations using paper-based or other alternative processes. User Downtime Procedures do not describe how to alert HP of a problem, nor explain how to diagnose or fix the problem. They only describe what to do in the absence of the system. Upon notification that the system is available, users must implement the recovery phase of the User Downtime Procedures. The intended audience for User Downtime Procedures includes departmental managers, application specialists and system users.
3.2 Communications/Escalation Plan
All notification of system failures must be made to the HP Customer Care Centre (Helpdesk) at Extension 4357. Upon notification of a problem, HP will initiate the escalation plan to alert the appropriate technical staff to respond to the problem, to update users, and to notify users when the system is once again available. The Escalation Procedures include a predefined list of messages for communication to users to ensure that they understand the impact and which downtime procedure to implement.
The intended audience for the Escalation Procedures is the staff of the Customer Care Centre.
3.3 HP/Informatics Recovery Procedures
HP maintains a separate Disaster RecoveryPlan to coordinate staff responsibilities in responding to a system failure, and to provide the necessary information to speed the recovery phase. This plan is triggered by notification of a failure via the Escalation Procedures, and ends when the recovery team notifies the Customer Care Centre that the problem is resolved. The intended audience for the HP Disaster Recovery Plan is HP and MSH Informatics staff.
3.4 Evaluation of the Downtime Procedure
An important part of downtime planning is evaluation of the event in downtime. Debriefing of the event is important to evaluate the strengths and weaknesses of the procedure.
4.0Definitions
4.1 Scheduled Downtime
Scheduled downtime is defined as a planned period of time when the system is unavailable for use. These are scheduled during a time when the system usage is low to minimize impact on end users. The established downtime window for all clinical applications may commence 23:30 hrs on the third Wednesday of the month until the 03:30 hrs of the third Thursday of the month as approved by the Technical Advisory Board (TAB) and the Change Advisory Board (CAB). Scheduled downtimes are used for updating the sytem with new hardware/software releases, fixes, or enhancements. User Downtime Procedures for scheduled downtimes are essentially the same as for unscheduled downtimes, but may involve some preparation before the downtime that could not be done if the downtime were not expected ahead of time.
4.2 Unscheduled Downtime
Unplanned failures caused by power failures, software problems, or hardware problems.
4.3 HP
Shared Information Management Services (SIMS) provides technical support for Mount SinaiHospital and the University Health Network (UHN).
4.4 Customer Care Centre
The Customer Care Centre provides Helpdesk support for desktop PCs and serves as a central point of contact for notification of any problem related to the computer systems. The Customer Care Centre service is provided by Compaq Canada, and is reached by dialing Extension 4357.
4.5 System Components
The microbiology department has identified 6 sets of downtime procedures. Each will be accessed separately.
Table 2
A / B / C / D / E / FComponents / HIS (Mysis-EPR Cerner or Meditech) / Healthlink Interface engine / HIS order entry (Mysis-EPR, Cerner or Meditech) / HIS ADT (Cerner or Mysis-EPR) / HIS results reporting (Cerner, Mysis-EPR or Meditech) / LIS
5.0 Downtime Procedures for HIS Systems (Cerner, Meditech or Mysis-EPR)
5.1 Purpose
The purpose of the HIS downtime procedures is to ensure that orders from the HIS (Cerner, Meditech or Mysis-EPR, ) continue to be processed and results from the LIS are sent out to the facility whose HIS (Cerner, Meditech or Mysis-EPR) is down and no data is lost.
5.2 Downtime Policy
In the event of one or more of the HIS systems going down, the Microbiology Department will continue to facilitate the entry of orders from the facility(s) into the LIS. Results will be phoned, faxed, printed or hand delivered to the facility (MSH, UHN or RougeValley) where the HIS is unavailable.
5.3 System Descriptions
HIS is used
- To send orders from the inpatient and outpatient wards from all of the facilities to the LIS. Orders are placed in Cerner, Meditech or Mysis-EPR. For Mysis-EPR, the specimens are ordered and sent to the Specimen Management Center (SMC) where they are collected/accessioned. The act of accessioning sends an order to the LIS. The specimens are sent from SMC to the Microbiology Department. For Cerner, the specimens are ordered which sends an order to the LIS. When the specimen is collected, the collection date and time is updated. Specimens are brought to the lab. For Meditech, the specimens are ordered collected and sent to the labs at RougeValley. Once ready a batch transmission is sent to the LIS and the specimens arrive by courier.
- To send ADT transactions to the LIS as patients are admitted, discharged or transferred within both MSH and UHN. ADT transactions are not received from RougeValley.
- To send results from the LIS to the HIS (Cerner Meditech or Mysis-EPR)
5.4 Downtime Procedures Grid
A / B / C / D / EStage / Mysis-EPR order/accessioned by SMC or Cerner and Meditech
order rec’d / HIS order no SMC accession or Cerner or Meditech order not received / Results reporting
1 / Less than 1 hour / AA1 / AB1 / AC1
2 / After 1 hour / AA1 / AB2 / AC1
3 / After 24 hours / AA1 / AB3 / AC2
4
5.5 Description of Downtime Procedures
5.5.1 AA Mysis-EPR order that has been accessioned by SMC. Cerner or Meditech order that has been received.
AA1 For Mysis-EPR, the downtime occurred between accessioning by SMC and the arrival of specimens in the microbiology lab. For Cerner, downtime occurred after the order was placed but the specimen has not been received in the lab. For Meditech, the downtime occurred after the batch transmission was sent but the specimen has not arrived by courier.
The specimens are received as usual in the LIS.
5.5.2 AB Mysis-EPR order that has not been accessioned by SMC. Meditech order that was not transmitted. A stat Cerner order that must be ordered and collected manually.
The Mysis-EPR downtime occurred between specimen collection and the arrival of the specimens at SMC. The Meditech downtime occurred after the order was placed and specimen collected but prior to the batch transmission for Meditech. A stat Cerner specimen would be sent with a requisition.
AB1 Immediately
All un-accessioned specimens from Ulitcare are sent to the Microbiology Lab. The specimens are held refrigerated for up to one hour. No other action is taken.
All non-transmitted specimens from Meditech are held by the facilities until the system is back up. The stat Cerner specimen would be entered manually into the LIS and follow the procedures under AB3.
AB2 After 1 hour
Mysis-EPR only:
- Blood cultures: keep a list of specimens with patient name, MRN, HIS number and bottle ID. Incubate off line in incubator. If the HIS is not back up, one hour before the lab closes, load bottles in the Bact/Alert by scanning the bottle ID.
- All other specimens: use the HIS specimen number for labeling plates and smears during planting. Save the specimens in a separate basket for recovery.
- Specimens that arrive with requisitions with complete patient information (MRN and visit #), generate the order in the LIS.
- Specimens that arrive with requisitions with incomplete patient information, use an internal numbering system. Place the internal lab number on the requisition and plant the specimens.
AB3 After 24 hours
All specimens will arrive in the microbiology laboratory with requisitions. The patient information on the requisition may be complete because the patient is already registered in the HIS or incomplete because the patient has just been admitted and the HIS is down.
- For those patients who are registered (MRN and visit #), generate the order in the LIS.
- For those patients without a MRN and/or visit number, use an internal numbering system. Place the internal lab number on the requisition and plant the specimens.
5.5.3 AC Results Reporting
AC1 Immediate and after 1 hour
No change. Call Critical results and force print important results to fax, print or deliver at the technologist’s discretion.
AC2 After 24 hours
- Notify the LIS officer to activate a print run in the LIS.
- Print LIS reports for appropriate status of culture results.
- For UHN, the secretary will fax/send printed reports to ward, clinic or office. For MSH, reports can be printed on inpatient wards and clinics and outpatient wards will be delivered. For Rouge valley, the reports can be automatically faxed or delivered.
5.6 Recovery Process
5.6.1 AA Mysis-EPR order accessioned by SMC, Cerner or Meditech order that has bee received.
AA1 No action required
5.6.2 AB Mysis-EPR order that has not been accessioned by SMC. Meditech order that was not transmitted and a stat Cerner order that must be ordered and collected manually.
AB1 Immediately
Take specimens from storage, receive in LIS and plant as usual. Specimens will arrive from Rouge valley and a batch transmission will have been sent.
AB2 After 1 hour
Mysis-EPR only:
- For all specimens: accession the specimen in the HIS, receive the specimen in the LIS and re-label plates, smears and blood culture bottles with the LIS order number.
- Blood cultures: for bottles incubated off-line, label and load bottles as usual. For bottles loaded by bottle ID, re-label bottle and use quick data entry in the Bact/Alert to attach the order to the corresponding bottles.
- For those specimens received with requisitions containing complete patient information, no action is required.
- For those specimens received with requisitions with incomplete information, generate an order in the LIS and re-label plates.
AB3 After 24 hours
- For those specimens received with requisitions containing complete patient information, no action is required.
- For those specimens received with requisitions with incomplete information, generate an order in the LIS and re-label plates.
5.6.3 AC Results Reporting
AC1 and AC2
Results will queue on the interface engine and as soon as the HIS is up, they will post to the HIS.
6.0 Downtime Procedures For Healthlink Interface Engine(s)
6.1 Purpose
The purpose of the interface engine downtime procedures is to ensure that orders from the HIS (Cerner, Meditech or Mysis-EPR) continue to be processed and results from the LIS are sent out to the facility whose HIS (Cerner, Meditech or Mysis-EPR) is down and no data is lost.
6.2 Downtime Policy
In the event of the interface engine(s) going down, the Microbiology Department will continue to facilitate the entry of orders from all facilities into the LIS. Results will be phoned, faxed, printed or hand delivered to the facility (MSH, Rouge valley or UHN) when the interface engine is down.
6.3 System Descriptions
Healthlink Interface Engine is used:
- As part of the link sending orders from the inpatient and outpatient wards from MSH, Rouge valley and UHN to the LIS. Orders are placed in Cerner, Meditech or Mysis-EPR. For Mysis-EPR, the specimens are collected and sent to the Specimen Management Center (SMC) where they are accessioned. The act of accessioning sends an order through the datagate interface engine to the LIS. The specimens are sent from SMC to the Microbiology Department. Batch transmissions from Meditech are received through the datagate interface engine to the LIS. When orders are placed the transmission is passed to the LIS via the cloverleaf interface engine.
- As part of the link sending ADT transactions to the LIS as patients are admitted, discharged or transferred within both MSH and UHN. The interface engine between Cerner (MSH) and the LIS is HCI and the engine between Mysis-EPR (UHN) and the LIS is datagate.
- As part of the link sending results from the LIS to the HIS (Cerner, Meditech or Mysis-EPR)
6.4 Downtime Procedures Grid
A / B / C / D / EStage / HIS orders / Results reporting
1 / Less than 1 hour / BA1 / BB1
2 / After 1 hour / BA2 / BB1
3 / After 24 hours / BA2 / BB2
4
6.5 Description of Downtime Procedures
6.5.1 BA HIS orders
BA1 Immediately
The specimens are ordered and accessioned in Cerner, Meditech or Mysis-EPR but no LIS order is generated. Transactions are queued to the interface engine. No action is taken. Specimens are stored appropriately.