2018-02 EPAS Closed Door Meeting Agenda

2018-02 EPAS Closed Door Meeting Agenda

EPAS Closed Door Meeting Agenda Jan10, 20171

1.Welcome, Call to Order, Introductions—Greg Park

2.Review WECC Antitrust Policy—Greg Park

All WECC meetings are conducted in accordance with the WECC Antitrust Policy and the NERC Antitrust Compliance Guidelines. All participants must comply with the policy and guidelines. This meeting is a closed session. The discussion should be limited to confidential information and should not extend to any matter that can be discussed in open session. You are also reminded that you have signed a non-disclosure agreement that governs any and all confidential information that is discussed during this meeting.

3.Approve Agenda

4.Review Jan 10, 2018 Minutes

5.Review of Previous Action Items—Greg Park

Action Items for Closed Events

  • Regarding the Aug 16, 2016 – Category 0 event, Tim Reynolds will continue to update the group as appropriate regarding updates on the third-party reviews.
  • Regarding the June 21, 2017 – Category 1h event, Tim Reynolds will include the recommended action for future follow up in EPAS meetings.

Action Items for Open Events

September 15, 2017 – Category 1a

Errors in settings and wiring resulted in misoperation of some relays during commissioning.

AI – Regarding the Sept 15, 2017 – Category 1a event, EPAS members will follow up with the following action items:

  • Tim Reynolds will follow up with Dean Spratt to know the results of the analysis done for Peer Review process.
  • The HPWG will look at the event to see if there are best practices or lessons learned about coordinating protection between generation and substation.

STATUS: 12/06/2017 – Information was shared at the meeting that this was a timing issues where generation personnel changed the relay drawings and failed to notify those in the project in charge of relay coordination before the commissioning. As a preventative measure, the company is putting into place a step to review and verify all future settings match the prints before commissioning. In addition, they will also add an extra step that final prints will be reviewed again for relay settings before it goes live.

STATUS: 1/10/2018 – The HPWG will continue analyzing the event for potential lessons learned.

October 23, 2017 – Category 1a

A high side breaker failed to trip in the required time and generated a bus lockout on a 115-kV bus, causing all breakers connected to the bus to trip and open ending of four 115-kV lines, one reactor, and the connection for a 230-115-kV transformer.

AI – Regarding the Oct 23, 2017 – Category 1a event, EPAS members will follow up with the following action item:

  • Tim Reynolds will ask the following questions:
  • Was the incident a miss-step or a flaw in the breaker (slow tripping breaker)?
  • Are there any follow-up actions that are going to take place or procedures that need to be corrected?
  • Have the test procedures been used successfully in the past?
  • What does “test” position mean for this breaker? What does “test” position do to the breaker?
  • Is there a standardized troubleshooting procedure?

STATUS: 1/10/2018 – An update on the event was provided. The high side breaker failure protection operated correctly at the high side breaker but did not open within specification due to a bent mechanical linkage. This correctly tripped the Transmission Bus Lockout.

AI – Regarding the Oct 23, 2017 – 1a event, Richard McLean will follow-up with the remaining four questions.

November 15, 2017 – Category 1h

Outdated file mistakenly used when performing an EMS update leading to loss of communications, visibility and control.

AI – Regarding November 15, 2017 – Category 1h event, EPAS members will follow up with the following action item:

  • The HPWG will look at the event to see if there are best practices or lessons learned.
  • Tim Reynolds/Katie Iversen will ask the following questions:
  • Why wasn’t the Model Propagation Guideline followed? Was it due to the procedure or human error?

STATUS: 1/29/2018 – Upon further investigation, contributing causes of the event were: Human performance in that existing guideline was available but not followed; and process deficiencies which include needed clarifications to the process and steps to remove the availability of old models.

  • Can the documented lessons learned be shared with EPAS and DEMSWG?

STATUS: 1/29/2018 – The list of potential lessons learned are:

•Model Propagation Guideline should include process diagram, defined actions, and expected results

•Model update file movement should be automated

•Back-up model push role not defined

•Attention to detail lacking

•Back-up ACE should be verified with each model push

•Operators should receive refresher training on use and location of Back-up ACE

  • Tim Reynolds/Katie Iversen will look at the possibility of posting examples of well-written Brief Reports.

November 26, 2017 – Category 1h

Loss of communicationslead to the loss of monitoring capabilities between two control centers for both the TOP and BA function.

AI – Regarding November 26, 2017 – Category 1h event, EPAS members will follow up with the following action item:

  • Tim Reynolds/Katie Iversen will ask the following questions:
  • There are several days between the first failure and the second failure. What happened between 11/24/2017 and 11/26/2017?

STATUS: 2/5/2018 – The fiber on the primary link could not be repaired without taking a major 500-kV line outage, so the dispatched Telecommunications Technician could not “work” on it immediately. Given that the communication link went down on the Thanksgiving holiday weekend, and that we had a functional backup in service (secondary communication link), Telecommunications Technician notified the on-desk system operator of the “new” configuration, and postponed the repairs to normal business hours.

  • Is there a process or procedure that defines manning of substations? Is there a procedure that provides direction for a specified amount of time to pass before the operator calls on the substation to be manned? Is switching to the third channel part of the processes/procedures?

STATUS: 2/5/2018 – The entity has an Energy Plan for Loss of Control Center Function that provides direction for staffing of substations. The Plan includes a checklist of items that are to be conducted prior issuing the dispatch of personnel to identified sites listed in the Plan. It also contains the personnel group to contact based on location of the site in the state. There is not a specified amount of time to meet, it is at the discretion of the Transmission Operators after fulfilling all the previous items associated with loss of SCADA Communication sources. The preceding items do include determination of failed sites (if site specific), notifications to internal personnel and external (neighbors, RC, etc.), and cancellation of pending and existing switching. Additionally, the individuals that are dispatched to substations will be required to report with the TOP at least every 30 mins.

The entity does not have a process (currently) that addresses a N-2 contingency (as defined in this event) for the communication link. The tertiary link will now be part of future processes.

  • What are local visibility and control modes? What do these modes do?

STATUS: 2/5/2018 – Visibility refers to the restoration of the EMS and DMS systems for visibility of the entire footprint, when the tertiary communication link was switched over. Previously, the desks were only receiving “half the picture”, and had visibility regionally (after a backup EMS server was restored to allow regional visibility was by an Engineer at 16:10.)

The “control” term would be referring to the loss of AGC capability for the BA desks, and due to this loss, we issued operating instructions to each Plant Operator to move to manual dispatch until further notice. Control of the transmission system was regained for the region when the backup EMS server was brought online.

  • Has the full root cause analysis been completed and can you provide the results?

STATUS: 2/5/2018 – A root cause analysis of the event was conducted and will be provided on February 8. Since the time of the event, we have identified additional future opportunities, and action items for the affected personnel.

  • What communication was lost from the fiber that was compromised?

STATUS: 2/5/2018 – For the loss of the secondary comm. link Corporate IT resources (External website, Outlook, SharePoint, VPN access, etc.), and data to EMS and DMS systems at the control centers was interrupted. The Call Center was unavailable to receive calls during this outage.

  • Jill Hoyt will review Peak’s data to see if there were any issues monitoring the area with their state estimator and RTCA.

6.Review of New Events since Last Call—Greg Park

November 28, 2017 – Category 1c

A line fault caused a RAS to arm a generator that should have tripped off but didn't. Error was found to be an AND gate where there should have been an OR gate.

January 5, 2018 – Category 1h

A third-party communications service provider cut fiber optic communications paths that route data from Remote Terminal Units (RTUs) on the entity's transmission system to the control center.

January 15, 2018 – Category 1h

RTCA failed to solve due to loss of two RTUs and higher than normal loads in the area.

7.Review of New Action Items

8.Review Upcoming Meetings

Closed Door Meeting

Mar 7, 2018...... Closed Door Webinar

Next In-Person Meeting

Feb28, 2018...... Salt Lake City, UT

9.Adjourn

Western Electricity Coordinating Council

Top View