Florida Reliability Coordinating Council, Inc. FERC Docket No. NP15-3-000 (October 30, 2014)
Reliability Standard: IRO-002-2
Requirement: R7, R8
Violation Risk Factor: High (R7, R8)
Violation Severity Level: Severe (R7), Moderate (R8)
Region: SERC
Issue: Florida Power & Light Company functions as the Florida Reliability Coordinating Council, Inc.’s (FRCC) agent for real-time and next-day planning RC functions. During August 8-9, 2012, a core router pair, which supported the communications network for the FRCC’s energy management system (EMS) at FPL’s primary control center, malfunctioned due to a buffer overrun. As a result, the EMS was rendered inaccessible from system operator consoles and FRCC was only able to fully access its EMS/Supervisory Control and Data Acquisition (SCADA) system three hours later. In connection with the incident, FRCC self-reported that, as it was unable to access the EMS/SCADA data and its Real-Time Contingency Analysis (RTCA) program was not able to function properly, it did not continuously monitor its RC Area or properly ensure that the System Operating Limit (SOL) and Interconnection Reliability Operating Limit (IROL) monitoring and derivations continued when its main monitoring system was not available (R7). FRCC also did not properly control its RC analysis tools or have adequate procedures to mitigate the impact of analysis tool outages. While FRCC had backup EMS procedures, those EMS procedures failed and FRCC did not have any adequate plans covering the simultaneous loss of both its EMS and RTCA (R8).
Finding: SERC determined that the violation posed a serious or substantial risk to the BPS reliability as, during the incident, FRCC lost the ability to continuously monitor its RC Area and to verify that the SOL and IROL monitoring and derivations remained. FRCC also lost control of its RC analysis tools and was unable to mitigate such loss. However, no actual harm to the BPS occurred as FRCC used other monitoring capabilities to maintain situational awareness during the incident. In addition, none of the FRCC BES elements were de-energized and no SOLs were exceeded during the incident. The IRO-002-2 violations occurred on August 8-9, 2012. FRCC neither admitted nor denied the violations. In approving the settlement agreement, the NERC BOTCC considered the fact that all of the violations constituted a serious or substantial risk to BPS reliability. However, these were FRCC’s first violation of the Reliability Standards at issue and the violations were self-reported. FRCC also had an internal compliance program in place, which SERC evaluated as a mitigating factor. In addition, FRCC was cooperative throughout the enforcement process and did not conceal the violations.
Penalty: $85,000 (aggregate for 4 violations)
FERC Order: Issued November 28, 2014 (no further review)
Western Electricity Coordinating Council, FERC Docket No. NP14-38 (March 31, 2014)
Reliability Standard: IRO-002-2
Requirement: 7
Violation Risk Factor: High
Violation Severity Level: Severe
Region: NPCC
Issue: The Western Electricity Coordinating Council (WECC) self-reported, that in an August 21, 2012 incident, its operators had difficulty loading the appropriate data into the Study Network Analysis application (STNET) and therefore WECC did not timely exercise its backup for its loss of the Real Time Network Analysis (RTNET) and Real-Time Contingency Analysis (RTCA) to identify potential System Operating Limits (SOL) and Interconnection Reliability Operating Limits (IROL) exceedances.
Finding: NPCC found that the IRO-002-2 violation constituted a serious or substantial risk to BPS reliability. As a result of the failure of its primary monitoring system and its inability to timely use its backup monitoring system, WECC was unable for 58 minutes (at which time the RTCA became operable again) to monitor pre and post-contingent conditions within the Western Interconnection. During that time, WECC would not have had the data available to identify the effects of potential contingencies. The IRO-002-2 violation occurred on August 21, 2012. WECC neither admitted nor denied the violations. In approving the settlement agreement, NERC BOTCC considered the fact that most of the violations constituted a serious or substantial risk to BPS reliability and NPCC concluded that, left unaddressed, the number and type of reliability violations represented a potential for serious risk to the reliable operation of the Western Interconnection. The violations also represented WECC’s second occurrence of violations for Reliability Standards COM-002, TOP-007 and IRO-005 (although it was not evaluated as an aggravating factor since the prior violations involved legacy procedures used by WECC’s predecessor). Ten of the violations were self-reported, which was viewed as a mitigating factor. WECC was also cooperative throughout the enforcement process and did not conceal the violations.
Total Penalty: $400,000 (aggregate for 19 violations)
FERC Order: Issued August 28, 2014 (no further review)
NP18-10-000: Electric Reliability Council of Texas, Inc. (ERCOT)
Region: TRE
| NERC Violation ID | Standard | Requirement | VRF/VSL | Discovery Method | Start Date | End Date |
|---|---|---|---|---|---|---|
| TRE2016016699 | IRO-002-2 | R7 | High/Severe | Self-Report | 7/7/2016 | 7/7/2016 |
| TRE2016016700 | IRO-003-2 | R1 | High/High | Self-Report | 7/7/2016 | 7/7/2016 |
| TRE2016016702 | IRO-005-3.1a | R1 | High/Severe | Self-Report | 7/7/2016 | 7/7/2016 |
Issue: IRO-002-2
On December 22, 2016, ERCOT submitted a Self-Report stating that, as a Reliability Coordinator (RC), it was in violation of IRO-002-2 R7. Specifically, ERCOT failed to continuously monitor its RC Area for approximately 47 minutes during an issue with its Energy Management System (EMS) on July 7, 2016. The violation was the result of the loss of monitoring or control at a Control Center such that it significantly affects the entity's ability to make operating decisions for 30 continuous minutes or more, specifically, this was a loss of operator ability to remotely monitor or control Bulk Electric System (BES) elements.
The root cause of this violation was that ERCOT lacked adequate controls to prevent the introduction of corrupt data into its EMS production environment. In particular, ERCOT did not technically preclude its personnel from inadvertently loading model cases into the EMS production environment or include cautionary messages or other prompts to prevent such actions.
IRO-003-2
On December 22, 2016, ERCOT submitted a Self-Report stating that, as a Reliability Coordinator (RC), it was in violation of IRO-003-2 R1. Specifically, ERCOT failed to monitor all Bulk Electric System (BES) facilities within its RC Area to ensure that, at any time, regardless of prior planned or unplanned events, it was able to determine any potential System Operating Limit (SOL) and Interconnection Reliability Operating Limit (IROL) violations within its RC Area for approximately 47 minutes on July 7, 2016.
The root cause of this violation was that ERCOT lacked adequate controls to prevent the introduction of corrupt data into its EMS production environment. In particular, ERCOT did not technically preclude its personnel from inadvertently loading model cases into the EMS production environment or include cautionary messages or other prompts to prevent such actions.
IRO-005-3.1a
On December 22, 2016, ERCOT submitted a Self-Report stating that, as a Reliability Coordinator (RC), it was in violation of IRO-005-3.1a R1. Specifically, ERCOT failed to monitor its RC Area parameters for approximately 47 minutes on July 7, 2016. This was a category 1.h.i event, loss of monitoring or control at a Control Center such that it significantly affects the entity's ability to make operating decisions for 30 continuous minutes or more, specifically, this was a loss of operator ability to remotely monitor or control Bulk Electric System (BES) elements.
The root cause of this violation was that ERCOT lacked adequate controls to prevent the introduction of corrupt data into its EMS production environment. In particular, ERCOT did not technically preclude its personnel from inadvertently loading model cases into the EMS production environment or include cautionary messages or other prompts to prevent such actions.
Finding: IRO-002-3
This violation posed a moderate risk to the reliability of the bulk power system. Specifically, ERCOT's inability to directly monitor its RC Area potentially increased the risk that it would be unable to identify and respond to a potential contingency in a timely manner.
IRO-003-2
This violation posed a moderate risk to the reliability of the bulk power system. Specifically, ERCOT's inability to directly monitor its RC Area potentially increased the risk that it would be unable to identify and respond to a potential contingency in a timely manner.
IRO-005-3.1a
This violation posed a moderate risk to the reliability of the bulk power system as ERCOT successfully followed emergency procedures in coordinating with other entities and there was no loss of Load and no SOL or IROL exceedances. Specifically, ERCOT's failure to monitor its designated RC Area parameters for 47 minutes increased the risk that it would be unable to identify and/or properly respond to a potential contingency in a timely manner.
Penalty: $140,000
FERC Order: Issued March 29, 2018
