NERC Case Notes: Reliability Standard CIP-003-3

Unidentified Registered Entity (URE), FERC Docket No. NP13-11 (December 31, 2012)

Reliability Standard: CIP-003-3

Requirement: 6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: SPP

Issue: URE self-reported that while it had established a change control and configuration management process for its CCAs, not all of its process documents had the required supervisory and managerial approval, a change implementation date, a notification date (for groups affected by change) and an implementation date. URE also self-reported that for software additions on its CCAs, it had not properly followed its change control and configuration management process. URE also had an additional instance (involving a virtual device residing on an existing server that was powered on inside an ESP) in which it did not follow its change control and configuration management process.

Finding: SPP found that the violation related to URE’s incomplete change control and configuration management process constituted a moderate risk to BPS reliability, but that there were mitigating factors in place. For example, URE had instituted a multi-tier approach to help ensure that some preliminary level of review and approval would occur before any system changes were made. Also, all changes requested during the violation period had received board approval. SPP found that the violation related to the software addition only constituted a minimal risk to BPS reliability since URE’s client configuration manager management server alerted URE’s information security personnel, who promptly uninstalled the offending software. SPP found that the violation related to the virtual device constituted a moderate risk to BPS reliability since the activation of the device (which had not been tested) within the ESP could potentially have introduced unknown vulnerabilities into the ESP. But, URE’s vulnerability scanner immediately alerted URE’s staff, who quickly responded to the device activation, and there were no adverse impacts caused by the activation. URE neither admitted nor denied the violations. In approving the settlement agreement, the NERC BOTCC evaluated URE’s violation history (which was considered as an aggravating factor for some of the violations), URE’s internal compliance program (which was not viewed as a mitigating factor) and the fact that some of the violations were self-reported. URE was cooperative during the enforcement process and did not conceal the violations.

Total Penalty: $107,000 (aggregate for 16 violations)

FERC Order: Issued January 30, 2013 (no further review)

Unidentified Registered Entity 1 (URE1), Docket No. NP13-12-000 (December 31, 2012)

Reliability Standard: CIP-003-3

Requirement: 6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: WECC

Issue: URE1 filed a self-report explaining that it did not follow its CCA/CA change control and configuration management process on 14 instances over two months associated with its control center and back-up control center.

Finding: WECC determined that the violation posed a minimal risk to BPS reliability, but not a serious or substantial risk. All of URE1’s CCAs are located with an ESP with controlled and monitored access. URE1 did not contest the violation. In determining the appropriate penalty, WECC considered that the violation was self-reported and URE1 ICP as mitigating factors.

Total Penalty: $5,000

FERC Order: Issued January 30, 2013 (no further review)

Unidentified Registered Entity (URE), Docket No. NP13-23-000, January 31, 2013

Reliability Standard: CIP-003-3

Requirement: R6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: RFC

Issue: URE self-reported a violation of R6 because it failed to carry out its configuration management activities (i.e. change control process and change management procedures) to document the changes it made to its firewall and to its anti-virus software.

Finding: RFC determined that the R6 violation posed a minimal risk to the reliability of the BPS because the company had a documented change management procedure in place at the time of the violation for changes to Critical Cyber Asset hardware or software. Furthermore, the violations represented isolated incidents, rather than systemic compliance issues. RFC and URE entered into a settlement agreement to resolve multiple violations whereby URE agreed to pay a penalty and to undertake other mitigation measures to come into compliance with R6.
RFC considered URE's ICP a mitigating factor in making its penalty determination. The violation began when the company installed the firewall into the ESP and ended when the company implemented its change management process and configuration management activities to document the changes it made to its firewall and ant-virus system. URE neither admits nor denies the R6 violation.

Penalty: $40,000 (aggregate for 8 violations)

FERC Order: Issued March 1, 2013 (no further review)

Unidentified Registered Entities 1, 2, 3, 4 (RFC_URE1, RFC_URE2, RFC_URE3, MRO_URE1) (collectively, RFC-MRO_UREs), Docket No. NP13-41-000 (June 27, 2013)

Reliability Standard: CIP-003-3

Requirement: 5

Violation Risk Factor: Lower

Violation Severity Level: Severe Region(s): ReliabilityFirst, MRO

Issue: RFC_URE1 and RFC_URE2 reported through self-reports violations of CIP-003-3 R5 upon finding that each had given a contractor access to a tool that identifies whether a Cyber Asset is a CCA, which is protected CCA information. The contractor had had access to the information for many years before such access was deactivated. RFC_URE1 and RFC_URE2 eventually reactivated the account allowing read and write access privileges. Due to technical issues, the ability to add security keys to inactive users had not been implemented and therefore the inadvertent access was allowed. The corporate security system of RFC_URE1 and RFC_URE2 reported the contractor’s access as “unapproved” and it was reported as such.

Finding: These violations were deemed to pose a moderate, but not serious or substantial risk to BPS reliability. The contractor only had access to determine which Cyber Assets were CCAs, but was not granted access to those CCAs. The individual, who had authorized access for many years prior to deactivation, completed cyber security training and did not attempt to access the tool which allowed access to CCA information. In determining the appropriate penalty, RFC found the RFC-MRO_UREs’ internal compliance program to be a mitigating factor; however, the repeat noncompliance was considered an aggravating factor. The CIP programs at RFC-MRO_UREs had been reviewed and revised after previous CIP violations to ensure compliance; however, the current violations of CIP-003-3 R6, CIP-007-3 R1 and R2 and CIP-004-3 R3 were found to be evidence of a failure of the CIP compliance program, which was an aggravating factor in penalty determination. Several of the violations were self-reported, which RFC found to be a mitigating factor.

Total Penalty: $20,000 (aggregate for 9 violations)

FERC Order: Issued August 26, 2013 (no further review)

Unidentified Registered Entities 1, 2, 3, 4 (RFC_URE1, RFC_URE2, RFC_URE3, MRO_URE1) (collectively, RFC-MRO_UREs), Docket No. NP13-41-000 (June 27, 2013)

Reliability Standard: CIP-003-3

Requirement: 6

Violation Risk Factor: Lower

Violation Severity Level: Severe Region(s): ReliabilityFirst, MRO

Issue: RFC_URE2 initially self-certified a violation of CIP-003-3 R6, and upon further review self-report further CIP-007-3 Reliability Standards violations. URE found through an internal self-assessment that a CCA router was missing a configuration baseline, which is required by CIP-003-3 R6. URE also did not follow established testing procedures on the router. In particular, URE did not carry out procedures for cybersecurity testing (CIP-007-3 R1.1); did not document that the testing was conducted in a way that reflects the production environment (CIP-007-3 R1.2); and did not document test results (CIP-007-3 R1.3). Lastly, URE did not document that only those ports and services needed for normal or emergency operations were enabled on the router, which left URE unable to show that the ports and services were enabled or disabled as appropriate (CIP-007-3 R2).

Finding: The violation was deemed to pose a moderate, but not serious or substantial, risk to BPS reliability. RFC_URE2 ensured that the router was secured with a domain identification and password, with authorization within the network and with membership requirement within the security group. Additionally, the devices supported by the network are built and operated using management templates to grant access to only the ports and services necessary for normal and emergency operations; though RFC_URE2 did not document that this had occurred for this router. In determining the appropriate penalty, RFC found the RFC-MRO_UREs’ internal compliance program to be a mitigating factor; however, the repeat noncompliance was considered an aggravating factor. The CIP programs at RFC-MRO_UREs had been reviewed and revised after previous CIP violations to ensure compliance; however, the current violations of CIP-003-3 R6, CIP-007-3 R1 and R2 and CIP-004-3 R3 were found to be evidence of a failure of the CIP compliance program, which was an aggravating factor in penalty determination. Several of the violations were self-reported, which RFC found to be a mitigating factor.

Total Penalty: $20,000 (aggregate for 9 violations)

FERC Order: Issued August 26, 2013 (no further review)

Unidentified Registered Entities, FERC Docket No. NP13-47 (July 31, 2013)

Reliability Standard: CIP-003-3

Requirement: 4 (2 violations – RFC and SERC), 5 (2 violations – RFC and SERC), 6 (2 violations – RFC and SERC)

Violation Risk Factor: Medium (4), Lower (5, 6)

Violation Severity Level: Severe (4, 5, 6)

Region: RFC and SERC

Issue: URE1 and URE2 (collectively, URE) self-reported that tickets in its change control system should have been classified as CCA information, but were not (4). In addition, URE did not have sufficient documentation of its access privileges or include links to defined or approved rules (as URE’s documentation did not clearly delineate which individuals are assigned to which roles or the access rights that they possessed). URE also did not properly assess and document, on an annual basis, its processes for controlling access privileges to CCA information (5). URE also did not properly document the entity or vendor-related changes made, pursuant to the change control process, to the hardware and software components of 60.04% of its CCAs. There were also several instances where URE’s business units did not follow the change control process. URE did not adequately establish and documents its configuration management process for adding, modifying, replacing or removing CCA hardware or software (6).

Finding: SERC and RFC found that the CIP-003-3 R4, R5 and R6 violations constituted a moderate risk to BPS reliability. For the R4 violations, it increased the chances of inappropriate access to CCA information. But, the risk to the BPS was mitigated since URE’s CCA information repositories are not publicly available and URE limits who has access to the information. In terms of the R5 violations, without the appropriate processes to control access to protected information, URE cannot guarantee that protected information is secured, which increases the chances of excessive or unauthorized access to URE’s system. But, the risk to the BPS was mitigated since authorization was required to access the information and URE limited access to individuals with a business need to access the information. For the R6 violations, configuration management is intended to ensure a secure environment and insufficient implementation and support of configuration management increases the risk of unwanted security vulnerabilities and unauthorized access points and can affect the availability of critical systems. The risk to the BPS was mitigated by URE performing a staged implementation in development environments before implementing the changes and that URE did have change management systems in place to manage any changes that would introduce new Cyber Assets into the ESP and the addition of new access points to the ESP. URE admitted the violations. In approving the settlement agreement, NERC BOTCC evaluated as aggravating factors URE’s compliance history and the fact that URE did not promptly prepare mitigation plans to remediate many of the violations. But, URE did self-report some of the violations and was cooperative during the enforcement process and did not conceal the violations. URE also had a compliance program in place when the violations occurred (even though URE only received partial mitigating credit as most of URE’s violations resulted from a lack of execution and coordination of programs). URE committed to perform certain actions that went above and beyond the compliance requirements. The CIP-002-3 R3 violations presented a serious and substantial risk to BPS reliability, whereas the other violations did not present a serious or substantial risk to BPS reliability.

Total Penalty: $350,000, $175,000 for each URE (aggregate for 62 violations)

FERC Order: Issued August 30, 2013 (no further review)

Unidentified Registered Entity, FERC Docket No. NP15-11-000 (November 25, 2014)

Reliability Standard: CIP-003-3

Requirement: R2/R3

Violation Risk Factor: Medium/Lower

Violation Severity Level: Severe/Severe

Region: ReliabilityFirst, MRO and SERC (collectively, "Regions")

Issue: URE self-reported that it did not document the delegation of responsibilities by senior manager to delegates which resulted in a cybersecurity manager approving extensions to cybersecurity exceptions on four occasions and senior management not approving cybersecurity exceptions annually on three occasions.

Finding: ReliabilityFirst determined that the violation constituted only a minimal risk to the BPS reliability since a documentation error caused the violation and only one of the cybersecurity exceptions related to a CIP matter. To mitigate the risk additional delegates were appointed including the cybersecurity manager, who was also qualified to review and approve extensions. URE later assigned the responsibility of reviewing cybersecurity extensions to the cybersecurity manager as part of URE's mitigation plan. URE neither admitted nor denied the violations. In approving the settlement agreement, the NERC BOTCC found that URE had a compliance program in place at the time of the violations. While none of the violations independently posed a serious or substantial threat to the BPS reliability, collectively they represented a significant risk to BPS reliability. Eighteen of the violations posed only a minimal or moderate risk to the BPS reliability and only one posed a serious or substantial risk to the BPS reliability. URE did have a history of prior violations; however, this was their first comprehensive CIP audit so the Regions did not consider their prior history an aggravating factor. URE self-reported several of the violations, performed several reliability enhancements and outreach programs and agreed to above and beyond compliance measures, including the implementation of an improvement program that is focused on improving its compliance with CIP standards and cybersecurity efforts. However, due to the duration of several of URE's violations, the Regions considered this risk negatively. URE was cooperative throughout the enforcement process and did not attempt to conceal the violations. There were no additional mitigating factors that affected the penalty.

Penalty: $75,000 (aggregate for 19 violations)

FERC Order: Issued December 23, 2014 (no further review)

Unidentified Registered Entities (UREs), FERC Docket No. NP15-15-000 (December 30, 2014)

Reliability Standard: CIP-003-3

Requirement: R6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: SERC

Issue: URE1 self-reported that it did not update malware prevention software on CCA according to its change control and configuration management process which requires that all changes to hardware and software on Cyber Assets are documented using a change control management ticket that includes testing, approvals and documentation.

Finding: SERC determined that the violation posed only a minimal risk to the BPS reliability as URE1 detected the oversight the next day and conducted subsequent testing (which indicated no issues) thereby reducing the risk of a malicious attack on its cybersecurity controls or CCAs. Additionally, URE1 had conducted cybersecurity and functionality testing on the malware updates, which demonstrated no adverse impacts on functionality or operations. URE1's EMS was monitored 24/7 by operators, who alert personnel whenever there is a reduction in system performance. In addition, system administrators are alerted by a security status monitoring program whenever there are signs of malicious software activity. Furthermore, the workstations at issue are protected within a PSP, reside in an ESP and only personnel with current PRAs and cybersecurity training can access them. UREs neither admitted nor denied the violations. In approving the settlement agreement, the NERC BOTCC considered the UREs history of violations an aggravating factor. However, the UREs did have an internal compliance program in place, which SERC considered a mitigating factor. While URE1 self-reported a violation, the UREs did not received mitigating credit since it was reported after it received a compliance audit notice. While three of the violations posed a serious or substantial risk to the BPS reliability, the majority of them (18) posed only a minimal or moderate risk. URE agreed to additional mitigating actions including: adding additional staff (including converting contractors to full-time); conducting human performance training at a cost of $50,000; installing firewall analyzer software at a cost of $485,000; and additional mitigating factors for five of the violations. The UREs were cooperative throughout the enforcement process and there was no evidence that they concealed the violations.

Penalty: $120,000 (aggregate for 21 violations)

FERC Order: Order on Review of Notice of Penalty, issued January 29, 2015. 150 FERC 61,051.

Unidentified Registered Entity 1, FERC Docket No. NP15-26-000 (April 30, 2015)

Reliability Standard: CIP-003-3

Requirement: R6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: ReliabilityFirst

Issue: URE1 self-reported that it did not follow its change control process (that required changes to be logged and approved by a change management advisory committee member) in one instance as a result of human error. ReliabilityFirst found that URE1 did not create and record a change control process and configuration management for (1) adding, modifying, replacing or removing CCA hardware and (2) identifying, controlling and recording all entity or vendor-related changes, as per its change control process.

Finding: ReliabilityFirst found that the violation posed only a minimal, but not a serious or substantial risk, to BPS reliability. The severity of the violation was decreased because the change was still approved by a staff member and URE1 mitigated its actions to prevent a recurrence. The UREs neither admitted nor denied the violations. In approving the settlement between ReliabilityFirst and five UREs in total, the NERC considered the UREs’ prior violations and the minimal to moderate risk posed by the current violations. As mitigating factors, the NERC found that among the UREs subject to the settlement agreement, individual URES had: (1) implemented pre-violation internal compliance programs, (2) self-reported violations, (3) cooperated throughout the enforcement process, (4) not attempted or intended to conceal the violations, (5) made significant progress in post-violation compliance. ReliabilityFirst found that no penalty was necessary since the violations posed minimal risks, the UREs demonstrated improvements in compliance, and the UREs that had prior violations were fully sanctioned for those violations in an FERC-approved settlement. URE1’s mitigation plan obliged URE1, among other things, to require validation of all change approvals prior to the start of work.

Penalty: $0 (aggregate for 22 violations)

FERC Order: The NERC did not further review the settlement and the settlement entered into effect on May 29th, 2015.

Unidentified Registered Entity 1 (FRCC_URE1), FERC Docket No. NP15-29-000 (May 28, 2015)

Reliability Standard: CIP-003-3

Requirement: R1

Violation Risk Factor: Medium

Violation Severity Level: Severe

Region: FRCC

Issue: FRCC_URE1 self-reported that a badge allowing a janitorial contractor physical access to Critical Cyber Assets (CCAs) was not timely revoked when the contractor began a leave of absence. The badge was given to a new and unauthorized contractor. FRCC_URE1 retrieved the badge two days later.

Finding: FRCC found that this violation posed a moderate, but not a serious or substantial, risk to BPS reliability because unauthorized access increases CCAs' vulnerability to physical attacks and alterations. FRCC found the risk was aggravated because the unauthorized janitorial contractor did not have a CIP-level personnel risk assessment on file and had not done cyber security training. However, the janitorial contractor would not have been allowed to physically contact the CCAs because authorized personnel staffed FRCC_URE1 at all times and would not have allowed such contact. To mitigate the violation, FRCC_URE1 (1) retrieved the unauthorized badge, (2) audited janitorial badges to ensure that individuals with badges were authorized, (3) retrained janitorial employees on security requirements, and (4) developed a Spanish version of cyber security training.

Penalty: $13,000 (aggregate for 2 violations)

FERC Order: FERC approved the settlement on June 26th, 2015.

Unidentified Registered Entity 2 (SERC_URE3), FERC Docket No. NP18-25-000 (August 30, 2018)

NERC Violation ID: SERC2016016029

Reliability Standard: CIP-003-3

Requirement: R6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: SERC Reliability Corporation (SERC)

Issue: SERC_URE3 did not implement an effective change control and configuration management program and deployed patches prior to the scheduled install date. SERC_URE3 submitted a Self-Report stating that it failed to test security patches under its existing change management program in several occurrences, which resulted in the early deployment of patches, as well as the unauthorized release of patches through the automated tool. SERC_URE3 deployed fifteen security patches early, each of which affected multiple workstations. Most of the workstations were located in the electric transmission system operations center (SOC), and some were located in the emergency system operations center (ESOC). SERC_URE3 discovered the noncompliance when reviewing an unscheduled workstation reboot, and thereafter resolved the issue by deleting the batch job within the deployment tool and verifying there were no adverse impacts several days later. One month later, however, SERC_URE3 again deployed security patches prior to the scheduled date, which affected additional workstations in both the SOC and ESOC. SERC_URE3 discovered this additional instance the day after it occurred and ensured the patch job was permanently deleted rather than placed in a hidden folder. In another instance, SERC_URE3 discovered unauthorized workstation security patches installed on energy management system (EMS) workstations. SERC_URE3 identified the root causes of the violations as weak operational technical controls, namely with its patch deployment server and application, inadequate training, and staff that was unfamiliar with the new tool and all of its new features pertaining to patch deployment.

Finding: SERC found the violation constituted a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS). By its failure to test security patches under the existing change management program, SERC_URE3 could have permitted malicious changes to occur that would affect security and operations of the local EMS. More than two thousand patches were released early, and more than two hundred patches were released without documented authorization, although all had been assessed and approved for eventual release. The duration of the violation started when SERC_URE3 deployed the first set of security patches early and ended when SERC_URE3 completed security controls testing after deployment to ensure no adverse impacts occurred. SERC considered SERC_URE3’s internal compliance program as a mitigating factor and SERC_URE3’s and its affiliate’s compliance history to be an aggravating factor. To mitigate the violation, SERC_URE3, among other steps, executed a task to validate that cyber security controls were not adversely impacted on affected workstations, consulted and worked with its vendor to evaluate the patch process and further analyze the extent of condition, redefined and documented the workstation patch process improvements, provided additional training to employees and backup product subject matter experts from the vendor, approved changes to the baseline for the patches that were implemented, and examined the extent of condition for two consecutive months following the violation.

Penalty: $220,000

FERC Order: Issued August 30, 2018 (no further review)

Unidentified Registered Entity 2 (SERC_URE2), FERC Docket No. NP18-25-000 (August 30, 2018)

NERC Violation ID: SERC2016016030

Reliability Standard: CIP-003-3

Requirement: R6

Violation Risk Factor: Lower

Violation Severity Level: Severe

Region: SERC Reliability Corporation (SERC)

Issue: SERC_URE2 inadvertently deployed fifteen security patches prior to the scheduled date of installation that affected several workstations. Following its discovery of the violation two months later as it reviewed an unscheduled workstation reboot, SERC_URE2 submitted a Self-Report citing a failure to implement an effective change control and configuration management program and identified the patch job problem as the root cause, subsequently deleting the patch job within the deployment tool. The workstations involved were located in the power generation Market Operations Center (MOC). Although SERC_URE2 believed it had resolved the issue a month after the violation, after it used the patch deployment service to delete the scheduled patch job, SERC_URE2 discovered a reoccurrence of patches deployed prior to the arranged date implicating new workstations in the power generation Emergency Market Operations Center (EMOC) and in the MOC. However, SERC_URE2 identified this issue the following day after it conducted a review of the CIP program for a particular security patch issue flagged by the corporate side of the company. In another instance, SERC_URE2 discovered unauthorized workstation security patches installed on 97.33% of its energy management system workstations. SERC_URE2 identified the root causes of the violations as inadequate training and weak operational technical controls that failed to recognize the operational issues with its patch deployment server and application.

Finding: SERC found the violation constituted a moderate risk and did not pose a serious or substantial risk to the reliability of the Bulk Power System (BPS). By its overall failure to test security patches under its existing change management program, SERC_URE2 could have permitted changes to occur that would affect security and operations of the local EMS and potentially create erroneous data models to which other entities may have reacted. The duration of the violation started when SERC_URE2 deployed the first set of security patches ahead of the arranged installation date and ended when SERC_URE2 completed security controls testing post-deployment to ensure no adverse impacts occurred. SERC considered SERC_URE2’s internal compliance program as a mitigating factor but found SERC_URE2 and its affiliate’s compliance history to be an aggravating factor in the penalty determination. To mitigate the violation, SERC_URE2, among other steps, consulted with its vendor to evaluate its patch process and determine whether a more effective process exists for deploying patches, validated that affected workstations’ cyber security controls were not adversely affected, and provided additional training for procedural changes, and lead product training for vendors.

Penalty: $220,000

FERC Order: Issued August 30, 2018 (no further review)

Unidentified Registered Entity, FERC Docket No. NP18-7-000

Region: Western Electricity Coordinating Council (WECC)

Issue: A third-party URE contractor exceeded its authorized access by improperly copying URE data from URE's network to the contractor's network, where it was no longer subject to URE's visibility or controls. While the data was on this contractor's network, a subset of live URE data was accessible online without the need to enter a user ID/password. This subset included over 30,000 asset records, including records associated with Critical Cyber Assets (CCAs). The CCAs associated with the data exposure included servers that store user data, systems that control access within URE's control centers and substations, and a supervisory control and data acquisition (SCADA) system that stores critical CCA Information. The records included information such as IP addresses and server host names. The usernames of the database were also exposed, which included cryptographic information of those usernames and passwords. The CCA related information was accessible on the internet for 70 days. The URE discovered this online data exposure due to a report it received from a white hat security researcher not associated with URE.

Finding: WECC determined that URE failed to implement adequately its program to identify, classify, and protect information associated with CCAs, as required by CIP-003-3 R4. This was caused due to the URE's failure to apply its information protection program to the CCA Information in its pre-production environment.

WECC also determined that the URE failed to adequately implement a program for managing access to protected information related to CCAs, as required by CIP-003-3 R5. Specifically, the URE failed to ensure that the contractor protected the CCA Information when it improperly copied the data from URE's network environment to its own, where it was no longer subject to URE's visibility or controls. The cause of this violation was URE's failure to ensure its contractor followed its information protection program and procedures on which the contractor was trained.

WECC found that the violations posed a serious/substantial risk to the reliability of the bulk power system (BPS). Exposure of the username and cryptographic information could aid a malicious attacker in using this information to decode the passwords, which increases the risk of the attacker gaining both physical and remote access to URE's systems. The attacker could then use this information to breach the secure infrastructure and access the internal CCAs by jumping from host to host within the network. Once in the network, the attacker could attempt to login to CCAs, aided by the possession of username and password information.

The following were considered in WECC's assessment of penalty: (a) the violations constituted the URE's first violations of the standards in concern; (b) URE had an internal compliance program at the time of the violation; (c) URE self-reported these violations; (d) URE was not fully transparent and forthcoming with all pertinent information detailing the data exposed in the incident (specifically, it didn't initially provide WECC with all the data fields exposed in the incident); and (e) the violations posed a serious/substantial risk to the BPS's reliability.

Penalty: $2,700,000

FERC Order: Issued May 30, 2018 (no further review)

Unidentified Registered Entity 2 (SERC_URE2) and Unidentified Registered Entity 3 (SERC_URE3), FERC Docket No. NP18-25-000

Region: SERC Reliability Corporation (SERC)

Violation Start Dates and End Dates:

Issues: SERC_URE2 and SERC_URE3 self-reported violations of the CIP standards set out above as following:

a) CIP-003-3 – a violation of CIP-003-3 (R6) occurred twice when SERC_URE2 deployed patches prior to the scheduled install date. 15 security patches were deployed early and affected several workstations, including some SERC_URE2 workstations and the majority with an affiliate. Even though SERC_URE2 had deleted the batch job, the issue occurred again because deleting the job moved the task into a hidden folder where it still functioned and triggered any available security patches to deploy. In another instance, SERC_URE2 discovered unauthorized workstation security patches installed on 97.33% of its and its affiliate's energy management system (EMS) workstations. SERC_URE2's use of a master workstation image to build out the EMS workstations resulted in the same unique identifier, used by the patching management tool, across the EMS workstations. Duplication of the identifier resulted in patch deployment outside of the change management process and ticketing system, which SERC_URE2 normally used to capture documentation of the patch assessment and approval, trigger the deployment schedule, and document the results.

b) CIP-007-6 – a violation of CIP-007-6 (R2; P2.2) occurred as SERC_URE2 did not assess 3 security patches for applicability within 35-day availability and assessed them 2 days late. The security patches were applicable to production AD domain controllers (DCs) classified as Electronic Access Control and/or Monitoring Systems (EACMSs) associated with eight High Impact Bulk Electric System Cyber Systems (BCSs).

c) CIP-004-6 – a violation of CIP-004-6 (R4; P4.1) occurred twice because SERC_URE2 did not implement a process to authorize access into the PSP as required after a PSP reconfiguration removed existing controls. In the first instance, SERC_URE2's parent company executed a planned change to modify 3 physically separated areas (of which 2 were PSPs) by removing interior chain-link fencing between each of them and creating one large cage PSP without interior barriers. However, post execution of such change, the now singular PSP retained the two PSP access doors, but removed a third (demilitarized zone) DMZ cage door. This permitted individuals who only had specific access permissions to one of the 3 areas to have physical access upon entry to Cyber Assets that SERC_URE2 had not authorized them for. SERC_URE2 permitted 8 unauthorized individuals access to Electronic Access Control and/or Monitoring System Cyber Assets within the firewall cage, and 18 unauthorized individuals access to multiple Physical Access Control System (PACS) Cyber Assets within the access control cage. The second time, SERC_URE2 erroneously granted a DBA electronic access to a PACS database which the individual was not authorized for.

d) CIP-010-2 – a violation of CIP-010-2 (R1; P1.4) occurred because prior to SERC_URE2 making a change that deviated from the baseline configuration, SERC_URE2 personnel did not determine the required cyber security controls in CIP-005 and CIP-007 that could be impacted by the change. Following the change, SERC_URE2 did not verify that required cyber security controls were not adversely affected and did not document such verification.

e) CIP-003-3 – a violation of CIP-003-3 (R6) occurred for SERC_URE3 on similar facts as for SERC_URE2 for its corresponding violation discussed in (a) above.

f) CIP-007-6 – a violation of CIP-007-6 (R2; P2.2) occurred because SERC_URE3 did not install or create a mitigation plan for security patches within 35 days of patch assessment. During 2 consecutive patch application cycles, 2.67% of SERC_URE3 workstations weren't communicating with the automated patching server and failed to receive 9 assessed and approved security patches.

g) CIP-010-2 – a violation of CIP-010-2 (R1) because SERC_URE3 did not update baseline configurations within 30 calendar days of a change (Part 1.3) and it did not validate cyber security controls through testing (Part 1.4). SERC_URE3 patched EACM database servers used to control remote access to Medium Impact BCSs. A week later, it patched a second database server. Although the initial SERC_URE3 patch of the first database server initiated the SERC_URE3 cyber security controls testing process, SERC_URE3 had not tested the security controls and updated the baseline for this change.

h) CIP-010-2 – a violation of CIP-01002 (R1; P1.1) occurred in 4 separate instances where SERC_URE3 did not develop baselines for all Cyber Assets within the CIP environment. The first instance involved a patch management server (this was a PCA within the High Impact BCS ESP). In the second instance, SERC_URE3's failure to include its EACM hosting servers in its baselines led to its corporate IT patching the operating system on the servers at irregular intervals. The servers were associated with High Impact BCS. The third instance involved SERC_URE3 failing to remove domain controller PCAs that it had installed within the High Impact BCS but never introduced into production as EACMs. After installing them, SERC_URE3 had decided to remove these from within the ESP but the domain controllers remained powered on and connected to the network within the ESP until they were discovered. The last instance occurred when SERC_URE3 made some desktop virtualization servers (DVSs) accessible to its EMS coordinator, but the teams completing the work did not document the baseline configurations and did not notify the SERC_URE3 EMS team (which was responsible for CIP Compliance activities).

i) CIP-007-6 – a violation of CIP-007-6 (R2; P2.2) occurred for SERC_URE3 on similar facts as SERC_URE2's corresponding violation discussed in (b) above.

j) CIP-004-6 – a violation of CIP-004-6 (R4; P4.1) occurred thrice for SERC_URE3. The first two instances involved similar facts as SERC_URE2's corresponding violation discussed in (c) above. The third instance occurred when a member of the access services team inadvertently granted a contractor physical access to all SERC_URE3 substations instead of the single substation the contractor was authorized to perform work.

k) CIP-010-2 – a violation of CIP-010-2 (R1, P1.4) occurred for SERC_URE3 on similar facts as SERC_URE2's corresponding violation discussed in (d) above.

l) CIP-007-6 – a violation of CIP-007-6 (R2; P2.1) occurred because SERC_URE3 did not effectively implement a patch management process for tracking, evaluating, and installing cyber security patches for multiple applicable Cyber Assets. It installed a software suite on application host servers used within its EMS, but omitted the software from its patch tracking and evaluation program, which feeds its baseline documentation. The host servers involved were Bulk Electric System Cyber Assets (BCAs) within High Impact BES Cyber Systems.

m) CIP-010-2 – a violation of CIP-010-2 (R4) occurred because SERC_URE3 did not implement documented plans for the management of removable media that included authorization and malicious code mitigation. A SERC_URE3 contractor connected a removable USB device to a workstation that was part of the SERC_URE3 EMS. The workstation was a BCA within a High Impact BCS. SERC_URE3 discovered that within 3 months' time, there were 7 additional such instances where personnel had connected removable media to 3 separate EMS workstations without proper authorizations. SERC_URE3 found USB ports had missing printed stickers that were placed to advise users not to connect unless authorized. SERC_URE3's EMS staff failed to investigate the first 7 alerts.

n) CIP-006-6 – a violation of CIP-006-6 (R2; P2.2) occurred when SERC_URE3 did not document visitor entry into and exit from the PSP for an individual in 3 separate instances on the same day.

o) CIP-007-6 – a violation of CIP-007-6 (R2; P2.2) occurred twice when SERC_URE3 failed to assess 3 security patches for applicability within 35-days of availability. In the first instance, an operating system vendor made a change to the digital certificate without notifying the users. SERC_URE3 had developed an automated script internally that it used to obtain and compile available security patches from the identified source. When the vendor made the certificate change, the automated pull no longer properly linked to the source. In the second instance, a vendor made a change in the entitlement certificate of a patching source repository, without notifying SERC_URE3. This change caused the patch repository to stop receiving security patch updates. Due to the certificate change, SERC_URE3 did not identify any available patches during its security patch assessments for two consecutive months.

Findings:

a) CIP-003-3 (R6) – SERC_URE2 identified the root cause of this violation as weak operational technical controls and inadequate training. It failed to recognize the operational issues with its patch deployment server and application, and did not have adequate internal controls to detect early patch deployments. This violation posed a moderate risk to the reliability of the bulk power system (BPS). SERC_URE2's overall failure to test security patches under its existing change management program could have permitted changes to occur that would affect security and operations of the local EMS and potentially create erroneous data models that other entities may react to. SERC_URE2's unfamiliarity with the functionality of the tool could have exposed its EMS to negative security modifications.

b) CIP-007-6 – SERC_URE2 identified the root cause of this violation as a lack of controls and human performance issues. This issue posed a minimal risk to the BPS's reliability. SERC_URE2's failure could have permitted known vulnerabilities to remain available for exploit, giving bad actors additional time to exploit and potentially degrade local operations or impact the BPS.

c) CIP-004-6 – SERC_URE2 concluded the root-cause in the first instance was a lack of training and controls. In the second instance, the root cause was a manual process that lacked detailed instructions and was commingled with non-CIP access requests. This violation posed a minimal risk to the BPS's reliability. SERC_URE2's failure to properly control access provisions could have permitted unauthorized individuals to access and possibly modify settings, either from unintentional or malicious actions, and cause operational impacts. However, everyone involved in both instances had received cyber security training and had a valid Personnel Risk Assessment on file.

d) CIP-010-2 – The root cause of this violation was a lack of complete work procedures and checklists, coupled with a lack of management oversight and appropriate internal controls. This violation posed a moderate risk to the BPS's reliability. SERC_URE2's failure to document the security controls that could be impacted by a change could have permitted changes to be made, resulting in unforeseen impacts to the security controls.

e) CIP-003-3 – SERC's findings in this regard for SERC_URE3 were similar to those for the corresponding violation by SERC_URE2 and are discussed in (a) above.

f) CIP-007-6 – SERC_URE3 determined that the root cause of this violation was an incomplete patching procedure coupled with a lack of controls. The patching procedure did not specify that the SERC_URE3 technician must validate that communications existed between Cyber Assets and the patching server upon commissioning. Thereafter, SERC_URE3 didn't ensure ongoing communications through controls that it hadn't established. This issue posed a minimal risk to the BPS's reliability. SERC_URE3's failure to implement security patches as planned could have permitted known vulnerabilities to remain available for exploit, giving bad actors additional time to exploit and potentially degrade local operations or impact the BPS.

g) CIP-010-2 – SERC_URE3 determined the root-cause of this failure was inadequate training. SERC found that this issue posed a moderate risk to the BPS's reliability. SERC_URE3's failure to test changes and update baseline documentation could permit changes that would impact cyber security controls to exist and remain available for exploit. Additionally, outdated baselines could allow SERC_URE3 to conduct inaccurate tests for new or additional changes, due to stale information about the state of the BCS or BCAs.

h) CIP-010-2 – SERC_URE3 determined that the root causes of the violations were training deficiency, insufficient processes and procedures, organizational weaknesses, and a lack of defined expectations. This issue posed a moderate risk to the BPS's reliability. SERC_URE3's failure to create and maintain baselines of Cyber Assets within the ESPs could permit patching failures to occur, leaving security vulnerabilities available for exploit. Additionally, baseline omissions could create operational impacts if necessary Cyber Assets were removed or added without proper controls and testing. However, none of the four instances involved BCAs.

i) CIP-007-6 – SERC_URE3 determined that the root cause of this violation was a lack of controls and human performance. This violation posed a minimal risk to the BPS's reliability. SERC_URE3's failure to assess security patches could have permitted known vulnerabilities to remain available for exploit, giving bad actors additional time to exploit and potentially degrade local operations or impact the BPS.

j) CIP-004-6 – SERC's findings in this regard for SERC_URE 3 were similar to those for the corresponding violation by SERC_URE2 and are discussed in (c) above.

k) CIP-010-2 – SERC's findings in this regard for SERC_URE 3 were similar to those for the corresponding violation by SERC_URE2 and are discussed in (d) above.

l) CIP-007-6 – SERC_URE3 determined the root cause of this violation was an organizational and procedural weakness. SERC_URE3 did not have a step in its change management program to ensure that SERC_URE3 personnel tracked and evaluated patches for all software being installed, and did not have a requirement for all software installed to be assessed to ensure patch levels were up-to-date. This violation posed a minimal risk to the BPS's reliability. SERC_URE3's failure to track and patch installed software could have permitted known vulnerabilities to be exploited, creating conditions where local operations could be negatively impacted resulting in more extensive BPS impacts.

m) CIP-010-2 – SERC determined that the root-cause was a lack of training and management oversight. This violation posed a moderate risk BPS's reliability. The use of removable media on EMS workstations could permit malware or viruses to become active within the high impact BCS, and possibly propagate through the network, resulting in a loss of command and control of the BPS, or control being ceded to individuals outside of the company who may have bad intentions.

n) CIP-006-6 – SERC_URE3 determined the root-cause of this issue was inadequate training. This violation posed a minimal risk to the BPS's reliability. SERC_URE3's failure to document visitor presence within the PSP could make any post-event forensics difficult and uncertain, since attendance in the PSP would be unclear.

o) CIP-007-6 – The root causes of the violations were a combination of operational control failures, human performance, organizational weaknesses and a lack of oversight. This issue posed a moderate risk to the BPS's reliability. SERC_URE3's failure to assess security patches could have permitted known vulnerabilities to remain available for exploit, giving bad actors additional time to exploit and potentially degrade local operations or impact the BPS.

The following were considered in the assessment of penalty for SERC_URE2 and SERC_URE3: (a) SERC_URE2's and SERC_URE3's internal compliance program; (b) SERC_URE2's and its affiliate's and SERC_URE3's and its affiliate's compliance histories. Additionally, SERC noted several specific circumstances in each violation that limited its impact, scope or duration.

Penalty: $220,000

FERC Order: Issued September 28, 2018 (no further review)

NP18-14-000: Unidentified Registered Entity

Region: RFC

Issue:  RFC determined that:

• (1) URE failed to document and implement a program for managing access to protected Critical Cyber Assets (CCAs) as required by CIP-003-3 R5, in three separate instances: in two instances, employees did not immediately pick up printed versions of CIP documents from printers, and in the third instance, URE inadvertently set the confidentiality classification level for a CIP process document to "public" view on its internal site; 
• (2) URE failed to document and implement a program for managing access to protected CCAs as required by CIP-003-3 R5 when a URE reliability assurance team member was able to access a file in a URE NERC compliance information folder, which contained CIP protected information;
• (3) URE failed to have minimum security management controls in place to protect CCAs: an analyst added a device to URE's logging tool and added the correct group to receive the alerting for the device, and the next day, another analyst deleted the device from the logging tool, and thereafter the logging tool was retaining logs for a certain device, but the device was not in the correct group to alert on the required conditions;
• (4) URE failed to have minimum security management controls in place to protect CCAs in two instances where URE did not follow the established change management process: first, when URE deployed a PACS intelligent controller into production, and second, when URE made changes to several assets; 
• (5) URE failed to have a documented Personnel Risk Assessment (PRA) program for personnel having authorized cyber or authorized unescorted physical access to CCAs, granting four employees unescorted physical access without appropriately documented PRAs;
• (6) URE failed to maintain lists of personnel with authorized cyber or authorized unescorted physical access to CCAs in three instances of URE not properly revoking access within the seven-day period, one instance where access for one URE employee located at another registered entity's facility was not properly revoked within the seven-day period and  one instance where access for one URE employee was not properly revoked within the seven-day period after changing job duties within URE;
• (7) URE failed to ensure that every CCA resided within an Electronic Security Perimeter (ESP), where URE did not identify and document an access point to the ESP; 
• (8) URE failed to document, implement, and maintain a physical security plan as required by CIP-006-3c R1, as during a routine inspection URE discovered that an air conditioning unit was an exploitable access point into an identified PSP; 
• (9) URE failed to document and implement the technical and procedural controls for monitoring physical access at all access points to the PSPs 24 hours a day, seven days a week, as the power supply to a security rack was shut off during maintenance work at one of URE's facilities, and for six days afterwards the facility was not communicating with URE's headquarters;
• (10) URE failed to ensure that new Cyber Assets and significant changes to existing Cyber Assets within the ESP do not adversely affect existing cyber security controls when a URE asset administrator performed a PACS modification, but did not complete the change and configuration management process documentation for a device that was replaced; 
• (11) URE failed to establish, document, and implement a process to ensure that only those ports and services required for normal and emergency operations are enabled, as URE had multiple undocumented services with ports enabled related to its PACS, and one of those ports was unnecessary for operations; 
• (12) URE was two weeks late in completing evaluations of security patches for two Cyber Assets included in CIP-007-6 Table R2-Security Patch Management., which were both PACS;
• (13) failed to establish, document, and implement a security patch management program for tracking, evaluating, testing, and installing applicable cyber security software patches for all Cyber Assets within the ESP when it failed to install three sets of patches in a timely manner, and no compensating measures were documented to mitigate risk exposure and failed to patch a certain server and failed to evaluate software supplied and installed by URE on associated devices;
• (14) URE failed to establish, implement, and document technical and procedural controls that enforce access authentication of, and accountability for, all user activity, and that minimize the risk of unauthorized system access when it did not change one password for a relay despite documentation stating the password was changed; 
• (15) URE failed to change passwords at least once every 15 calendar months for a shared account that could be used for interactive user access for two Cyber Assets, both PACS; 
• (16) URE failed to ensure that all Cyber Assets within the ESP, as technically feasible, implement automated tools or organizational process controls to monitor system events that are related to cyber security when they commissioned four devices and inadvertently created firewall rules that disallowed logging and when certain programs were not sending failed login attempt notifications to URE's logging tool; and 
• (17) URE failed to perform an initial risk assessment and subsequent risk assessments of its transmission stations and transmission substations that meet the criteria specified in CIP-014-2 when URE did not assess one substation pursuant to Section 4.1.1 of the CIP-014-2.

Finding: RFC determined that fifteen of the violations posed a minimal risk, six posed a moderate risk, and one posed a serious and substantial risk to the reliability of the bulk power system (BPS). RFC determined the violations do not involve and are not indicative of programmatic issues across URE's CIP compliance program. URE implemented internal controls that identified many of the instant violations. While most of the violations were short in duration, or relatively short, several of the moderate risk violations had longer durations (up to two years), indicating a potential weakness in detective controls in these areas. Nevertheless, these moderate risk violations generally involved isolated systems or assets, and thus, did not involve programmatic or systemic issues. The serious risk violation of CIP-006-3c R5 provided the opportunity for undetected compromise of an unmanned, critical substation and showed URE's inability to respond due to lack of situational awareness. While the risk was somewhat mitigated because certain assets were being monitored via an alert and monitoring program, which would have detected unauthorized changes, and local physical access controls were working, URE's headquarters could not monitor or communicate with the site and thus would have been unaware of and unable to respond to an intrusion. 

Penalty: $180,000

FERC Order: Issued May 31, 2018 (no further review)

NP19-4-000: Unidentified Registered Entity

Please search for this docket no. here ››