The North American Electric Reliability Corporation and the Electricity Information Sharing and Analysis Center recently released a lessons learned report tied to a late 2023 grid security exercise, GridEx VII.
Scott Corwin, President and CEO of the American Public Power Association, and Adrienne Lotto, Senior Vice President for Grid Security, Technical & Operations Services, at APPA, participated in the exercise, which took place on Nov. 15-16.
More than 15,000 participants from approximately 250 North American organizations -- including the electric industry, cross-sector partners from gas and telecommunications, and U.S. and Canadian government partners -- took part in a two-day exercise, which tested operational and policy measures that would be needed to restore the grid following a severe cyber and physical attack.
Using core planning and exercise materials developed by E-ISAC’s GridEx team, participants tested their response and restoration capabilities by customizing the core scenario to meet their actual operating environment.
GridEx VII concluded with a day-long Executive session during which industry executives and government leaders from the United States and Canada convened in-person in Washington, D.C., as well as virtually, to explore strategic and policy implications presented by the scenario.
The lessons learned report released in early April includes key recommendations “to industry’s success during a severe security event”:
- Explore opportunities to improve the transmission of critical data between control centers
- Evaluate alternative technologies for voice communications necessary to operate the grid
- Increase participation and collaboration between utilities, local, state and provincial governments
- Conduct further discussion between industry and government regarding restoration priorities and supply chain concerns
- Evaluate options to manage the grid reliability impacts of energy market disruption over an extended period
Additional Details on Recommendations
Executive Tabletop Recommendations Overview
The report said that industry should evaluate technologies and processes that could be used to increase the resilience of InterControl Center Communications Protocol telemetry exchange between control centers.
While ICCP systems are highly reliable and supported by layers of redundant infrastructure and cyber security protections, the severity of the Tabletop scenario prompted participants to consider that the current ICCP infrastructure may not be sufficiently resilient against certain single-point-of-failure or common-mode vulnerabilities. The electric industry should consider the potential impact of a complete loss of ICCP functionality and develop recommendations for alternatives that would provide comparable capabilities.
Industry should review these alternate technologies and determine if they could be applied across North America. Industry should evaluate options to maintain basic grid operation using minimal data and manual methods.
The report also said industry should evaluate opportunities to employ alternate technologies for operator voice (i.e., interpersonal) communications essential to operate the grid. While operator voice communications are highly reliable and have robust backup facilities, the Tabletop scenario exceeded this capability.
Industry should coordinate to clearly identify what specific aspects related to resilient voice communications may be improved and leverage past efforts such as those undertaken by the ESCC Resilient Communications Working Group. If necessary, industry should evaluate the need for alternate voice technologies with a focus on essential operator-to-operator communications.
NERC will increase the resilience of its Reliability Coordinator Hotline, and the E-ISAC will evaluate hosting a centralized satellite phone book for ESCC members.
The report also said that industry and government should continue discussing how to consider government priorities during a complex and prolonged power outage scenario as part of the electric industry’s established restoration procedures.
“RCs and electric utilities have well-practiced plans and resources in place to restore the grid and supply power to customers on a prioritized basis. However, a large-scale crisis that affects electricity and other critical infrastructure providers over extended periods will reveal new and conflicting priorities,” the report said.
“Industry should determine the need to develop an improved restoration framework that considers government requests that may conflict with pre-established restoration priorities and recommend guiding principles for coordinating with other critical infrastructures as needed. Industry should leverage its supply chain efforts and determine the need to improve processes to address equipment and supply shortfalls during a large-scale crisis and identify where government authorities can resolve supply chain issues.”
Industry should evaluate options to manage the grid reliability impacts of electricity market system or data unavailability over an extended period. Given the importance of electricity markets as an integral part of reliable grid operations, market operators and participants should review their market rules to ensure a common understanding of how generation dispatch and financial settlements would be administered through an extended period of market system or data unavailability. Industry should coordinate to develop best practices to manage long-term unavailability with a focus on maintaining reliable grid operations. FERC and the equivalent authorities in Canada should contribute to these reviews by considering how regulatory waivers and emergency tariffs applicable to the electric and natural gas industries may help support these efforts.
Distributed Play Recommendations Overview
With respect to recommendations from the distributed play element of the exercise, the report said that non-federal government partners and electric utilities should advance coordination efforts.
GridEx is an opportunity for government to collaborate with electric utilities, increase mutual understanding, and identify critical interdependencies.
While it is important for regional government partners to remain proactively engaged in emergency response with electric utilities, GridEx VII Distributed Play saw an overall reduction in the number of government entities that participated.
Municipal government entities, such as city, town, and county governments, as well as state energy offices would benefit from greater involvement in emergency response planning, training, and exercises, such as GridEx, the report said.
The report also said that communications and response in a hybrid work environment should be further refined. Organizations are still identifying best practices for hybrid communications since the onset of the COVID-19 pandemic. By providing an opportunity to exercise in-person and virtual response protocols, GridEx VII helped participating organizations identify challenges with hybrid response and interoperable communications with internal and external response partners.
The report also provided additional details on the following recommendations:
- Response planning should be augmented to ensure comprehension of technical information across functional teams and external response partners.
- GridEx should continue to evolve to provide additional support for planners from organizations of varying sizes and with different levels of experience.
- Cyber and technical components of the GridEx scenario should continue to be developed and expanded for future iterations of GridEx.
Further Collaboration Already in Progress
Further collaboration and coordination between the E-ISAC, industry and government is already in progress with action plans in development based on recommendations in the report.
Feedback received from participants will be incorporated into the planning process for GridEx VIII in 2025.
