Roughly two months before Hurricane Harvey pummeled Houston, inundating hundreds of thousands of homes and leaving 300,000 Texans in darkness, a group of 200-plus participants from the federal government and the electricity and telecommunications industries convened at the Department of Energy to work through a hypothetical tabletop exercise: How would industry and government respond to a major hurricane strike on Houston?
The exercise was the fifth in an annual series, called Clear Path, that focuses on disaster preparedness and response across all levels of the energy sector. “The people in the room during that tabletop exercise included officials from the local government of Houston, gas industry, telecommunication industry, and electric utility industry,” said Joy Ditto, president and CEO of the Utilities Technology Council, a trade association serving utilities and critical infrastructure providers. “When Hurricane Harvey hit, it absolutely helped to have that additional knowledge of going through that exercise together, of knowing each other and establishing priorities in advance.” For Houston, exercises like this and lessons from past hurricanes, including Ike in 2008, spurred investments into improving grid operations. So even when Hurricane Harvey dropped more than 4 feet of water over the city in two days, electricity kept flowing to more than 90 percent of its residents.
Together as an industry, public power, rural cooperatives, and investor-owned utilities have a good track record of collaborative mutual aid, research and development, and education programs. And as the grid has gotten more complex, threats have evolved, and storms have intensified, the need for a united effort across the sector and with federal agencies has become more critical.
“When you look at the North American electric grid, it is one big machine with thousands of owners and operators,” said Scott Aaronson, executive director of security and business continuity at the Edison Electric Institute, a trade association representing investor-owned utilities. “We’re critical to the life, health, and safety of Americans, and to our national and economic security.” When electric power does not work, neither does almost anything else. Most financial, telecommunications, transportation, and water networks depend on electric power at some point.
“It’s hard to say that something as devastating as superstorm Sandy was beneficial,” said Aaronson. “But it really crystallized, for both the government and the industry, the value of a group at the CEO and senior government official level that can do blue-sky planning but can also be a center of gravity to respond when incidents happen.” What emerged was the Electricity Subsector Coordinating Council, a collaboration between the electric utility industry and the federal government at the highest levels, for grid resiliency and grid security. CEOs from 30 electric utilities — five public power, five cooperative, and 20 investor-owned — participate in the ESCC, which is proportioned based on the number of customers served by these utilities in the United States. Aaronson serves as the ESCC secretary.
Disaster response and coordination is a key mandate for the ESCC, and 2017 — with its string of devastating hurricanes — has served as a test of its value. “During both Harvey and Irma, there were daily calls during the critical parts of those storms and their recovery,” explained Kevin Wailes, CEO of Lincoln Electric System in Nebraska and a member of the ESCC leadership. “Secretary [of Energy] Perry, his chief of staff, and Department of Homeland Security representatives were on the bulk of those calls from beginning to end, asking, ‘What do you all need that you don’t have?’”
Access to the highest levels of government meant ESCC could help entities on the ground swiftly cut through the red tape. For instance, during Harvey, when Houston’s CenterPoint Energy wanted to check on a submerged substation without endangering utility personnel, ESCC arranged for a waiver from the Federal Aviation Administration so CenterPoint could deploy a drone and then worked to have temporary flight restrictions on airspace lifted within a few hours so assessments could be done and power restored expeditiously.
Even long-standing mutual assistance programs worked more efficiently, with the ESCC helping the industry to coordinate the 60,000 additional mutual aid and contract crews that poured into Florida following Hurricane Irma. “When you have, on a Monday night, 7.8 million electric utility customers knocked out, and the following Saturday you have 800,000 more, and in a period of five days you’re able to restore power to 7 million of those customers, it demonstrates that it could not have happened without collaboration, not only within the industry but also with the government,” said Wailes.
“In extreme events, public power CEOs connect with the highest level of the federal government,” said Nathan Mitchell, senior director of electric reliability standards and security at the American Public Power Association. “This happens because we keep an open dialogue with senior government leaders on how to prioritize power restoration during major events, and how we can efficiently respond when the next disaster strikes.”
“We weren’t the ones restringing lines, repairing poles, or restoring power directly, of course, but what we were able to do was provide a coordinating function across all segments of the industry,” said Aaronson.
Preparing for cyberattacks
In recent years, as cybersecurity has emerged as a critical risk to the quality and reliability of the power infrastructure, ESCC’s focus on resilience through mutual assistance has come to extend beyond natural disasters to include manmade incidents. In November 2015, a month before Ukraine’s distribution system suffered a cyberattack, the North American Electric Reliability Corporation conducted GridEx III, an extensive cyber and physical security exercise involving more than 4,400 participants from 364 industry and government organizations in the U.S., Canada, and Mexico. It culminated with a discussion between senior leaders from utilities and the U.S. government. They were asking, “Do we have the capacity to respond to a massive cyber incident? Do we have the surge capacity? Do we have the human resources and capability to ramp up in the event of a cyberattack?” recalled Aaronson.
From that, the Cyber Mutual Assistance Program formed to help electric companies restore systems in the event of a regional or national cyber incident. “In a very short span of time, we have developed a playbook for it, and we have grown to 127 companies participating in CMAP — companies that cover more than 80 percent of customers in North America,” said Aaronson. “That gives you a sense of the urgency that the ESCC can create and the breadth and scope of programs it can develop.”
The list of ESCC programs covers the spectrum, from deploying proprietary government tools and technology for the safety of the grid to getting security clearances for the industry to improve the flow of information on potential threats, and developing equipment-sharing programs and incident response exercises.
Perhaps the greatest value of this coordination between the industry and the government is that “it’s really a place to look over the horizon at some of the threats facing the industry and then develop plans and processes, and deploy technology, at times classified, to make sure we are as secure as possible,” said Aaronson.
“Through participation in the ESCC, public power utilities have a stronger voice on grid resiliency,“ said Mitchell.
Consider the example of high-impact, low-frequency threats, such as an electromagnetic pulse attack. “The Electric Power Research Institute is doing a three-year study on EMP,” explained Wailes. “And through the ESCC, they are getting data cooperation from the Department of Defense.”
Another program, officially called Supplemental Operating Strategies but nicknamed “the MacGyver Project,” explores whether the electric grid can be operated under suboptimal circumstances in case of an incident. Or as Aaronson described it, “How do we hold the grid together with bubblegum and duct tape?” Within this program, the ESCC is working with grid experts to explore whether resorting to manual operations, engaging in planned separations, leveraging secondary and tertiary backup systems, or operating in other degraded states are response measures that can be planned for and practiced in advance of an incident.
One more area is supply chain security, which is the security of all critical hardware and software that enables utility operations. “The critical infrastructure you deploy is not manufactured by your electric utility; it’s a product you buy,” explained Ditto. “We’ve seen example after example over the last 10 years of very well-known manufacturers who have had security breaches in their products.” The ESCC, in response, invited security and technology vendors to participate in the group along with industry and government stakeholders to identify and address threats to the supply chain.
While it is still the electric utilities, the security firms, and the NERC standards that work on the ground at a granular level, a collaborative effort like the ESCC is what keeps the wheels moving, even if there are roadblocks. “Think of the ESCC as an umbrella organization that works across all manners of threats and all segments of the industry, including associations like the American Public Power Association, who in turn work with their members,” explained Aaronson. “It doesn’t matter why the lights are out — physical, cyber, act of war, or act of God — we want our entire grid to be as resilient as possible if the unexpected, or maybe even the expected, happens.”
Prepare for and Respond to Cyber Threats
The Cyber Mutual Assistance Program is a pool of professionals dedicated to keeping the grid secure. Learn more about the program.