Bad actors are out there, probing utility systems thousands of times each day. Yet, to counter these threats, many public power providers have too few employees to make cybersecurity anyone’s full-time job. On the upside, utility staff who are juggling cybersecurity with multiple demands can get involved in the many industry groups and government programs designed to help them address cybersecurity efficiently and tackle their guardian duties. Public power utilities may have operational technology (OT) – which gives the ability to automate or operate our grids remotely and with greater situational awareness – in addition to their information technology (IT) – which gives the ability conduct business with our customers and others through platforms such as websites, billing systems, and databases. Partnerships help utilities prioritize the vital tasks involved in keeping critical systems and assets protected.
Bridging the Gaps
Stowe Electric in Vermont is about to get a new IT person, and the people managing cybersecurity to bridge the gap between the new IT person and the previous one are looking forward to having an expert on staff. Michael Lazorchak is the utility’s manager of regulatory affairs, and Trish Waugh is the business and customer manager. They’ve been devoting about a third of their time to cybersecurity, and they’ve linked up with several organizations to help them get up to speed.
Those organizations include the American Public Power Association, the Electricity Information Sharing and Analysis Center, or E-ISAC, and working groups with the Department of Public Service for Vermont. “One of the reasons we’ve tried to be more active in the APPA working groups is because they help with information sharing,” said Lazorchak. “I have no background in cybersecurity, but I can ask the group what we need to focus on. These groups are very helpful.” Stowe also gets security alerts from multiple agencies, and the alerts come with varied frequency: weekly, monthly, and even daily when appropriate.
Dustin Moore, operational technology superintendent at Riviera Utilities in Foley, Alabama, also values the information he gets from cybersecurity alliances. He uses that information as a tool to be ready for anything. “We want to stay ahead of what’s going on in the industry,” he said.
Partly, this is because his utility offers a variety of commodities: gas, water, wastewater, internet, and cable TV. If, in the future, regulators decide that any device touching any operational technology network has to comply with standards from the North American Electric Reliability Corporation, or NERC, Moore’s utility would be ready. “It’s better to be proactive than reactive, so when we set out any of our equipment, we set it out meeting today’s requirements. If some regulation comes down on the gas or water side, we’ll be able to say we’ve already checked the box,” Moore said.
Riviera Utilities gets support from E-ISAC, the Multi-State Information Sharing and Analysis Center, or MS-ISAC, the APPA’s Cybersecurity Defense Community, and a working group with the Alabama Municipal Electric Authority, a wholesale power provider for Riviera Utilities and 10 other cities across the state. Moore values the timely alerts such partners provide. “If they come out and say there’s a security patch vulnerability on this type of substation equipment, we can go ahead and upgrade our firmware. Alternatively, if we’re getting different hits on our network, we can push that information through our partnerships and inform others,” he said.
Dangerous Discoveries
Stowe in Vermont has its systems monitored by MS-ISAC, as well, and Lazorchak called the results eye-opening. “Our web domains are attacked on a regular basis,” Waugh noted. “Attackers are trying to access or replace PDF files on our website so that when people click on them, they’ll get a virus. There are constant phishing attacks, mostly for the purpose of taking our information and holding it for ransom.”
The number of attempts a utility routinely sees is surprising, even for a smaller utility like Stowe, which has some 4,400 residential and commercial customers. For instance, the report Waugh reviewed for the week ending July 10 had more than 303,000 hits to the website on it, and a couple dozen of them were malicious attacks. Those are the attacks the utility sends on to its outsourced IT management firm so that it can block malicious IP addresses or take other appropriate action.
The rest, Waugh said, were probes. “A lot of these attempts are scouting, looking for vulnerabilities. They’re not trying to attack anything at the moment,” she said. For those that do pose a threat, Waugh is able to submit the information to her IT management vendor. The things she submits include IP addresses, domain names, the country of origin for the attack, and the IP addresses for the utility domains or devices that have been affected.
Moore also uses some of the alerts he has received for training purposes. “We train our users based on the stuff other utilities are seeing because an adversary’s biggest asset is an untrained user,” he said. “We tell people, ‘This isn’t just a training [scenario]. It’s an actual attack that has happened.’”
Shared Needs
Another way smaller utilities can gain cybersecurity knowledge was trialed through a regional municipal energy agency that offered 24 cities the opportunity to take part in a Shared Cyber Analyst program. To participate, each city had to complete APPA’s Municipal Cyber Academy, which covers practices detailed in the public power cybersecurity scorecard. Then, based on each city’s needs, a dedicated cybersecurity analyst could split time across the participants to focus on ransomware readiness, scorecard improvements, and incident response.
Twenty-three of the 24 cities that could participate did. Participation meant meeting with an analyst individually at least twice during the yearlong trial to focus on their specific needs, attending a monthly cybersecurity working group for information sharing, honing their incident reporting skills, learning about planning and activities to recover from a ransomware attack, and receiving end-user training to help protect against phishing threats and more.
“Cities mentioned that they did not know where to start,” noted a report on the program, which was funded through a cooperative agreement between APPA and the Department of Energy. “Providing a few recommendations at a time gives them a good place to begin and allows them to make positive changes without becoming overwhelmed by a complete list of items to accomplish.”
After all, once a city gets one potential vulnerability scratched off the list, other new threats can appear.
Valuable Learning
Both Moore and the Stowe team have learned plenty from alerts and interactions with industry groups, for both cyber and physical security concerns. “After we learned about a California utility where someone was taking shots at a substation, we reached out to our town police and put more cameras up around town. We learn about things to be aware of so we can design plans for mitigation,” Lazorchak said.
Moore noted that his various connections alerted him to the reality that some criminals have been targeting older exchange servers. “Microsoft is one of the best tools an adversary can use,” he said. “Breaking into a router or other device is too cumbersome and takes too long, but an adversary can look at the latest patch provided from Microsoft. If it’s on an exchange, a server or even an administrative tool, the hacker can reach out and see if anybody hasn’t patched against it.”
Waugh values the connection with experts at other utilities, people she has met and bonded with via industry groups. “It’s great to have utility contacts. If something comes up, you have somebody you can call and talk to,” she said.
Beyond colleagues, monitoring and alerts, there are other resources, too. Moore said his utility leans heavily for guidance on the Public Power Cyber Incident Response Playbook, which provides protocols for things such as asset identification, patching, firewall rules, secure system configuration, data recovery, incident response, application security, and more. “A lot of our utility success has come through the policies and procedures that APPA built as kind of a template,” he said.
Step by Step
Ed Krieger, power system director for Piqua, Ohio, has been shoring up his utility’s systems with help from his joint action agency, American Municipal Power, which provides electricity to 132 communities in nine states. To keep power flowing to Piqua’s 11,000 customers, Krieger has used two assessment tools in the past couple of years: the Public Power Cybersecurity Scorecard, which is based on the Department of Energy’s cybersecurity capability maturity model, or C2M2, and CIS Controls, which is a prioritized set of security actions from the Center for Internet Security.
Among the topics covered by the public power cybersecurity scorecard are risk management, incident response, operational resilience, monitoring of cyber systems, supply chain, workforce management, training and more. Experts from AMP led the Piqua team through the assessment.
“It was a very lengthy process to evaluate our posture and existing condition in all 14 areas,” Krieger said. To work on this project, he brought in people from his IT department, operations, two technology specialists, and representatives from the water and wastewater utilities. These individuals were interviewed for the assessment and had to pull together information for the assessors from AMP to review. Krieger estimates the process took as much as 200 manpower hours to complete.
After the information was gathered, Krieger noted that the AMP team reported on where his utility stood and offered recommendations for improvement. “They went down through the list and gave us a percentage grade,” he said.
Only two areas showed up as real problems. His utility earned a zero on information sharing and communications, which includes talking to both employees and the public about the incident. “We really hadn’t thought a lot about that,” Krieger said.
The other area where his utility needed to spend some time was in “external dependencies,” where the town earned a grade of 50%. “We needed to tighten up how we interacted with the supply chain and how we could be impacted by disruptions with one of our larger suppliers,” said Krieger. “What happens if the supplier is offline, like the pipeline that got shut down in a ransomware attack? Do you have a backup?” Piqua does now.
“We’re still working toward the finish line on some things, but we’re a lot better off than we were when we identified where some of the holes were,” Krieger said. Finding those holes was easier for his utility with help from other organizations, including APPA, AMP, and CIS.
“If you’re only looking at ideas from within your organization, you may be missing out,” Moore said. “Sometimes it takes an idea outside of your organization to see a procedure, operation, or plan that can help you be more successful.”
