What Gets Measured, Gets Done
Originally published March-April 2012
In the September 2011 Public Power article "Safety—An All or Nothing Proposition," we discussed the merits of creating a culture of safety by actively pursuing a safety best practices program. The central thesis was that adopting and applying best practices across a utility can lead to a culture of safety and thereby reduce incidents.
In this article, we will provide further data analysis from the American Public Power Association's Reliable Public Power Provider (RP3) program to show how a long-term commitment to adopting and applying safety best practices can improve overall safety at your utility. To that end, we strive to go beyond setting out safety checklists to providing direct analysis of impacts associated with adoption and use of safety best practices.
The RP3 program scores utilities for their adherence to best practices in four areas of utility operations—reliability, safety, work force development, and system improvement. Utilities seeking RP3 recognition submit an application that covers each area of utility operations and are awarded points based on the merit of their responses. If a utility scores 80 out of 100 possible points, it earns recognition as a Reliable Public Power Provider. To maintain the RP3 designation, utilities must reapply every two years.
The data used to test our observations were collected from two cycles of utility applicants, RP3 Cycle 5 (2009) and RP3 Cycle 7 (2011). These two cycles were the most recent with utilities re-applying for RP3 designation. Many of the utilities that applied in Cycle 7 were the same utilities that applied in Cycle 5.
In our observations of electric utility safety programs, it is clear that utility employees are safer when managers and supervisors focus on safety. This means going beyond simply complying with safety regulations and focusing on creating a culture of safety. In the field of public administration, there is an axiom that guides this: "what gets measured, gets done." We believe that by participating in best practices programs and striving to achieve best practices over time, utilities programmatically improve their safety records.
Sixty-seven of the utilities that applied for RP3 recognition in Cycle 5 reapplied in Cycle 7 and provided an OSHA (Occupational Safety and Health Administration) incidence rate. These 67 utilities provide a good data set to test our general observations.
The data from both cycles were compared to help determine whether the RP3 program's focus on best practices was helping utilities create a culture of safety and, by virtue of measurement, making utilities pay more attention to safety.
The analysis showed a lower average incidence rate in Cycle 7. In total, the Cycle 7 applicants had 52 fewer incidents than the Cycle 5 applicants. In addition, the average and median incidence rate for the Cycle 7 group was lower than for Cycle 5.
This means that, on average, an RP3 utility reduced its incidence rate by 27 percent from Cycle 5 to Cycle 7. The use of averages limits what we can say about how the improvement programs help, so it is important to examine the data closer to determine if the difference is due to random chance or some other factor. The observed result could mean the average utility in 2011 had a better ‘safety' year than in 2009. On the other hand, it could mean that the escalated emphasis on safety in order to meet the RP3 program goals is having a positive impact on these utilities' organizational commitment to safety. To test this assumption, we dug a little deeper.
First we plotted the incidence rates of the applicant utilities for the utilities applying in both Cycle 5 and 7. As shown in Figure 1, there is a clear general pattern to the data. The incidence rates from Cycle 7 tend to be lower and more closely grouped than the incidence rates for the same utility group in Cycle 5. This can be expected from data shown in Table 1, but it puts it in perspective to see the data by utility.
In Figure 1, the red triangles representing the incidence rates for the utility in Cycle 5 are typically higher and more spread out as compared to the Cycle 7 group. As utilities embrace best practices, we would expect to see the incidence rates fall. The lower number of incidents for each utility in Cycle 7, represented on the graph by the blue diamonds, confirm our expectations.
The second measurement of overall difference was made by putting each cycle in rank order from lowest to highest incidence rate and lining up the two cycles to see any observable differences. As can be seen in Figure 2, the Cycle 7 incidence rates are generally lower. Keep in mind that Figure 2 does not line up on a utility-by-utility basis, so it does not reflect where individual utilities got better or worse. When the incidence rates are ordered for each year without regard to the utility, the difference between the groups becomes clearer.
This is compelling evidence for programmatic safety improvement, but to further test our assumptions, we sought to determine if the two cycles of RP3 utilities experienced different incidence rates due to random chance or some other factor. To examine this, we conducted a T-test.
The T-test showed that there is a 3 percent chance that the differences between the two groups are due to random chance alone. This meets the measure for consideration as significant by statisticians. Put another way, there is 97 percent likelihood that the differences in average incidence rates between the two years of utility applicants is not due to random chance. This gives us confidence that following best practices in safety leads to measurable improvement over time.
Though we cannot say with certainty that RP3 participation in the long term causes all of the observed improvement, we believe at least some is attributable to the organizational commitment to safety best practices set out in the RP3 program.
As can be seen in Figure 3, 54 percent of utility applicants reapplying to the RP3 program had lower incidence rates in cycle 7, compared to two years earlier. Interestingly, the average RP3 score on the whole application for each cycle remained similar, 91.2 for Cycle 5 versus 91.75 for Cycle 7. We believe this shows an overall trend of improving utility safety performance in real terms.
RP3 is another way to help ensure that best practices are adopted, and it is important to note best practices make a difference only when everyone concerned takes them seriously. Getting involved and staying involved is a pathway to success. Many utilities have found RP3 designation to be valuable for public relations, operational self-check, and as an indicator of their choice to pursue best practice operations. We encourage all utilities to find a way for everyone at the utility to be involved in measuring safety because "what gets measured gets done."
Please Sign in to rate this.
Members of the American Public Power Association receive Public Power magazine as part of their annual dues payments. The subscription rate for non-members without the annual directory is $100 per year in the United States and $130 per year outside of the United States. A subscription that includes the annual directory is $200. The annual directory alone can be purchased for $150.
Public Power is published eight times a year by the American Public Power Association. Opinions expressed in single articles are not necessarily policies of the association.
APPA works with IPA Publishing Services of Lancaster, Pa., which sells reprints of Public Power magazine articles. It has come to APPA's attention that from time to time other companies approach APPA members to sell plaques and other merchandise to memorialize APPA awards or articles in APPA publications. These other vendors are not affiliated with, or endorsed by, APPA.
David L. Blaylock
Senior Vice President, Publishing
Jeanne Wickline LaBella
Robert Thomas III
- House passes pipeline review, electric transmission bills
- Report catalogues state net metering, DG actions in the second quarter
- Lawmakers hear about capacity market flaws, rising grid costs
- Hamilton Utilities’ urban forestry program boosts safety, reliability
- Kansas City BPU exceeds 45 percent renewable energy threshold
- Officials urge public power utilities to be prepared for cyberattacks
- Public power utilities recognized for high customer satisfaction
- SMUD board approves new Time-of-Day standard residential rates
- Lawmakers hear about capacity market flaws, rising grid costs
- Report sees more than seven million plug-in EVs in U.S. by 2025