The Christmas 2020 bombing in Nashville, Tennessee, that took out a major telecommunications center should prompt a reassessment of design considerations, network vulnerabilities and risk tolerance for utilities and other users.
This incident illustrated how a single point of failure can take out mission-critical applications developed for those we designate as first to respond and react to these disasters. It also took out internet, phone and wireless services across the provider landscape for varying durations spanning nearly a week.
These are all critical linkages that utilities depend on in some way. It was a wake-up call for those reliant on others to support delivery of their core services to customers.
For the utility industry, there is almost zero tolerance for any disruption in delivering its basic service — electricity. Now that we can step back and evaluate what happened in Nashville, it’s time to ask: What else should we now be addressing to mitigate risks and single points of failure in our networks?
Dissimilar Business Models
All telecom carriers follow roughly similar business models. They make significant capital investments in infrastructure assets, then optimize the system to serve as many subscribers as possible. This could be a router, switch, section of fiber-optic cable, dense wavelength division multiplexing (DWDM) system, generator and battery load, processing power, server use, or tower capacity. No matter the physical or logical asset, the intent is to maximize returns by optimizing utilization — minimizing overheads, contingencies and redundancies — to drive down costs per customer to the lowest point achievable.
All those accountable for optimizing for shareholder value — such as boards of directors with fiduciary responsibilities — would make the same or similar decisions.
The historical business model for most carriers has been to design large hub-and-spoke networks that ultimately run through large data centers. Like the facility in Nashville, these facilities control switching for large regions of the country. Given the amount of infrastructure that terminates or runs through these facilities, they carry geographical risks of failure from a single catastrophic event.
In contrast, utilities do not make return to shareholder in the same way and therefore do not view deployment strategies and risks in the same light. Utilities ultimately optimize for a different solution set. They operate under an obligation-to-serve compact that allows reasonable returns on investment, as long as the utility provides safe, reliable, cost-effective electric service to its customers.
Utilities have long planned power networks to achieve very high levels of reliability and resiliency. Optimizing around this problem set requires designing a high level of redundancy into all aspects of their infrastructure, including the high-voltage systems and the communications that control them. This allows systems to quickly recover from any disruptive event, sometimes in milliseconds and often without their customers knowing. Utility-grade communications systems are the critical infrastructure that links the system, making near-real-time decisions on the operations of the grid possible. Reliance on them for making real-time decisions is increasing rapidly.
Misaligned Interests
It is understandable that carriers’ systems are designed fundamentally differently from utilities and optimized to solve different problems. But it is becoming clear that these objectives are misaligned with the interests of utilities — especially for those utility applications that are deemed critical.
To reemphasize the point, utilities must align infrastructure performance requirements with their communication systems. Single points of failure and network traffic leaving a service territory should now be under a spotlight. Imagine a scenario in which a utility customer experiences an outage or slow service restoration because of an issue with a third-party provider system a few states over and not even in the utility’s service territory. That is the unfortunate reality. It illustrates what we mean when we say they have optimized for a different problem. That piece of telecom infrastructure had capacity and that is why it was selected for this critical application.
For utilities, the question becomes: Why should we outsource or trust a third party with communications when it has become so critical to our core business? If we earn money based on providing extremely reliable service, shouldn’t our core communications network be equally resilient, reliable and redundant?
Utility-Grade Vs. Carrier-Grade
Whether for utilities or common carriers, communications networks often are built with identical equipment, sharing many of the same protocols and technical standards for reaching networks. Yet, they aren’t designed for the same purpose or built the same way.
Though there are a host of technical reasons, the differences between utility-grade and carrier-grade systems boil down to reliability and resiliency. How much redundancy is designed into the network and its major components? How vulnerable is the system to a single point of failure? Does the failure impact the delivery of electricity or the safety of the community? It has to be a risk-based approach.
Utilities have traditionally relied on the common carriers for at least portions of their communications functionality. However, as utilities increasingly design systems to accommodate many new demands of our changing power system, communications are regularly considered part of utilities’ critical infrastructure, much like substations, power lines, transformers and switchgear.
Utility-grade communications systems are built as diversified networks with the intent of creating failover capacity to route communications traffic in the event one segment of a network goes down. And that is just the start. They are built with multiple diversified power sources and backup, diversified fiber and microwave paths, redundant building entrances, failover sites for network operations and controls, environmentally hardened devices, and so on. These are purposely designed and operated network systems, solving for one thing: keeping the lights on.
Achieving Control
Control is the key enabler needed by utilities. A key principle of effective control is diversification of risk during planning, giving the utility the ability to influence desirable outcomes during a crisis. In today’s environment, even major carriers are questioning the prudence of relying on major data centers to serve as hubs of national and international telecommunications.
If outsourcing communication systems is a must, utilities should look for a partner that speaks the same language. All work must be done through honest collaboration with all interests closely aligned. When a partner makes promises, utilities must be ready to ask the hard questions and dig into the details. Near the top of the list should be questions on whether the communications provider shares the utility’s priority of keeping traffic local and redundant.
Defining the meaning of reliability and resiliency is important, as is rewarding the outcomes. Too often the stick is used before the carrot. The partnership should be structured so that all parties own the elements they are good at and do not take on areas outside of core competencies. How do all parties view utility control? If the answer isn’t the same, then perhaps the partnership isn’t right.
Utilities do not tolerate outages. They work tirelessly to ensure they are avoided and minimized. They align their investment to their tolerance of risk. They need partners that think and invest in the same manner. Providing safe and reliable power is fundamentally different than providing reliable retail and wholesale consumer and business communications service. While it may be annoying to be without cellular or internet service for a few hours, it can be life-threatening to be without power for that same period.
LTE communications infrastructure can greatly improve utility control and monitoring.