As we all continue to cope with the COVID-19 pandemic, we are witnessing the increasing infrastructure challenges (and potential vulnerabilities) facing companies and remote workers. Even without these recent seismic shifts in the workforce, certain risks have always been important to consider as a part of your IT strategy.
What’s Your Power Outage Business Continuity Plan?
Natural disasters, nefarious acts and equipment failures can all be contributors to power failures. The potential for more extreme weather during the summer months, coupled with the stress of the current pandemic, has us asking these questions:
- Is there a documented power outage business continuity plan?
- What would be the impact of a day without power?
- How, and to whom will you communicate the power outage?
- What are the second and third order issues including communications for employees and customers?
How Should Businesses Plan for Power Outages?
Of course, power outages are just one part of a comprehensive Disaster Recovery (DR) or Business Continuity Plan. However, the frequency of power outages is now all too commonplace. A 2018 survey by Frost and Sullivan in conjunction with S&C Electric found that nearly half of the companies surveyed experienced a power outage at least once per year, and nearly a quarter of the respondents experienced outages at least once per month.
These respondents were the key decision-makers who select electricity commodity suppliers across the manufacturing, data center, healthcare, small franchise, and education sectors. They were screened and selected according to their relevance and authority to make power-related decisions. Their 251 companies represent a balance in size, with their monthly energy consumption ranging from 10 MWh (megawatt hours) to more than 50 MWh. As a frame of reference, 10 MWh is enough energy to power the average American home for one year.
The Impact of a Power Outage to Businesses
According to Uptime’s 8th annual Data Center Survey, 31% of respondents experienced downtime in 2018, an increase of 25% over the previous year. Nearly half the respondents reported downtime in the past three years, the number one cause being power outages. Monetarily speaking, some studies show the cost of an outage in a U.S. data center can be as high as $8,851 per minute.
This begs the following question: Do you know your actual costs per hour of IT Data Center outages/downtime? Of course, this varies by industry and the amount of mission-critical (and especially real-time) applications.
Companies who have real-time revenue-generating activities such as financial and commodity trading, manufacturing, retail, and education, can range to thousands of dollars per hour and reach millions of dollars very quickly. Although not the result of a power failure, the stark reality of the COVID-19 shutdown across the country in terms of lost-revenues (and lost jobs) certainly qualifies as a “black-swan event.”
There are many large data centers that store and manage data for a wide range of companies. When they go down, a whole suite of businesses that rely on them for their mission-critical data storage is impacted. Today’s data centers have a few technical solutions in place to reduce the impact of power outages, such as data center mirroring or cloud computing. However, the frequency of outages continues to increase.
Real-Life Examples of the Impact of Major Power Outages
Let’s look at a couple of real-life examples of the impact loss of power had on businesses.
A 2018 lightning storm in Texas interrupted grid power to Microsoft’s San Antonio Azure data center. Though lightning storms are common, this storm was part of a slow-moving system that broke rainfall records by more than 7 inches. The lightning strike caused a voltage swell which knocked cooling systems offline, damaged equipment, and brought down Active Directory and Visual Studio Team Services for almost a full 24 hours.
In 1996, there were two Western North America blackouts six weeks apart (July 2 and August 10), affecting millions across Western Canada, the Western US, and Northwest Mexico. Both incidents are thought to be the result of faulty power lines. Though the power outage did affect millions of people, its impact is considered to be minimal.
Disaster Recovery Best Practices for Widespread Power Outage
Infrastructure redundancy (backup for power and cooling systems, data, and the entire data center) does not address a site-wide power outage. Even the largest public cloud providers such as Amazon and Microsoft have experienced well-publicized outages affecting thousands of clients. The best protection is to replicate to a fully redundant, mirrored system at a data center on a different power grid. For example, companies in the 13 states on the Western Power Grid could look to have their replication site in Texas or Minnesota.
Regardless of the specific approach, Sanity Solutions is happy to assist our clients with Disaster Recovery or Business Continuity assessments and provide a checklist with common best practices.
As an example, here are four broad areas to consider:
- Planning for the impact of an unexpected or catastrophic event on your business.
- Assessing your data and technology needs in the event of a failure in operations.
- Communicating your plan to your employees and vendors
- Coordinating with external organizations and helping your community.
Protect your business’s most valuable asset – your data. Contact Sanity Solutions to discuss your options today.
