A survey of Belgian, quoted companies, commissioned by LCL, shows that only 3% of the targeted companies ever test their power backup systems by actually turning off the electricity. Meaning that they will only learn whether or not the power backup systems work when there is a power cut. That’s like buying skis and not trying them on before you actually hit the snow. Or going hiking with brand new boots, straight from your favourite online shop. The only guaranteed result is sore feet.
We’ve all read and heard what deficient power backup systems can lead to. Remember the power cut at Eurocontrol? The business world couldn’t believe the company shut down just like that, by lack of well functioning backup systems.
We knew that many companies are only theoretically prepared for the worst-case power scenarios. But we never expected it to be that many. 97% of the companies plug their power backup and pray; that’s like: as good as everybody. In France, they expect to have an electricity shortage of 5 GW next week. Knowing that we generally import electricity from France, next week could represent a live test for the companies concerned…
Another astonishing fact is that 53% of the surveyed companies doesn’t have a second data center. Meaning, that in case of any disaster, not just a power cut, they have a big problem. More over: only a minority of companies interviewed said they were planning to set up a second data center.
This shows that data security is not seen as essential within IT governance, not even with quoted companies. How many Board members are aware that data security is taken so lightly in their company? More and more, ICT is on the Board’s agenda, and rightly so. All we need to do now, is educate Board members so that they can evaluate the security systems in their company/ies, and make sure that they really are as safe as they should be.
Today, if clouds don’t spoil it for us, we’ll get to see a solar eclipse. On a sunny day, an eclipse translates into 40% less electricity from solar panels. Given the shaky state of the electricity supply in Belgium, I do hope you have a backup server room or data center, just in case we should get the long feared electricity blackout after all.
Earlier this week, well before the eclipse, we experienced two power cuts in our data centers, shortly after one another: one in Aalst yesterday, and one at the beginning of the week in Diegem. We have ample backup systems, of course. Our clients never noticed there was a (short) blackout. When data center activities are not your core business however, you can never protect your data and assets in the same way a professional external data center can. So a power cut represents a threat.
Even if you do have a backup solution, a second data center, at a distance of, say, 5 km: how relax can you be about potential disasters such as a power cut? I would keep my mobile close by if I were you. If your server rooms/data centers are only 5 km apart, both are very likely to suffer from any power cut that occurs on the grid. Even if you have the right backup facilities in both your data centers, your staff will have to be in both places at the same time to monitor. A double power cut represents an extra risk, there’s no doubt about that. In practice, your data centers need to be at least 20 km apart to avoid any unnecessary extra risk.
In short: a lot of companies are not as safe as they think they are. A condition for growth is that you can focus on your core business. And that your backend is no source of worry…
The government’s plan to shut down parts of the electricity supply temporarily during harsh wintry conditions is likely to have a major impact on businesses – just try working in industrial premises with no lighting. Employees will have to find new ways to maintain internal and external communications, and I will not even mention the impact on the main day-to-day operations of the business or what should be done if all the business processes grind to a halt. As a data center business, LCL is a specialist in redundancy with backup facilities for every part of our offer. We also perform a power-outage test once a month, Since we are keen to share our extensive experience, this blog contains a number of tips and tricks that you can use before, during and after a power outage.
The digital economy
It is claimed that businesses will not encounter too many problems as a result of the announced blackouts. Blackouts are expected to take place in the early evening, at around 5 p.m., when most businesses conclude their day-to-day activities. However, a power outage can have major consequences for business-critical information even if every precautionary measure has been taken, and this fact seems to have been ignored. Servers that run in the background to support business processes may shut down. The impact of this cannot be underestimated. Moreover, the implications of power outages for data centers and lines of communication is quite easy to overlook. Data centers and server rooms are often located in places that are not seen by the rest of the business. Besides planned power outages, unexpected outages may also occur. If consumption is too high for the network to cope with, power may be lost instantly.
Can this be anticipated? Existing legislation on critical infrastructures only covers energy, transport, the financial sector and digital communication. According to the law, IT is not a critical infrastructure. However, nothing could be further from the truth. Information technology is found everywhere, such as the simple PLC used in a coffee maker, IT used in locks in canals, and factory production lines that are run by desktop and mainframe computers. Lawmakers have not taken today’s digital economy into account at all, as the past year’s cyber attacks show.
Given this, I recommend that businesses with a server room or a data center make proper preparations. They can do this by performing a number of tests based on the following questions. Does the emergency power supply work if the network power supply is interrupted? What plan of action should be followed in the event of a power outage and what steps need to be taken when the power supply comes back on?
As great deal depends on design, does the business have a few servers in one server room or a comprehensive data center with ten racks or more.
Tips for the server room If your business stores its data in a server room, the best thing you can do first is to check whether the battery capacity is sufficient to tide the server room over during a power outage. You can perform a short discharge test to measure the batteries’ actual capacity. You also need to check that only business-critical equipment is connected to the backup system. It has been stated that the blackouts may last for up to three hours. For server rooms with batteries but no emergency power generation, this means that the servers will shut down. The autonomy of most backup systems is ten minutes. This is sufficient to cope with a short power outage, but not enough to cover one lasting three hours. Moreover, the cooling system will also shut down unless it is connected to the emergency power supply. This can lead to the servers’ processors becoming overheated, causing defects and a loss of data.
Businesses need to make sure that their cooling system is connected to their generator if they have one. The cooling system will briefly shut down if there is a power outage, but it will start-up again once the generator starts producing power.
As many data centers have grown organically, it is vital that actual power consumption is measured. Frequently, consumption exceeds UPS capacity due to the connection of non-critical equipment to the UPS. If the UPS is overloaded, it is very likely to malfunction and break down.
The same is true of the emergency power generator. Is it able to cope with the full load of the servers and the cooling system?
Check how old the batteries are and perform a discharge test.
Ensure you have a generator and enough fuel to cover three days’ use. Prepare an emergency plan for longer power outages.
Turn off all servers that will not be used.
Some UPSs allow your servers to be ‘shut down gracefully’. Use this option if it is available in your UPS.
Check whether all equipment is connected to the emergency power supply. The switchover from the normal power supply to the emergency supply must also go without a hitch.
Ensure that all installations are serviced before the winter starts.
Tips for the data center A data center has more racks containing business data than a server room. Usually, the facilities are better than those found in a simple server room. Nevertheless, it is still important to test batteries, compare capacity consumption to the power provided by the installed batteries, and ensure there is enough fuel for the emergency power generator.
That said, the complexity of larger data centers makes for a longer checklist:
Is the access control system connected to the emergency power generator?
Are the servers connected to two independent power sources?
Can the cooling system handle the switch from the normal power supply to the emergency supply provided by the generator?
Are alarms registered properly?
Are all the alarms properly connected?
Have the A and B UPSs been tested, and does the fallback UPS work properly?
I recommend simulating a power outage. Check the installation and turn off the main switch to see what happens. Many businesses are afraid to perform this exercise as they have never tested the installation or they are unable to assess the implications. If so, this is precisely the right time to perform the test.
Always opt for winter diesel when ordering fuel for the generator. Summer diesel is delivered as standard during the summer, but it freezes during cold winters, and power outages are most likely to occur when the temperature outside is very low.Connect everything to the redundant supply. Use source transfer switches, etc. for non-redundant equipment.
Keep the wiring diagram of your electrical installation close to hand in case you have any problems. A data center is a living installation, and the more documentation you have, the better.
Check which applications are considered business-critical in a power outage and prepare a plan so that these applications can be started up again first.
As a preventative measure, replace old servers if possible before the plan to shut down parts of the electricity supply temporarily comes into effect (i.e. now!). Years of use may have resulted in wear and tear to the power supply and it is likely that the servers will not start-up again.
Fluctuations in the electricity network may come from either direction, so think about overload protection.
In the event of a power outage, disconnect the fuses from the servers and isolate the servers from the network.
What to do after a power outage
The UPS batteries must be charged before power can be delivered to the servers. The main fuse may blow if the servers are started up while the batteries are still charging.
Do not allow the servers to start-up automatically when the power supply returns. Use the BIOS settings to configure the servers so that they do not start-up automatically when the power comes back on. Try to have the servers start-up sequentially. The startup current may also trip the fuse. The network is often unstable following a power outage and fluctuations in the network may lead to the server shutting down again and becoming defective. The server and storage infrastructure can only be safely started up again if there has been a stable power supply for 15 minutes.
Prepare a plan for starting up the infrastructure again. Document everything as well as possible. Identify which servers can be started up again first and check them for defects.
Consult the instructions for your installations and the suppliers.
Prepare a good backup plan of the configuration of your switches and routers. This can prove invaluable if the data stored on your equipment is lost.
Based on experience… Our years of experience with three data centers has led us to build very redundant installations that can run on an emergency power supply for months. There is enough fuel available for 72 hours of emergency power and this can be constantly topped up.
The data centers were built with all the redundancy necessary to cope with interruptions in the power supply. The engineering of a large installation is important, and we have taken unforeseen circumstances into consideration. For many years, we have provided our technical staff with training on dealing with interruptions in the power supply.
We test the entire installation and simulate a real interruption every month. During this exercise, the entire data center is powered by emergency power generators. The exercise takes almost an entire day and is carried out by two people.
Our employees are fully aware the importance of business continuity, and because they are dealing with this matter every day they think in terms of redundancy. Practice pays off, as LCL Belgium’s employees will agree. They know better than anyone how installations ought to respond, and have developed a knack for them. LCL’s entire Operations Team is working to ensure that our three data centers continue to operate without any problems during the coming months and years, irrespective of any power outages that may occur. Laurens van Reijen – Managing Director of LCL Belgium