Yesterday saw one of the moments you spend a great deal of time, money, planning, and preparing for – but rarely happen. I was on the phone at the time, however, there is something very magical that happens when the power dies, it goes, humanity pauses… like a deer in headlamps. The penny does not drop straight away. Especially in this kind of environment… as computers stay on, phones stay on, the while of rack upon rack of enterprise hardware not too far away continues…. you are just plunged into darkness without warning.

Clunk. Lighting is gone. A pause follows, then a cheer: These small victories.

These rare events, these victories, usually go unsung. The most you are likely to hear about is the odd obscure advisory about a ‘lack of protection’, maybe a ‘load bank test’ or the Stranger-Things-Esque sounding ‘black building test’. The unspoken goodness that goes into the preparation of the thing we probably take most for granted and is a sprawling and automated wonder that keeps it that way.

So how does it happen, and what is going on in the background without me knowing about it?

Well, I am glad you asked – let me introduce these three wonders of modern technology:

– Transfer Switch;

– Batteries;

– Generators.

This trio allow us options to continue service and reduce impact to our hardware, and in their many guises can be found throughout the industry – and in other applications such as hospitals.

For those of us interested in an uninterrupted power supply (UPS – usually referring to a physical device and out of this context) – then mains failure panels are out what you really need is a Transfer Switch.

The Transfer Switch.

This is where the smarts happen. These take two feeds, one from the utility (the feed your building gets) and one from your generator, and supply two outputs – a chosen power feed, and a signal to start a generator.

The device has enough smarts to figure out how long if the power is off for, and if it is “long enough” to start the generator. Timing very much depends on how much you want to avoid burning diesel and how long your batteries can support the load.

When the utility power fails, a timer starts. At a given point it will start a generator by signalling it, wait for it to deliver stable power, then switch the feed to off, then to a generator.

When utility power returns it will wait a defined amount of time, and then switch off, then to the utility.

Needless to say, these big chunks of no power are not ideal for computers – so something downstream from the transfer switch needs to catch the load… which is where the UPS comes into things.

 

The Uninterruptible Power Supply

In simplistic terms “a big box of batteries” and a box of magic – the Uninterruptable Power Supply (UPS).

Now – as you will appreciate if you are running racks of servers and cooling, then there will need to be more than a few D cells, in fact, more than a few truck batteries.

Equally – you will appreciate there will need to be some magic happening, as batteries give out DC power, and all but network equipment run on three or single phase AC power. So one of the jobs the UPS does is convert the DC power into AC power.

With all this clunking and switching upstream, you may find the power arriving at the UPS really is rather noisy. So it will likely also filter the power you receive, and remove any harmonics (caused electrically noisy neighbours).

If you ask your average ‘Joe Q. Public’ how the mains is in the UK is delivered they will likely say 240Volt 13Amp. However what you will tend to find is that the voltage goes up and down a great deal throughout the day in relation to demand, and also in the short term as draw changes second to second. The frequency however – remains the same. The UPS can also take all the excitement out of this for the servers – ensuring a filtered and steady feed arrives at the power supply. We reduce the voltage down to a stable 220v for example. To be able to do this high-efficiency rectifiers (AC to DC) and inverters (DC to AC) run full time to smooth this out. This is Online or double conversion operation.

The most vital part happens between these two. When the power is good – they gain their power from each other. When the power is gone, then it takes this from the batteries. How long the run time depends on the nature of what you are protecting, and how often you get power disturbances (as when the power returns to the UPS be it Generator or Utility it needs to recharge the batteries – so every time you switch between them a chunk of charge is taken away until they are empty).

Note: UPS’ are key to the current season of Mr. Robot. As a rule of thumb it is best not to let Christian Slater have access to your UPS.

For example on one of our sites with two giant ferry engines and a lake of fuel under a car park – that has not had a power incident in a decade… their batteries may run for minutes. In situations where you are more remote, you may be half an hour or more.

This doesn’t sound long – and it is not – however what you need to appreciate is that as soon as the power has gone and that magical “defined period” has passed – something else is turned on to provide the power.

That something else would be generators.

 

The Generator

Big ones. Really big ones. Somewhere between single big turbocharged tractor engines, or ferry engines… they all have the same goal – turning an alternator to produce AC power that feeds into the transfer switch. Be they many switches, many generators, one of each, whatever the concept is the same.

So how fast will a generator power up? Heating jacket? Check. Full-time battery maintenance? Check. Static magnet self-energising? Check. 5 seconds to power maybe.

Working like the regenerative brakes on a hybrid these are diesel engines turning electromagnets against each other. The braking effect as current flows through them slows the engine – and a smart governor adjusts the throttle to maintain a given RPM. The RPM effects the frequency of the power. Bad things (in a full-on ghost-busters-cross-the-streams way) start to happen if the frequency is not maintained as the load increases.

So how much oomph does that require? Well, to take one example – a straight 6 Cummings 6.2 Litre turbo diesel at 1500rpm attached to a Leroy Somer Alternator will chew through about 40 Litres an hour on StandBy (flat out for a short period) – but you would be mad to run one at anything apart from well below Prime (maximum comfortable load). It has to save the day after all – why give it a harder life than it has to have right?

Next, comes the joy of fuel. How long do you have – 40L an hour means 10 hours is 400L … and that is quite a small generator and quite a lot of fuel to be sitting around going off? How much do you have in the tank? Tanks are BIG. How much have you got on site? If it is a lake of fuel it may need cleaning (polishing) before it can be used – do you have one of those? Day tank? can you run for hours… or days? Can you refuel when it’s running? Can you switch to another generator if you need to – what about a plugging in a (relatively) portable one? So many decisions – again – as with each of the other steps – the times between operations – how long to run on for on return should it drop again – and so on. There is a LOT to think about…. oh and MAKE SURE SOMEONE KNOWS IT HAS HAPPENED!

So what happens when we pull it all together?

So – the generator comes to life – it energises with a great deal of smoke and thunder – power happens, the transfer switch sees power, switches through dead to generator (which curiously deliver much better power than the utility in terms of stability – who knew right?) – the UPS then sees the power – ‘does magic’ and once it is happy with the power – takes the batteries slowly off-line introducing more and more load to the alternator (as opposed to the shock of everything at once).

On the return of power, the transfer switch makes a call to switch back to Utility Power. The process occurs again – and the UPS is none the wiser. First the off. The UPS steps in with its batteries and catches the load, then starts to feed the load back into what it sees feeding it, until the load is returned without a sudden jolt back to the utility power. The generator runs on for a given period ‘just in case’ so it is ready should it drop again now with no load, cooling down before shuddering to a stop and silence.

Short outages will just be caught by the UPS and batteries alone. With no more activity than a flurry of emails and SMS alerts.

…. oh and cheers from engineers.

Shared Hosting, Virtual Machine, Dedicated Hosting – all of these are backed by power protection on all of our sites. It increases the life of the hardware, reduces bills, means that we don’t live in fear of those holes in the road, people on pylons, or some unseen squirrel making a poor life choice late on a Monday afternoon.

It’s just not worth taking the risk – it is a cost of doing business within the hosting environment, redundant, and doubly redundant platforms – and still you hear of things going wrong and the dreaded phrase ‘cascade failures’. Take a long hard look at the server on site – that box in the corner – that quarter rack with the 10-year-old UPS unit in the bottom with the battery alarm needing to be turned off periodically… it may be time to think about moving that liability off-site, colocation, hosted, something.

Power and connectivity are all too often overlooked for local on-site solutions. Have a word with our engineers or sales team to see if we can take any of those concerns away from you.

Here is to the plans we make for the worst case scenarios…. and the simple joys of when a plan comes together : )