Acts of God: Google Loses Data in Lightning Strikes

Written by Steve Ross

A lightning strike in late summer wiped disks in Google’s Belgian data centre. What was the extent of the damage – and what does this say about the way we protect our own information?

Has Google angered the technology gods? You’d be forgiven for thinking so if you’re an employee of the company’s St. Ghislain data centre, where lightning struck, not once, but four times in August. Despite the multiple protections in place, some user data was permanently wiped.

Google pointed out that many of the initially-affected disks in the centre were recovered and repaired after the lightning strike and that the data lost amounted to 0.000001% of that stored in its system. On top of that, it looks like customers affected were able to recover their data using personal back-up technology.

Being struck four times by lightning is unlucky, but could Google have done more to protect its users’ data?

Power Failure & Battery Drain

As you might expect, the problem isn’t as simple as a surge of electricity frying chipboards and, fortunately, Google has been refreshingly open about what happened. The company revealed that the lightning didn’t actually strike the Google data centre, but a local power grid south of Brussels. With power from the grid knocked out, storage hardware was supposed to switch to auxiliary power. However, in this case, recently written data was stored on “systems which were more susceptible to power failure from extended or repeated battery drain”.

Google also pointed out that when the event occurred, it was in the process of upgrading the hardware affected by the lightning strikes – upgrades which would have insured against this very problem. Those upgrades are obviously now being pursued as priority.

Threat Combinations

All this goes to show that even tech giants like Google are susceptible to threats – which may come from any angle and even manifest as a combination of unfortunate circumstances. There’s an important lesson here: threats to your data are, by definition, unexpected. Sooner or later your system will face a threat and when it does, its impact will depend on how prepared you are. So, what can we learn from this incident?

Firstly, Google avoided a larger disaster by incorporating redundancy into its storage system. The Google cloud set-up includes multiple cloud regions, with isolated zones within each region allowing users to deploy their stored data with ‘failover’ security should one zone go down. Google’s cloud infrastructure, known as Google Compute Engine, is currently made up of three regions: Central US which holds four zones, and East Asia and Western Europe which hold three zones each.

Secondly, companies of every size should consider the practical effects of a power outage on their networks. The extreme unlikelihood of the Google lightning strike should serve as proof that your company isn’t immune to this kind of threat – could you continue to do business if your power failed? Do you have back-up generators in place? Is your data backed-up on both on-site and in the cloud?

Finally, it’s important to work with clients regarding their data security. Whether you have a cloud-only approach to storage, or opted for a hybrid strategy, by working with you we can ensure you have the right kind of protection – and increase the chances of a full recovery should the worst happen.

Could Google have done more to protect its client data? Are you prepared for unexpected threats? Share your thoughts with us here…