Home » Media Coverage

Disaster Recovery: Before The Bad Things Happen

Data Recovery Article: Backbone Magazine – November 10, 2003

By Andy Pedersen

Put yourself in Henry Gin´s shoes.

You work in the Toronto office of a company called Genesys.

With headquarters in Montpellier, France, and offices in more than a dozen other cities including San Francisco, Berlin, Milan, Singapore, Hong Kong and Melbourne, the company is a world player in audio and video teleconferencing. As vice-president of IT infrastructure, it’s your job to make sure the networks is working and that your customers aren’t having any trouble talking to each other.

Today it is a beautiful but hot day in August.

Then the power goes out.

“We were blind,” Gin said. “If there was a problem with any of our clients anywhere in the world, our ability to deal with that problem was…limited.” But Gin was a man with a plan. A disaster recovery plan, to be specific. As much of eastern North America entered the biggest power outage in the history of North America, Gin calmly declared emergency for Genesys. His staff members immediately abandoned their darkened offices, got into their cars and made their way toward the city´s outskirts.

A little more than 20 minutes later, they reassembled at a plain but fortified and heavily wired building in Mississauga, Ont., that thrummed to the low rhythms of two 1,500-kilowatt diesel generators. A small sign out front said Fusepoint Managed Services, one of the small but growing number of companies that are in the business of protecting other businesses from disaster.

While most of the rest of Toronto hunkered down to a night of heavy heat and candlelight, the Genesys team was back online, making sure its worldwide systems were still working properly.

“(Fusepoint) allowed us to get back into normal operation as quickly as possible,” Gin said. “And we didn´t have any complaints from our customers, that´s the really important thing.”

Few people think a disaster will befall them. Tornados and forest fires, sabotage and terrorism: these are things you see on the TV news, not stuff you have to deal with as part of your working day. But bad things happen. And even if they don´t happen to your city and your business, the world is now so interconnected that a catastrophe in one part of the world can trigger a shockwave that spreads all the way to your part of the world.

And yet, a poll conducted by Info Tech Research Group in the wake of August´s massive power outage found that fewer than 40 per cent of companies had a plan similar to Gin´s and Genesys´. The result, according to another study, this one by Dynamic Markets found that businesses in New York City alone were bleeding revenues to the tune of US$36 million every hour the power was out.

“A lot of companies learned an important lesson during the blackout,” said Robert Offley, CEO of Fusepoint Managed Services, a company that offers secure facilities with guaranteed backup power and redundant phone and Internet connections.

They learned they weren´t ready. But why were so few prepared?

“I think there´s the perception, especially in mid-sized businesses, that protecting themselves is going to cost,” Offley said.

“They think it´s going to take thousands and thousands of dollars to develop a plan and then millions to implement it. But it´s just not so.”

FOUR STEPS

So now, take yourself out of Henry Gin´s shoes and get back into your own. It´s time to start developing a disaster plan for your company. The good news is that the most basic and important parts of that plan will cost you very little.

Step 1: Gather the home phone numbers for your employees and management team and make sure those numbers are accessible.

“I´ve talked to companies that don´t even have a list of people´s home numbers,” Offley said, amazed. “If something like the blackout happens, how do you contact your people?” How do you let your systems guy know that the system´s down? How does the night shift let the supervisor know there´s a problem that needs immediate attention?

“This stuff is crucial,” Offley said.

Step 2: Protect your data. If a business doesn´t regularly back up its data, such as customer lists, orders and receivables, it can be devastated by a dramatic event like a hurricane or a flood, or the end can come from a simple power surge that fries the server.

“Many of the most panicked phone calls we get are about power surges,” said Bill Margeson, president of Markham, Ont.´s CBL Data Recovery Services. “So let the word go out: many businesses need better surge protection.”

To protect yourself from surges, your best bet is a piece of equipment called an uninterruptible power supply (UPS). Basically, it´s a set of batteries (not unlike the 12-volt in your car) that not only keeps your lights and computers running if the power goes out or dips, it also filters out the spikes or surges. (For more on UPSs and generators, see the sidebar.)

Step 3: Margeson´s advice doesn´t end there, though. “Let me also put the call out for better backups. Every company should be backing up its data regularly. And for heaven´s sakes, don´t store those backups on site!”

Again, this should be fairly painless to set up. There are countless backup systems that will regularly record all the information on your server to tape or disc.

Perhaps even better are the systems that take regular snapshots of your server info and transmit them securely to offsite electronic storage for just a few hundred dollars a year.

Step 4: Sit and think. From here, planning for disaster means rolling up your sleeves, taking a close look at your company and imagining the worst. Not that you should get bogged down in the morbid details of the disaster itself, it doesn´t really matter if you´re struck by fire or airplane or the sudden death of your sales VP. What you have to think about is the aftermath: how quickly you must recover, and how long that recovery will actually take you.

“There was a study done not long ago by the University of Texas,” said Bob Miano, CEO and president of Agility Disaster Recovery in Mississauga, Ont. “It found that 30 per cent of small and medium-sized businesses go out of business less than a year after suffering some sort of catastrophe. And that´s not all.

Of the businesses that do survive that first year, 50 per cent are out of business by the end of the second year.”

The killer, he said, is the downtime during which you can´t communicate with customers, giving your competitors a chance to steal them.

“You´ve got to do what I call a business impact analysis,” Miano said. “Most companies can do without accounts payable for, say, 35 days. You could probably survive without receivables for 50 days. But for most companies, if you lose your ability to take orders for 10 days, less even, you´re losing customers and that´s the kiss of death.”

BUYING INSURANCE

But how can you keep taking orders if the power´s out?

If the server hosting your Web portal is down? If your office is in ashes?

The solutions here run from the exotic and expensive to the makeshift but effective. The one that fits your company depends on what you can afford and how long you think you can safely stay out of touch.

Agility Recovery, for example, rents mobile offices that can be towed anywhere by a tractor-trailer and be set up and operating within a matter of hours.

“We bring in a trailer filled with desks, computers and phones that are already set up to be useful to the client,” he said. “It´s got satellite hook-ups and a generator. The entire city around you can be totally without infrastructure and our mobile units can keep you working.”

This level of mobility comes at a premium, though.

A less flexible, but also less expensive, solution is to rent what the industry calls a “hot site.” For as little as $500 a month, Fusepoint can deliver a safe, powered and connected environment from which a company can keep doing business.

“It´s like buying any insurance,” said Fusepoint´s Offley. “You can take out the triple-A five-star coverage or you can take out a policy that is more affordable and covers the things you know you really need covered.

But I think everybody knows it´s not prudent to take out no insurance.”

It can be overwhelming, trying to shore up your company´s disaster defences. And if you´re in the business of selling widgets online, you don´t want to divert too much attention from your core competencies.

Fusepoint and other companies, like Q9 Networks in Toronto, offer another elegant solution: they build the fortified buildings that have backup power and redundant communications lines and you simply rent server space inside.

“Why should you become an expert in backup power?” asked Q9 CEO Osama Arafat. “We already have that expertise, let us do that for you.” One of Q9´s biggest clients is Indigo.ca, and Arafat points out that the book company didn´t suffer any downtime during the blackout.

“It´s not as expensive as you might think, either,” he said.

“We offer service bundles that start at $1,250 a month. And for that you´re hosted in the same centre, with the same HVAC (heating, ventilation and air conditioning), the same fire suppression and the same trained personnel as Indigo.”

But no matter the solution you choose for your company, all the experts will tell you that a disaster plan is practically worthless if you don´t test it regularly and rigorously. Just look at what happened when Air Canada´s nerve centre in Toronto lost power during the blackout. The backup generator kicked in. And then failed. The airline had to cancel 500 of its 700 flights that day, and lost millions.

“You can´t just develop a plan, put it on the shelf and then dust it off when you have a disaster,” said Fusepoint´s Robert Offley. “It´s more like a culture that you have to develop inside the company. You have to encourage everybody to think about protecting the company, and train them to protect the company.”