Home » Media Coverage

Blackout Tests Enterprise Disaster Recovery Plans

Data Recovery Article: IT Business – August 20, 2003

By Shane Shick

For once, data was backed up and diesel generators were ready to kick in, but no one thought they’d have to run this long. We talk to RBC and others on how the outage expands the scope of their strategies

Terrorist attacks, ice storms and a number of lesser emergencies gave Canadian enterprises a good education in disaster recovery that prepared them for last week’s blackout, experts said Tuesday.

Five days after an unexplained grid failure darkened most of Ontario, the provincial government asked businesses and communities to continue conserving power in order to avoid rolling blackouts. Most private sector firms are now back in business, but are working with fewer lights and less air conditioning. Despite the severity of the blackout, which experts have called the worst in history, companies responsible for recovering lost or corrupted data reported little change in their day-to-day business.

“There were no servers that caught fire or tragic issues that way,” said Bill Margeson, CEO of Toronto-based CBL Data Recovery Labs. “Things got managed this time.”

The situation after the blackout is much different from that following the 1998 ice storm in Montreal, where server failures and major data loss temporarily crippled some businesses, Margeson said. This time, CBL has seen a 50 per cent increase in the number of calls it receives, but Margeson said they mostly concerned surge or power supply-related issues around a single drive in a user’s laptop.

Although there are some concerns that a failure to conserve may see the lights go out again, Margeson said user awareness about disaster recovery has grown considerably.

“We’ve been standing in the wilderness for 11 years — now people are approaching us, it’s on their mind. They’re thinking about backups,” he said. “If they pass the initial wave of problems — brownouts — I’m sure they’ll be able to weather any minor problems that may occur.”

At RBC, Canada’s largest bank, the Toronto data centre and a backup data centre at an undisclosed location outside the city are now drawing only 10 per cent of their power from the electrical grid, according to RBC senior vice-president and chief information security officer Diana Burke. The remainder is coming from diesel generators, which themselves have backup generators. Burke said the bank hasn’t experienced any backup and recovery issues and the data centre and backups can be run three to five days solely from the generators.

Burke admitted that the terrorist attacks of Sept. 11, the ice storm and the Year 2000 problem showed how necessary disaster recovery strategies are.

“Every incident like that helps you review your process and plans,” she said. “It’s unfortunate . . . (but) there’s no question they helped us.”

Veritas Software Tuesday tried to further that education process by releasing a five-point overview of disaster recovery basics. Fred Dimson, general manager and director of operations at Veritas Canada, said the blackout would likely see many enterprises reconsider where their data centre and backups reside.

“The replication of data centres was a thing about migrating to Ottawa or somewhere else in Ontario,” he said. “Now we’re looking at wide-area replication issues where you have to think hard about where you may want to put secondary sites. A few years ago, it was one building to the next.”

The government’s request to continue power conservation may also expand the scope of some disaster recovery plans, Dimson added.

“Most of them already have diesel and backup generators,” he said. “In reality, the data centres weren’t in bad shape, but a lot of them hadn’t thought of it going to be off for two or three days.”

While it may be difficult for some enterprises to conceive of the worst-case disasters, Dimson recommended they think of the financial impact of failing mission-critical applications.

“People don’t really spend the time to look at their operation and say, ‘If I don’t have HR for a week, I can survive, but what about payroll?” he said. “Here they had a test.”

Burke said RBC would be continuing to use diesel power until the bank is comfortable there will be no more blackouts and is monitoring the situation by the day.