Friday, August 29, 2003
Thanks to KL for this link to a thoughtful essay on the process of bereavement. Worth a look, and a moment's thought.
Shipped a piece on disaster recovery and business continuity the other day, so here it is:
September 11 was the ultimate, iconic disaster: destruction of life, property, communications and data - the lifeblood of business. Ever since, the world has felt less safe for all of us. For the system administrator who guards the life-blood of companies against fires, malicious denial of service, virus attack and a million other hazards, the challenge is maintaining business continuity in a war zone.
Business as Usual – during Disaster
It’s a mental image that causes nightmares for every system administrator. Most sysadmins are early birds, happy to know the network and all vital servers are up and running before the staff come into work. Normally, the car park is pretty empty at this hour, but on this particular morning it has a conspicuous occupant – the fire brigade, parked next to a smoke-blackened building that is home to everything that system administrator – and every employee – holds very dear.
At this moment, the future of the company is literarily dangling by a thread. Most studies show that about 60% of businesses that suffer major disaster go out of business within 5 years. Some of the more pessimistic results show that only 6% of companies that suffer catastrophic data loss actually survive. Source: (Eric Savitz in Fortune, Nov 2001). Insurance carries the victim limping past the point where a new office and new hardware is in place, but while “disaster recovery” has been achieved (in the barest literal sense of “getting a replica of what was there before up and running”), the underlying goal of “business continuity” has been critically, fatally broached.
Think of the disruptive effect of the average fire drill (“everyone to Assembly Point A”), then imagine a fire drill that goes on for several working days. Then imagine that (thanks to your disaster recovery plan) the office you return to, at a new address, while nominally a replica of what was there before, in fact has nothing anywhere you expect it to be: “where are the servers, the e-mail, the printers?”. In the meantime, customers have turned into angry customers, and angry customers have turned into lost business. Prospects have turned into missed opportunities, and staff, worried by media announcements along the lines of “Major fire puts jobs at risk”, are restless and jumpy. The prospect of being one of the 6% that survive looks slimmer and slimmer.
And the bad news is: this describes the situation for the company that actually has a disaster recovery plan in place. Exactly how does it stand with you?
An EMC/NOP Technology poll of 254 executives at major corporations and other large organisations in seven countries found that 40% of business leaders and 44% of the IT executives felt their important business information is very vulnerable to being lost.
The shared feelings of vulnerability are in sharp contrast to their American counterparts where a large gap exists in the perceptions between U.S. business and IT leaders. Of 274 U.S. executives surveyed, only 14% of business executives felt their data is very vulnerable to being lost, compared to 52% of the IT executives.
When asked how long it would take to restore data to resume normal business operations following a disaster, European business executives again share the same expectations as their IT counterparts. A quarter of the business (25%) and IT executives (23%) in Europe say it would take three days or longer. In contrast, only 9% of the American business executives surveyed felt recovery would take three days or longer, while 23% of U.S. IT executives, the same as their European counterparts, felt it would take in excess of three days to recover if a disaster struck.
Andrew Warriner, technical manager, ON Technology notes: “It isn't a surprise that after 9/11 we have seen a huge investment in disaster recovery, and many companies in New York last week reaped the rewards of careful planning when they lost electricity. However, not all companies have given such careful consideration to their disaster recovery needs. Whilst many have set up recover centres and taken steps in the right direction to protect their business critical data, companies need to be asking tough questions about whether the measures they have put in place will give them true business continuity in the event of a disaster. “
“It would appear that the lines of communication in European organisations are more open when it comes to disaster preparedness,” said Richard Jameson, Managing Director at NOP Technology. “However, resuming normal business operations in three days or more would be very costly in terms of lost revenue and customer satisfaction. Both the business and IT executives understand the risk and share a common realisation that it is a problem.”
One message that could be taken from these figures is that American business has spent more comprehensively on hardware such as remote, mirrored servers, and outsourced rack space, while Europeans have been more focused on planning and establishing lines of communication. In modern business continuity planning, teleworking and mobile communications have a key role. It is good for peace of mind to know that all your computer data can be siphoned back – over time – from remote rack-space. It is even better that an organized swarm of mobile, laptop-toting managers and IT staff are immediately in action and in constant contact. This virtual management team can use multiple layers of communication, already set-up and tested: mobile phone, Internet, free-mail accounts, texting, WiFi.
Dave Dignam, Business Development Director at Synstar has faint praise of the Irish IT sector’s awareness of business continuity: “In European terms the medium-to-large enterprises based in Ireland are relatively advanced with disaster recovery and continuity thinking. However, recent statistics show that many organisations restrict their strategies to developing recovery plans for IT in isolation to the rest of the business.”
Alan Yaverbaum, Storage consultant, DataDurability, is even less up-beat: “There is still a major shortfall in awareness. We have found a disturbing number of medium sized companies do not have any disaster recovery plan in place. Some large enterprises have a level of disaster recovery such as tape or optical backup which is protected in a fire proof safe or archived off site. However, the same companies that invest in disaster recovery often overlook business continuity.”
“Business today cannot afford to be without their data for very long before the business itself is at risk of collapse. Irish businesses need to be aware of the time it takes between a disaster and resumption of normal activities.” Irish businesses dependent on IT need to become more aware of how to operate in 24 x 7 environments that can achieve 100% uptime with technologies such as replication, clustering and automated availability management.
Mr. Sean Conway, CEO, Orlogix, which offers easy-to-use backup programs, suitable for home office and small networks, is even more sceptical of Irish IT preparedness: “There would still seem to be a large shortfall in the extent to which companies are even adequately protecting their data; a full disaster recovery plan, with a "hot site" to facilitate business continuity is even rarer and probably limited to larger, multinational organisations.”
Chris Casey, Service Manager, of PFH Computers is even more scathing: “The vast majority of SMEs have a high awareness of potential disaster situations. They take a very ad-hoc approach and are probably more likely to deal with a disaster in try-it-and-see fashion.”
Whether you are the company that just needs a sensible offsite backup policy or a full-blown always-on hot site, it all comes down to people and process when it’s hammer time on your watch. Real responsivity and agility grows out of team familiarity with the disaster recovery plan itself when things go Code Red. It’s not much good if the plan only exists in a few systems administrator’s heads. (Question: How widely known is your disaster recovery plan in your organisation?).
Tony Andreucetti, Business Continuity Project Manager at Synstar Ireland, provides a succinct roadmap for developing a business continuity or disaster recovery plan: “People – how will you manage them? Processes – what are the key processes of the company that we need to recover? Premises – where will you put your people and systems? Providers – what infrastructure support, including IT, do you need?”
Like all process documents, a disaster recovery plan has someone who administers it, tracks version numbers, who contributed, and where the document is stored (including multiple off-site copies, naturally), and contact information for the various back-up copies. You don’t want to be looking for a piece of paper while the facility is going up in flames about your ears!
The first section describes the conditions for the plan’s activation, who is authorised to push the Big Red Button, and what to do to prevent loss of human life. All this part looks much the same as your fire evacuation procedure.
From an IT perspective, the next stages involve moving essential business activities or support services to alternative temporary locations. Ideally, this may simply mean moving an application to another host. Resumption procedures describe the actions to be taken to return to normal full business operations, usually at the original location.
Derek Ashmore, Head of Design and Consultancy, ALLnet, notes that a disaster plan rates assets according to the criticality of the business processes that they each support. A real-time transaction system, for example, is worth the expense of “hot standby” servers in a remote location, ready to swing into action at the jolt of a fire alarm. In other areas, simply protecting the data on a daily basis may be sufficient.
You certainly don’t want to be doing all this for the first time it’s Code Red for real, any more than you want to test your fire drill on a real blaze. The disaster plan should contain a test schedule, which describes how and when the plan will be tested.
These basic components of a disaster recovery plan, and the cost of their implementation, will obviously vary in extent according to the size of the enterprise. Basic site issues outside the scope of IT are covered, such as water, power, seating plan, transport issue, physical security, staff communication and training – tools such as handbooks, an internal company web page, a help line, may all be considered to promote awareness.
During a real life Black Friday Apart the system administrators life is devoted to overseeing the recovery of vital data, and the continuity of critical applications. Further complexity is added by third party issues, such as dealing with the ISP, the telephone company, and teleworking or contract staff. Pre-set agreements and levels of service contracts are a godsend here. All of this is happening at a time when prayers are being said for the reliability of redundant IT infrastructure supporting external connectivity and business application services, such as Internet, RAS, and the corporate web site.
The next challenge is the provision of a “mirror” of the old local IT infrastructure, i.e. routers, switches, DNS, DHCP, business application services, desktop and peripheral devices need to be available in the structure in which employees are used to seeing them. It’s not much good having everyone back at desks if no one can use email, find a file server, or print anything. Network configuration issues will be a headache: for example, IP naming and addressing may be chaotic as people switch from one network to another.
The new infrastructure needs to be as secure as the old one, so all the old barbed wire fencing of password authentication, anti-virus, encryption, filtering of incoming content, logging and monitoring of user activities needs to be racked up fast at this vulnerable time. Data replication and backup for each network and business application level service need to be fired up pretty quickly too. The system is in a perilous situation – similar to driving in a car where the spare wheel is already in use and the damaged wheel hasn’t been fixed yet.
It gets more complex as the employee numbers grow. “Industry needs to make the shift from highly available IT infrastructures to always-available business processes”, says Tom Byrne, Business Recovery & Continuity Services Manager, IBM Ireland.
In the larger company, you might be talking about dealing with 100 to 200 people arriving at the site at a few hours notice. Moreover, many large organisations can run a diverse portfolio of applications on a range of hardware and software platforms. Its not untypical for a company to require 3 x AS/400s, 4 Sun Servers, a HP 9000 , and 20-30 Intel servers for a single test. Not to mention the fact that they will require 200 desktops setup with three different images, ISDN communications, ATM networks or the latest generation of MPLS network such as Eircom's BIP. Such complexity needs enormous planning and coordination between business continuity provider and customer.
It’s all worth it when it pays off. AIB Capital Markets comprises AIB Group's Global Treasury, Investment Banking and Corporate Banking businesses. The division operates worldwide with offices in Ireland, UK, USA, Europe and Asia Pacific. On Thursday 26 September 2002 a fire broke out at approximately 6:45 pm on the fourth floor of Ashford House, one of AIB's offices in Dublin. It took just 40 minutes to extinguish the fire, but the damage proved to be far greater than first envisaged. The building's fourth floor, home to the bank's global treasury operations, had been very badly damaged by fire and smoke. The floor below had suffered serious water damage.
The business continuity team, resourced from Synstar's Dublin Business Recovery Centre (BRC), swung into action and overnight set about preparing a recovery room for staff to use the following day. The incident management team gathered all staff in the company restaurant at 9.00 am the following morning.
At midnight a group of selected individuals from AIB's treasury business support service (TBSS) began testing the new PC builds to ensure that they were ready for immediate use in the recovery suite the following morning. The business continuity team then turned its attentions to staff management issues such as co-ordinating employees in the morning, making plans to communicate to staff details of the site's facilities and ensuring that everyone was comfortable in the new environment. By 2.00 am most preparations were complete.
Next morning, before the start of the new business day, staff were briefed on the events of the previous night, the preparations that had been made overnight, and the way in which AIB's invocation plan would guarantee business as usual for the company's clients. Every staff member was able to perform his or her role perfectly using the outsourced facilities, just as they would have done in their own corporate office environment. And, most importantly, customers remained completely unaware that there had ever been a crisis, even though the temporary arrangements lasted a fortnight.
Mentally compare this benign scenario with the idea of a “Closed due to Fire” sign in front of a world-wide treasury operation for two weeks. And if you are one of the silent, guilty majority without a disaster recovery and business continuity plan, start in on writing one now.
End of article
PS I must spawn off a separate blog for pieces like this, as I'm sure embedding articles probably pisses off readers (if there are any?).
Had the clamps taken out yesterday, so no longer look like Arnie after some minor repairs. Had some long sleep-ins so the body is generally repairing itself and returning to normal. Vack to normal work on Monday.
Monday, August 25, 2003
It's still (a) painful (b) good to be alive.
Saturday, August 23, 2003
Just out the far side of a hernia operation: it's (a) painful (b) good to be alive.
Wednesday, August 16, 2003
Came across a cool bit of html, let's you generate a popup over a piece of text - originally inventer to give popup explanations of acronyms. It looks like this (just hover the mouse over the acronym): [W3C]
Off to hospital for a spot of elective surgery: the bright side is getting some value back out of years of paying out to VHI. Back out and blogging again tomorrow, hopefully.
On a fairly related note, I see the NHS are giving out cannabis to help relieve post-surgical pain.
Tuesday, August 16, 2003
A study in contrasts: this page, a review of a new book by Trainspotter's Welsh is a paean to Scottish cuisine (i.e. everything that lies still is deep fried in batter), links to this page on the Slow Food movement.
One of my workmates found this piece which is David Byrne (ex-Talking Heads), bizarrely, enthusing about PowerPoint. Never thought I'd live to see the day, but the guy has a point - power point, even (boom boom).
Looks like China is making a move to block all non-Chinese software from government contracts. Music to the ears of the open source crowd, but it hardly sounds like model WTO member behaviour, does it?
Monday, August 18, 2003
I wrote my first little review over at blogcritics.org. Jonathan Raban's Old Glory. Here's the piece I came up with.
Old Glory, by Jonathan Raban
If you haven't read anything by Jonathan Raban, Old Glory is a good place to start. Raban's books fall into that uneasy space between autobiography, travel, history and philosophy that seem to define the late twentieth century travel genre - for example, his "Passage to Juneau" is not only about sailing the Inner Sound between Vancouver Island and the mainlaind, but about being divorced by his wife because he is for ever off sailing and writing books.
Old Glory draws its inspiration from a childhood fascination with the image of Huck Finn, on his raft, going down the Mississippi. As a grown man (if self-confessedly not a fully adult one), Raban "lights out for the terrority" in a utilitarian launch with an outboard motor - no raft for him. Sometimes the river is hauntingly empty, sometimes frightenly crowded with massive ferrys. It can be idyllic or life threating, with logs, eddys and boils of turbulent water to avoid. As ever, Raban meets people all along the way, well off, humble, even dirt poor, because while the river flows through America's prosperous heartland, it also flows through an alternative world of bums, bikers and old black guys fishing for catfish. Read this, and Raban and Old Glory will become friends to you, in the end you will dread the inevitable time when the river loses itself in the sea, and Raban turns back to losing himself in the crowd.
Piqued by the ongoing hoo-ha about smoking ban in pubs, and hearing of yet another pub shooting in Dublin, I combined the two into this modest proposal in a letter to the Irish Times. Let's see if she's printed:
A MODEST PROPOSAL
The Irish Hospitality Industry Alliance (IHIA) would have us believe a smoking ban in pubs cost tens of thousands of jobs in New York. Meanwhile, Chicago is marketing their advantage: "come to Chicago as we have everything New York used to have", according to Minister Cullen, (Times, 12 August).
Rather than have our visitors and Irish natives lured away to Berlin-style smoky venues, we should draw on another Chicago heritage: using firearms on licensed premises. Leveraging the already healthy upsurge of shootings in Irish pubs, we are in an excellent position to combine two vibrant traditions: the Right to Smoke and the Right to Bear Arms, in selected and qualified Irish pubs.
Dedicated smokers could then enjoy a rip-roaring environment where they can indulge the beloved vice, and better still, defend their rights with hot lead. Any disputes could be settled in the summary fashion of the old West. If the situation gets out of hand, the bartender emerges from under the counter with a pump-action shot gun and clears the house.
Eat your heart out, Chicago! These rip-roaring Wild West theme pubs would attract aficionados of hard-core "craic" from all over the world. Consider the benefits for the undertaking trade, which could offer Wild West-themed funeral products. Wild West pubs would also remove - permanently - many long-term and expensive sufferers of chronic smoking-related ailments from our hard-pressed medical services.
Friday, August 15, 2003
The home PC got nabbed by MS Blaster on Wednesday, here's a good advisory for any victim.
Thursday, August 14, 2003
Looks like end of the line for Microsoft Outlook Express. Well, I guess there's only so much you can do with an e-mail client, and Express for me is the close to perfect e-mail app. Ain't broken ...
Wednesday, August 13, 2003
Found an excellent source of winning new blogs: at blogcritics.org. Emailed them to sign up as a volunteer critic, help keep the community thriving ...
Monday, August 11, 2003
A hilarious flame-war erupted among the Linux kernel community when some-one did a global replace of "flavour" to "flavor". You would be surprised at how worked up people get about US vs. Euro spelling - well, cancel that, if you've worked in US companies with a European operation, or vice versa, you won't be a bit surprised.
Friday, August 8, 2003
The Reg, provocative as ever, predicts the death of the IT industry. Well, as we know it, anyhow - think commodotized, ultra-cheap everything.
Fog screens and whispering windows - Minority Report looms closer ...
Lite on the blogging at the moment - workload is more than usually hellacious and I don't have a mouses kick left in me after the working day.
Thursday, August 7, 2003
The heat goes on ... tho twas a little cooler last night.
Wednesday, August 6, 2003
But does anyone really know the time? Not according to this piece about the diverging systems we have for telling the time. I didn't know, for example, that airplane time is on a different clock to the rest of us. And because the Moon acts as a drag on Earth roation, every so often we have to stop all the clocks for a couple secs ...
The Customer Support story from Hell - or rather, Dell. Bloggers are hoping that enough people link to this that Dell maybe replaces the lemon laptop they sold him. We're having our own grief with Sky at the moment. No telly for well over a month.
Good essay on the blogging process. Now if only it was really like that.
Dirty rotten rats can in fact be quite sweet. These ones are trained mine detection experts.
Good piece here on freelance tech support. Good to know there's a demand for something any techie should be able to do.
Tuesday, August 5, 2003
The terrorism futures market runs and runs: this Reg piece has the skinny on the Extropians - the space cadet end of the extreme right-wing of the blogosphere.
From the IT poll, a contributor thus: "Margaret Thatcher was known as "Daggers", short for Dagenham, Dagenham being three stops beyond Barking (on the District Line). "
VB 6.0 is the preferred programming language of baboons. And though it looks like a hoax, the study actually does seem to be scientifically respectable. I love the shot of the alpha male with his wee laptop, while the presumably less intelligent types look on admiringly ...
Big fires in Europe after record heat-wave conditions. France sets new record of 45 C. The globe is pretty warm already ...
One long hot Bank Holiday weekend. Long walk on Saturday, over Cushbawn. Didn't succeed in working out a way to walk on down to Billinaclash, but got a nice mid-level hillwalk worked out that gives excellent views of the Wicklow hills and the South East generally.
On Sunday we had a picnic in the Vale of Clara. Monday was spent bashing the weeds out of the front drive and border: the sight of weeds was getting a bit downspiriting. Stiff and sore today after the work.
Friday, August 1, 2003
This sounds like one for me: the ape diet.
A new word: phishing. It's the practice of cloning a page from, say, Amazon, to con people into giving you their credit card details. The bit I like: "The name appears to have no connection to the band Phish, an FBI spokesman said".
I like the ominous tone of this one: ""We are expecting something sooner rather than later," Ingevaldson said. "But there's no horizon on some of these things."
What Ingevaldson's talking about is the well known security flaws in Microsoft stuff. And in the context of "threat to Homeland Security". Seems it might be OK for MS to establish a bloated monopoly - but selling warez that could let (towel-wearing) hqrz onto America personal computers - maybe even re-format yr HARD DRIVES - now that's something else.
There is more than a whiff of rodent about the way the US is setting up the rules for rolling out mobile licenses in Iraq. Watch this space! Anyone care to take a bet the Yanks don't try to shoehorn cdmaOne (the American standard owned by Quallcomm) rather than GSM (the standard which works seamlesslessly everywhere in the world except America - and S. Korea).
What was the tag line again from "One Foot in the Grave": "I don't believe it." Well, that's about how I feel about Bush's decision to renew sanctions against Iraq because "it continues to pose a threat to national security". Wait a minute: I thought they invaded the place and occupied, I mean liberated it. Surely this is a blatant admission they have't succeeded in occupying it?
Some bloggers made the news yesterday for getting a story onto newsprint: "Silvio Berlusconi demands return of Ireland's Caravaggio, says Irish 'ignorant'". All too believable!
posted by A Seeker after Knowledge 2:46 AM