Unannounced DCP testing a no-no

If you are tempted to spring an emergency drill or contingency test on the organization without properly pre-announcing it, be prepared for the emotional fallout from those who are duped into believing the incident is real ... especially if your scenario involves a gunpoint hostage seige ...

Plan B includes not being able to get to work

A former director of FEMA, the US Federal Emergency Management Agency, promotes the value of planning for employees being unable to get to the office in an emergency.

"Everyone will tell you: I have a risk manager, a safety manager, we have contingency plans in place for their business. What plans do they have in place for the workforce? Because if those people cant get to work, those other plans dont do them any good. One of the things that federal government does and state government does is they really try to drive home this concept of being prepared at home. I think businesses should do the same thing, regardless of the size. The better prepared employees are in the neighborhood they live in, the more likely they are to get back to work quicker, the more likely they are to be more loyal to you because youve helped them be more prepared in the neighborhood where they live."

Does your DCP cover frozen hydrazine tanks crash-landing?

A US spy satellite "the size of a bus" (the SI unit of satellite size) that went out of control shortly after being launched a year ago, has been blasted by a US missile over the Pacific Ocean. They aimed (literally) to blow the satellite to smithereens (the SI unit of satellite size following missile impact), ostensibly to prevent the frozen hydrazine fuel tank smashing to Earth and giving someone a nasty surprise. Any secret weaponry allegedly on board would also, presumably, have been destroyed.

What if the missile had missed its target or they had not been able to fire the missile for fear of creating an international security incident amid fears of the Star Wars initiative? And what if the spy satellite had landed, intact, on your data center? What if the missile landed on your data center? What if ...?

Now I don't expect your contingency plans to mention falling spy satellites, frozen hydrazine or missiles explicitly, but that's really not the point. The point is that your plans perhaps ought to mention and should definitely cover commonplace and credible disaster scenarios, but should also cover the more extreme, outlandish and incredible incidents too, the nature of which is presently unknown and, in fact, unknowable. That is the essence of true contingency planning: "We don't know exactly what might happen but we are as ready as we can ever be to cope with any disaster that comes our way."

The US military's contingency plan for the spy satellite going out of control presumably reads:
- Have large missiles available in strategic locations worldwide
- Launch large missile at satellite
- Handle PR nightmare as well as can be expected given circumstances
- Reassure Chinese and Russians that WW3 is not declared
- Fire designers and builders of out of control spy satellite

For you and me, a specific contingency plan to cover the spy satellite scenario might read something like:
- See flaming ball of fire approaching at 22,000 mph
- Take cover under large immovable object, quickly
- Hear flaming ball of fire explode, releasing no-longer-frozen hydrazine gas
- Hold breath
- Crawl out from under large hot immovable object
- Staunch bleeding, dampen fires
- Seek fresh air
- Call insurer to make incredible claim

A more general plan might read:
- Have large immovable object or similar, under which to take cover
- Have first aid kit with all essentials
- Have disaster survival kit with all essentials
- Have insurance policy
- Watch for news of imminent disasters, Google "hydrazine" and refine/enact plan accordingly

Do your contingency plans cover mice and snakes?

Physical security incidents are one class of incident that virtually all contingency plans cover, but are your plans broad enough to cater for the full range of potential physical security incidents? Here are some classic photographs of actual incidents that might make you re-think your approach:
- Mice nesting inside a system, using a handy computer manual as nesting material
- A snake living inside a nice warm system box
- Lightning/storm damage to electronics
- Inept maintenance and repairs
- Equipment overheating

There are more photos of this nature at the Microwave Mortuary if you need something to spice up your awareness program.

BCP auditing the IIA way

"During their planning cycles, many companies around the world evaluate how prepared they are to handle disasters as well as the effectiveness of their business continuity and disaster recovery plans. As part of this process, internal auditors can help organizations establish effective business continuity management (BCM) programs. To do this, auditors need to understand what is involved in developing a BCM program and the steps they should take to evaluate the effectiveness of existing programs that incorporate necessary business continuity, disaster recovery, and crisis management efforts."

I'd like for you to be able to read what the Institute of Internal Auditors, or more precisely author Mark T Edmead of Control Solutions International, advises IT auditors to look for when reviewing business continuity arrangements. Unfortunately, the IIA article has dropped off the Web in the past few days. Sorry.

Mark's advice is sound but stops well short of the audit-style Internal Controls Questionnaire provided in this month's NoticeBored security awareness module. Still, it validates and summarizes the approach detailed in our ICQ and is an interesting piece.

A modern Doomsday

Middle-Eastern Internet services have been severely disrupted by the failure of an undersea cable linking Egypt to Italy. There are backup connections, of course, including satellite and other cable connections but their capacity is limited, hence Internet traffic in some countries in the region is experiencing delays and probably failed connections due to timeouts.

Thanks to packet switching technology and multiple routes, the Internet as a whole is highly resilient. Undersea cables can often be repaired within days or weeks. But imagine what would happen if the Internet went down, and stayed down. Not 'stayed down for a few minutes' or hours or even days, but for an extended period perhaps indefinitely.

There are various horrific scenarios that could cause this to happen e.g.:
- Widespread technology failure, disrupting the packet switching backbone;
- Deliberate action by one or more nation states in wartime, severing critical connections and/or injecting massive amounts of spurious traffic at multiple points to disrupt;
- Natural events such as solar flares/X-ray emissions from the sun, storms etc. damaging critical equipment and links;
- Cyberterrorist attacks on the Domain Name Systems or other critical elements of the Internet, perhaps combined with conventional terrorist attacks on key nodes, cables and satellite ground stations;
- Worms or other malware, in other words, software agents swamping or damaging the network;
- "Something else" - the classic contingency planning scenario. We don't know exactly what might happen. It could be something completely novel and unanticipated or a chance combination of more than one type of event, known as 'bad luck'. For true contingency planning purposes, the exact cause and nature of the incident is irrelevant: we need to be ready to cope with whatever actually happens.

With a moment's thought, the horrendous consequences of such an incident start to become clear. The developed nations are highly reliant on the Internet and would suffer economic and social consequences very quickly. Developing nations are also actively using the Internet for eCommerce and communications with the rest of the world. The Internet has penetrated even the least developed third-world countries, and disruption to first world aide programs would have consequences there too.

We're hardly on the same scale as Google, eBay and Amazon but at a local level, our own small business would suffer within days if the Internet went down. We use the Internet for marketing and promotion, sales and delivery, research and communications. There are fallback delivery mechanisms - sending CD-ROMs in the post or direct dial-up access - both of which are limited, wouldn't work very reliably and would increase our costs. We could resort to old-fashioned research methods but would miss the ready, free access to up-to-date information security news from around the globe. Our marketing and sales would suffer the most as conventional print, TV and radio advertising is far more expensive and limited in scope. That, in a nutshell, is our own risk assessment.

Larger e-enabled businesses (such as the entire financial services industry) would su=ffer immediate problems, others might hardly notice at first, at least until their suppliers, partners and/or customers started to fail. Government departments and utilities would suffer quite quckly, causing knock-on effects as the national infrastructures started to unravel. If petrol companies and airlines were disrupted, well we'd have to get used to walking or cycling to work, if indeed work existed. Civil disruption could have serious consequences for personal safety and security.

We're just a few paragraphs into this very brief overview but the 'worst case scenario' is shaping up badly. This is starting to sound like one of those science fiction doomsday stories.

On the upside, TV, radio and print media would be severely disrupted too so we might not get to hear too much about the civil disruption outside our barricaded front doors. Some of us will retreat to our caves.

What kind of contingency plans would or could you make for "the Internet is down"? Some of the more obvious things might be to retain or stockpile ordinary modems (assuming that the telephone networks are running ... but, oh dear, they are using VOIP and, no doubt, sharing a lot of the Internet technologies and links) and generally retain (or rather rebuild) the ability for non-electronic commerce and communications.

More resourceful organizations might build their own private networks to run in parallel with the Internet - such as the financial services, military and other special purpose networks. These are expensive but the greater concern is to ensure they are adequately isolated from the Internet in fact. Supposedly private bank ATM networks have been known to crash due to Internet worms so finding and closing those worm-holes must be a priority. That's definitely something we can do today.

What else would you suggest in the way of contingency measures? Any ideas you'd like to share? Just post a comment ... while your Internet connection is still running, please.

Plan B

Despite our best intentions and investment in a range of preventive security controls, serious incidents and disasters may still interrupt IT systems and impact the business processes which they support. As some say, **it happens. Just when everything is running sweetly, something unanticipated occurs, revealing that Plan A is not quite so perfect after all.

Contingency planning (Plan B) puts us in a better position to survive any disaster by:
1) Managing the immediate crisis professionally and confidently;
2) Keeping the organization’s essential processes and systems running despite the event through resilience and continuity planning; and
3) Recovering non-essential processes and systems as soon as possible thereafter disaster recovery planning.

The time to plan for a disaster is now, when things are going well: planning during a disaster will be too late.

As always, this month’s NoticeBored module provides a range of high quality security awareness materials aimed at staff, managers and IT pro’s. We found it relatively easy to write a detailed 9-page white paper on Disaster Recovery for IT and a 5½-page management briefing on Plan B. Crunching the key facts into one page staff, management and technical briefings was harder, and doing so without losing the plot was quite tough. Our solution was to put the subject in context for each audience:
- We encourage ordinary employees to find out about their department’s contingency plans and draw up their own personal Plan B;
- For managers we point out their governance responsibilities and highlight the risk management advantages of thinking ahead and preparing for the worst;
- Technical aspects of high availability systems architecture and DR are of interest to IT people, and it doesn’t hurt to emphasize IT’s critical role in keeping the average corporation on the air.

Privacy in the 21st Century

This week is the third annual Global Security Week. This year's topic is Privacy in the 21st Century. For information on GSW events, free awareness materials to download and links to further privacy resources, visit the GSW website.

There's also a GSW blog: I've just posted the following item to the GSW blog and there are contributions from supporters of GSW.

Does your organization have a policy on promptly informing those affected by privacy incidents and, where necessary, disclosing breaches to the proper authorities? If not, a privacy incident at John Hopkins Hospital might make you think again:

"A desktop computer containing the personal information of 5,783 patients was stolen from Johns Hopkins Hospital in mid-July, and the hospital waited more than five weeks to inform the patients or their families of the theft. The computer, taken from an "administrative work area" in a building on Johns Hopkins' main campus the night of July 15, contained patients' names, Social Security numbers, birth dates, medical histories and other personal information, according to Hopkins officials. Another computer and a projector were also stolen."

Another suggestion is to make sure your organization's contingency plans cover privacy and security incidents, giving management a blueprint to help them deal with a crisis in the most efficient and professional manner possible under the circumstances.

Of sloping baths and disk drive failure

Disk drive manufacturers quote MTBF (Mean Time Between Failures) of around a million hours under ideal conditions, suggesting a failure rate of less than 1% per year, but some studies show significantly worse performance (2-10% failure p.a.) in the Real World™. It seems the “bathtub” reliability curve has a sharply upward sloping or even stepped bottom, not the long flat period of stability often assumed.

Thanks to George Spafford's Daily News for both the above links :-)

If your data are vital and their availability is critical, the studies suggest the value of monitoring drive age, error rates and temperatures carefully. Also techniques such as RAID will help. However, the unpredictability of disk failure also implies the need to have contingency plans, backups and hot-swappable drives. Or, if money is no object, solid state disks might be the way to go (plus cosmic ray shielding!).

More availability resources

Contingency planning for small businesses

The Australian Attorney-General, no less, has released a booklet of advice for small businesses called Good Security - Good Business. Contingency and continuity planning is the main subject with a little risk analysis thrown in for good measure.

Whilst I have no problem with the government producing useful booklets, I hope they are doing rather more than that to promote good security practices.

More contingency planning links

BCP lessons from hurricane Katrina

A report published by the Federal Financial Institutions Examination Council (FFIEC) does a great job of distilling the key disaster management and contingency planning lessons learned from hurricane Katrina. The report deserves a wider audience than the financial services industry since the lessons apply more broadly:
- Some organizations may not have anticipated or prepared for the extensive destruction and prolonged recovery period resulting from Hurricane Katrina.
- To be realistic, disaster drills should include all critical functions and areas.
- Anticipate disruptions in communications services, possibly for extended periods of time.
- Critical staff may not be able to reach their assigned recovery location.
- People are essential to the recovery of operations.
- Replacement supplies may be difficult to obtain during a protracted recovery period.
- Financial institutions' facilities could be damaged or destroyed, creating a need for alternate facilities.
- The location of any back-up site can be critical to successful recovery efforts.
- Processing transactions may be extremely difficult.
- Be prepared to operate in a "cash only" environment.
- The financial industry is dependent on numerous critical infrastructure sectors that potentially have competing interests.
- A financial institution's involvement in neighborhood, city, state, federal, and non-profit or volunteer programs can facilitate a community's recovery from a catastrophic event.
More contingency planning links

A solid information security manual

NIST Special Publication 800-100 "Information Security Manual: A Guide for Managers" is a 174-page draft released in June 2006 for public comment. It refers throughout to [US Government] agencies but in fact is broadly applicable, containing sound guidance on important areas such as information security governance, investment and metrics, planning, contingency, C&A, incident management and, of course, awareness training and education. It's a good-un, well worth a serious look.
More infosec laws, regulations and standards

Disaster management standard

NFPA 1600 is an American standard for Disaster/Emergency Management and Business Continuity Programs - a new one on me but first published in 1995 and most recently updated just a couple of years ago. Four of its 46 pages form the core, the rest being index and appendices with additional explanation. Its stated aim is to "establish a common set of criteria for disaster management, emergency management, and business continuity programs".
More disaster contingency links

New Orleans disaster predicted

An article published last year by the Natural Hazards Center effectively predicted the New Orleans disaster currently plastered all over our TV screens. What if Hurricane Ivan had not Missed New Orleans? describes with uncanny foresight the damage and disruption that would ensue if the levees were breached and a significant proportion of the population was unable to evacuate due to lack of transportation. There are some hard lessons here for contingency planners everywhere. Global warming undeniably changes the threat horizon for anyone located near the sea.
More contingency planning links

Slow patchers hit by worms

Systems at CNN, ABC, the New York Times, DaimlerChrysler and others were reportedly either hit by the Zotob-family worms or were taken offline to apply the Microsoft patches. The decisions about whether and when to apply security patches are especially difficult in the case of critical business systems. It sounds like some organizations either didn’t get the right answers from their risk assessments or simply fouled up implementing the patches. However their contingency plans (presumably at some point involving the command ‘apply those **** patches, NOW!’) seem to have limited the damage, so far, although companies that were infected with Zotob now have to deal with the threat that their systems may perhaps be 0wn3d with keyloggers and other nasties quietly doing their stuff.
More change management resources

Contingency plans in action

I'm waking up this morning to news of three contingency situations. First of all, an Airbus A340 aircraft failed to stop on the runway on landing at Toronto. The Air France emergency evacuation procedures worked pretty much as designed with only relatively minor injuries, we hear.
Secondly, the space shuttle crew are about to undertake a 'delicate task', cutting away some ceramic spacer strips protruding between the shuttle's tiles using a makeshift tool. The tool and cutting process are themselves the product of a well-rehearsed contingency process (the Apollo 13 film is a popular case study for contingency situations).
Finally, today's Handler's Diary from the SANS Internet Storm Center recounts a power incident involving the partial failure of a standby generator and office UPS units. It seems the generator has insufficient capacity for the full startup load, and some of the UPSs were incorrectly installed by users, raising questions about the system design, installation and testing procedures.
Otto von Bismark said "Only a fool learns from his own mistakes. The wise man learns from the mistakes of others" - I'd rephrase the last part slightly: "The wise man learns from the successes and mistakes of others, and makes his own contingency arrangements."
More crisis management and contingency planning resources

Data recovery hardware

An interesting suite of products from Germany protects key system files against unauthorized modifications. WatchIT presumably takes a backup copy of boot files and other key data from the disk. If the files are corrupted (e.g. by a virus) or accidentally deleted, the originals can be restored in a flash. Sounds ideal for classroom and many corporate situations where users have a tendency to corrupt their own systems from time to time.
More contingency planning links

London cellphone network resilient under stress

The BBC is reporting that cellphone networks in London are coping adequately with higher-than-normal call volumes arising from the bomb incidents at lunchtime today. Cellphones have becomeas much a part of the critical national infrastructure as the "Plain Old Telephone System" (POTS). Wireless networks like their wired ancestors are designed with resilience in mind, including spare capacity, alternate routing and 'intelligent' real-time switching protocols. This is mostly to cope with the diurnal peaks and troughs of demand, partly for continuity through abnormal periods such as bombings, planned maintenance and unanticipated system failures.
More on crisis management and contingency planning

Security awareness on crisis management

We published a special NoticeBored Classic module on crisis management and contingency planning, inspired by the emergency services' amazing response to the bombing of London last Thursday, along with a special newsletter. [These materials are no longer online]
More crisis management and contingency planning links

Patch Tuesday

Yesterday was 'patch Tuesday' meaning that millions of PCs running Windows Update are slavishly downloading the latest patches from Microsoft. The explanation of "cumulative security update for Internet Explorer", just one of this month's patches, indicates that unpatched PCs accessing 'malicious Web pages' could be completely compromised by bugs in IE's handling of DHTML and URLs, potentially giving an attacker 'complete control of an affected system' through 'remote code execution'. In case you missed it, this important snippet of information is buried under the (normally unexpanded) vulnerability details section of the detailed bulletin accessible from the information page about the fix included in the latest set of patches ... how many of us bother to follow the trail through three web pages? What's more, today's Handler's Diary at SANS Internet Storm Center (which we blogged yesterday) reports that "A proof-of-concept exploit for this vulnerability is already publicly available from FrSIRT. The availability of the exploit is likely to increase the severity of this patch for most organizations.", a point which Microsoft neglected to mention explicitly. (FrSIRT notice here)
Watch out for a forthcoming NoticeBored security awareness module on 'security in information systems development' which will mention the patching treadmill as a contingency measure following the release of buggy software.
More Internet security resources

Microsoft's approach to incident response

A Microsoft paper gives the inside track on how they deal with infosec incidents.
More incident management and contingency planning links here

The Oops List

The Oops List is a collection of images of (mostly) aircraft disasters. Warning: these are truly graphic images - not much blood and gore as such but undoubtedly passengers or crew were injured or killed in at least some of them. A few look like fakes or set-ups but, subject to any copyright restrictions, they would make fascinating slides for your contingency planning presentations.
More contingency planning links here