IT Projects: Expert Witness, CIO, In-House Counsel & Litigation Forum

Seven Key Lessons to Keep in Mind When Communicating an IT Failure

Posted March 4th, 2008 by Jonah Paransky
ttp://www.stacksafe.com/blog/2008/03/04/seven-key-lessons-to-keep-in-mind-when-communicating-an-it-failure/

IT Failures Happen.

While business and customers often expect 100% uptime, they don’t always receive it.

When failures happen, often the quality of the communication process does not match the technical resources brought to bear to solve the problem.

Here are seven key lessons to keep in mind when communicating an IT failure.

1. Have a communication plan in place and ready to go

All IT service delivery teams should maintain an active business continuity or disaster recovery plan. The time to develop your communications plan is not during the outage or service failure when the entire organization if focused on dealing with the problem at hand. Any credible continuity planning process should include the development of canned messages and a communications playbook that can be used during the failure. Does yours? If not, start the process of updating it as soon as possible.

2. Direct Communication with your customers is the number one concern

Customers would much rather hear about a problem directly from you, rather than from the media or discovering the problem themselves. If you are experiencing a major service failure, let your customer base know ASAP.

IT failure at Heathrow T5: What really happened

April 7, 2008, by Michael Krigsman @ 7:48pm, See http://blogs.zdnet.com/projectfailures/?p=681

Anyone having the misfortune to pass recently through Heathrow’s new Terminal 5 knows the huge project debuted as a case study in full-blown systems failure. Most reports have focused on stranded customers, a crazed supermodel, and Queen Elizabeth’s grand opening speech, which called the terminal “a 21st Century gateway to Britain.”

Here’s my rundown of the major failures and why this debacle happened.

Why this blog? Why now? Why YOU?

“IT Projects: Expert Witness, CIO, In-House Counsel & Litigation Forum

... engaging IT best practices, turning around IT projects, & understanding IT litigation issues” http://blog.wsrcg.com

Despite great changes and/or advances during the last 20 years in the shape of the world (it’s flat again), science, medicine, publishing, distribution, communications, the price of gasoline, the world-wide web and 9/11, two related areas of technology have failed to keep pace: IT system project success and the IT Law that supports it.

Did you know that:

1. The success rate for software development/implementation projects has not exceeded 35% (per the Standish Group’s “Chaos Reports”) during that period?

2. Scrapped software projects accounted for $55 billion in the USA – just a few years ago (per the National Institute of Science & Technology)?

3. The promised cost, schedule and functionality benefits from outsourcing IT have not been realized in 50% of the North American & European companies that use it?

4. Lawsuits between many large software companies, their Fortune 500 customers, and the giant IT consulting/systems integrator firms have grown from virtually nothing into a steady stream of specialized litigation?

Is there nothing we have learned during the past 20 years to improve the odds of a successful implementation that meets functional, quality, cost & schedule requirements and ROI, tolerable risk levels & stakeholder expectations? 

Just One More Time, "Why Do Systems & Software Projects Fail?”

In this new millennium, it is still true that approximately 29% of all large-scale systems projects are successful, 53% are challenged (with average overruns of 84% in time, 56% in dollars and only providing 67% of the required functionality), and 18% are scrapped and written off altogether. Although these stats show a small improvement over the statistics from a decade ago, and even with all the reported advances in technology, methodology, and software development and implementation, this is a poor showing for the Information Technology Industry [Source: Standish Group “Chaos Reports”].

For the past 36 years I have been a computer systems consultant and computer crisis consultant, including 20 years as a large-scale systems project turnaround specialist and an expert witness in large-scale software and systems implementation failures in the United States, Canada, the Caribbean, Europe and Asia. If I have learned anything, it is the following:

1. Virtually all systems failures stem from the same handful of root causes and lack of “-abilities” (e.g., cap-ability, respons-ibility, suit-ability, etc.). And this is true everywhere in the world!
2. All parties in a large scale systems or software project failure or dispute typically share some of the blame, but with some bearing much more blame than others for causing the failure(s) to happen, allowing them to happen, or being unaware that they are happening – with each party contributing anywhere from 0%-100% in each failure category.
3. Many, if not most, of the problems (sand their causes) could have been avoided, mitigated, or managed if the parties followed a few simple, but not easy, rules.
4. Putting in proper project policies, procedures, resources, monitoring metrics, tools, and project team communications to address these root cause areas could have either: (1) allowed the project to be successful in the first place if they were implemented and used from the outset of the project, (2) been used to turn around a runaway project midway if not originally implemented.
   
   Alternatively, determining how and how much each party contributed to these root causes is extremely powerful in helping judges, juries, and arbitrators assess the comparative guilt of the parties in litigation or dispute resolution for failed systems.

I developed the chart below to summarize my findings regarding why systems and software projects fail. This chart has been published in several peer-reviewed scholarly journals with relatively few changes over the years because of the persistent applicability and relevance of these root cause failings through today.

Please send me your experiences and insights on Why Systems Fail -- and What to Do About it?

More Software Bugs: 2003 - 2005

Trading on a major Asian stock exchange brought to a halt in Nov. 2005, reportedly due to an error in a system software upgrade. The problem was rectified and trading resumed later the same day.

May 2005 newspaper article reported that a major hybrid car manufacturer had to install a software fix on 20,000 vehicles due to problems with invalid engine warning lights & occasional stalling. In the article, an automotive software specialist indicated that the automobile industry spends $2 billion to $3 billion per year fixing software problems.

Media reports in Jan. 2005 detailed severe problems with a $170 million high-profile U.S. government IT systems project. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. In March 2005 it was decided to scrap the entire project.

July 2004 newspapers reported that a new government welfare management system in Canada costing several hundred million dollars was unable to handle a simple benefits rate increase after being put into live operation. Reportedly the original contract allowed for only 6 weeks of acceptance testing and the system was never tested for its ability to handle a rate increase.

Millions of bank accounts were impacted by errors due to installation of inadequately tested software code in the transaction processing system of a major North American bank, according to mid-2004 news reports. Articles about the incident stated that it took two weeks to fix all the resulting errors, that additional problems resulted when the incident drew a large number of e-mail phishing attacks against the bank's customers, and that the total cost of the incident could exceed $100 million.