Confirm? - Conform!

Confirm? - Conform! Adam Donlin, SE3H

Introduction

It's a shocking fact that nearly 75 of all major software projects either fail to reach completion or the software produced by them never gets used. Every year, millions of insert your favourite currency here is thrown down the drain as projects run aground or just explode in mid-air!

One of the most recent examples of such a failure was the development of the CONFIRM travel industry reservation program. Over a 3 year project duration, nearly 125 million was invested in a project that was never completed. There follows a summary of an article, by Effy Oz, in which I outline the development of the CONFIRM Information System (IS) and subsequently outline the lessons that should be learned from it's failure. The report is concluded with my own short critique of Oz's article, conclusions and viewpoint in general.

The CONFIRM Story

The inception, design and development of CONFIRM spans a fairly wide period of time and is perhaps best addressed by examining the chronological ordering of events. The story begins in early 1987 when American Airlines Corporation (AMR) discovered a gap in the combined Airline, accommodation and rental market. AMR found an imbalance in the ratio of the number of hotel reservations that were made through a central clearing house, some 20 , in comparison to the number of airline bookings, approximately 80 , made through an equivalent system.

In March of 1987, a special spinoff company of AMR, AMRIS, was setup and approached the major hotel chain, Marriot, with the CONFIRM proposal. The body of the CONFIRM proposal consisted, mainly, of a series of promises that AMRIS would, for a fee, construct a system that ``would be superior to any current reservation system, while not more costly to use''.

Marriot and a series of other companies, including Budget Rent-A-Car and the Hilton Hotel Group, entered into negotiations with ARMIS and in the October of 1987, the partners formed the Intrico Consortium and then began to plan the detailed functionality behind CONFIRM. Eight months later, AMRIS announced that it had entered it's initial design phase, which was originally planned to take a further seven months and would be followed by a 45 month development and construction phase.

The Intrico agreement was a contract that formally set the roles of the 4 partners. These roles were, firstly, that AMRIS would be the ``Managing Partner'' taking responsibility for all design and development whilst the remaining partners would each appoint special ``interface'' teams to provide any relevant input to the project.

The design effort continued throughout the remainder of 1988 and the early parts of 1989. AMRIS produced a draft base design in December of 1988 which, when presented to Marriot, received a dubious reception. Marriot claimed that the functional specifications in the base were not adequate, a fact that was later backed up by AMR. During this design phase, the cost of the project was increased from an original 55 million to 72 million. The partners didn't withdraw from the project, however, as AMRIS produced some projected financial documentation that showed the project was still of value. Later events showed the financial statements, that had convinced the partners not to withdraw, were fabricated. The design ``proper'' appears not to have been announced complete and as late as February, AMRIS was still making design decisions even although it had moved onto it's development phase.

Marriot and Budget began to express concern with regards the progress of the project in mid 1990 and despite reassurances from the AMRIS management that the project was on schedule, the two companies feared the AMRIS management was ineffective. AMRIS completed the first development phase shortly after, although they refused to release any deliverables to Marriot when requested.

From then on, the overall development process on CONFIRM seriously broke down. AMRIS had previously employed unethical and strong-arm tactics with it's employees and as a result, more than 50 of employees had either requested re-assignment, been re-assigned or dismissed. In 1991, AMRIS produced a re-plan with a new price tag of 92 million and, despite all the warning bells, Marriot and partners continued with the project.

In October 1991, AMRIS's president resigned and by April 1992, AMRIS finally admitted the scale of the problems it faced. A letter to the other three partners, stated that AMRIS would be unable to complete the project on schedule and offered a full refund to any partners who wished to withdraw. A tangle of accusations followed and were given voice in various lawsuits lodged by and against AMRIS who eventually reached out of court settlements with all the partners for around 160 million. Initial reports suggested that the damage could have reached nearer 500 million, had the cases gone fully to court.

The Post-mortem

Systems fail all the time and, in the majority of cases, bad design can be shown to be the source of the problem. The commonality of failure has meant that many cases sidestep widespread attention and only the high order ``disasters'' receive attention. CONFIM was placed under public scrutiny for one reason, the amount of resources, that it managed to consume, without reaching any degree of completion.

With public attention, comes an inevitable search for understanding and the main questions that were raised after the CONFIRM failure were as follows.

Firstly, in the event of a problem arising, should the client be notified? If so, how serious should the problem be to warrant the notification of the client. Finally, If unethical decisions are made by a higher echelon, is it the responsibility of the individual to raise an exception? Many of the questions raised call upon the professionalism and ethics of the people and organisations involved which, itself, raises the question, what is professionalism?

One definition of a professional is ``an expert whose services are required because of increase in technological or other speciality demands'' but computer systems engineering is a wide and varied field. Applying a generalised definition of professionalism may not guarantee the desired quality of product. In fact, the situation may arise where an individual is in a development situation where an employer has one set of professional values and the individual is also a subscriber to the values of a professional organisation. A double set of values may work to complement each other but, when push comes to shove, may do the exact opposite, adding fuel to the fire... Many professional computing organisations exist, each with their own variants of values and it is far from uncommon for an employer to have a customised code of conduct.

With these questions in mind, the following are a series of principles that aim to reduce damage in an IS development.

Managers should be realistic when determining project schedules and plan in enough slack to accommodate reasonable delays. As a rule of thumb, it is best to avoid taking shortcuts or beginning work on another phase of the project when the phases it depends on are not yet complete. The project schedule should be viewed as a part of the contract with the client, and given a suitable status as such. If a problem arises, it is unwise for executives to attempt to make covering blanket statements as they often just become more fuel for the fire. In the long term, dishonesty and in-accuracy will only serve to damage your own reputation.

Managers play a large part in shaping how a project will progress and grow, but the input of employees is equally valid. Therefore, when an employee realizes a problem exists, they should communicate with the management structure who should, of course, listen.

Clients should play a central role in their system. For example, they provide the ``concrete'' that the development managers have to work with. If the concrete is of the wrong consistency, then the end result may be weak or useless. Clients should therefore communicate, clearly, their requirements, realising simultaneously that change will invariably result in delay. Clients should also pay close attention to the progress of their product, watching for any tell tale signs of problems. If clients are unable to perform any of these functions in-house, then it makes long term sense to draft in a consultancy firm to serve as a bridge between client and developer.

The above conclusions are drawn by analysing commonalities between documented failures. There is no real detailed data available to base solid empirical analyses on for reasons of client/developer confidentiality.

Analysis

In general, and on reflection, I feel that Oz has produced a comprehensive breakdown of the events that lead to the CONFIRM failure, a non-trivial task considering the lack of available data and the unreliability and mis-focussed nature of what was available from press clippings and court proceedings. When addressing Oz's conclusions and principles, however, I find I have some doubts regarding whether some of the conclusions are valid and whether certain aspects warranted more attention than they received.

To begin with, she mentions that ``management should be deeply involved in the progress of large-scale projects'' without explicitly determining how this would solve any problems. Conversely, I believe that a deep management involvement would actually be counter-productive and impractical. There is a danger of ``too many cooks spoiling the broth'' and she fails to take account of the level of in-depth knowledge required for such deep involvement.

Oz mentions that it is wrong to ``start phase n before completing phase n-1'' and while this is quite true in the case where development is done in a linear progression from state to state. In non-trivial design and development scenarios, there is a high likelihood of using a non-linear development model, therefore having to manage parallel development processes. Imposing strict linearity in large projects would more than likely be counterproductive and restrictive. Perhaps a more suitable suggestion would be to examine carefully the interdependencies within the project development, ensuring that interdependent states are processed in the correct order.

A major area that I feel was not suitably addressed in the original article was the dilemma of the individual. Oz seems to go to great pains to spell out in great detail exactly how management should behave in terms of existing codes of professionalism. However, to all extents, she fails to offer any resolution to the moral dilemma that faces the individual. Management shape and lead a project, but it is a bad manager who fails to address the needs of their staff, and in such situations, a project will invariably suffer. I believe that the article would have been better served if Oz analysed the possible routes available to the individual, with the same fervour as she pursued a management resolution.

Rules and regulations are all fine and well, in theory, but often break in real world practice. The solution to this problem, I believe, involves a careful analysis of the problems, giving due attention to all the players.

The last major area I feel could have been covered by the article were mechanics for ensuring more information was available for better understandings of project failures. Oz takes the time, at various points in her article, to assert the lack of information on which to give her arguments a solid basis. I feel that highlighting the problem only serves to complete half the job, the second half being the proposal of solutions to the problem. Possible solutions to the information problem could involve imposing legislation and the setup of specialised ``postmortem'' boards with the relevant authority to analyse the remains of a dead project.

Having taken such care to highlight the lack of data, I feel that Oz has actually discredited some of her arguments but basing them on a series of conjectures and ``recommended practices''. An argument based on conjecture is, after all, not really an argument at all.


Adam Donlin / adam@dcs.ed.ac.uk