Starting out Right:
Measurable Requirements
Ian Alexander
Independent Consultant
Summary
Systems engineering projects that involve large amounts of software continue to run a high risk of failure. There is good evidence that many problems begin right at the start of such projects, during the requirements phases. This paper argues that while elaborate metrics may not be appropriate so early in the life-cycle, simple metrics can give clear warning of risks ahead. These correspond directly to the steps in an ideal scenario for starting out right in a project.
1 Introduction: How Not to Begin
The phrase 'Requirements Engineering' has quite a ring to it, but perhaps there is really nothing very technical about its basic purpose, which I will express in the less glamorous phrase 'Starting out right'. The situation today for system engineering projects is still quite insecure, from all points of view.
There is, first, personal experience. How do such projects look? How is it to work on one? I see people working hard, usually long hours and often in uncomfortable conditions. I see managers often not quite grasping the situation, and salesmen willing to launch risky projects. I see software tools and consultancy being bought in, often long after they could really help. And there is 'dilution': problems which are quite evident to the people on the ground become paler and paler towards the top, if not completely invisible.
Secondly, there is the evidence of the newspapers: a steady stream of big-name projects that are going terribly wrong. For example, on the 24th of March 1999, Ian Burrell in the Independent wrote:
A 'too ambitious' £77m private contract to install the system at the xxx has led to months of delays…
The National Audit Office said that government departments should carefully consider whether such computer projects were achievable even where prospective suppliers made enthusiastic bids for the work. xxx's computer project is already lagging 14 months behind schedule…exacerbated by the problems of relocating xxx's offices…
The aim of the project … was to switch from a paper-based to a computer-based system to speed up decisions…[but] problems with the computer project began after it was decided to abandon plans to use existing Information Technology packages and instead introduce tailor made software.
[There] could be more problems if the timetable slipped further because most of the limited software in use was not yet year 2000 compliant. David Davis, chairman of the Commons Public Accounts Committee said "Whilst there has been substantial transfer of risk to the contractor, ultimately if the project is delivered late, or not at all, the taxpayer will foot the bill".
Well, the story is only too familiar, and the department concerned is probably no different from many other organisations in its approach to large systems projects.
Thirdly, there is ample academic and business research to document the problems and risks. The Standish Corporation (http://www.standishgroup.com) is justly famous for its clear reports of continuing disaster. One of its strongest findings is the high percentage of problems that are related to requirements:
Project Challenged Factors % of Responses
* 1. Lack of User Input 12.8%
* 2. Incomplete Requirements & Specifications 12.3%
* 3. Changing Requirements & Specifications 11.8%
4. Lack of Executive Support 7.5%
5. Technology Incompetence 7.0%
6. Lack of Resources 6.4%
* 7. Unrealistic Expectations 5.9%
* 8. Unclear Objectives 5.3%
9. Unrealistic Time Frames 4.3%
10. New Technology 3.7%
Other 23.0%
Source: The Standish Group
* Factors related to requirements
What are the key elements of the scenario that leads step by step to banner headlines of 'IT Chaos'? The NAO identifies two:
· failure to 'consider' whether goals were achievable
· suppliers (and clients?) over-enthusiastic
and the watchdog identifies two more:
· change of plans
· failure to manage risk realistically.
As it happens, the paper published in the same issue a 4,600 year old example of the same scenario: the great circle at Stonehenge. The archaeologist Aubrey Burl argued that far from making epic trips dragging giant stones from Wales, the builders simply dug up stones carried by an ice-sheet. Unfortunately:
At Stonehenge, the discoverers [of the stones] may have ambitiously planned a concentric circle for the 83 holes, but when the last bluestone was unearthed and the countryside scoured, no more were found … [and] the scheme was modified into a less impressive single circle of about 57 stones.
In both the ancient and the modern projects, it seems there was no feasibility study, and certainly no accurate assessment of the risks. Enthusiasm held sway. And when the plans inevitably changed, the already great problems were compounded.
What is to be done? The message has to reach the planners, managers, consultants, and salesmen who call for ambitious projects. Why do people expect to be able to move offices, re-engineer their business processes and run major automation projects simultaneously? Any one of these is a major task carrying serious business risks.
Our message, then, must not be overly technical. There are numerous technical approaches available to help people engineer their requirements better, but first, project initiators and risk-takers need to be encouraged to look for help. Perhaps what such people most need is a simple list of warning signs, and some equally short suggestions as to what to do about them or, better, to pre-empt them. Here is my shortlist:
· Identify the Agents
· Fix the Scope
· Assess Feasibility
· Evaluate the Risks
· Baseline the Requirements
Each item is a warning if taken negatively: e.g. failure to identify the Agents could be disastrous; or an exhortation if taken positively. Let us consider each item in turn.
2. A Better Scenario
The list of virtuous steps forms a rough scenario for the goal of starting a project right: a top-level script for engineering a project's requirements. The overall goal is the usual one, to deliver what the customers want, on time, and to budget. Here, we will look only at the first subgoal in that picture, namely, creating an agreed and realistic set of requirements.
Each step is described with the headings:
· Negative Side: what a project can expect if the step is not taken.
· Positive Side: the benefit to a project from taking the step.
· Simple Metrics: what to look for. Each metric is scored simply Pass or Fail; any failure may be critical to a project, but a project with three or more failures is certainly a risky proposition.
The methods mentioned are documented in the References section.
2.1. Identify the Agents
Negative Side
It is impossible to define what a system is supposed to achieve for its users if we do not know who those users are. Lip-service is often paid to the need for a User Requirements Document but in practice this is often a thinly-disguised System Requirements Document or even a Design and Interface Specification.
Ian Graham's SOMA method divides Agents into (human) Actors directly involved and playing a part in achieving the desired goals; External Agents, indirectly involved (whether human or machine); and Systems, machine agents involved with the Actors in achieving the goals. The Scenario Plus method uses the same approach.
Positive Side
The stakeholders who are prepared to attend meetings to define a future system are likely to include at least some of the Actors and External Agents in that system's context. They are probably ideally placed to put together an Agent Interaction model. This is a simple object-oriented model, readily expressed as a diagram, which indicates the interactions (Ian Graham calls them 'Conversations') that achieve the stakeholders' goals. It is a modern descendent of the Context Diagram.
Simple Metrics
Are the Actors (roles), External Agents, and Systems involved clearly defined?
2.2. Fix the Scope
Negative Side
Developments with undefined scope are doomed to difficult meetings where one stakeholder tries to have something included while another tries to have it excluded. If the stakeholders have roughly equal power and can enlist allies, such a debate can continue throughout a project, causing endless replanning and zigzagging between conflicting objectives.
Positive Side
Scope is readily indicated as a freehand boundary drawn onto an Agent Interaction diagram. Is maintenance part of the system or not? The boundary will make it clear. Any interaction that crosses the boundary means an external interface. Any system or agent outside the boundary is assumed to be already present, or at least, someone else's responsibility.
Simple Metrics
Is there a definite of which roles and goals are included and which are excluded?
Is this scope agreed by all stakeholders?
2.3. Assess Feasibility
Negative Side
Projects which go ahead without knowledge of their feasibility are assuming that the technical, commercial, and other risks are negligible.
Positive Side
Every project needs to know how long work will take, how much it will cost, what resources will be needed, and whether those resources can possibly be provided. The earlier that an accurate estimate becomes available, the better. Richard Stevens in his book 'Systems Engineering' argues persuasively that projects should go through a succession of gates, so that cost and risk are known with enough confidence for each level of spending to be undertaken safely.
The European Space Agency's SOCRAT approach takes each function believed to be necessary, and seeks to calibrate the effort required for it against previous projects. The required work can be predicted with good accuracy if most of the functions can be matched with equivalents on completed projects (especially if costs can be interpolated rather than extrapolated).
Various other methods have been proposed, such as Function Point Analysis for software. These methods cannot be perfect, but are valuable for giving an indication of the size of a problem. Technical feasibility is likely to require more direct methods of assessment, such as prototyping.
Simple Metrics
Is there a report that evaluates the feasibility of completing the project successfully?
Does it consider all the major issues?
Is the cost known to within 20%?
2.4. Evaluate the Risks
Negative Side
If the risks are unknown they are likely to overwhelm the project. For example, the failure of the $25m software for Denver Airport's baggage handling kept the airport closed for over a year at a cost of at least $200m.
Positive Side
The evaluation of risk goes well together with the analysis of goals into individual tasks, and hence into requirements and constraints. A key question to ask is 'what can go wrong here?'. This question identifies Exceptions (for instance, a dog or a fly triggers the burglar alarm), and hence to the analysis of the goal 'to handle this exception'. The result is a hierarchy of tasks, where exceptions form additional branches to the 'normal' tasks. These exceptions will have to be handled in the product, whether that is hardware, software, or both (as in the burglar alarm). Scenario Plus offers a simple and effective way of modelling tasks and exceptions.
Similar thinking can be applied to the implementation processes of the development project itself, resulting in a hierarchical hazard analysis for the project (as opposed to the product). Michael Jackson says simply that the care taken on a project should be proportional to the risk of a disaster times its size.
Simple Metrics
Is there a report that lists all the major risks to the project?
Are the risks realistically considered?
Does the project know what it would do to handle each risk?
2.5. Baseline the Requirements
Negative Side
If different stakeholders have differing opinions as to what is to be developed and what is being paid for, disputes and misunderstandings are inevitable. Users may feel they can slip changes in unnoticed, and developers may feel they can ignore some of the requirements as they may not be missed.
Positive Side
A baselined set of requirements gives developers confidence that work can proceed without interruption. It gives users and customers confidence that they will get what they want (or at least, what they have asked for). And when changes come, there is a precisely defined baseline to move forwards from.
Simple Metrics
Is there a reviewed, agreed, and baselined set of requirements?
3 Conclusions
System engineering projects continue to fail much too often. The problem seems often to be related to poor requirements engineering at the start. A series of simple and well-understood steps are suggested as a basis both for starting projects off better, and for identifying projects that need urgent attention. These steps are to: Identify the Agents, Fix the Scope, Assess Feasibility, Evaluate the Risks, and Baseline the Requirements.
Projects will not suddenly become risk-free if the requirements engineering steps listed here are taken. There is no 'rocket science' here. But perhaps the problems that do arise will be smaller, less frequent, and better prepared for. Many will appear as normal tasks and challenges: project plans will have allocated realistic resources for them, and project managers will be able to react to them quickly and appropriately. As for the sort of project discussed in the Introduction, perhaps rather more of them will be cancelled when their feasibility studies and risk analyses land on Managing Directors' desks.
References
Scenario Plus
http://www.scenarioplus.com
SOCRAT
http://easyweb.easynet.co.uk/~iany/consultancy/socrat.htm
Software Metrics Association (Function Point Analysis)
http://www.uksma.co.uk
(SOMA/SOMATiK)
Requirements Engineering and Rapid Development
Ian Graham, Addison-Wesley, 1998
http://www.bezant-ot.com
Software Requirements & Specifications
Michael Jackson, Addison-Wesley, 1995
http://www.awl-he.com
Standish Group
http://www.standishgroup.com
Systems Engineering, coping with complexity
Richard Stevens et al., Prentice Hall, 1998
http://www.complexsystems.com
7