Optimism Bias in the
Appraisal of Fire Projects
Research Report
Optimism Bias in the Appraisal of Fire Projects Research Report
March 2007
Department for Communities and Local Government: London
On 5th May 2006 the responsibilities of the Office of the Deputy Prime Minister (ODPM) transferred to the Department for
Communities and Local Government.
Department for Communities and Local Government
Eland House Bressenden Place London SW1E 5DU
Telephone: 020 7944 4400
Website: www.communities.gov.uk
© Crown Copyright 2007
Copyright in the typographical arrangement rests with the Crown.
This publication, excluding logos, may be reproduced free of charge in any format or medium for research, private study or for internal circulation within an organisation. This is subject to it being reproduced accurately and not used in a misleading context. The material must be acknowledged as Crown Copyright and the title of the publication specified.
Any other use of the contents of this publication would require a copyright licence. Please apply for a Click-Use Licence for core material at www.opsi.gov.uk/click-use/system/online/pLogin.asp or by writing to the Office of Public Sector Information, Information Policy Team, St Clements House, 2-16 Colegate, Norwich NR3 1BQ.
Fax: 01603 723000 or email: .
If you require this publication in an alternative format please email
Communities and Local Government Publications
PO Box 236
Wetherby
West Yorkshire
LS23 7NB
Tel: 0870 1226 236
Fax: 0870 1226 237
Textphone: 0870 1207 405
Email:
or online via the Communities and Local Government website: www.communities.gov.uk
March 2007
Product Code 06PPI04374
Contents
Chapter 1 Terms of Appointment 4
Chapter 2 Key Research Issues and Methodology 5
Chapter 3 Project and Sample Selection 9
Chapter 4 Collation of Cost and Duration Estimates at OBC and FBC Stages 11
Chapter 5 Analaysis of Deviations 28
Chapter 6 Cost and Duration Uplifts – Providing New Guidance 36
Appendix A – BCIS Data Analysis 39
Appendix B – Selected Bibliography 41
3
Chapter 1. Terms of appointment
1.1 In March 2005, DTZ Pieda Consulting was appointed through the Office of the Deputy Prime Minister (ODPM)1 to undertake research in optimism bias in the appraisal of regeneration, decent homes and fire projects.
1.2 The key aim of this study is to review and further develop Communities and Local Government’s/DTI’s methodological approach to accounting for optimism bias in the appraisal and approval of regeneration, decent homes and fire projects, and to provide numerical information and guidance to inform optimism bias adjustments, based on investigation of past experience of projects in these areas.
1.3 The objectives of the research were to:
(i) Review Communities and Local Government’s/DTI’s proposed interim methodology for accounting for optimism bias in regeneration, and advise on: a) any changes required to the approach following the research; and b) on the application to decent homes and fire appraisals.
(ii) Investigate the project records of the English RDAs (including LDA), English Partnerships, organisations delivering decent homes outcomes; and records associated with central government fire/civil resilience projects, to gather data on past optimism bias in projects. This should include optimism bias in costs, receipts/revenues, and (to the extent possible) project timetables and outcomes.
(iii) Use the information gathered at (ii) to develop average optimism bias “mark-up” and “contributory factor” data for costs, receipts, timetables and outcomes for a range of types of project at relevant approval point(s).
(iv) Present the data, and the finalised approach to their use in future project appraisals, in a Guidance Note suitable for dissemination to front line project appraisers in agencies and authorities.
1.4 This research report covers fire projects procured through the Public Finance Initiative
(PFI) only.
1.5 The case studies examined as part of this study are mainly Pathfinder schemes involving a variety of delivery outputs and activities. Many of the schemes currently underway have delivered tangible benefits for local fire and rescue services and their key stakeholders. However, due to the varied nature of these schemes, both Communities and Local Government and DTZ have found it difficult to draw general conclusions at this stage. Because of the difficulties involved, secondary performance data has also been used to augment the study. The intention is to build on the current study as more data and project records become available.
1 The responsibilities of the Office of the Deputy Prime Minister (ODPM) were taken over by the newly created Department for
Communities and Local Government on 5 May 2006.
4
Chapter 2. Key research issues and methodology
2.1 The Green Book (2003) significantly revised HM Treasury guidance as to how risk and uncertainty should be dealt with in public sector appraisals. Underlying these changes was the decision to ‘unbundle’ the discount rate, and set it at the lower level of 3.5 per cent real (the social time preference rate). Factors that were previously ‘bundled-up’ in the old discount rate are now to be dealt with explicitly and separately. Risk and optimism bias fall into this category, and the principles underlying the new approach are outlined in the main text of the new Green Book:
‘There is a demonstrated, systematic tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both private and public sectors … To redress this tendency, appraisers should make explicit adjustment for this bias.
Adjustments should be empirically based, and adjusted for the unique characteristics of the project in hand. Cross-departmental guidance for generic project categories is available, and should be used in the absence of more specific evidence. But if departments or agencies have a more robust evidence base for cost overruns and other instances of bias, this evidence should be used in preference.’2
2.2 Supplementary guidance and material annexed to the main Green Book text elaborate on these principles, and outline the techniques to be used in applying these rules. In particular capital expenditure, operating costs and benefits, and time profiles are addressed in a special note which links methodology to the generic evidence base established by the Mott MacDonald study Review of Large Project Procurement in the UK.3 The findings of the Mott MacDonald research were subsequently used in the Treasury’s August 2004 Quantitative Assessment User Guide, which guided practitioners through the standard mandatory spreadsheet to be used for the Value for Money (VfM) assessments submitted to central government departments for project approval.4
2.3 The Quantitative Assessment User Guide provides departments with a set of standard uplifts and assumptions to be used in the calculation of optimism bias. However it also notes:
‘Any estimates used should be informed by sector specific evidence and draw on cross-departmental experience where appropriate, but should be updated on a project specific basis. Through their Estates Agencies, a number of sponsoring Departments already maintain good record both of the movement in estimated costs of projects between the OBC and FBC stages, and particularly for conventionally procured projects, of subsequent changes to outturn costs (i.e. between the FBC and the completion of the relevant asset). Where information is not readily available Departments should determine what steps should be taken to assemble a strong evidence base upon which to draw their sector specific conclusions on optimism bias.’5
2 HMT (2003), p. 29.
3 Mott MacDonald (2002).
4 HMT (2004).
5 HMT (2004, pp.22-24).
5
Optimism Bias in the Appraisal of Fire Projects
2.4 This report outlines how the evidence base for fire related PFI projects has been assembled.
2.5 Based on DTZ Pieda Consulting’s experience with other public sector bodies, a five stage research methodology was agreed for all three study streams (regeneration, fire and housing). This is shown in Figure 2.1.
Figure 2.1: Study Methodology
2.6 For fire projects, the five stage research methodology consisted of the following activities.
STAGE 1: STUDY MOBILISATION
Inception Meeting
2.7 Following the inception meeting held on the 1st April 2005, we completed several tasks required to prepare the inception report.
6
Optimism Bias in the Appraisal of Fire Projects
Consultations
2.8 Initial consultations were held with representatives from the following organisations:
· Office of the Deputy Prime Minister, Central Economic Advice; and
· Office of the Deputy Prime Minister, Fire and Rescue Service Directorate.
Central Data Audit
2.10 An investigation of data currently held by ODPM and collated for the purpose of this study was completed. This included previous evaluation studies and OBC and FBC appraisals.
Review of Existing Optimism Bias Guidance
2.11 The study team also reviewed optimism bias guidance that has been produced by other central government bodies, including:
· Department for Transport;
· Department of Health;
· Scottish Executive; and
· Northern Ireland Executive.
STAGE 2: PILOTING/FIRST ROUND DATA COLLECTION
2.12 The key objective of the pilot stage was to provide an assessment of the quality of data available from relevant authorities. The key output of this stage of the work was a primary research strategy for the full data collection, which outlined the methods of data collection and proposed sampling strategy.
STAGE 3: INITIAL REPORTING
2.13 The stage involved the production of a research summary report, which was endorsed by members of the project steering group.
STAGE 4: FULL DATA COLLECTION AND ANALYSIS
2.14 The key objective here was to complete the fieldwork component of the study, and collate the responses in a database that protected the anonymity of individual programmes and projects. As agreed with the steering group, the work involved fieldwork visits to four Fire Authorities.
7
Optimism Bias in the Appraisal of Fire Projects
STAGE 5: REPORTING AND DISSEMINATION
2.15 There were to be two key outputs of the research.
· A research report for the ODPM; and
· A Guidance Note for the assessment of optimism bias in relation to regeneration projects.
2.16 An element key to the success of the project was the provision of workshops with both practitioners and funders.
2.17 The research was underpinned by the adoption of a simple five step-procedure for assessing optimism bias:
· Step 1: Selection of relevant projects to assess.
· Step 2: Collation of cost, duration and output estimates at OBC and FBC stages.
· Step 3: Analysis of the deviation between the estimates and actual figures.
· Step 4: Assessment of the contributing factors to the deviation.
· Step 5: Provision of guidance on using the findings of the research.
8
Chapter 3. Project and sample selection
3.1 It was clear from the outset of this project, that sample selection would be a crucial issue. As the DfT Guidance on optimism bias made clear, in statistical analysis, data should be a sample from a larger population, and the sample should represent the population properly.6 These requirements are ideally satisfied by drawing the sample by randomised lot. Randomisation ensures with high probability that non-controllable factors are equalised. A sample should also be designed such that the representation of subgroups corresponds to their occurrence and importance in the population. In studies of human behaviour, however, where controlled laboratory experiments often cannot be conducted, it is frequently impossible to meet these ideal conditions. This is also the case for studies of optimism bias and a different approach therefore has to be taken to sampling and statistical analysis.
3.2 Accordingly, the projects for the sample of fire projects were selected on the basis of data availability. The primary requirement was to identify projects that had sufficient information on estimated capital costs, revenue costs, works duration and outputs at the OBC and FBC level, and if possible actual figures at project completion stage. All of the projects examined in detail for this study were pathfinder or round one projects. There is a clear danger that experience and best-practice learned since the first waves of these projects means that observed levels of optimism bias might be higher than those projects assessed in Rounds 2, 3 and 4.
3.3 There is also an important issue relating to optimism bias assumptions for the public sector comparator. The PSC option is defined by HM Treasury as procurement through conventional approaches that use public funding (for example, letting a design and build contract for the construction of an asset, and then letting annual operating and maintenance contacts for the ongoing maintenance of that asset). So in addition to the PFI experience, the investigation would be looking for a sufficient quantity of projects managed under conventional procurement routes to underpin the creation of relevant public sector comparators (PSC).
3.4 The original research strategy was based on an in-depth review of four PFI applications. It was clear that given these samples sizes it would not be sufficient to provide enough examples to undertake a robust analysis for either costs or outputs. This issue has also affected work undertaken for other central government departments. Research commissioned by the Department for Transport into optimism bias7 had used cost information collected on 252 projects, including North American and European schemes
(172 road projects, 46 rail projects & 34 fixed link projects). In contrast, the Mott MacDonald study was confined by data availability to using a sample of 50 major public procurement projects as the sample.
3.5 The ‘population’ of Fire PFI projects available for study included 7 Pathfinder projects,
3 Round 2 projects, 1 Round 3 project, and 1 Round 4 project.
6 Department for Transport (2004) p. 19.
7 Department for Transport (2004).
9
Optimism Bias in the Appraisal of Fire Projects
3.6 The four projects studied in depth for this project included a scheme to provide bespoke training facilities for fire fighters (Project A), two schemes to rationalise and replace existing premises (Projects B and C), and a scheme involving the lease, maintenance and management of equipment (Project D).
3.7 It is important to note that since the inception of the Pathfinder projects, there has been a great deal of policy, technical and practical development in appraising and managing PFI schemes. In particular Communities and Local Government has made concerted efforts to spread knowledge, experience and best practice through such mechanisms as the regular practitioners’ workshops. In addition, the provision of procurement support through the 4Ps initiative, the growth in the number of private sector ‘PFI specialists’ and specialist advice from bodies such as Partnerships UK have changed the intellectual landscape for PFI projects and their potential managers. Similarly, HM Treasury’s introduction of a minimum PFI capital threshold of £20m is having an impact on the shape of projects now put forward for approval. This means that many of the lessons and early difficulties noted in the four sample projects are less likely to be repeated.