Evaluation of Public-Private Partnership Projects: Initiatives and Lessons Learned From a Survey of Ex-Post Evaluations

By

Thematic Working Sub-Group on Evaluation of Public-Private Partnerships

Editor: Mehmet Uzunkaya

Summary

This paper has two main objectives: first, drawing attention to the importance of ex-ante and –in particular- ex-post evaluation of public-private partnership projects (PPPs), which have become a common alternative to traditional public procurement in both developed and developing countries; and second, presenting the findings of a study conducted by the PPP Thematic Working Sub-Group launched by the European Evaluation Society (EES). While properly designed and implemented PPPs offer advantages over conventional procurement, complexities inherent in PPPs and risks involved make them prone to sub-optimal resource use or even failures. The delicate balance between public and private interests, as well as among costs, benefits and risks call for careful evaluation of PPPs, ex-ante and ex-post, at both project and program level. It seems, however, that evaluation of PPPs has not been given the desired level of attention, particularly in the developing world. To contribute closing this gap, a Thematic Working Sub-Group on Evaluation of Public-Private Partnerships was established within the EES as a part of a Thematic Working Group (TWG) initiative launched during the 10th EES Biennial Conference in Helsinki in October 2012. The ultimate output of the sub-group was set forth in the annual work programme, a discussion paper, which reflects findings and lessons learned from a sample of ex-post evaluation studies.

The main findings of the paper can be summarized as follows: In the preparation stage, PPPs bring complexities as compared to conventional public procurement, but costs incurred due to these complexities can possibly be compensated by benefits in the following stages. A common finding in this respect is that PPPs are more likely to be completed on time and on-budget, the latter being predominant when completion risk is transferred to the private sector. However, for toll-road projects, rather high forecasting errors and optimism bias in traffic projections are reported. A common criticism is the inflexibility of PPP contracts. Standardization in procedures and documentation is especially emphasized; there is also strong support that value-for-money considerations should be the policy focus for PPP promoters. PPP model-project matching, strong procurement and negotiation skills on the public sector side, transparency, competition, efficient risk transfer and maintaining a long-term vision are other key issues reported. On the other hand, their political popularity and off-balance sheet treatment makes PPPs prone to risks when they are used in otherwise unviable projects. Last but not the least, assuming identical economic benefits generated by the two methods (public procurement vs PPPs), the ex-post evaluation studies reviewed have not been able to draw a conclusion, ex-post, as to which method offers a lower whole-life cost.

Given these findings, the first and probably the most important lesson from the review is that public sector must build capacity (such as in finance, economics, law, project evaluation, procurement and negotiation) in-house or learn how to utilize external advisors in all aspects of PPPs. This capacity would not only ensure minimizing costly mistakes in PPP processes, but also would help public authorities to be able to objectively decide whether PPP is the right choice for the case in question relative to other procurement methods. Second, balanced risk transfer and sharing is critical. Failing to achieve an optimal point in risk allocation results in increased costs and inability to realize full potential. Strong feasibility studies at the initial stages is another must for successful PPPs. Creating and maintaining effective incentives through appropriate contract measures, legal arrangements and stakeholder management and cooperation is another crucial point. Well-designed incentives would reduce the effects of market failures and opportunistic behaviour while securing ecological, social and economic sustainability. Incentives should focus not only on the technical (legalistic and contractual) aspects of the partnership, but also on the qualitative aspects. This can be done through a greater emphasis on shared culture (shared vision, effective relationships, trust etc) cooperation and stakeholder management in order to foster more collaborative partnerships.Governments and donors should also ensure that wider societal goals are not undermined in the partnerships, given that PPPs are typically oriented towards financial issues. Flexible partnership options should also be retained by public authorities to relax constraints that could emerge due to changing conditions and sustainability goals. Affordability for the poor, especially in sectors such as water utilities, should be taken into account for the financial sustainability, as tariff increases are not uncommon for cost recovery. Alpha-risk (non-business or soft risk) should not be undermined with too much focus on beta risk (business or hard-risk). Ensuring competition, transparency, accountability and public consultation are other crucial factors. Finally, creation of a regulatory body which is independent from the preparation and implementation stages of PPPs and therefore can monitor, evaluate and regulate the whole process emerges as an important measure towards better PPPs.

  1. Introduction

PPPs have become common in the provision of public investments in both developed and developing countries. Increasing demand for better infrastructure on the one hand, and constraints on government budgets on the other, has boosted the use of this method. However, whether it has created value for money is a focus of intense debate among academicians and practitioners.

In a PPP, public and private actors combine their respective strengths, comparative advantages and interests so as to realize a public-interest venture on the basis of a long-term contract that shares the risks in such a way that each party assumes the risks for which it is better equipped to handle.

Interpretations of PPPs differ among countries and international institutions. For instance, while some organizations and countries consider privatizations as PPPs, others do not. However, all definitions recognize the long-term contractual nature of PPPs between public and private agents in pursuit of public mission. PPPs in the European Union, for example, refer to “forms of cooperation between public authorities and the world of business which aim to ensure the funding, construction, renovation, management or maintenance of an infrastructure or the provision of a service”[1].

That said, PPP arrangements span a wide array of sectors and methods. Traditionally, they are applicable to sectors such as energy, transport, water and sewage, where the return to capital investments is obtained during the operation period through user charges or -in some way or other- through government purchasing. Recently, PPPs have also emerged in the construction of public buildings (schools, hospitals, prisons, etc.), the provision of equipment and the delivery of environmental services (water/waste treatment, waste management)[2]. In addition to the sectoral diversity, PPP methods also span a wide spectrum, ranging from service contracts to build-own-operate contracts. In this range, private party involvement and responsibilities vary considerably depending on the contract terms, making contractual details crucially important.

Properly designed and implemented PPPs offer advantages over conventional procurement with three main motivations: investment in infrastructure, greater efficiency in resource use and commercial value generation from public sector assets[3]. PPPs promise acceleration of infrastructure provision, faster implementation, reduced whole life costs, better risk allocation, better incentives to perform, improved quality of service, additional revenue generation and enhanced public management[4].

At all stages of the project cycle, PPPs display sophisticated procedures and complex interactions among stakeholders. This reflects the need to reconcile differing objectives and incentives. While the public sector is mainly interested in generating net socio-economic benefits at micro and macro levels, the private parties are driven by the profit motive. These complexities along with the long term contractual nature of PPPs make them subject to a multiplicity of risks, especially for large projects. These need to be managed properly and strategically so as to realize all the benefits offered by the method.

Given the complexities inherent in PPP arrangements and risks involved, PPPs are prone to sub-optimal resource use or even failures, if they are not properly managed. Additionally, the delicate balance between public and private interests as well as among costs, benefits and risks call for careful evaluation of PPPs, ex-ante and ex-post, at both project and program level. It seems, however, that evaluation of PPP projects and programs has not been given the desired level of attention, particularly in the least-developed world and in emerging market countries.

To contribute closing this gap, a Thematic Working Sub-Group on Evaluation of Public-Private Partnerships was established within the European Evaluation Society (EES) as a part of a Thematic Working Group (TWG) initiative launched during the 10th EES Biennial Conference in Helsinki (October 1-5, 2012). The working group on PPPs is part of the Private Sector Evaluation TWG.

The group started its activities by identifying the group members and taking stock of the areas of specialization to make sure that the group adequately represents capacity that is consistent with the group aim. In this respect, the sub-group includes members from a variety of organizations throughout the world with diverse backgrounds on evaluation and PPPs. After the group formation, an annual work programme was drafted upon the contributions of the members and was finalized in August 2013.

The work programme sets forth as the ultimate output of the sub-group activities a discussion paper to be presented at the 11th Biennial Conference in Dublin as a collective product. The paper would aim at drawing lessons from a representative sample of ex-post evaluations on PPP projects towards finding the principles of successful PPPs and making methodological recommendations for better evaluations.

The sub-group collected evaluation studies from around the world to form the sample. As a general principle, evaluation studies from various levels and classes were searched, namely,

  • from developed, developing and least-developed countries,
  • greenfields and brownfields,
  • medium-large projects as well as those in which the special purpose vehicle (SPV) can be considered as a small-medium enterprise (if available)
  • successful, problematic and failed projects.

The search process resulted in evaluation studies, covering projects from a variety of countries and regions, including Europe-Central Asia, Africa, East Asia-Pacific, Latin America-Caribbean and Middle East-North Africa. The evaluation studies obtained were reviewed to extract lessons towards achieving successful PPPs. The results of the review effort and lessons learned from the sample of evaluation studies are presented in the following section.

  1. Lessons Learned From the Survey of Ex-post Evaluations

In this section, a sample of ex-post evaluation studies and cases from different parts of the world is examined. Other supplementary material was also reviewed to complement the analysis. The analysis starts with evaluation studies from Europe, conducted mainly by the European Investment Bank. Then, UK’s experience is examined, as UK represents a pioneering country in mobilising private finance inpublic services. UK’s experience is followed by a review of PPPs in Latin America, which raises the issue of sustainability. African PPP experience is examined particularly for the urban water services sector, among others, which provides valuable insights specific to the water sector and important for PPPs as a whole. A review conducted by the World Bank is another valuable study focusing on PPP performance in water utilities, this time on the global developing country scale. As one of the active countries in PPPs, projects in South Africa were particularly interesting and discussed in somewhat more micro level. Finally, a very recent evaluation conducted by the Independent Evaluation Group of the World Bankincludes in-depth evaluation of a large number of “targeted interventions”throughout the world by the World Bank Group, which aimed at improving the enabling environment and facilitating specific PPP transactions.

EIB’s first evaluation performed by Thomson and Goodwin (2005) includes in-depth evaluation of 10 PPP projects, (after a desk- review of 15 projects) which were then either fully operational or close to completion based on the Bank’s data and information. Bank’s standard evaluation criteria (Relevance/Efficacy, Efficiency and Sustainability) were used in the evaluation.

According to the study, the main reason for using the PPP method was to launch, “within a reasonable time”, investment programmes, which would otherwise be impossible under public sector budget. It seems that this intention was achieved, since the evaluation showed that the projects evaluated were mainly completed on-time, on-budget and to specification.

However, the PPP projects brought complexity to promoter public sector authorities as compared to conventional public procurement, especially in respect of contract negotiation and contractual relationships; mainly because of the differences between the frequencies of exposure by the public and private sector to such projects. The asymmetry between the quality of advice used by the public and private sectors is also mentioned.

When it comes to project implementation, there were cost overruns and construction delays in some of the projects; however overall, the study concludes that, ex-post, PPP projects are more likely to be completed on-time and on-budget. There is indeed a “complexity cost” associated with PPP projects, but this seems not to create a substantial cost disadvantage even in the present case, where contract standardization is not yet widely used. Measures such as contract standardization would reduce this cost further.

In terms of project operation, the study is not able to draw a conclusion as to whether long-term benefits of PPPs in operating and maintenance materialize, since the projects evaluated were generally in their first years of operation. In terms of revenues, traffic forecast errors turn out to be significant in real-toll road PPPs (realized traffic being below projections) at least for the early stages of operation. In these projects, pricing and “willingness to pay” seem to be critical issues, as any mismatch between them may either inhibit infrastructure use or cause renegotiation of the PPP contract, or even “buying out” the Provider’s rights by the Promoter. Revenue projections seem to be satisfied in projects that were based on service availability.

The study finds no evidence that private sector management and implementation skills are introduced to the public sector; on the contrary, the possibility of technical skills transfer from the public sector to the private sector is highlighted. In terms of innovation in design and operation, it is argued that it is too early to draw a meaningful conclusion. Lenders intervention, on the other hand, is found to produce better deals, due to the external discipline imposed.

Overall, the study finds that, as compared to conventional public procurement, PPPs are more likely to be completed on time. Additionally, when the completion risk is on the provider and with no change in the project definition, PPPs are completed on-budget. However, because of methodological constraints and the limited number of projects in the sample, the study cannot draw a conclusion as to which method offers, ex-post, a lower whole-life cost, given the assumption that economic benefits generated by the two methods (public procurement vs PPPs) would be the same. Thus, ex-post, the study cannot answer the question as to whether or when to use PPPs as an alternative to public procurement.

Another evaluation of EIB projects was conducted by Bain (2009) including 66 projects, construction of which were completed and operation started. The evaluation methodology includes three elements, first involving the analysis of the performance of projects on the basis of the Bank’s project and credit monitoring documentation. The second element includes a literature review on the lessons gained by other organizations globally from PPP projects. Third, a series of semi-structured interviews was conducted with the Bank staff to gain insights on the lessons learned from their PPP project experience.

According to the Bank’s project and credit monitoring documentation on PPP performance, 85% of EIB’s PPP projects completed within budget, which is considered as evidence of price certainty. This rate is found to be consistent with UK’s National Audit Office and HM Treasury’s findings of 79 % and 80%, respectively. Ignoring minor delays, 80% of the projects were found to be completed within time, again, consistent with the findings of UK National Audit Office (79%) and HM Treasury (88%).