Title: Title of Report

FutureTest Webinar Report

Contents

Executive summary

1.Introduction

2.The Webinar

3.The webinar format

4.Q&A Discussion

5.Summary of key issues discussed:

6.Next steps and proposed way forward in researching equity and BER

Appendix 1Webinar Programme

Appendix 2Webinar Participants

FutureTest Webinar Report

Executive summary

The webinar highlighted a number of general lessons and guidance for good practice in VfM for BER:

Success in VfM assessment is connected to good programme design. This includes understanding the context in which reforms are being designed (including the political economy context) and drawing links between programme outputs, outcomes and results.

A sound theory of change, which identifies key assumptions, defines programme attribution and connects causal changes to broader results is essential if results are to be measured and VfM accurately counted.

Indicators should be clearly linked to programme work streams.

The sustainability of BER is a key element in VfM assessment. This requires a sound assessment of the impact of reforms on market and government systems, and can only be successfully undertake some time after the completion of a programme.

Within the World Bank Group, projects with a high development effectiveness rating were found to have had a higher value for money.

It should be recognised that measures to achieve equity may have an impact on programme efficiency. For this reason, equity measures should be developed to articulate clearly the value placed on these objectives.

VfM has a strong “work in progress’ nature. It involves a lot of experimentation and a great deal of patience is required. It takes time to learn from these experiences. In addition, there is a need to separate the Results Agenda from VFM: it is important to properly measure results, before applying VfM to these results.

There is clearly a strong interest in better understanding and applying equity measures within BER programmes. The report ends with suggestions as to how this work can be taken forward.

1.Introduction

On 28 September 2016, the Business Environment Reform Facility (BERF) hosted its first webinar. This focused on measuring value for money (VfM) in DFID-supported business environment reform (BER) programmes. This report provides a brief summary of the webinar, the issues discussed and the way ahead.

The webinar built on from a BERF Evidence and Learning Note on ‘How to Measure VfM in DFID’s BER Programmes’ published in July 2016 which provided practical guidance to DFID Country Offices (COs) on how to conceptualise and measure VfM in DFID’s BER programmes, in particular those that are delivered by the International Finance Corporation (IFC).[1] Most of DFID’s BER Programmes are implemented in partnership with World Bank Group. The Evidence and Learning Note reviewed the IFC’s recently developed investment climate VfM framework and DFID’s conceptualisation of VfM based on the 3E approach.

The Evidence and Learning Note presented a series of qualitative and quantitative VfM indicators from which DFID COs could choose in designing VfM metrics for BER programmes and mapped each of the key efficiency and effectiveness indicators to the IFC BER work streams. It also looked briefly at a fourth criterion: equity. It offered suggestions on how gender, one equity metric, could be measured as part of a VfM assessment.

Feedback on the Evidence and Learning Note from DFID COs and IFC officers identified a desire to better understand and apply equity measures when conducting a VfM assessment of BER programmes. Hence, this BERF webinar sought to flesh out this topic further and to identify areas and relevant topics related to equity, which could form the basis of further research following the webinar.

2.The Webinar

Following the publication of the BERF Evidence and Learning Note, BERF organised the webinar on 29 September 2016 on ’How to Measure VfM in DFID’s Business Environment Reform Programmes, including those delivered by the IFC’. The objective of the webinar was to share lessons learnt in measuring VfM in DFID’s and other donors’ BER programmes using evidence from a range of sources including BERF’s Evidence and Learning Note.

Appendix 1 contains the agenda of the one-hour webinar.

The webinar targeted DFID staff members, both in the UK and in COs, as well as other donor agencies, consultants and BER programme managers. About 24 people participated in the webinar –– see Appendix 2.

Before the webinar BERF’s Evidence and Learning Note on VfM in BER was circulated to registered webinar participants.

The Adobe Connect web conferencing platform was used for the webinar. Adobe Connect is best known as a virtual meeting tool with a strong security focus influenced largely by its work with governments and commercial entities. It has a flexible user interface and includes a range of features for presenters and participants.

3.The webinar format

Angela Strachan, BERF Evidence and Learning Coordinator, opened the webinar, welcoming participants and setting out the purpose of the meeting. She introduced the speakers and provided technical information on how participants can use the Adobe Connect web conferencing platform.

Kamal Siblini, Senior M&E Specialist and VfM Expert at the World Bank Group, gave the first presentation on ‘Measuring VfM in BER and Investment Climate Programmes: Challenges and Opportunities; The World Bank’s Perspective’. His presentation provided a practical framework for understanding and applying VfM in BER programmes and considered a wide range of challenges and lessons that can be learned from the field.

He described how VfM is calculated during project design, using budgets and results targets, and applied at project completion, using expenditures and actual results. VfM is computed differently depending on the type of indicator used. For example, measuring the impact of BER typically focuses on private sector compliance cost savings, while measuring the reach of reforms focuses on the number of private firms benefiting.

Among the challenges identified were the financial challenges, where financial management systems typically only allow project teams to budget by cost categories and not by programme component costs. In most cases, the high-level nature of those cost categories make it difficult to accurately determine programme costs. In addition, there are many challenges arising from cross-country BER programme comparisons. This includes variations in the availability of private sector data, the political economic context and the level of programme intervention and country preparedness for reform.

Finally, a range of lessons learned was presented. Mr Siblini stressed the importance of clearly defining attribution and preparing a context-specific theory of change in the design phase. VfM assessments can only be fully undertaken when results have been achieved––it is not possible to calculate VfM where data have not been collected (e.g., jobs quantity and quality, economic growth).

Ed Hedley, senior consultant from Itad, and Gulden Bayaz, independent VfM Expert, gave the second presentation, which was based on the BERF Evidence and Learning Note.

Ed Hedley provided an introduction to the DFID and NAO 3E conceptual framework for assessing VfM. To this framework a 4th E, equity, is now frequently added. Mr. Hedley noted that DFID uses quantitative and qualitative approaches to assessing VfM and that there is a trend in DFID programmes to develop VfM indicators which are linked to the programme logframe. Mr. Hedley then introduced the IFC approach to assessing VfM in BER programmes in terms of reforms introduced, outreach and compliance cost savings. The framework has been developed from the experience of 200+ ongoing and completed IFC programmes across the IFC’s main workstreams. Mr. Hedley then noted that the IFC is working on a similar VfM framework that seeks to develop benchmarks for other outcomes of BER, specifically cost of job creation.

Ms. Bayaz discussed some of the example cost-effectiveness indicators presented in the Evidence and Learning Note, including cost per job created and cost per pound of increased value of increased sales, which were mapped to relevant IFC workstreams. Ms. Bayaz noted that there is an interest from DFID COs in using ToCs to explain impact pathways in BER programmes and presented an extract of a ToC to explain how such a tool can be used to link reforms to expected headline programme results. She covered a number of expamples: for instance an evaluation of BRICK Kenya indicated that clearer and better-evidenced links in the programme ToC were needed to break down a number of key assumptions such as how and why regulatory reforms have an impact on large firms and whether these firms are creating jobs for poor beneficiaries.

Ms. Bayaz covered some of the feedback from IFC and DFID officials which emerged from the consultations for the Evidence and Learning Note:

-The need for greater disaggregation of project budgets and financial reporting.

-Better understanding of government opportunity costs (which is also a DCERD recommendation).

-Political economy analysis during programme design and implementation.

-Better understanding of who benefits from BER interventions (equity).This could include for example breaking down impact by income quintile of beneficiaries and gender. Also relevant is a deeper understanding of how reforms have affected women’s social position.

Finally, Simon White, independent consultant and coordination of the Business Environment Working Group (BEWG) or the Donor Committee for Enterprise Development (DCED) spoke and presented a range of preliminary thoughts on how equity can be measured in a BER VfM assessment. Simon highlighted the link between programme design and VfM assessments. While equity measures may be treated as a cost driver, because reaching poor, disadvantaged and marginalised groups can be more expensive than dealing with the general population, a well designed programme will integrate equity concerns across the programme. The models of inclusive growth, which focus on the pace and pattern of economic growth, provides a useful means to achieving this integration.

Four dimensions to measuring equity in BER programmes were presented for discussion:

Gender: this involves disaggregating data by sex (firm creation, employment), considering the impact of reforms on sectors where women are highly represented, and considering the extent to which the reforms have addressed power imbalances in the economy;

Informal economy: this includes the extent to which BER has supported the formalisation of informal firms and increased productivity of informal firms, as well as the impact of reform results on the formalisation and improvement of working conditions and arrangements;

Spatial: this involves an assessment of how BER results have affected people living in specific locations and measuring how reform results affect spatial pockets of poverty and the extent to which reform results have addressed the specific determinants of poverty based on location (e.g., land tenure); and

Sector: measuring results in sectors where poor women and men are highly represented (e.g., agriculture, garments, services).

In conclusion, Simon spoke of forthcoming work that is anticipated on this topic, and highlighted the need to consider equity measures within the context of DFID’s 2014Strategic Framework for Economic Development for Shared Prosperity and Poverty Reduction.

Following the three presentations, Simon White moderated a question and answer session.

4.Q&A Discussion

A number of questions were raised and responded to following the presentations. These are summarised below:

Question 1: Outputs are produced and can be measured based on economy, efficiency and effectiveness, and fit for purpose. Should we focus on Outputs when doing VFM?

Kamal Siblini: Agrees––this is a starting point. However, it is important to remember the issue of quality when thinking about outputs. Outputs occur within a specific context and it is important not to overlook this context. Thus, it is better to focus on results, being the changes to the system, than just on outputs.

Question 2: How practical are equity measures in the context of national BER programme?

Simon White: Local and sector-oriented BER typically provide an opportunity to look at reform results in a more nuanced and detailed manner, which includes the use of equity measures. National reforms can be general in nature (i.e., by definition they can be very broad) and this can sometimes make it different to achieve equity outcomes. However, this should not make it impossible to measure equity results across a range of BER programmes, regardless of their scale.

Kamal Siblini: As with all BER programmes, we need to get attribution right and theory of change right, before we started measuring VfM. Thus, it is important to get the design of programmes right if equity is to be achieved and measured.

Question 3: What is the role of VfM in achieving sustainability?

Kamal Siblini: VfM assessments take time. They are best measured later in the programme and often after its conclusion.

Gulden Bayaz: The indicator, jobs created and sustained employment, is a useful measure.

Ed Hedley: As with work in market systems, it is important to measure change after the programme to understand how reform has changed systems over time.

Question 4: We heard during the Evidence Note consultations that no DFID programme implemented by IFC had used the IFC-proposed framework yet (and that BRICK had plans to do so). Is any participant aware of such an attempt and the results they found?

Kamal Siblini: IFC began applying this framework in March 2016. A few have been done, but it is still early days. We are all learning about this as we go. We are still at the lower level of thinking and experience. We are not yet at the point where VfM becomes a decision-making tool for management. We are at VfM 1.0, and versions 2.0 and 3.0 are still to come.

Question 5: How can we discount political disruption and consider this in VfM of BER programmes?

Kamal Siblini: This is about the political economy. These are issues that have to be considered at the beginning of the project. The links between political stability and BER need to be considered at the early stages of the project. It is useful to monitor the political situation over the course of a project and to see how this affects BER results.

Question 6: Why don’t we measure changes in tax revenue and increases in the number of taxpayers?

Kamal Siblini:It depends on what we measure. Beyond cost-savings it is possible to measure improvements in firm management. Data on registration is relatively easy to measure. With tax rates, its easy to measure registration, but more difficult to measure how rates affect performance. With inspection it is possible to see how many firms are inspected and how this affects the firms. Thus, there is a need to define these changes in a broader theory of change and to define indicators within this.

Ed Hedley:Depends on the expected results the programme has been designed to achieve.

Gulden Bayaz:Tax payment can be included in an outreach measure.

5.Summary of key issues discussed:

The webinar highlighted a number of key issues. Some of these form general lessons and guidance for good practice in VfM for BER:

Indicators should be clearly linked to programme work streams.

Within the World Bank Group, projects with a high development effectiveness rating were found to have had a higher value for money.

Overall, the webinar was considered a success. While there are technical issues to address within the Adobe Connect platform, the webinar format appears to be a useful and efficient means for communicating information to programme managers and partners. BERF hopes to conduct another webinar in late 2016 to early 2017.

6.Next steps and proposed way forward in researching equity and BER

There is clearly a strong interest in better understanding and applying equity measures within BER programmes. If there is sufficient demand, BERF suggests preparing a follow-on Evidence and Learning Note on the topic of measuring equity in BER programmes. Based on the discussion during this webinar, a suggested range of topics to cover in the Note would include: