UK Headline Report

Envirowise IA 2005

UK Headline Report

February 2007

MTP Additional study: process and international comparison

Draft report

June 2012

Contents

1Executive summary

1.1 Introduction

1.2 Pixie dust summary findings

1.3 MTP process challenges and solutions

1.4 International comparison findings and implications

2Introduction

2.1 Context and objectives for the additional research

2.1.1 Investigating the ‘pixie dust’

2.1.2 International study exploring MTP equivalents

2.2 Approach

2.2.1 Defining the ‘pixie dust’

2.2.2 International approaches

3Defining ‘pixie dust’

3.1 Clarifying the process

3.2 Key findings

3.2.1 What PAMs bring to the process

3.2.2 Extent of the PAM skills sets and access to them

3.3 Implications for Defra and the MTP contract

4MTP process challenges

4.1 Process challenges

4.1.1 Independence

4.1.2 Working within the Eco-Directive

4.1.3 The extent of UK commitments

4.1.4 Prioritisation

4.1.5 The 2007 contract split

4.1.6 General budget limitations

4.1.7 PAM quality and diversity

4.1.8 Data availability

4.1.9 Evidence building vs. modelling effort

4.1.10 Defra input

4.1.11 Output quality

4.2 Implications for Defra and the MTP contract

5International learnings

5.1 Equivalents

5.2 Structures

5.3 Core team profile and resources

5.4 Requirements

5.5 Particular challenges

5.6 Implications for Defra and the MTP contract

6Appendices

6.1 ‘Pixie Dust’ / process topic guide

6.2 International comparisons topic guide

6.2.1 Introduction

6.2.2 Background

6.2.3 Addressing challenges

6.2.4 Close

MTP Additional study: process and international comparison

Draft report

June 2012

1Executive summary

1.1Introduction

This research was undertaken as an additional project to the main evaluation of the Market Transformation Programme (MTP). The project comprised two elements:

‘Pixie dust’ study: A study to explore the uniqueness of PAM contributions to the production of MTP outputs, termed the ‘pixie dust’;
International MTP comparison: A study to explore how other countries are set up to develop similar outputs to MTPm what challenges they encounter / mitigate approaches.

Pixie dust: For the first element, the study aimed to explore the perception of exclusivity/ scarcity around the MTP consortium contractor (PAM) skills set and knowledge; in particular:

What specific knowledge and content does each PAM contribute to MTP outputs overall and to specific outputs?
To what extent to these skills / knowledge banks exist in the UK and worldwide? And where are they to be found?
What has been the process for identifying and recruiting appropriate contractors? i.e. how wide was the net cast, how many individuals were identified, and how were these screened?

These findings could then be used to assess:

The value added by independent contractors to the programme;
The potential for the programme outputs to be more efficiently produced and the programme to be more flexible in managing workloads.
Further learnings for Defra on process challenges within the MTP current contract.

International study: Regarding the second element, in light of process issues highlighted in the first element of the study, it was felt to be useful to explore similar set-ups in other countries to:

Provide insight into the way they operate and the rationale behind that.
Explore whether issues similar to those encountered in the MTP are experienced, and:

If so, what if anything has been done to address these.
If not, why not.

It was expected that this would provide valuable lessons for Defra for the future running of the new MTP contract and mitigation of issues. In addition, it will demonstrate what MTP does well in comparison to other set-ups and should therefore look to maintain.

Both elements of the study comprised qualitative telephone interviews with a selection of PAs and PAMs within the MTP structure and representatives – in other national governments – of the nearest equivalents to the MTP programme.

1.2Pixie dust summary findings

Pixie dust: Overall, the PAM’s ‘pixie dust’ appears to be a mixture of a relevant academic background, longstanding professional experience and a network of contacts to draw upon. Importantly, many PAMs also access market data and publications that – in theory – PAs could be accessing themselves. However, PAMs are felt to be more efficient in sourcing these and better able to interpret and apply the data in them.

What PAMs bring to the process: The general perception amongst PAs is that the PAM input is absolutely essential to production of the required programme outputs. Their inputs include not only technical data but also evidence-based assumptions, policy sensitivity and advice on the impact of particular policies for particular product groups or sectors.

PAM availability: The issue of PAM numbers or lack of availability delaying the delivery of MTP outputs was not as widespread as previously reported[1]. However, the findings have highlighted some ways in which Defra / MTP can guard against this risk, which is apparent for some product groups:

-Look ahead to new and emerging product groups and ensure that there are sufficient resources in place to address data requirements for these.

-Ensure recruitment efforts look outside of Europe to the large numbers of experts in, for example, North America.

-Help to ensure that the skills and development sector are encouraging and producing individuals with a strong grounding and academic background in subjects relevant to this area.

-Do not consider tying PAMs into prioritising MTP work through contract or Framework stipulations. Most PAMs seemed keen to produce work for MTP without this; indeed, restrictive stipulations on PAMs could lead to some dropping out the Framework.

-Ensuring, where possible, that there is regular work of sufficient size to keep PAMs actively interested and engaged in the work.

1.3MTP process challenges and solutions

Through the discussions with stakeholders about PAM inputs, further challenges in the MTP process were highlighted:

Independence: Interviews with PAMs revealed that manufacturer data is sometimes used within outputs, often where there is poor market data from independent sources. This raises the question of whether Defra needs more visibility of data and assumptions used in developing outputs to mitigate potential risks.

Timescales: Timescales for production of impact assessments is often tight. Although little may be able to be done to influence the Commission to alleviate this, respondents noted that Defra has increased the amount of review time required internally, which is likely to be negatively affecting outputs.

Extent of UK commitments: Defra does prioritise efforts on a product group basis to some extent; however it does still consider a wide range of groups. Some respondents questioned whether some product groups (e.g. those with low carbon and energy saving potential) need to be included[2].

Prioritisation: Several respondents reported that all outputs Defra require appear to be ‘high priority’ and questioned whether this really was necessary. Defra could consider whether the current team structure exacerbates this, as each product lead has independent responsibility for different product groups but tend to be working with the same contractors, which can lead to bottle necks.

Contract split: The 80/20 contract split led to inefficiencies in delivery from the PAM perspective, particularly for smaller work. This re-emphasizes that this method of delivery should be changed for the next contract, seeking alternative approaches which are flexible enough to encourage a wide pool of expertise without introducing administrative burden.

Evidence building vs. modelling: Market models, no matter their sophistication, are only as good as the evidence and assumptions used within them. For some product groups there was felt to be an imbalance of effort towards the modelling with questionable added value. Defra should assess the quality of key market evidence prior to assigning modelling resource to a product group.

Defra input: Turnover within the Defra team was reported as an issue within this study, as within the main evaluation. It was recognised that this has not been as much of an issue more recently, but it is a problem particularly as this is a highly complex policy area, requiring deep understanding of product groups and markets, which takes time to learn.
Economist input: Notwithstanding previously reported issues with regards to economist capacity within AEA, economist input earlier in the process would be beneficial in terms of improving both PAM and economist understanding of needs for IAs and the rationale behind detailed assumptions.

1.4International comparison findings and implications

The international comparison of MTP equivalents reported the following findings. Implications for MTP delivery are also considered.

View of UK approach: In several ways, the UK MTP model was envied by other states, particularly as it enabled the UK to (a) produce evidence-based negotiating positions for policy setting and (b) secure prominent involvement in a number of international programmes e.g. IEA 4E. No respondents felt that the UK approach was less rigorous or less comprehensive (across product areas) than their own.

Challenges: A number of the challenges and issues encountered through the MTP process, such as quick turnaround times, technical expert / economist disagreement, were recognised by respondents in their processes.

However, there are a number of differences from the UK approach in some countries; differences which appeared to yield benefits for the country in question:

-Direct management of experts (PAM equivalents) by the government department or agency rather than through a consortium manager; in such cases, excessive bureaucracy or contract management time were not cited as significant issues. Respondents believed the direct contact helps to:

‘Train’ experts as to what information government needs and in what format they prefer it;
Enhance the skills of the departmental team through exposure to working with sector / product experts.

This was previously reported within the main evaluation and the challenges faced by Defra in undertaking direct commissioning of experts were discussed. However, this should be raised again as we have found no similar equivalent contract managers in other countries. Expert contractors are either directly managed, or in some cases recruited into the department in-house.

Should direct commissioning or in-house recruitment not be possible, Defra should consider some other practical issues, which directly affect output quality, including:

How to find and recruit PAMs for new product areas within MTP.
How to assess the need for requested enhancements to the existing models and analysis.
Decide how and when the Defra team and particularly economists will input to the process, and what that involvement should comprise.
Finally, several government teams / agencies perform have their own testing lab (in one case because they also fulfil a regulatory and enforcement role on compliance). This enables ad hoc in-house testing and primary research to be conducted, and also builds the skills and understanding of the team on various product areas. Defra could consider a similar arrangement with the NMO.

2Introduction

2.1Context and objectives for the additional research

The research additional to the main evaluation of the Market Transformation Programme (MTP) comprised two elements:

A study to explore the uniqueness of PAM contributions to the production of MTP outputs;
A study to explore how other countries develop these outputs and encounter / mitigate challenges.

2.1.1Investigating the ‘pixie dust’

One of the key process issues identified through the evaluation of the MTP was that of delayed and / or resource intensive turnaround of data requested by Defra to feed into Commission negotiations.

The challenge of meeting such deadlines was in part ascribed – by key stakeholders within the consortium - to the short notice provided by the Commission (and therefore by Defra). However, the short notice was cited as an issue because of a lack of immediately available contractor resource within the timescales required.

When probed further as to why the pool of contractors could not be expanded, the picture painted by a number of consortium stakeholders was that they were already using all the available experts for particular areas. The suggestion was that for some product groups, there were simply only a handful of individuals with specific expertise across the globe.

This also explained – according to stakeholders - why it might be difficult for the MTP to tie these in-demand experts to contracts or agreements that made them more available for MTP requirements.

Discussion of these findings led to questions around exactly what the MTP contractors contribute to MTP outputs (as opposed to elements that could be picked up by staff with more general modelling / IA calculation skills) and to what extent the capability to provide these contributions is rare.

This study therefore aimed to explore the perception of exclusivity around the MTP consortium contractor (PAM) skills set and knowledge; in particular:

What specific knowledge and content does each PAM contribute to MTP outputs overall and to specific outputs?
To what extent to these skills / knowledge banks exist in the UK and worldwide? And where are they to be found?
What has been the process for identifying and recruiting appropriate contractors? i.e. how wide was the net cast, how many individuals were identified, and how were these screened?

These findings could then be used to assess:

The value added by independent contractors to the programme;
The potential for the programme outputs to be more efficiently produced and the programme to be more flexible in managing workloads.

2.1.2International study exploring MTP equivalents

The core evaluation and additional process study also highlighted potential for intended outputs and outcomes to be delivered more efficiently and effectively. The core evaluation also highlighted a number of set-ups – at both national and international levels – designed to deliver similar outputs and outcomes to the MTP i.e. informing policy positions on product standards and labelling for that country. The use of the term ‘set-up’ is deliberate, as many did not appear to be as sophisticated as MTP (in terms of being distinct organisations with contractual arrangements).

In light of the process issues highlighted – and the forthcoming renewal of the MTP contract – it was therefore felt to be useful to explore these set-ups to:

Provide insight into the way they operate and the rationale behind that.
Explore whether issues similar to those encountered in the MTP are experienced, and:

If so, what if anything has been done to address these.
If not, why not.

2.2Approach

2.2.1Defining the ‘pixie dust’

This element of the study comprised a review of the evaluation data and further interviews with consortium representativesand contractors to fully clarify:

The extent - and content - of contractor (PAM) input into MTP outputs i.e. the ‘pixie dust’.
Whether or not this data could be collected in any other way. Their views on the added value of contractor input – what is the counterfactual? What could be picked up by staff with more general skills? What would be missing and what is the value of that?
AEA efforts to identify and recruit contractors, as well as the content of subcontracts between the two parties; what level of commitment does the contract require?
Any efforts to expand the available contractor pool and the success of these efforts
Awareness of others providing / capable of providing this input.

We envisaged that ten interviews should provide sufficient representation of MTP managers and contractors.

2.2.2International approaches

This additional part of the study comprised:

Desk research, liaison with Defra and review of evaluation information to identify organisations / set-ups comparable to MTP in terms of intended outputs / outcomes.
Qualitative telephone interviews with stakeholders or representatives of 8-10 organisations / set ups (depending upon the number identified) to explore operations and processes, successes and weaknesses.

3Defining ‘pixie dust’

3.1Clarifying the process

To show how the PAMs fit into the overall process for production of MTP outputs, the theoretical process for producing outputs relating to the Eco-Design Directive is as follows:

A product comes up for standards and labelling policy negotiation
Defra inform AEA of the need for an impact assessment (this should already have been anticipated through annual plans)
AEA PAs engage the subcontractor experts (PAMs) to obtain the data required
Experts (PAMs) obtain and provide data pertaining to the product
AEA PAs collate and interpret the data and provide an impact assessment to the Defra team
The output is reviewed and discussed between the Defra team (including economists) and AEA (who may also refer back to the PAMs where necessary).
The output is finalised and the Defra team then use this to underpin policy positions negotiated at the Commission level.

3.2Key findings

3.2.1What PAMs bring to the process

The PAM’s tend to share the following attributes:

Academic backgrounds that are relevant to assessing product energy performance e.g. physics or engineering: “I have a Bachelors and Masters specialising in renewable energy, later transferring to energy efficiency.”

Longstanding professional background working in both manufacturing and consultancy sectors: “I’ve been working in this product area for over thirty years and have been technical advisor on ECA scheme modelling in this area. I also provide advice to the Carbon Trust and retailers.”One PAM had been involved in the design of particular products.

The way in which PAMs are invited to contribute to the process of developing MTP outputs varied slightly depending upon the product, level of PA expertise, and type of data required. However, perceptions amongst both PAMs and PAs as to how and why PAMs should be involved in the process centred around the following inputs: