M&E Capacity Building Needs Assessment

Implemented by Financed by

DMI Associates Consortium the European Union

Monitoring & Evaluation Capacity Building Needs Assessment

STE MissionREPORT

Project Partner/Beneficiary is the Coordination Bureau of European and Euro-Atlantic Integration of the Secretariat of the Cabinet of Ministers of Ukraine (SCMU)

Direct local beneficiaries: seven Sub-Committees of the Cooperation Committee/Ministries chairing PCA (ENP AP) Sub-committees

Inna Bayda, Project Expert

Kyiv, 30 January 2009

Table of contents

List of Acronyms 4

Executive Summary 5

1Introduction6

1.1Objective of the assignment

1.2Approach and methodology used

1.3 Target groups interviewed

2Mission Results by Tasks8

2.1Quantitative analysis of results obtained through questionnairs8

2.2Qualitative analysis of results based on questionnairs and interviews 12

3Conclusions20

4Recommendations/Identified areas for improvement21

5Identified areas for further work22

5.1Target groups built around the proposed M&E capacity building plan 22

5.2.Tentative module-based M&E capacity building plan26

ANNEXES

ANNEX ITerms of Reference

ANNEX IICV of the Expert.

ANNEX IIITentative mission plan

ANNEX IVList of stakeholders contacted

ANNEX V Documents consulted

List of Acronyms

APAction Plan

CSP Country Strategy Paper

EC European Commission

EU European Union

LFMLogical Framework Matrix

M&EMonitoring and Evaluation

NPI New Practical Instrument

OOOverall Objectives

OVI Objectively Verified Indicators

PCA Partnership and Cooperation Agreement

PEProgramme Estimate

PCM Project Cycle Management

PPProject Purpose

SCMU Secretariat of the Cabinet of Ministers of Ukraine

TA Technical Assistance

ToR Terms of Reference

EXECUTIVE SUMMARY

Conclusions

1. The previous and current level of knowledge and experience in M&E (in evaluation in particular) is generally low among relevant stakeholders. The level of willingness to learn can be generally assessed as above average.

2. None of the ministries interviewedand MDCSU have a special unit on M&Eresponsible for European integrationprocess. Existing units, which have some M&E functions have general administrative functions and do not correspond to actual needs in M&Erelated to European integration process.

3. The level of understanding of M&E tasks, methods and importance varies depending on the level of administrative power. More understanding has been observed at the execution level. The higher level officials of administrative hierarchy (heads of departments) pay less attention to these issues due to busy schedule, which results in the absence of formalization of institutional framework for M&E functions.

4. M&E functions and responsibilities go far beyond the boundaries of departments of European integration formally responsible for these activities in line ministries. Active involvement of other relevant departments providing information to the coordinating one is critical for M&E system’s proper operation.

5. Clear specific instruction are required for ministries from the Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine on concrete M&E tasks to be incorporated in daily activities of the relevant specialists aimed at improvement of M&E activities.

Recommendations

• To involvedepartments of the line ministries which provide required information on the status of Ukraine-EU Action Plan and further - New Practical Instrument implementation to the departments of European integration, in M&E capacity buildingactivities.
• To pay particular and urgent attention to capacity of relevant departments of the line ministries in charge of filling in the structure/formulating tasks of NPI.
• Although evaluation is generally not considered by stakeholders interviewed as a priority area for project support, it is advisable to build the capacity in this area in a view of introduction of the New Practical Instrument.
• To discuss with the project beneficiary the issue of gradual official introduction of new duties into the relevant structures of coordinating and line ministries (interim evaluation of programmes, for instance) aimed at facilitation and improvements in implementation and monitoring of the NPI.
• Clear specific instruction are required for line ministries from the Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine on concrete M&E tasks to be incorporated in daily activities of the relevant specialists.
• In order to maximize the impact from M&E capacity building, it is recommended to send training/seminars materials to participants well in advance for preliminary study, preparation of questions etc.
• Individual consultation, short-term on-the-job training, counselling etc. have to be decided with each particular institution. It is recommended to send a letter from the project asking provide subject, timing and other arrangements for such activities.
• For individual consultations, specific thematic trainings, counselling etc. to consider distance learning method.
• To consider further planning and carrying out capacity building programmes in cooperation with the School for Senior Civil Service Executives of the Main Department of Civil Service of Ukraine.

1 INTRODUCTION

1.1Objective of the assignment

In accordance with the ToR, developed for the short-term assignment on M&E Capacity Building Needs Assessment, the objectives of the mission were:

The Overall Objective: improved capacity in the area of monitoring and evaluation of the Ukrainian civil servants and other representatives of various project beneficiaries and partners through a proper assessment of their capacity building needs as the first step in forming a coordinated project response through training, study tours and other capacity building activities.

The Specific Objective: to create a consolidated inventory of the capacity building needs in the area of Monitoring and Evaluation (M&E) through analysis of individual needs of particular target groups among Ukrainian civil servants and other representatives of various project beneficiaries and partners involved in European integration of Ukraine.

The final outcome of the present short-term assignment is the identified areas/priorities for project training interventions in 2009 formulated in the proposed tentative M&E module-based capacity building work plan.

The aim is to make the proposed capacity building plan client-oriented taking into consideration the level of knowledge on M&E among relevant stakeholders, busy schedule of civil servants and increasing importance of using M&E tools and techniques in a view of introducing New Practical Instrument replacing Ukraine-EU Action Plan.

1.2Approach and methodology applied

The two-side approach has been used for this assignment based on the results of questionnaires and interviews with relevant stakeholders:

(1)what stakeholders think they need to know about M&E, the level of depth of this knowledge, further use of the knowledge to be obtained in day-to-day work, areas of particular interest/needand

(2)what stakeholders actually need to know about M&E, further use of the knowledge to be obtained in day-to-day work, areas of particular interest/need in a view of forthcoming challenging changes in requirements to Ukraine in the framework of the New Practical Instrument and future Association Agreement towards EU integration.

Two major components – institutional and personal – have formed the basis of the approach for the present M&E capacity building needs assessment.

The methodology applied has been based on logical sequence of activities (see Tentative Mission plan, annex III to this report), starting from studying the relevant documents and structures of institutions, sending adjusted questionnaires to the relevant stakeholders/target groups, carrying out interviews with target groups’ representatives with a focus on the Bureau of European and Euro-Atlantic Integration of the Secretariat of the Cabinet of Ministers of Ukraine (SCMU) and Secretaries of Sub-committees from the Ministries chairing PCA (ENP AP).

These preparatory and implementation stages have been followed by thorough analysis of the information received in the form of filled in questionnaires and interviews.

Qualitative analysis followed by quantitative analysis of M&E capacity building needs were used as major methodological tools. The outcomes from the analysis are presented in conclusions and recommendations and have resulted in tentative module-based M&E capacity building plan.

1.3Target groups interviewed

Target groups for M&E capacity building needs assessment are represented by the project beneficiary, line ministries which are in charge of seven PCA (ENP AP) sub-committees and other relevant state institutions:

• Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine, Project Beneficiary, which plays the main coordination role in the process of European integration in general and responsible for coordinating M&E activities in particular;
• Ministry of Economy of Ukraine, chairing three PCA (ENP AP) Sub-committees (N 1 – “Trade and investment”; N 2 – “Economic and social affairs, finance and statistics” and N 3 – “Enterprise policy, competition, regulatory cooperation”);
• Ministry of Fuel and Energy of Ukraine, chairing PCA (ENP AP) Sub-committee N 4 “Energy, transport, nuclear safety and environment”;
• State Customs Committee of Ukraine, chairing PCA (ENP AP) Sub-committee N 5 “Customs and cross-border cooperation”;
• Ministry of Justice of Ukraine, chairing PCA (ENP AP) Sub-committee N 6 “Justice, freedom and security”;
• Ministry of Education and Science, chairing PCA (ENP AP) Sub-committee N 7 “Science and technologies, research and development, education, culture, public health, information society and media”;
• Ministry of Foreign Affairs – in charge of coordination of New Practical Instrument development;
• Main Department of Civil Service of Ukraine – in charge of capacity building for civil servants and a number of EU-funded programmes relevant to M&E.

The whole list of stakeholders interviewed is presented in Annex IV of this report.

Target groups/participants for particular module-based capacity building activity will be formedfrom the above list of stakeholders around specificpriority areas identified through the analysis presented in the further sections of this report.

2 MISSION RESULTS BY TASKS

2.1Quantitative results obtained through questionnaires

Eight organizations were interviewed in total in the framework of the assignment.

Bureau of European and Euro-Atlantic Integration of the SCMU was represented by two relevant departments (Department on Monitoring of Ukraine-EU relationships and Department of Strategic Planning of European Integration).

Departments of European integration in each of the line ministry responsible for sub-committees of the Cooperation Committee established in the framework of the PCA/ENP AP (Ukrainian parts of the joint Ukraine-EU bodies) represented the key relevant stakeholders for interviews.

Other relevant stakeholders interviewed were represented by:

• the Ministry of Foreign Affairs, which plays one of the key coordination functions in the process towards EU integration, and currently is responsible for New Practical Instrument formulation, which will replace Ukraine-EU Action Plan in few months;
• Centre for adaptation of the civil service to the standards of the European Union
• School for Senior Civil Service Executives of the Main Department of Civil Servants of Ukraine.

Both structures have a function of advanced trainings for civil servants in the fields related to the all aspects of EU integration.

Table 1 summarizes the number of questionnaires filled in by each organization. The detailed list of interviewees with names and positions is presented in annex IV of this report.

Table 1

N / Organization / Sub-committee / Number of questionnaires filled in
1 / Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine / N/a (coordination function) / 12
2 / Ministry of Economy of Ukraine / 1,2,3 / 2
3 / Ministry of Fuel and Energy of Ukraine / 4 / 5
4 / Customs Committee of Ukraine / 5 / 2
5 / Ministry of Justice of Ukraine / 6 / 3
6 / Ministry of Education and science / 7 / 7
7 / Ministry of Foreign Affairs of Ukraine / N/a (coordination function) / 1
8 / Main Civil ServiceDepartment of Ukraine, namely: / N/a / 2:
Centre for adaptation of the civil service to the standards of the European Union / 1
School for Senior Civil Service Executives / 1
Total number of questionnaires received / 34

Institutional aspects have been included in questionnaires in order to determine the role of M&E functions in day-to-day activities of the relevant stakeholders (which have been the subject of current needs assessment) and how M&E capacity is built by the institution concerned.

The key results obtained are summarized in the Table 2 below[1].

Table 2

Organization / (Declared) availability of a separate M&E unit / Intention to establish/expand a separate M&E unit / (Declared) N of experts in charge of M&E / Capacity building activities undertaken by organization
Monitoring / Evaluation
Ministry of Economy of Ukraine / No / No / 2 / Planning
Data analysis
Reporting
Project monitoring / No
Ministry of Fuel and Energy of Ukraine[2] / Yes / No / 3 / Planning
Data analysis
Indicators development
Reporting
Project monitoring / Preliminary
Current
Ex-post
Project evaluation
Customs Committee of Ukraine / Yes (?) / No / 0 / No / No
Ministry of Justice of Ukraine / Yes / No / 2 / Planning
Indicators development
Data analysis
Reporting
Project monitoring / Current
Ex-post
Project and
Programme evaluation
Policies evaluation
Ministry of Education and science / Yes / Yes / 7 (in total) / Planning
Data analysis
Indicators development
Reporting / Current
Ex-post
Programme evaluation
Ministry of Foreign Affairs of Ukraine / Yes / No / 5/6 / Data analysis
Reporting
Policy monitoring / Preliminary
Current
Ex-post
Policies evaluation
Centre of adaptation to EU standards, Main Civil Service Department of Ukraine / Yes / Yes / 8 / Planning
Data analysis
Indicators development
Reporting
Project and
Programme monitoring / Current
Project and
Programme evaluation
School for Senior Civil Service Executives, as above / Yes / No / 3 / No / No

It can be stated that out of 8 organizations analyzed, seven ones have a separate unit responsible for M&E. However, qualitative analysis presented below shows imbalances in practical performing of M&E functions.

Personal experience and knowledge of M&E has been assessed from retrospective point of view (past experience), current level of M&E experience and knowledge expressed in the course of needs assessment and areas of further capacity building in the field concerned.

Table 3 presents retrospective summary of the personal M&E experience and knowledge of the relevant interviewed stakeholders per organization.

Table 3

Organization / Number of specialists involved in previous capacity building activities on monitoring/total N of interviews / Number of specialists involved in previous capacity building activities on evaluation/total N of interviews / Percentage of the total number of specialists interviewed
Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine / 5/12 / 2/12 / 42% - monitoring
16,7% - evaluation
Ministry of Economy of Ukraine / 1/2 / 0/2 / 50% - monitoring
0% - evaluation
Ministry of Fuel and Energy of Ukraine / 0/5 / 1/5 / 0% - monitoring
20% - evaluation
Customs Committee of Ukraine / 1/2 / 1/2 / 50% - monitoring
50% - monitoring
Ministry of Justice of Ukraine / 1/3 / 0/3 / 33% - monitoring
0% - evaluation
Ministry of Education and science / 2/7 / 0/7 / 29% - monitoring
0% - evaluation
Ministry of Foreign Affairs of Ukraine / 0/1 / 0/1 / 0% - monitoring
0% - evaluation
Main Civil Service Department of Ukraine / 2/2 / 1/2 / 100% - monitoring
50% - evaluation

The declared average percentage of previous experience in monitoring capacity building activities is 38%, which is not too high.

However, it is obvious that participation in the past capacity building on evaluation is considerably lower as compared to the past capacity building experience in monitoring and constitutes only 17% in average.

The current status of average personal M&E experience and knowledge is summarized per organization as follows[3]:

Table 4

Organization / Average experience and knowledge in monitoring (as declared) / Average experience and knowledge in evaluation
(as declared)
Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine / Yes / Partial
Ministry of Economy of Ukraine / Yes / No
Ministry of Fuel and Energy of Ukraine / Yes / No
Customs Committee of Ukraine / No / No
Ministry of Justice of Ukraine / Yes / Partial
Ministry of Education and science / Partial / Partial
Ministry of Foreign Affairs of Ukraine / Yes / Yes
Main Civil Service Department of Ukraine / Yes / Yes

It can be concluded that none of the stakeholders have adequate knowledge in evaluation. Further analysis on the current status of M&E knowledge and experience is provided in section 2.2 of this report.

The previous andcurrentexperiences in M&E and capacity building activities in these fields have formed the basis of the assessment by interviewees of the general areas for further project support the most useful for current work. It is summarized as follows:

Table 5

Organization / The most useful areas for project support[4]
Monitoring / Evaluation / Both
Bureau of European and Euro-Atlantic Integration of the SCMU of Ukraine / +
Ministry of Economy of Ukraine / + / +
Ministry of Fuel and Energy of Ukraine / + / +
Customs Committee of Ukraine / +
Ministry of Justice of Ukraine / +
Ministry of Education and science / +
Ministry of Foreign Affairs of Ukraine / +
Main Civil Service Department of Ukraine / +

2.2 Qualitative analysis of results based on questionnaires and interviews

Qualitative conclusions of the present analysis to the large extent come out of quantitative analysis of the stakeholder’s feedback presented above plus analysis of the responses, comments and opinions expressed during the interviews.

As can be seen from the Table 1, the response of stakeholders in terms of number of questionnaires filled in by each organization differs. This dependson whether the questionnaires were filled in mostly by the specialists of departments of European integration or they were distributed to other departments which provide information to the departments of European integration.

Usually, departments of European integration play coordination role at the level of the line ministry while other relevant departments are responsible for actual implementation of particular articles of Ukraine-EU Action Plan. It has been concluded that as compared with departments of European integration which have some knowledge on M&E (on monitoring in particular) other relevant departments have no or very wage idea (as a best scenario) about M&E.

Assuming that these departments will continue working on implementation of tasks in the framework of New Practical Instrument, which is currently being formulated by the same departments/people of the line ministries, their involvement in M&E capacity building activities is critical.

This issue is particularly relevant at the current moment during the NPI formulation process, as each task of the NPI should be formulated in such a way that allows objective verification, quantitative and qualitative assessment of the status of achievements, indicator-based monitoring and criteria-based evaluation in a longer run.

Some line ministries have already understood the importance of the above-mentioned necessity of involvement of other relevant department in M&E capacity building activities (Ministry of education and science, Ministry of fuel and energy) and their feedback on project needs assessment was therefore more pro-active.

Institutional M&E capacity

Table 2 summarizes the current institutional aspects of M&E in line ministries in charge of sub-committees and other relevant stakeholders. Although it looks like the most of the ministries interviewed formally have separate M&E units, in fact these units/divisions quite often have nothing to do with M&E as such but control the execution of certain tasks and activities of the ministry and report to the top management in line with hierarchy structure.