Version 13 August 2012
Paper presented at the IV Seminarof the Brazilian M&E Network
Indran A. Naidoo, Director: Evaluation Office, United Nations Development Programme, New York
Topic: The importance of developing national monitoring and evaluation capacities and the role of the leadership in this process
Greetings
Distinguished guests, ladies and gentlemen, a very good morning to one and all of you. I begin my address this morning by acknowledging the following persons who grace this event. They are:
Mr. President of the BNDES, João Carlos Ferraz;
Mr. President of INMETRO, JoãoJornada;
Mr. Sub-secretary of the Secretary of Strategic Matters of the Presidency, Paulo Jannuzzi; [ARMS1]
Mr. President of the Institute of Municipal Urbanism Pereira Passos, Ricardo Henriques
- Introduction
The independent Evaluation Office, responsible for the oversight of the global programme of the United Nations Development Programme (UNDP) that covers [ARMS2]166 countries,considers this invitation an honorand it is our privilege to support this impressive event. Brazil is a leader in Mercosur, in the region and increasingly in south-south cooperation. Inthe field of evaluation it has set an example of the value that can be derived from sound evaluation capacity and networking. The economic growth of the country is increasingly attracting attention, with people asking how this country gets it right. It has a vibrant democracy and media, the performance of the state is under scrutiny from many quarters, all of which reflects an educated society demanding to know what the results are from taxes paid, as well as evidence of good governance. Evaluation as a philosophy and a function, profession and practice, serves to guide and align a commitment between policy and practice – as evaluation will show up the discrepancies and spur action for correction between what is promised and delivered.
The product of a sound M&E infrastructure, based on the bedrock of democratic valuesand a commitment to reducing inequality, is a visible development programme within which marginalized groups are recognized and advanced and key development indicators, such as the Millennium Development Goals (MDGs) show progress. Keeping to the MDG goals requires authentic M&E which all stakeholders can be confident about. This requires leadership that must be exercised from different quarters; government, civil society, academia and evaluation professionals. The form of leadership varies but the permeating factors are:
a)A commitment to national ownership and thebuilding of local capacity so that evaluations are informed by contextual, so that accountability, analysis and recommendations are informed by contextual specifics.
b)Using evaluations to advance learning, with an emphasis on continuous improvement through capacity building and ownership, rather than sanction
It is essential that evaluation partners work in a way that is complementary as capacity development requires social consortium as no sector can do it on its own. The development and evaluation agenda must be shared, as sector performance affects all stakeholders as they individually and collectively are also citizens with democratic aspirations and interests.
- Evaluation leaders must subscribe to the underlying philosophy of evaluation
In my brief discussion today I will explicitly address the role of leadership in developing national M&E capacities, and do so from the premise that conducting good evaluation stems from a deeper commitment about how the world should work; it is a commitment to be and receive critical appraisal, to respect the role of others in the field and work with them towards a common good, and a commitment to improve, irrespective of the challenges posed by resources and situations.
Evaluation as one of the most dominant movements in the past 20 years has brought to the public domain more evidence of government performance than ever before, leading to a heightened accountability and potential better alignment between the political promises and actual practices.
The evaluation event helps to mediate discussions between the service deliverers and the service uses – generally government and citizens. It is a powerful tool which is reflected in the number of continental, regional and country evaluation associations and networks, each of which help support on of the most dominant new disciples of our times. Given the prominence of evaluation it is necessary that it is credible and objective – and for all evaluators that the concern of independence is addressed. This is because whilst we may address the technical dimension of evaluation it needs to be recognized that it is an inherently political activity which deals with the contentious issue of “judgment”. In doing so, issues of allegiance, bias and objectivity surface, hence the need for structural independence of the evaluation function. I am pleased to share with you that the EO of the UNDP is an independent office that heports directly to the Executive Board and not to the administrator of UNDO. It has an evaluation policy that ensures the independence of the function and invite delegates to discuss it as you seek to strengthen your own evaluation capacities.
- External interventions offer the opportunities to build decentralized capacity
I return to the question of how evaluation practice can improve evaluation capacity, by illustrating how a global evaluation programme helps to develop an evaluation discourse and evaluation practice through its regular work. The EO covers all UNDP programmes at global, regional and country level. The evaluation office is able to achieve greater success in its work – especially our country level evaluations - the Assessment of Development Results (ADRs) of UNDP country programmes,due to an approach which develops evaluation capacity through an engaging evaluation process and the increasing use of country level capacities to support the evaluation. The former relates to a process that seeks engagement to get impressions about UNDP performance from stakeholders at the country level on how successful initiatives [ARMS3]have been. In the past decade, we have conducted over 60country level evaluations [ARMS4]inall 5 UN regions, and in many of these, where there was limited evaluation literacy, helped introduce the discourse and practice of evaluation. Evaluation needs to germinate and any evaluation practice, irrespective of whether it is introduced in a country or region, externally or internally, by donors, beneficiaries, government, civil society or the media, redefines the environment by introducing concepts such as accountability and transparency. In practice, this means greater expectations and obligations for good governance – and evaluation thus serves a very democratic enhancing function.
The EO recognizes the importance of local expertise, and has sought, found and developed evaluation capacities by using local expertise to conduct country level evaluations. In this way capacity has been built through demand, and the seed for further germination has been helped. This practice also means that expertise is recognized as existing within countries; but needs to be identified, advanced and supported, so that self-determination – a UN value – is advanced. We have found where there has been meaningful engagement with local capacity and governments the propensity for getting results used is greater; as in Sri Lanka, Costa Rica and Liberia this year. After such evaluations, one leaves a country, a region with greater ability to strengthen [ARMS5]its capacity, as they have the knowledge, experience and legitimacy to nourish the new saplings that have emerged from such experience. More, of course, needs to be done, and the responsibility now falls on the levels of leadership within government, the academic sector and civil society to ensure that the momentum continues after evaluation events.
- Who are the leaders in governance?
It is important to situate evaluation advocates within the broader conceptual landscape of governance. We live in a world where, due to globalization and membership of organizations such as the UN, a common governance terminology has emerged. The UN seeks a unity amongst nations which is enjoined by democracy and civil liberties, better social services and a respect for the environment. Its visibility seeks to go beyond crisis prevention and recovery and the holding of elections; indeed, it seeks to entrench sustainable solutions so that there is self-governance allowing for its exit. In talking about values there needs to be concrete steps to measure. Leadership needs to respond by ensuring that it promotes capacity development through various actions.
a)Political leadership [ARMS6]
Evaluation promotes transparency, and hence accountability; it isthusnecessary that there is political commitment to the practice, as implicit in accepting an evaluation function within government is an invitation for critique where evaluation is seen as an ally to improve performance. Political commitment, on its own, is not enough. It needs to be followed throughwith resources and/or a commitment to mobilize the resources to build capacity.
Political leadership must commit itself to a process which involves dialogue with critical partners, such as civil society, the private sector, beneficiaries, academia, the evaluation professionals and even the media. Successful examples include events such as this, and we look forward to co-hosting with Ministry of Social Development and fight against hunger the next International Conference on National Evaluation Capacity in Brazil next year. The emphasis shall remain on advancing consensus on strengthening capacity for evaluation.
There is also the symbolic importance when political heads commit to evaluation in that it shows pride in democratic governance.
b)Administrative leadership
This tier of leadership, generally located within the higher decision-making structures of government, is critical as it has to enact the political commitment to evaluation by making resources available and ensure that policies support accountability. It is interesting to note that many ministries across the globe have or are in the process of entrenching evaluation units and policies. This is an important step in self-determination, and marks a shift from a bygone era, where the only evaluation of performance that was seen as credible was an externally initiated and conducted one. Some of the classic arguments against internal evaluation functionrelates to the question of lack of independence and hence the lack of credibility. However, if properly located and capacitated, with the correct operational policies, internal units can be credible – and it is important to build internal evaluation capacity. Both forms are needed and can complement each other. The practice of peer reviews, advisory boards, having evaluators properly trained and participating in the events of evaluation associations help to build the credibility of evaluation functions, irrespective of where they are located. Administrative leadership must support the build of its own evaluation capacity, as this will build sustainable capacity and skills that cannot be obtained by parachuting consultants into situations where local capabilities installed.
In practice, this means engaging, actively, with evaluation associations and training institutions to develop capacity/needs assessments and put in place strategies that improve the evaluation literacy and competence of all officials. Less commission, more hands-on, is required. Efforts need to be put in place to change mind-sets of politicians, administrative leaders and public servants that it is better to be self-critical in order to improve, than be externally critiqued, when it is too late and there are adverse consequences. The internal environment must be receptive to capacity development, which means it must be open to evaluation – call for and embrace critique so that the evaluation function is not just symbolic, but a genuine function with structural independence and authority to perform credible evaluations.
c. Leadership of civil society organisations
In many countries and contexts civil society has been the original evaluators – working with communities to identify and direct their concerns to decision-makers. We need to commend the role of the private sector supporting corporate social responsibility work, calling for evaluation of their investment. These private and corporate institutions serve as examples of promoting shared responsibility in achieving development outcomes. Examples are the partnerships developed through the global compact for the promotion and achievement of MDGs.
The challenge now is to better align civil society and private sector resources and skills towards the common development agenda and thus complement the work of government.
d.Media
The media uses evaluations results to make political points; and often results are sensationalized, as part of the performance deficits are highlighted. This tends to often attract an adverse reaction from government. However, in a democracy, this is to be expected and respected. The media can play a role in helping to educate citizens and government of their service delivery rights and obligations. Evaluation cannot function without democracy, and democracy is enhanced by evaluation which brings with it transparency, accountability and learning. [ARMS7]
e.Evaluation institutions
There is generally weak formal offering in academic institutions to provide evaluation training, which may be traced to historic factors such as academic independence, amongst others. Academia should recognize the demand for evaluation training respond with practical curricula, that takes into account the fact that many public officials would not be able to take long periods of time away from work for study, and more creative options from in-service training to formal certification, grounded in evaluation and development should be developed. However, as more programmes and policies[ARMS8]require evaluation a collective supply from consultants, donor networks, public servants and academia shall be required as demand will outstrip supply. To make supply more effective there is no need to replicate institutions with similar mandates, limited resources suggest that it is better to consolidate funding into institutions so that synergies are achieved and duplication reduced.
[ARMS9]
In conclusion I thank you for your time and trust that what we have presented helps to stimulate debate on how to genuinely address capacity development in evaluation. The Evaluation Office shall continue to watch progress in the region with a view to supporting its development.
Paper presented at the IV Brazilian M&E Network by Indran A. Naidoo
Version 13 August 2012[Type text]Page 1
[ARMS1]He may be present just in the last day. I’m confirming the table with Marcia and will give you the details tomorrow.
[ARMS2]I though it was 166, the last time I checked
[ARMS3]It is better to use initiatives, because often latinamericans are too proud to accept the UN to come and “intervene” in the country…,especially this the labor party in place…
[ARMS4]I know you mentioned before but people will get lost if interpreters translate it different or just refers to the acronym it may be better to use Country level evaluations
[ARMS5]Avoid using the word build, government often is offended when they perceive that the UN thinks they are here to teach Brazil how to do things that they believe they already know… Brazilian arrogance call for the use of words such as strengthen, further develop…
[ARMS6]Something very problematic in Brazil is that often the bureaucracy is such, even within UNDP, that evaluation processes (with ridiculously long recruiting processes) take so long that the timing of the results of the evaluation is less than optimal. Often the management responses I got from evaluations from government is that it was too late to change anything because the budget for that round was already approved…and other similar excuses. So the timing issue may be something important to raise because often evaluations are considered useless because they come too late in the game. And that goes back to a problem related to administration organization, commitment and capacity. Especially if the evaluation I critical, they will sit on it, politically purposefully to avoid interference in elections or budget…I’ve seen an evaluation literally hidden and numbers manipulated in order not to affect an ongoing legislative process …
[ARMS7]Just so you know, the media in Brazil is extremely powerful, one sole TV station, GLOBO, is strong enough to elect a president and take him down, as it did with Collor, the impeached president before Lula. The soap operas are an incredible tool to manipulate the mass. Their quality is so good that people only watch that ONE channel, so Globo can convince enough people to believe whatever GLOBO says. The UN is often trying to train the media to avoid problems, but often doesn’t help much. They constantly make a mess with the HDRs and MDG reports and put the UN in very complicated situations. Helen once visited Brazil but Lula refused to see her at the last minute because of the interpretation the media gave about our HDR that affected Dilma’s election… But training for the media does help better for writing material where they provide more detail, but not TV.
[ARMS8]The most popular evaluations in Brazil are policy evaluations linked to programmes such as the cash conditional transfers, education for adults... The problem is that they are mostly marketing tool to promote the administration. One of the reasons I left Brazil was because government wanted to use and manipulate evaluations’ results to make political arguments in their favor. Quite often government gets upset at the evaluators if they presented poor results, as if they were the implementers. But slowly this is changing, one has to hope.
[ARMS9]We can show the map as well