Right or Wrong?What values inform modern impact evaluation?

(Agenda - tentative)

28thJanuary 2015,

Refreshments from 0830am to start promptly at 0900am

Eliot Room,

British Library, London

All practice – whether evaluations or development interventions - is underpinned by particular value systems. This event creates a space to explore what these value systems are and how they are related. In recent years the field of impact evaluation within international development has become largely driven by methodology and empiricism. It has lost touch with the ‘value’ dimension of evaluation (Picciotto 2014) in that values are now primarily understood in relation to rigour: the scientific generation of facts or truths which are assumed to be self-evident and universally valid.

Debates on ethics have tended to be limited and focused on ‘care of the subject’ to the exclusion of other ethical issues. Evaluators tend to either follow protocols advocated by funders (and which are sometimes inappropriate), or their own instincts (which leads to inconsistent practice). Evaluation societies and professional bodies have also largely responded to ‘ethics’ by providing universal guidelines that may be hard to use in practice.

The aim of this eventis to open up the debate on ethics and explore how it can become more relevant to the field of impact evaluation. The event considers three core themes: Firstly, a theme that explores new ways in which evaluation might challenge what we consider to be ‘good’ development by characterising and exploring the relationship between development values and evaluation values; Secondly, a theme around universality and plurality, which highlights the situated nature of ethics within evaluation practice; And finally, a theme that highlights the next generation of ethical challenges that may face evaluators. Each theme is outlined in the agenda below.

Agenda

Timing / What / Who
8.30 – 9.00 / Coffee and Registration
0900-0915 / Introduction to the event / Dr. Chris Barnett, Director of CDI;
Dr. Laura Camfield, Senior Lecturer, UEA
0915-0930 / Tracking the debate:Development Values , professional ethics and evaluative judgements / Elliot Stern, Emeritus Professor of Evaluation Research, Lancaster University; Honorary Research Fellow, SPAIS, University of Bristol
Theme 1: The Relationship between Development Values and Evaluation Values
Empiricism and analytics alone won’t solve our knowledge gap about what works, for whom, and why. A new generation of development policies and interventions are increasingly complex, often overlapping and interrelated, and in contexts that are changing and uncertain. Excellent research methodologies – while a key part of sound impact evaluation – only take us so far, as ultimately, some form of ‘judgement’ is required. How then, do we create the ethical and policy ‘space’ where evaluators can challenge the assumptions underpinning evaluations, for example, the focus on outcomes, or question development outcomes? Does the present system of commissioning limit such opportunities (e.g., as output-driven contracts dominate evaluation practice)?
0930-10.15 / Framing Ethics
The link between moral philosophy, ethical theory, and evaluation practice is often weak – and perhaps increasingly so as ‘professional evaluators’ become trained with a narrower and narrower focus on methodologies and analytics and ethics are framed in terms of universal guidelines. What are the key theoretical frameworks that might provide ethical guidance to evaluators? Can approaches such as democratic evaluation, participatory impact evaluation or systems evaluations offer new directions? How are they compatible (or incompatible) with each other? / Rob D. van den Berg, Visiting Fellow of CDI and President at International Development Evaluation Association
10.15-11.00 / What ‘impacts’ do we value, and how can evaluators better challenge development choices?
What is it about the present political structure and organisation of evaluation system that leads to a focus on questions and impacts that fail to address the critical issues in development? How can we shift from a project-by-project basis, to evaluating ‘good’ development more broadly? Specifically, how can evaluators go beyond a narrow focus on assessing stated objectives, and better assess the impact on capabilities, well-being and human security? What different ways can evidence be deliberated – to empower citizens and other stakeholders to better understand evidence, and it a way that makes it count? / Professor Allister McGregor, Research Fellow, IDS
11.00-11.30 / Coffee
11.30-12.15 / Ought Implies Can? Reflections on an evaluator’s duty to society
What role do evaluators have in public debate? What different ways can evidence be deliberated – to empower citizens and other stakeholders to better understand evidence, and it a way that makes it count? How can evaluators contribute to the broader knowledge system, to the reanalysis and reuse of evidence? / Dr. Richard Palmer Jones, Research Associate, UEA
12.15-12.30 / What have we learned so far? / Elliot Stern, Emeritus Professor of Evaluation Research, Lancaster University; Honorary Research Fellow, SPAIS, University of Bristol
12.20-13.30 / Lunch
Theme 2: Universality and Plurality: Ethics as a Situated Practice
Universal guidance is widespread, and often idealised. Achieving all such principles establishes an unachievable benchmark that is rarely met. Guidance is often abstract, and it is not always clear how such principles should be applied in particular contexts. As such, ethical principles only really become realised when situated in practice and where real decisions have to be taken (e.g., where one person’s ‘right’ principle may need to be offset against another equally valid claim for ‘rightness’). How then do we understand such principles and negotiate between them in practice? How can we achieve this in impact evaluations where structures are often complicated, dispersed and overlapping, and where there are many different stakeholder interests? What should govern an evaluator’s ethical behaviour, and how far should an evaluator’s duties extend to citizens and a broader society?
13.30-14.30 / Negotiating Ethical Commitments:
Evaluation as a Situated Practice
How can evaluators make the link from theories and concepts, to the principles and procedures that inform their work? How are different stakeholder interests taken into account, and what is the role of the evaluator in negotiating between the powerful and the marginalised? / Dr. Michelle L. Bryan,
Senior Associate, Bellwether Consulting, Associate professor, University of South Carolina
Theme 3: New Challenges
What is it that is changing in the way we do development? What emerging ethical challenges are stimulated by new modalities (such as ‘Pay by Results’ or ‘Results-based Financing’) and the use of new technologies and methodologies (big data, data collection by mobile phones, crowdsourcing, etc.)? How should evaluators respond?
14.30-15.15 / Behavioural Experiments in Development: Ethical ‘moments’ and oversights
Possible topics:
New methodologies - Evaluators are able to increasingly make use of new technologies and methodologies to collect and analyse data. Big data (gathered by the private sector), mobile phones, and other innovations provide evaluators with new opportunities but also new ethical dilemmas. What are these new challenges, and how best can evaluators respond? / Dr. Vegard Iverson, Senior Research Fellow, Manchester University
15.15-15.45 / Coffee
15.45-16.30 / Ethical considerations with respect to evaluationofresults-based financing
Possible topics:
There has been a more recent shift towards various forms of results-based financing, whether forms of social investment or payment by results. Within such modalities, evaluators are often tasked with a very narrow role – typically one of validating results in order to trigger disbursements. What new ethical challenges does this raise, and how can evaluators be better equipped to take on these challenges? / Burt Perrin,Independent Evaluation Consultant (specialist in planning, research and evaluation)
16.30-16.45 / Concluding reflections: Values in impact evaluation today / Elliot Stern, Emeritus Professor of Evaluation Research, Lancaster University; Honorary Research Fellow, SPAIS, University of Bristol

Centre for Development Impact (CDI) |

CDI Seminars:

CDI Practice Papers:

Page 1