The Singer, not the Song
The Vexed Questions of Impact Monitoring and Social Change
Reflections on an exploration between some European donors and some southern African development NGOs, entitled: Impact Assessment and Impact Monitoring – How do we know we’re on the Right Track?
“Acts have their being in the witness. Without him who can speak of it? In the end one could even say that the act is nothing, the witness all. … If the world was a tale who but the witness could give it life? Where else could it have its being?”
Cormac McCarthy
facilitators and narrators:
allan kaplan and sue davidoff
may 2011
Framing the Reflections
During November 2010 a few European donors and a much larger number of southern African – mostly South African – leaders of NGOs came together for a two and a half day conversation which was entitled Impact Assessment and Impact Monitoring – How do we know we’re on the Right Track?; about 25 people in all. The idea was conceived about a year earlier by the EED programme person for South Africa, as a way of stimulating some critical thinking around … critical thinking on the part of the NGOs concerned. He was concerned that NGOs take their own self-reflection processes seriously, and wanted to encourage this by getting organisations, through their leadership, to come together and share their own experiences, insights and questions around the challenging question of organisational and strategic effectiveness.
By and large the NGOs involved are social change NGOs, concerned with long- term social transformation, with the transformation of socio-political relationships, with the facilitation and activation of community-led movements and associations engaged with various sectors of social functioning. Participants are, by and large, social and political activists and leaders of long standing, and their NGOs have been involved in the field of social change for many long years. Their work is focused, primarily, not on doing things for others, but on accompanying people’s own efforts of social transformation.One could say that the overall objective of their efforts is the much deeper work of shifting people’s perceptions of themselves and their possibilities, of the social dispensation of which they are a part, and also of accompanying people and communities in the search for more inclusive and socially just social structures, policies and processes. Despite this, though, these organisations do implement particular projects with material outcomes; and some of the communities they work with value this “assistance” higher than any other.
We have been asked to facilitate the conversation. We are, to an extent, trusted by most of the participants who know us, or know of our work, to enable the conversation to move into contested and even conflictual terrain without flinching and without prejudice. We are also development practitioners as are most of those present, with our own practice, and we share the same issues as those present with regard to assessment and weighing of the impact and value of our work, and we are not given to reducing the complexity of what we all face in our work.
The idea is not to arrive, in these short days, at a common method or technique of impact assessment suitable for all organisations, practices or situations, but rather to engage seriously with each other and with the subject matter in order to ascertain whether there is sufficient stimulated interest amongst participants to engage in a longer-term process of collaborative investigation and experimentation with respect to the monitoring and ongoing reflection on, and assessment of, the value of our work. The provided space is premised on a sense that many of us engaged in development work could do far more than we currently do with respect to such ongoing assessment; still the space is an entirely free opportunity for everyone to reflect and interrogate and learn, and the space has been provided – made free – by EED, without condition and without prejudice, and it is to be valued on this ground alone.
But things are never so simple.
The background to these discussions, the context within which they take place, is of an increasingly managerial and bureaucratic approach to development work (or social relief and change work more generally). Results-based management, an emphasis on prediction and control, the assumptions that underlie a mechanistic and instrumental approach to development and social intervention, are in the ascendancy. Funding, globally, is getting tighter, rules and regulations are getting tighter, criteria for assessment of work done in the field more inflexible and more specified. And set further and further from the field itself.
Within a context where poverty is growing rather than diminishing, despite all funding efforts to the contrary, those NGOs reliant on funding face faceless hierarchies of funding provision – the governmental and institutional ‘back-donors’ who hold many of the purse strings and have become arbiters of much of prevailing donor culture – which transfer a corporate mind-set of short-term results and log-framed logics and distinct material projects, as opposed to more theme-based and responsive programmatic approaches. One could say that as the rich get richer and the poor get poorer, the terms of engagement - the modern rules of largesse perhaps - dictate ever more stringent criteria and regulations and acceptable projects with respect to the provision of funding, and provide that funding with a financial investment angle as the bottom line. Such the context.
Up till very recently, EED’s funding was free of these looming constraints. Free to engage with ‘partners’ within an approach based on trust and the reciprocal features of human relationship, and on ongoing intelligent conversation interspersed with programme evaluations from time to time, and the provision of organisation development support where deemed necessary. Very recently, in fact between the time that these discussions were first mooted and the conversation itself, EED has been constrained to radically tighten its regulations with respect to impact assessment. Now, the narrative and discursive approach to assessment of ‘field value’ with respect to particular funding will no longer do, there is a demand to know more rigorously and precisely exactly what difference an organisation is really making in the field with respect to particular projects completed within a particular timeframe. EED has become party to a new discourse in which words like indicator and outcome and impact and assessment predominate.
There are, then, dynamics raised by the contradictory role in which the donors find themselves: on the one hand friends of the NGOs they fund, offering an opportunity for (self) reflection, and on the other hand messengers of bad tidings, bureaucrats come to lay down the new procedures which NGOs might deem a tightening of the noose around their necks. There are, as well, dynamics between the NGOs themselves; some regard the new funding specifications as something that simply has to be lived with, a ‘necessary evil’, and want to learn how to fill in the required spaces correctly, others want to discuss the ‘evil’ itself, and regard the required tickboxes as activists would any unjust or unacceptable dispensation. There are also some people here who do not have problems with the new dispensation, and are eager to learn how to master it.
There is then, in what follows, no attempt to follow the proceedings of the conversation with regard to sequence, particular process or specific point. No attempt to be exhaustive. Despite, and in part because of, the dynamics, much emerged of great relevance, but most likely no one participant would agree with what we found to be key aspects of relevance. We have chosen, therefore, in writing this report, to write our own appreciation and perception of the relevant aspects of the conversation that struck us as true in the sense of being a true reflection of the conversation as a whole. At every moment, as facilitators, we had to listen both for the individual position and for the emergent flow and position of the conversation as a whole. For us the conversation turns out to have great importance, given the context in which we all find ourselves, as activists.
Opening Ripostes
The very opening of the conversation configures the terrain that will follow. Are we here to talk about the problem of impact assessment, or are we here to collaborate and learn from each other about social change strategies? Or, are we here to talk about critical and reflective enquiry into our practice? Are these in fact the same thing? Or do they, indeed, actively contradict each other?
The task faced by the EED programme person in opening the space is reflective of the ambiguities we all find ourselves in. He is trusted and respected by all present for the depth, intelligence and openness with which he engages the people he works with, his ‘partners’; he has the right mix of endurance, commitment and irony, and he in turn respects the complexity of social change processes. He genuinely dignifies his partners by the giving of his trust; at the same time he constantly questions the real value of the work that all of us are doing, himself included; and deep down we all question it too, despite our somewhat assertive rhetoric. We all believe we’re doing the best we can, but ‘results’ are not easy to see, society is changing, patterns are entrenching or disappearing around us, and our practices are not always robust and accurate enough to hold and guide the unfolding skein of the social fabric.
We all despair, and we all remain inspired, and the programme person thinks that perhaps if we were more rigorous with respect to self-reflection our practices might become more robust and accurate, and produce some really tangible ‘results’. He really does manage to occupy the position of ‘accompanier’, with material resources to back this up. And up till now, up till the organising of this very conversation, EED has enabled this space to stand.
But things are not so simple.
His opening has to include a section that goes something like this:
We have planned this workshop for quite some time. In our first letter to you we stated our purpose, to explore different ways of impact monitoring with our partners. We also sent out a second letter and I would like to give you some background to that second letter: EED receives funds from the German government (since 1962) and for this year we received 100 million Euro. The government is now asking EED how we monitor the impact of our work. We have signed a memorandum with the government that states the following: it says what we cannot do, it lays out our funding principles, and it gives reporting requirements; most especially meticulous reporting is required financially.
We have obligations of stewardship. EED is responsible for the selection of the right partner organisations; however EED is not responsible for the success of a project. This approach has worked for almost ten years but now the government also wants more impact monitoring. The OECD has introduced the Paris Declaration = monitoring for results. Now we get pressure from the Paris Declaration, the Federal Court of Auditors and also the German public. The Federal Budget code changed and led to discussions so many things have since changed. Thus there is a need for more impact monitoring. We have to comply.
With impact monitoring we have to have a goal, a project objective, activities and outputs. The new thing about impact monitoring is that we have to look at how to measure the outputs. We have to come up with indicators. This can be a very mechanical way of looking at things. The indicators have to be specific, measurable and realistic.
Each project needs to have one or two objectives and each of these objectives needs to have two or three indicators. But this is not our main focus in this workshop, not at all.
Let us accept that it is legitimate for government to ask for impact monitoring. And organisations individually want to know their impact. What we are wanting to understand here is in what sense Impact Monitoring is a relevant issue, what are its challenges, and do we wish to work further together in this ongoing journey of discovering together whether we are on the right track, and if so, what would be the best way of doing this?
Already, in this last paragraph, the ambiguity of intentions and relationships is manifest in these alternative ways of framing the enquiry, an ambiguity that was entrenched in the very title of the conversation. Two separate phrases masquerading as one. “How do we know we’re on the right track?” is an open-ended question, unbounded, inviting conversation; it’s a discursive question, unthreatening and non-judgemental yet potentially rigorous and very challenging. It’s an inviting question, an invitation to the consideration of a complex conundrum that may have no one answer, or whose answer may change through the very tackling of the question. Most of all, it implies dialogue. All this means too that there is no end to the question, and no final arbiter.
On the other hand, however, Impact Monitoring and Impact Assessment is not a question at all, but a statement or name or command. It has a ‘should/ought’ quality to it, a boundedness, a normative insistence. One must engage with a particular procedure or technique, which seems as though it is collectively understood and agreed to (though in reality it is anything but). The word ‘impact’ implies particular kinds of results, and the words ‘monitoring and assessment’ have a judgemental quality to them, and the implication of a standard against which to judge. The concept carries a feeling of righteousness about it, a feeling for clear definition and discrimination, thus cutting its way – perhaps helpfully – through the complex interweaving of unbounded processes and dynamics, and demanding clarity.
These two ideas – of the imperative around impact, monitoring and assessment, and of reflective enquiry into practice – have come together in the title of this conversation and in the opening address, and perhaps they can inform each other, or perhaps, through understanding the dynamic of their relationship, we may inform ourselves as to how to proceed in the tricky terrain of social change. In a sense, the struggle between these two conceptions frames the conversation itself quite precisely.
So we will pursue these particular lines of enquiry through the course of this paper. In the first instance, we look at the reaction to the new EED constraints, to the new emphasis on ‘Impact Monitoring’, which entails new rules of engagement. Though elements of this response run through the entire conversation, we try to capture the main aspects as strands of one argument (Possible Impacts of Impact Monitoring). We then move onto the question of how we actually ascertain whether we are on the right track with our change efforts, or not; how this relates to impact monitoring, how it relates to the work we are doing and to our organisational processes and cultures (Tracking our Traces). It becomes clear that the dynamic between these two ideas creates a helpful framing for a central challenge of our work (The Enigmatic Dance between Achievement and Emergence). And so to a conclusion that arises after the conversation has ended, as we, facilitators, sit around the glowing embers of the fire late into the night, watching the wind rustle the coals into wakefulness (The Singer not the Song).
Possible Impacts of Impact Monitoring
We have not come here to discuss the new EED impact monitoring and assessment regulations and specifications, which themselves are compliant with the new German Government regulations, which in their turn respond to the Paris Declaration. We are told that we will have to comply strictly with these new regulations in the future, but they are not the point here. The point is to discuss the real task of making sure that we are aware of the value, or lack thereof, in our work, and to use such information to build the knowledge required to improve our effectiveness. It’s a pity that these two EED ‘agendas’ coincide at this moment, but let us not get sidetracked by the compliancy regulations. Surely the necessity for holding ourselves accountable to a few indicators of impact cannot be debilitating, and may even help to ‘keep us on track’. And certainly, those who donate the money have a right to ascertain whether their donations have any value.
It is hard, though, not to get sidetracked by the compliancy regulations; for most of the participants in this conversation they are as a red rag to a bull. Some participants do not quite follow the fuss, or can yet view the new regulations as a necessary evil, and they are here in fact to learn how to set indicators, how to translate objectives into outcomes, how to tell the difference between an outcome and an indicator, how to develop adequate outcomes and indicators for empowerment programmes, for capacity building and for long term social change. For most of the participants, however, such technical and bureaucratic questions are a very real part of the problem of impact monitoring, and for these it is the new emphasis on impact monitoring itself that is the problem.