ORGANISATION-CENTRED EVALUATION:
APPLYING PARIS IN EVALUATING COMMUNITY-BASED ORGANISATIONS
Tracey Konstant
Karel Stanz
University of Pretoria, School of Economics and Business Management, South Africa
In partnership with the AIDS Consortium, South Africa[1]
Summary
The Paris Declaration on Aid Effectiveness (Paris) commits the international development response to enhancing ownership among development partners, and to aligning with and building development partners’ priorities and systems. There has been concerted effort to apply Paris in government-to-government development assistance. While this formal national alignment is applauded, the organisational fabric of local government and community-based organisations (CBOs) ultimately carries development implementation. How then, do the principles of Paris apply at local level, and particularly, how does Paris apply in evaluation for CBOs?
Evaluation systems for community organisations are generally designed by the developed, and driven by the power distribution of a donor:recipient relationship. Firstly, the assumptions and implications of power inherent in money are seldom questioned. Secondly, externally driven evaluation systems tend to require that sophisticated written communication is imposed on those who may be competent development practitioners, but have limited formal education. Thirdly, evaluation tends to rely on criteria for success devised by those outside of the community level development setting. Together, we see processes designed at the convenience of the developed partner, with their control over criteria, wielded by financial authority, placing power firmly in the hands of the developed.
Applying Paris to CBOs requires a fundamental shift in assumptions and approaches. This paper emerges from action research with HIV and AIDS support CBOs in South Africa. It explores alternatives for evaluation communication which meet the call for accountability, while building intrinsic capacity for organisational reflection, decision-making and representation. We have designed and trialed methods based on organisation-centred success criteria and communication formats. As development evaluators we need creative solutions towards being accountable to our own final clients: developing communities.
1. Introduction
1.1 Paris
Ninety one countries signed the declaration at the High Level Forum on Aid Effectiveness in Paris in March 2005 (Paris Declaration, 2005). The Paris Declaration stipulates a set of principles to which members of the Forum committed their countries. These principles ask that funding agencies and country partners support partner ownership of development; ensure alignment with the agendas of partners; are held mutually accountable; manage for development results; and that funding agency relationships are harmonised.
1.2 Community-Based Organisations
Development must ultimately occur at the level of households and communities. Its final delivery can only take place through the local institutions of the public sector, such as schools and clinics, and through community-based organisations. Although they are recognized as critical partners at the front-line of development delivery, CBOs present some interesting challenges to the application of the Paris principles.
Many CBOs own their services and activities very deeply, being largely driven by the passion of local community members to effect change where they see deprivation and need immediately around them (Kaplan, 2002; Strode Grant, 2004). Their programmes are generally fluid and highly responsive, but strategically rather vague. Their systems are necessarily loose, organic and opportunistic. They tend to rely on their knowledge, observation, intuition and good sense in making decisions, rather than a documented and formally justified evidence base. They are likely to consider satisfying and well-attended activities as results, and may not be sensitive to outcomes or impacts. They therefore offer an organisational style that has little that is concrete with which to align (Bornstein, 2006; Gasper, 2000; Kaplan, 2002; Mebrahtu, 2002; Yachkaschi, 2006).
In addition to being informal, the CBO environment is highly complex, with convoluted and often unsynchronised accountability relationships. CBOs are internally accountable to a volunteer workforce that is both their key resource and they first client. They are accountable to the communities they serve, but without mechanisms for being held to account by these clients, and not always in synergy with their other lines of accountability (Edwards and Hulme, 1996; Edwards, 1999; Ebrahim, 2003; Gray, Bebbington and Collinson, 2006). They may also be accountable for funding from donor agencies and government, which are not always in alignment with each other or with CBOs’ other clients. When we consider issues of mutual accountability, we need to ask ‘to whom?’ and consider the compromises between different accountability demands.
Heinrich (2001) talks about the monitoring and evaluation of civil society as being like “trying to nail a pudding to a wall”. Applying Paris to CBOs is like attempting to dance with the pudding.
2. Research Methods
We have attempted to explore how community organisations best represent their abilities, capacities and contribution through an action research process with a selection of volunteering CBOs. Our aim has been to isolate some of the principles of organisation-centred evaluation and aid effectiveness. We have aimed to explore the emergence of effective communication rooted in the experience and capacity of organisations. The researcher took the role of facilitator, while observing, reflecting and self-evaluating through practice.
Six organisations have each participated in an intensive day of organisational reflection and affirmative enquiry, drawing on their strongest skills and competencies as communicators. The analysis that we present in this paper highlights some of the insights that have emerged during this research, while also drawing on the development effectiveness discourse in the literature.
3. Situation and rationale
3.1 Who designs evaluation systems for community organisations?
In the absence of accepted systems for the CBO context, evaluation systems and rules tend to be adapted from more formalised settings. These rules and conditions are dictated by the funder, with varying leniency and openness among funder cultures. As an example, in the last several decades predictive, ‘logic-based’ models have been the norm for planning and evaluation (Gasper, 2000). Organisations are engaged in time-bound, outcome-oriented commitments, which are funded on the basis of anticipated impacts, indicators and targets. Indicators of performance against these predictions must be defined in advance of implementation. This is the way in which ‘managing for results’ is generally interpreted.
This highly structured, linear design is directly co-opted from frameworks used in military, engineering and private sector contexts in the late 1960s (Gasper, 2000). There is little to support the assumption that the same logic is appropriate in complex social situations. While these paradigms have grown through sound logic and thoughtful conceptualisation in their original context, there are many examples of unexpected negative impacts as a consequence of forcing these methods of planning and evaluation into social development situations (Bornstein, 2006; Gasper, 2000; Gray, Bebbington & Collinson, 2006; Ebrahim, 2003).
A fundamental flaw in the linear, logical approaches of the results chain is that the crystal ball gazing used to define the outcomes and indicators of a project at the outset, need not produce realistic or valid predictions. In practice, impacts are usually far wider and more complex than could have been imagined, and are quite likely to be different from those predicted. An organisation which does not predict accurately may be considered a failure by its funding agency, and deemed unworthy of further support on the basis of the variance between its achievements and its predictions (Gasper, 2000). The capacity to predict well is rewarded more enthusiastically than the capacity to serve community interests.
These predictive, linear systems persist despite this lack of logic in the ‘logic model’. Development is contradictory, unpredictable and emergent (Kaplan, 2002; Soal, 2004). Inertia, crisis, revolution and consolidation are more typical of development processes than predictability or attributable cause and effect (Quinn Patton, 2002). The very concept of an indicator is incomprehensible in a local setting. How will I know that I have had an impact on my client? They might not die, or they might die with dignity. They might smile more, or they might be more assertive. Their family might accept them, or they might move to another town. They might take their medication, or have personal reasons not to. They might have access to the clinic, but might require other social services more urgently.
Since there is little logical link between ability to predict the future and the impact of CBO relationships, many successes are lost from learning, and many questionable and arbitrarily selected results are masqueraded as achievements. Predefining indicators in the context of local community development is as meaningful as trying catching a selected drop of water from a sieve.
3.2 ‘Lack of capacity’ exposed
Where community organisations do not succeed in applying these systems they are regarded as lacking capacity. Capacity building therefore focuses on training community organisations to use systems designed for a different context. We see an emphasis on skills around governance and boards, strategic planning or prediction, and financial management. These may be valuable, but were not identified as areas where lack of capacity was felt to limit effectiveness by any of the participants in our study. Capacity needs that were identified included team work, fund raising, leadership, management and motivation for unpaid volunteers.
Externally imagined capacity building does not necessarily prioritise capacity areas perceived as necessary for performance by CBOs. There is a risk that most of capacity building is appreciated but not owned. This may be part of the reason for organisational learning and the institutionalisation of training into organisations being one of the eternal challenges of capacity building programmes.
3.3 The power of money
In many CBOs little or no funding is required for their early work. In the HIV and AIDS sector, for example, CBOs most often advise and counsel, share information on available services and encourage clients to access those services, and provide basic hygiene and medication support. To a massive extent they are staffed by unpaid volunteers. The day-to-day survival of their staff is provided through the independent sources to each individual, possibly a child-support grant, a family member’s pension or one employed person attached to a household. This model of unpaid volunteers providing social services in a context of poverty and unemployment is difficult to conceive as possible. Nevertheless, approximately two thirds of the members of CBOs in this study received no compensation for their work, and the great majority of others receive stipends approximately equal to the minimum wage.
CBOs soon reach a ceiling in the service they can offer to their clients without funding, and begin to experience a gulf between the needs of their clients and the services they offer. In the HIV context, for example, patients need transport to reach hospitals and clinics, and many deaths from AIDS are a result of the lack of bus fares. Their clients need food and shelter, only some of which can be negotiated through donations in kind, partnerships and public sector social services. In order to grow to become sustainable and semi-professional organisations, CBOs need an income stream to retain and remunerate staff and to provide formal premises. As these needs are experienced, CBOs become interested in entering the funding game, often only slightly aware of the costs.
3.4 Power play
We suggest that development is essentially the more equitable distribution of power. Development can be seen to have been effective if the net power held by a developing community to self-determine and achieve a suite of basic human rights, has increased. Power is complex and paradoxical. It is desirable and yet it corrupts. Power raises power, and yet it also undermines itself through the distance, ignorance and delusions of the powerful (Kaplan, 2002). Power is essentially a perception (Sen, 1987; Kaplan, 2002; Ebrahim, 2003). Social conditioning, including perception of power, is constructed and embedded by society and culture requiring the collusion of both the powerful and the powerless (Kaplan 2002; Kilby 2006). While power is certainly gained by an organisation that is funded, it is also lost.
Relationships between community-based programme implementers and those who claim the right to hold them accountable revolve around the distribution of power and perceptions of power and powerlessness. The power dynamics of global society are lived out in development relationships. As development practitioners, our least surmountable challenge is influencing the social structure in which we ourselves are embedded.
The balance of power is profoundly affected by the processes through which organisations engage with each other. Formal research, intimidating terminology, complicated quantitative approaches, impersonal checklists and dictated requirements give an impression of power (Miraftab, 1997; Kilby, 2006). A consequence of playing the funding game is that capable, intelligent, locally knowledgable development practitioners driving community organisations, begin to spend their energies attempting to invent indicators, grappling with the fine distinctions between outcome and purpose, and spending vast proportions of their time writing reports for which they themselves see little relevance or value (Bornstein, 2006). They do this because their compliance is a condition of funding.
This power distortion tends to warp the organisational psychology of grassroots development organisations (Bornstein, 2006; Gasper, 2000; Gray, Bebbington & Collinson, 2006; Ebrahim, 2003). Appeasing and wooing funders begins to take precedence over a focus on understanding and meeting the needs of their beneficiaries (FAHAMU CAE, 2004; Hearn, 2000; Jaime Joseph, 2000; Ebrahim, 2003; Kotze, 2004). Organisations lose sight of their purpose, dilute their integrity, and mould their projects to suit the expectations of those with financial power (Kaplan, 2002; Bornstein, 2006; Lewis Sobhan, 1999). Utopian vision, the capacity to question and oppose, radical criticism, political activism and control over their own administration are compromised when they become financially dependent (Bebbington, 1997; Lewis & Sobhan, 1999; Hailey, 2000; Jaime Joseph, 2000). When funders control criteria for success, dictate processes for evaluating success, and use financial opportunities to maintain this authority, power is placed firmly in the hands of the developed. The target audience of the community organisation becomes the wealthy (the funding agency), rather than the poor (their community clients).
More insidiously damaging, organisations learn to respond to funders demands through ‘creative reporting’, subtle deception, manipulation and selective emphasis (Bornstein, 2006). They may attempt to preserve their moral integrity with arguments of this being a means to achieve their legitimate development goals. Organisations may shift their focus towards their achievements, and deny their challenges with the incentive of funding. In so doing, they lose opportunities to learn, their self-respect and sense of personal power are eroded, and development is undone. The very process of manipulation humiliates, wastes time and emotional energy, instils fear and dilutes internal authority (Bornstein, 2006).