Innovative Evaluation Tools, Community Capacity Building and Youth Engagement

Innovative Evaluation Tools, Community Capacity Building and Youth Engagement

Annotated Bibliography

Contents

The Meaning of Community 1

Criterial Evaluation Tools 3

Critique of Evaluation 6

Interpretive Evaluation Tools 7

Participatory Evaluation 9

Measuring Community Capacity 13

Youth Engagement 17

Planning Practice 26

Appendix 1: 29

Appendix 2: 30

The Meaning of Community

I start this bibliography with a review of some of the literature that underlies the community capacity theory that has inspired the Engaging Neighbourhoods Initiative.

Putnam, Robert. (2000) Bowling Alone. Simon & Schuster, New York.

This foundational book explores the concept of social capital. Putman explores how community health, educational achievement, local economic strength and other measures of community well-being are dependent on the level of social capital that exists in a community. Social capital is defined by the quality of the relationships and the cohesion that exists among its citizens. As part of his research, he studied a number of Italian communities to try to understand why some were more democratic, more economically successful, had better health and experienced better education achievement. The relatedness that existed among its citizens was the one thing that distinguished the more successful from the less successful towns. Social capital is a term widely used in the field of community development which is sometimes referred to as the glue that holds communities together or “the networks, norms and trust that facilitate coordination and cooperation for mutual benefit”.

Kretzman, J. and McKnight, J. (1993) Building Communities from the Inside Out: A path toward finding and mobilizing community assets. Center for Urban Affairs and Policy Research, Northwestern University, Evanston, IL.

This foundational text on community building outlines an asset-based model of capacity-focused development. Central to its argument is that while the traditional development models have focused on communities’ problems and needs as a way to design interventions, the asset-based model starts by looking at what individual, community and relationship assets a community has with which to build a development strategy. It is an internally focused strategy that builds on the agenda and capacities of local residents, associations and institutions. It is also a strategy that places the upmost importance on relationships as a determinant of community capacity. It is by focusing on the gifts of citizens and the relationships between citizens that community capacity is built. In further reading I have discovered many attempts to focus on or measure community capacity at a determinant of health.

Block, P. Community: The Structure of Belonging. (2008) Berrett-Koehler Publications, Inc. San Francisco.

Peter Block identifies the need to create a structure of belonging due to do isolated nature of our lives, our institutions and our communities. What he calls an “age of isolation” is caused by western culture’s individualistic narrative and the inward attention of our institutions and our professions. Although many intuitions are working hard under its own purpose, our communities are separated into silos that do not overlap or touch. Block sees the results of this fragmentation in low voter turnout, struggles to sustain volunteerism and the large portion of the population who remain disengaged. “The key to creating or transforming community, then, is to see the power in the small but important elements of being with others”.

Alexander, B. (2008) The Globalization of Addiction: a study in poverty of the spirit. Oxford University Press, Vancouver.

I was surprised to find that the primary message of this book written by a psychologist focused on addiction was that the solution to addictions laid in the health of our communities. Alexander became famous through his “Rat Park” experiment where he demonstrated the falsity of the belief that simple exposure to narcotic drugs can cause addition. Only the rats in isolation became addicted, while those that lived communally and in pleasant surroundings chose water instead of drugs. His more recent work has been exploring the causes of the current worldwide proliferation of addiction, not only to drugs, but to a great variety of other habits and pursuits. Through this work he has concluded that at the root of addiction is what he calls “dislocation” or an absence of psycho-social integration. The free-market society subjects people to irresistible pressures towards individualism and competition, tearing rich and poor alike from close social and spiritual ties. Addiction becomes an adaptive measure as people try to substitute for a sustaining social and spiritual life. The solutions lie in the health of communities through increasing our social support networks, reviving community art and engaging in social action.

Criterial Evaluation Tools

This section provides an overview of a number of tools and techniques that are prominent in the evaluation and monitoring field. The review includes some of the most common program evaluation models, and in particular logic models and the “Splash and Ripple” model.

Mark, M. Henry, G and Julnes, G. (2000) Evaluation: An integrated framework for understanding, guiding and improving public and Non-profit policies and programs. Jossey-Bass, San Francisco.

This text offers tools to help organizations make programming decisions based on systematic methods. It outlines four purposes of evaluation: to review the merit of programs and their value to society, to improve the organization and its services, to ensure program compliance with mandates and to build knowledge and expertise for future programs. The book outlines how to define evaluation purpose, how to plan an evaluation and how to describe, classify and analyse results. It classifies evaluation into four inquiry modes: description, classification, causal analysis and values inquiry and explores the relationship between key realist concepts and each inquiry mode. Mark et al. (2000) point out that sometimes our senses do not tell us the full picture. Therein lays the need for systematic and controlled processes which provide us with an accurate understanding of a program. Mark et al. describe how evaluation techniques can support our natural abilities to make sense of the world.

W.K. Kellogg Foundation, (2004) Logic Model Development Guide, Michigan, January 2004

The Logic Model concept is central to outcome measures or the Splash and Ripple Model. A logic model is a systematic and visual way to present and share an understanding of the relationships among program resources, activities and desired changes or results. The logic model breaks down evaluation into steps involving inputs, activities, outputs, outcomes and impacts. The purpose of a logic model is to provide stakeholders with a road map describing the sequence of related events connecting the need for the planned program with the program’s desired results. A logic model can use a theory approach, an outcomes approach or an activities approach. This guide provides a number of exercises including how to describe your results and actions, logic model checklists, evaluation guides and an indicator development template.

Trousdale, W. (2005) Module four: Have we Arrived? Step 9: Monitor and Evaluate. Promoting Local Economic Development through Strategic Planning, UN-Habitat and Ecoplan International.

In Trousdale’s 10 step model of strategic planning for community economic development, the 9th step is monitoring and evaluation. Trousdale speaks of the challenge that monitoring and evaluation is often perceived as nothing more than a way to find fault and be critical of a program which will possibly have adverse effects on funding. As a result, it is a step in strategic planning that is often overlooked. In contrast, monitoring and evaluation can be used as a way to enhance learning and capacity-building throughout community development processes. Therefore, the underlying question in a monitoring and evaluation scheme should be: what can we learn by monitoring and how will this help us make decisions next time? Like many other articles, this chapter focuses on the importance of collaboration in evaluation and explores ways that stakeholders can be integrated into the process either directly in design, in data collection or in a promotional capacity. Trousdale borrows from the logic model of examining outputs, outcomes, impacts and reach. He stresses the importance of aligning monitoring and evaluation tools with the overall objectives and ideologies of the organization. Key also to his argument is that monitoring and evaluation should never be an afterthought, but be built into any programming from the beginning.

Cox, P. Kozak, S. Griep, L and Moffat, L. (2002) Splash and Ripple. PLAN:NET LIMITED, Calgary.

The primary metaphor underlying this approach is that of a drop of water being the activity, the splash being the outputs and the ripple being the outcomes and impacts. The approach is one of “outcome measures” that is described as one that is clear about what we are doing and what we are changing. Every splash and ripple is an experiment; with the hope that we get better at predicting our splashes and ripples with time. The main difference between this model and more traditional goal orientated models is that rather than looking at what is being done, this model looks at what is different; therefore, more focus on the change programming creates rather than zooming in on the actions of the organization. This model uses indicators at three levels: outputs, outcomes and impact. Output is the direct result of an organizations programming (for example: youth learns about a program), while outcomes are the results of the output (youth joins program and learns about leadership). The impact is the overall long term result of the program (youth becomes community leader). Indicators aim to capture effects at all three of these stages. Indicators are designed to be cost effective, accurate and useful and ideally involving participant or beneficiary groups.

Cox, P. (2001) “Splash and Ripple”: Making Results Based Planning and Measurement Useful within Social Development Organisations. Master’s Thesis, University of Calgary, January 2001.

Cox describes Slash and Ripple as a new trend that attempts to link resources, activities and their deliverables to more systematic long-term results. A central challenge identified to evaluation is that there are two main goals to evaluation and those goals are not necessarily compatible. One is to measure and to be accountable to funders. Another is to determine the effectiveness of a program is order to better manage it. Efforts to demonstrate results in social development have had partial success because when “measurement” and “accountability” are the main points of emphasis it works against organizations investing in evaluation for their own management purposes. For so many organizations, their focus is not expert-led technically oriented solutions, but rather more indirect capacity building approaches wherein the distinction between the helping organization and the beneficiary is blurred.

In order for accountability and management to be compatible a few things must be in place: a) an understanding that knowledge is constructed by people and that nobody has a monopoly on what is right and what is best b) a willingness to accept that causal logic cannot fully account for the complexity in our world, it can only be suggestive. Cox does believe that by helping people inside organizations and communities generate contextualized, logical connections between activities and desired results, results-based planning can help create a conceptual roadmap or framework to serve a project’s management requirements. In order for this to happen the plans must be easy for people to participate in and subject to revision; overall the process requires dialogue and flexibility.

Earl,S. Carden,F and Smutylo, T. (2001) Outcome Mapping Building Learning and Reflection into Development Programs. INTERNATIONAL DEVELOPMENT RESEARCH CENTRE

This approach focuses on behavioural change outcomes rather than large scale impacts of community development programs. When referring to “impact,” development organizations usually mean significant and lasting changes in the well-being of large numbers of intended beneficiaries. These changes are the results for which donors expect accountability. This is problematic because the complexity and fluidity of development processes mean that achieving such impacts requires the involvement of a variety of actors, often over a considerable period of time. When large-scale change — or impact — manifests itself, it is often the product of a confluence of events over which no single agency has control or can realistically claim full credit. Outcomes are defined as changes in the behaviour, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly. As development is essentially about people relating to each other and their environments, the focus of Outcome Mapping is on people.

When donors and recipients try to be accountable for achieving impact, they are severely limiting their potential for understanding how and why impact occurs. The drive to claim credit interferes with the creation of knowledge. Pressure to demonstrate, measure, and be accountable for impact has led donors to conceptualize, implement and evaluate programs using tools and methods which seek a linear cause and effect relationship between a problem and the identified “solution” to that problem. However, experience tells us that development is a complex process that takes place in circumstances where a program cannot be isolated from the various actors with which it will interact. While it is necessary to simplify to some extent in order to plan and implement programs, methodologically one needs to acknowledge the contextual reality of which programs are a part. Outcome Mapping deals with the problem of how to attribute impact by increasing the value and attention placed on results achieved “upstream” from impact. It does this by focusing on the changes that are clearly within a program’s sphere of influence. While, at first glance, this appears to suggest concentrating on easier, less important, short-term achievements, in fact it does the opposite. It focuses attention on incremental, often subtle changes, without which the large-scale, more prominent achievements in human well-being cannot be attained or sustained.

Critique of Evaluation

The following texts explore some of the challenges associated with criterial evaluation techniques.

Westley, F. Zimmerman, B and Patton M. (2006) Getting to Maybe: How the World is Changed. Random House Canada.

“Getting to Maybe” describes the process of social innovation as one of ongoing experimentation, learning and adaption. In this text, evaluation is identified as a major barrier to social innovation. It explores the problem that premature demands for accountability can shut down social innovations and eat up precious resources. The traditional narrowly focused, bottom line-oriented goals based model just doesn’t work when the ultimate goal is to “change lives”. Nonetheless, taking stock of how things are going is crucial. Therefore, there is a need for evaluation tools that help organizations “articulate their passion and commitment”. An alternative they identify is what the authors call “developmental evaluation”. Developmental evaluation is described as integrating creativity and critical thinking by asking probing questions and tracking results to provide feedback to improve programming. This method is especially appropriate in the chaotic, explorative or reorganization phase of social innovation.