The Impact of Rccs Research Project; Interim Report 2

Rural Community Value:

Assessing the Impact of the Work of Rural Community Councils

Draft final report to Defra

Malcolm Moseley, Stephen Owen and Phil Johnson

University of Gloucestershire

Gary Craig, Sally McNamee and Mick Wilkinson

University of Hull

2006

Contact addresses:

Table of contents

Acknowledgements

Executive Summary

1: The context for the study

1.1 The purpose and focus of the research

1.2 The structure of the report

1.3 Rural communities: current debates

1.4 The voluntary and community sectors and local governance

2: The evaluation of rural developmental work: issues and concepts

2.1 The impact of the voluntary and community sectors

2.2 Conceptual and methodological issues pertaining to impact measurement

2.3 Measuring the Impacts of Community Development in rural areas

2.4 Measuring the Economic Outcomes of local economic development initiatives

2.5 Current debates within RCCs

3: Developing the Tool – the research and development work

3.1 Introduction

3.2 The Intensive Case Study Phase

3.3 The Creation of the Draft Tool

3.4 Validating the Draft Tool

3.5 Making Final Revisions

4: The Economic Outcomes Tool: A Good Practice Guide for RCCs

4.1 Preamble

4.2 An Overview of the Tool

4.3 The Tool - Stage by Stage

4.4 Annexes to the Tool

5: Conclusions and recommendations

References

Appendix 1: Websites and other useful resources

Appendix 2: The Scoping Survey E mail Questionnaire

Appendix 3: A Listing of Some RCC activities

Acknowledgements

This study, completed in a relatively compressed timetable and either side of the disruption of a General Election, depended on the cooperation and support of a number of people and organisations and we would like to acknowledge, with thanks, the help they gave.

The project was supported by a Steering Group, chaired with a helpfully light touch by Janet Gawn, Head of Social Research, Rural Economies and Strategy Division in Defra. Those who took part in the work of the Steering Group at various points in the study were:

Janet Gawn

Tony Poole

Holly Yates

Iain Southall

Max Boelhoff

Emma Bevis

Chris Athey

all of Defra

Lorna Sambrook (East of England Development Agency)

John Hazelwood (Gloucestershire Rural Community Council)

Sylvia Brown (ACRE)

David Atkinson (Commission for Rural Communities).

All Rural Community Councils (RCCs) took part in an email survey and provided documentation; Sylvia Brown of ACRE was most helpful with wise advice and access to ACRE’s records; and six RCCs acted as case studies for developing and trialling the analytical tool which is described in this report. The counties served by the RCCs were:

Bedfordshire

Cumbria

Dorset

Durham

Gloucestershire

Nottinghamshire.

t. WThey are anonymous in this report but they know who they are and we are grateful for the time they gave at short notice to help with this study. Representatives of most RCCs and some other key partner agencies took part in two workshops, in Loughborough and York. We hope the results justify the commitment of all those who participated in this study.

EXECUTIVE SUMMARY

Executive Summary

Preamble

The Impact of RCCs research project, to which this report relates, ran for much of 2005. Its aim was to develop a practical tool that would enable England’s 38 RCCs (Rural Community Councils) to better assess and articulate the impact of their work – especially any work which might have local economic outcomes. The research was undertaken by the Universities of Gloucestershire and Hull in close association with the RCCs – six of which were actively involved in detailed case study work – and with ACRE, their national association.

The research arose from a growing appreciation that voluntary organisations such as RCCs need to demonstrate more clearly what are the ultimate outcomes of the work they undertake. This will enable a closer focusing on ‘outcomes’ in the management of their work and a greater ability to demonstrate to partners, funders and other stakeholders what outcomes have occurred as a result of particular activities. It is accepted that the RCCs’ missions typically embrace a range of social, cultural and environmental aims – as well as some relating specifically to the local economy – but often economic outcomes can and do flow from a host of activities not overtly focused on the economy.

The ‘tool of analysis’

Thus the main product of this research has been a detailed tool of analysis to be used by the RCCs themselves. The tool is designed for application to specific RCC activities, which may be discrete projects (for example establishing and running a ‘wheels to work’ scheme) or broader programme of ongoing work (such as the provision of advice and support to those who manage village halls within a specific county).

The proposal is that in relation to specific activities to be addressed one at a time, RCCs should work systematically through seven stages of analysis culminating in a short report of about four pages which might be termed ‘A Report on the Economic Outcomes of Activity X.’ This report would include quantified information where appropriate data are both available and meaningful but would be largely qualitative in character. It is estimated that about three person-days of staff time, plus time contributed by external ‘validators’, would typically be needed per activity evaluated, though this commitment of staff time could be reduced if appropriate monitoring procedures were put in place at the outset of the various activities to be scrutinised.

Briefly the seven stages of the tool are as follows.

1 Preparation. This stage involves defining the specific RCC activity in question (e.g. managing the Wheels to Work scheme in County X) and the precise period of its operation to be placed under scrutiny (e.g. the calendar year 2005) albeit accepting that related outcomes may in part have occurred subsequently. Also to be defined at this stage is the target zone (e.g. the rural parts of County X) within which any outcomes will be deemed relevant. Preparation also involves the early clarification of data sources and resolving ‘who will do what and how?’

2 Elaboration of the Activity. The next stage is to set out concisely just what the RCC did with regard to the chosen activity during the period under scrutiny. What support was given, meetings addressed, newsletters written, representations made etc? What did other agencies do that might have complicated the picture from an analytical point of view?

3 Outlining the Outputs. This stage involves attempting a comprehensive listing, with some degree of quantification where possible, of all the principal outputs of the activity elaborated in the previous stage. Some of these outputs may be economic; many will not be. But all should be listed as a prelude to focusing on the economic outcomes in the next stage, below. At this ‘outputs stage,’ an eclectic range of source material will need to be gathered and appraised.

4 Assessing Economic Outcomes. The task here is to estimate what longer term benefits of an economic nature have accrued within the target zone as a result of the activities and outputs defined earlier. Most of these economic outcomes will fall into one or more of five categories; increased employment / reduced non-employment; increased business activity; increased / improved ‘capital’ of various kinds; the avoidance or delay of expenditure by the state; the influencing of other agencies with regard to their support of local economic activity.

5 Checking for Over- or Under- Statement. Here an attempt is made to correct for five possible sources of error that often undermine analyses of this sort. Three of them commonly lead to an exaggeration of the real economic outcomes that have occurred; these relate to ‘attribution’, ‘deadweight’ and ‘displacement’. Two relate to typical sources of understatement, namely ‘multiplier effects’ and ‘delayed or protracted outcomes’. Advice is given on how these sources of error might be addressed, albeit pragmatically and imperfectly.

6 External Validation. It is now time to seek the validation of a draft ‘Report of the Economic Outcomes of Activity X’ based on the above research. Up to three well-placed external commentators should be recruited for this purpose; each will know the activity well and be able to give a reasonably objective view in a face-to-face interview. The validators are asked to comment in turn on sections of the draft report, addressing in turn the activity, the claimed outputs, the claimed outcomes and the suggested summary of over- and under-statements.

7 Conclusion and Final Edit. It remains to revise the draft report and present it to the board of the RCC as an aid to management, including staff management, and to the relevant external partners and funders of the RCC. A template is set out to help RCCs frame these reports.

Developing the above tool

In order to devise and refine the above tool of analysis the research team had undertaken a series of exercises which are written up at some length in the report but may be briefly summarised as follows.

1 An analysis of relevant literature pertaining to

existing evidence on the impact of the voluntary and community sectors in general – an analysis that established the considerable scale and complexity of the impact of these sectors in England

a range of conceptual and methodological issues pertaining to impact measurement – dealing with the treatment of such concepts as outcomes, attribution, deadweight, capital etc, and with the temptation to deem important that which is measurable rather than to attempt to measure that which is important.

the assessment of impact specifically in the domain of rural community development – a review that established inter alia that although there is now a body of experience pertaining to the evaluation of rural community development projects, there is very little guidance on how to evaluate the impact of rural community development per se

measuring the economic outcomes of local development initiatives. There is now a considerable literature on this, emanating from a host of national and local agencies, giving useful pointers on how to deal with the familiar range of issues around outputs, outcomes, deadweight, displacement, double-counting etc

2 A review of current RCC experience in this domain An analysis of a sample of RCC documentation (e.g. business plans and annual reports), supplemented by an e mail survey of RCCs, indicated both a growing interest in the general subject of evaluation and impact assessment, and also a wide range of RCC activity that would appear to produce economic outcomes of some kind. This review also revealed, however, that there is as yet no commonly agreed methodology for this work – a state of affairs often attributed in large part by the RCCs to the very different expectations and requirements of the wide range of agencies and organisations that purchase services from them.

3 A programme of Research and Development extending from June to November 2005 and undertaken by the research team in close association with six RCCs. This R&D involved

intensive case study work (involving the RCCs which serve Bedfordshire, Cumbria, Dorset and Nottinghamshire) in which various links between selected RCC activities (18 in all) and their outputs and apparent outcomes were hypothesised, explored with reference to available information and in large measure validated with external involvement.

And then, after the subsequent formulation of a draft tool of analysis,

the ‘road-testing’ of this draft tool by Durham and Gloucestershire RCCs – each RCC having identified two of their own recent activities (4 in all) for careful scrutiny. Again this validation and further development of the tool was undertaken with the involvement of local external validators.

two workshops involving about 30 RCCs plus various external agencies with an interest in RCCs as ‘delivery agents’, to receive and critique what by then had become a strong proposal regarding the form of the tool.

Conclusions

The main product of this research has been the tool– summarised above and set out at length in the body of the report. It is also being published as a freestanding Good Practice Guide.

But looking at the whole research exercise retrospectively a number of specific conclusions are drawn:

concerning the context – that RCCs need, and are generally ready, to demonstrate more convincingly the outcomes of their work both economic and non-economic;

concerning economic outcomes – that this is a difficult concept to grapple with both in theory and in practice and that it is a delusion to imagine that such outcomes can be easily expressed in simple arithmetic or cost terms;

concerning the nature of the proposed tool - that it has been found in this research to be user-friendly, relevant, useful and timely and that RCCs generally, and many of their stakeholders, have expressed their support for its application;

concerning its use by the RCCs – that there are important issues to be addressed by them regarding their information gathering and storing, the involvement of appropriate personnel within and outside the organisation, the exigencies of validation, and the embedding of the process into normal work practices. There are also important issues pertaining to the cost of staff time to be expended in undertaking the work.

concerning the transferability of the tool – that it should be capable of modification for use (a) in relation to the more social or community outcomes that characterise much of RCCs’ work, and (b) by other voluntary or community organisations facing broadly similar challenges.

1: Introduction: the context for the study

1.1The Purpose and Focus of the Research

This research project was commissioned by Defra early in 2005, with the aim of developing a practical tool for assessing the impact of England’s 38 Rural Community Councils.

The brief for the research stated that:

‘ Defra is interested in providing support to the RCC network to demonstrate their impact….we want the research to advise on how impact information can be captured as well as what information to capture. Defra therefore wishes to commission research

to identify the impact of RCC activity (with special account of the economic impact) and
to develop indicators to assess the outcomes from the funding they receive.’

It was agreed at the outset that the work would focus primarily on the economic outcomes of RCC work, while recognising that such outcomes tend to flow from a wide range of RCC work including that of a more explicitly social, environmental or community nature. It was also agreed that in no sense would the research be an evaluation of RCCs; rather it would attempt to develop a tool for the RCCs themselves to apply whenever a rigorous appraisal of their outcomes might be useful.

There have been four main phases of work:

a preparatory phase, involving a literature review and a scoping survey of RCCs,
a case study phase to explore in depth the work of just four RCCs and its apparent outputs and economic outcomes,
the subsequent drafting of an ‘outcomes tracing tool’,
the testing of that tool by two further RCCs – and its subsequent modification,

Thus the RCCs and their national association, ACRE, were active and enthusiastic collaborators in each phase of the work but responsibility for this final report rests with the research team alone.

1.2 The structure of the report

The research has involved bringing together three key dimensions to current policy debate. The first is the nature of rural communities in the UK and, in particular, the issues surrounding deprivation on the one hand and the social and economic revival of these communities on the other. Secondly, it examines the work of a particular set of agencies, which privilege the values of community development – of local involvement in the identification and meeting of social and economic needs. Thirdly, it is situated in the growing debate about ways in which the impact of work to deliver ‘human services’ – broadly defined – can be evaluated.

In this introductory chapter, we look at the first two of these dimensions; Chapter 2 will examine current debates on evaluation of public services with a particular emphasis on services that are set within a community development paradigm. In Chapter 3, we describe the methodology of the study, which led to the development of an analytical tool to be used by Rural Community Councils (RCCs) themselves as a means of looking at the impacts of their own work. The tool was focused on economic impacts – again broadly defined – but although most respondents to the study suggested that it was possible to separate out economic and social impacts of the work of RCCs, and indeed the brief given us by Defra implied this very strongly, there was also a strong view that economic and social impacts are very closely related and one may lead to or be associated with the other. Chapter 4 outlines the tool itself. Although this is focused very much on the work of RCCs, comments by other partners in this field of work have suggested that much of the structure of the tool may be applicable to other similar areas of work.