Value for Money and Policy Review – Probation Projects
Value for Money and Policy Review
Report on
Projects Funded by the Probation Service
May 2008
Value for Money and Policy Review – Probation Projects
FOREWORD FROM THE DEPARTMENT OF JUSTICE EQUALITY AND LAW REFORM
This Value for Money and Policy Review of Projects funded by the Department of Justice, Equality and Law Reform, through the Probation Service provides a solid analysis of the current position and maps, through a series of time bound recommendations, a forward strategy that will provide better governance, better management information and a more effective and efficient delivery to the client group of the community based projects in receipt of State Funding.
This Review was undertaken during a period of significant change within the Probation Service, which has culminated in a strengthened senior management structure, focussed deployment of resources and a systematic review of how it does its business. It is not surprising in this changed process that the focus of some projects may not be fully aligned to the aims and strategic objectives of the Probation Service.
There is no simple solution to dealing with offending behaviour. The partnership approach of the Probation Service has evolved and developed over many years within communities whereby local links and projects are established to meet identified needs for a cohort of offenders, ex-offenders or persons engaged with the Service in those communities.The partnership approach provides important support and is a very valuable resource for the Probation Service in its drive to enhance community safety through bringing about behavioural change in this group. Working in partnership with local communities is at the heart of what the work of the Probation Service is all about and is an acknowledged strength of the Service. This Review will help enhance that bond by putting in place key reforms to support and sustain best practice.
It is intended that this report should act as a catalyst to ensure that the required realignment of Project focus with Probation Service objectives occurs. The Department and the Probation Service fully accept the findings as set out in the report. They are comprehensive and challenging and the recommendations provided are designed to provide a time bound framework to addressing the key findings. The Probation Service, through the assignment of dedicated resources, and in partnership with the Department and the funded Projects is committed to implementing the findings of this report within existing resources.
1
Value for Money and Policy Review – Probation Projects
Table of Contents
Chapter Page
FOREWORD i
EXECUTIVE SUMMARY
Introduction
Methodology
Findings
Summarised Findings and Recommendations
1. Introduction
Background to the Value for Money and Policy Review Initiative
Structure of the Review Programme
Expenditure Review Steering Group
Terms of Reference
Methodology
Structure of the Review
2. Background: The Probation Service
Status
History and Recent Developments
Working Within the Community
Funding of the Probation Service
Funding of Community Voluntary Bodies
Context and Selection of Topic for Review
3. International Trends and Experience - Literature review and Research
Northern Ireland and Community Involvement
New Zealand – A Focus on Outputs
United Kingdom– Impact of Employment
United Kingdom–Evidence Based Development
Conclusion
4. Project Profiles and Probation Service Objectives
Probation Service Objectives
Services Provided
Project Aims and Objectives -
Project Outputs
Are Projects meeting the Objectives of the Probation Service?
5. Profile of Projects Funded by the Probation Service
Years in Receipt of Funding
How are Projects Funded?
What is the Funding Received from the Probation Service Spent on?
Operation of the Project
Existence of a Strategic Plan
Communication of Objectives
Strategic Alignment
Routine Reporting and Monitoring Systems
Reporting of Effectiveness and Outcomes
Review Meetings with Probation Service
Comparing Results to Previous Periods
Comparing Results to Other Similar Projects
Research on Best Practice
Key Performance Indicators Defined
Cost per Client
Multi Annual Funding
Formal Aftercare
Client Recidivism
Contractual Relations with the Probation Service
Access to Services
Service Availability
Rapid and Effective Service
Level of Demand
Networking with other Similar Services
Aggregation of All Responses
Positive Progression
Difficulties Facing Projects
Throughput and Capacity
6. Performance Indicators
Introduction
Performance Indicators
Performance Monitoring and Evaluation
Performance Framework
Appendix I Management Structure Probation Service
Appendix II Probation Service Projects and Grants Approved 2005 – 2007.
Appendix III Selected International Experience
Appendix IV Selected Bibliography
Appendix V Terms Of Reference
Appendix VI Recidivism in Ireland Press Release
Appendix VII Joint Prison / Probation Accreditation Panel - Accreditation Criteria
Appendix VIII Copy of Questionnaire
Appendix IX Profile of Selected Projects
1
Value for Money and Policy Review – Probation Projects
EXECUTIVE SUMMARY
Introduction
This report presents the results of avalue for money and policy review of the programmes and projects supported by the Probation Serviceand was carried out during 2007 by the Department of Justice, Equality and Law Reform (“the Department”) with the assistance of Petrus Consulting Limited. The draft review was independently assessed byMr. Brendan Shiels. A steering group comprising senior staff of the Department along with representatives of the Probation Serviceoversaw the production of the report in accordance with the defined terms of reference.
Methodology
The methodology consisted of reviewing internal documentation along with previous reports including in particular a series of six preliminary reports prepared on a sample of projects representing small, medium and larger sized (in terms of funding) projects. Further visits or telephone interviews were carried out with these projects subsequently. A detailed questionnaire was circulated to projects in receipt of funding and meetings with Probation Officers and officials were held. A review of relevant international literature was carried out covering a number of countries including Northern Ireland, the United Kingdom and New Zealand.
Findings
The Probation Service is going through a process of change and refocusing that will give it a more strategic direction and allow it to play a more significant part in the criminal justice family and enable it to act as an important component in the Government’s approach to tackling crime in the 21st century.In parallel with the transformation currently taking place in the Probation Service, similar change is required in the management and operation of the funded projects.
Many probation projects were set up at the instigation of the Service to supplement the work of the Service and to provide programmes for offenders. This involvement has been reviewed and is being put on a more secure foundation i.e. agreed conditions of grant, specifying the expectations held of the projects is underway. There is now a clear emphasis on defined outputs, rather than required inputs. Some management committees have been revitalised, and some projects that were not delivering or were no longer appropriate in modern conditions have closed.
International experience shows us that there is developing a degree of consistency of approach to assessment, evaluation of probation interventions. Best practice in probation includes a focus on accountability and monitoring and evaluation of the outcomes associated with interventions.
Over time, new programmes, projects, therapies and interventions will be proposed and trialled. Any such new interventions need to be carefully assessed and monitored in terms of their effectiveness and cost. Furthermore, for the forseeable future, existing projects are likely to incur the bulk of the expenditure. These projects also need to be rigorously assessed in terms of their impact, effectiveness and cost.
In 2006, the Probation Service had a gross budget of approximately 50 million euro (2007, €59 million) and approximately 42%of this was used to provide capital and operating funding to a range of projects that support the work of the Service (2007, 36%). Funding is made available through the Probation Service for a range of facilities and programmes that support the work of the Service in the management of offenders. The Service currently supports 66[1]projects nationally that offer programmes to offenders.
The Probation Service has as its primary objective a desire to bring about a positive change in the behaviour of offenders, a reduction in re-offending and a reintegration of ex-offenders. In turn it wishes to develop processes, standards and structures with partner agencies and projects to ensure good practice, value for money and to ensure value added.
Questionnaire Response
A questionnaire was issued to all the projects funded fully or partly by the Probation Service and 56 completed responses were received representing a response rate of 82% by number. The projects responding also account for 93% of the value of grants allocated in 2006. The responses to the questionnaire are self-reported and may therefore present a more positive interpretation of activities than an objective assessment.
- Projects responding to the survey dealt with a total of 4,913 referrals during 2006 and, of this number, 3,533 individuals were directly assisted or joined the projects involved. Of those individuals referred to projects, 1,380 (28%) did not commence or engage with the project.
- In addition, the number dropping out after commencement was 804 or 16% of the total number of referrals.
- The waiting time to join projects was approximately 2.2 weeks on average.
- The 56 projects which responded to the questionnaire have 1,839 places available and an annual capacity to deal with just over 3,000 individuals.
- The average duration of participation is 14 weeks for addiction type projects, 27 weeks for accommodation and counselling type projects and 45 weeks for education, work training and placement projects.
- Projects are operating close to capacity and, on average, at any time there are 82 places available representing less than 5% of available capacity. This low level is a good indication of the activity level of the projects as unfilled capacity is a waste of resources.
Objectives, Outputs and Outcomes
At an overall level, the objectives of the projects are difficult to identify and are defined more in terms of the nature of the service that is provided rather than the desired outcome to be achieved. This leads to difficulties when the projects are evaluated because there is a lack of clarity regarding the linkage between the specific objectives of the projects and the overall objectives and mission of the Probation Service. Projects need to have clear, quantitative objectives that are aligned with the objectives of the Probation Service.
A large number of the projects are not accustomed to considering and reporting the outputs and outcomes from their projects and are more familiar with dealing with the inputs and their activities. There needs to be a system whereby the outputs and outcomes associated with projects, individuals and types of intervention can be tracked.
The Probation Service needs to work with the projects to develop and monitor specific, measurable objectives and outcome indicators that will allow subsequent evaluation to take place. An important outcome in this respect is the recidivism rate but other changes such as attitudes to offending behaviour and acceptance of the need to change are also crucial.
In general, there is no evidence-based, quantifiable, linkage or relationship between the objectives of the Probation Service and the actual achievements of the type of project or individual projects.
The projects in the main do not specify outcome related objectives, their recording systems do not track the necessary information and their reporting systems deal mainly with input and activity type information.
Performance Measures
The performance measures used by projects at present are predominantly related to the level of inputs such as financial resources used or the number of referrals handled. While these are valid measures they are insufficient on their own to allow the Probation Service to identify what has been achieved for the resources used or to assess the overall achievement of desired outcomes.
The Probation Service needs to balance the measures currently used with measures and indicators related to the outcomes achieved by individual projects. Ultimately, this means tracking how well individuals participating on the projects have progressed over time measured in terms of behavioural change and recidivism rates.
The indicators to be used will need to be carefully defined and larger projects should be encouraged and supported to establish procedures to evaluate their own performance. Smaller projects should have a simpler set of measures suitable to their activity but which can be linked to the strategic objectives of the Probation Service.
There is therefore a need to develop a tailored but consistent set of performance measures for the individual projects and also, by aggregation, for the overall population of projects supported by the Probation Service. The performance measures need to be tailored to the specific circumstances of each project bearing in mind particularly:
- The size and resource capability of the project
- The suitability or otherwise of any existing performance measures produced by the project
- Whether it is fully or partially funded by the Probation Service.
For projects that are partially funded by the Probation Service there is a need to collaborate with the other funding providers, such as FÁS and the Department of Health and Children, to develop an agreed set of performance measures that meets the needs of the Probation Service.
Cost Information
The average cost per individual project participant is €5,519 taking into account only the funding from the Probation Service.
The average cost of €5,519 per participant suggests that projects would need to achieve a reduction in average recidivism rates of at least 14% in order to justify the expenditure when compared solely to the likely future cost of imprisonment.
Further research and evaluation on the costs and benefits associated with specific projects and interventions types should be carried out in order to identify a mechanism to allocate funding to those projects which can demonstrate reductions in recidivism and positive behaviour change in excess of the associated cost. Further academic research is required on recidivism rates and an appropriate baseline is required.
It is recognised that it is difficult to compare recidivism rates since different studies can use widely different approaches and time periods. There is a need to agree on a definition of recidivism which can be applied to funded projects in a consistent manner.
Management of Projects
As part of the next round of funding, each project should be required to provide a copy of their strategic objectives showing their alignment with the strategic objectives of the Probation Service. Specifically, this should include the objectives associated with positive behaviour change and the objectives associated with quantifiable reductions and improvements in recidivism rates.
A contract or service level agreement would form the basis of an agreement between the projects and the Probation Service setting out,inter alia, the requirements and objectives of the Probation Service, detailing the monitoring and evaluation procedures, specifying performance indicators linked to key objectives, scheduling review meetings and covering legal, administrative and governance requirements.
Funding should be based on service level agreementswhich should be regularly reviewed with a detailed annual review performed whereby the release of funds under a multi-annual funding agreement would be considered in the light of progress and achievements against key objectives. The provision of funding should be linked to the achievement of specific performance targets so that there is a clear incentive for projects to achieve the Probation Service objectives.
1
Value for Money and Policy Review – Probation Projects
Summarised Findings and Recommendations
Finding / Recommendation / Target / Priority / Effort / TimingSome projects are unclear about the objectives and requirements of the Probation Service. / Probation Service to provide clear communication of core policy goals and objectives of its work emphasising the need to focus on outputs and outcomes to all projects in a seminar and written format as appropriate. / Hold seminar and/or provide written guidance to all sole funded projects in attendance and majority of the others. Probation Service to communicate requirement and call for funding applications. / High / Medium / Q3 – 2008
All projects do not routinely report on measurable performance criteria. / The Probation Service to advise and set criteria for Projectsto have clear, quantitative objectives that are aligned with the objectives of the Probation Service. / Develop a minimum set of measures (5-10) for all sole funded projects to report on quarterly and consult with other funding agencies where relevant. / High. / Medium / Reporting in place by Q3 - 2008
Need to build on the requirement to submit Business Plans introduced last year. / Funding applications provided to the Probation Service to include updated business plans, with strategic objectives and outcomes aligned to Probation Service objectives. / All projects submit objectives and outcomes aligned to Probation Service goals. / High / Medium / Q3 2008
Monitoring and reporting of performance is not consistent for all projects. / The Probation Service , in consultation with the Department of Justice, Equality and Law Reform to develop an agreedcentral database to enable the funded Projects to report monthly/quarterly/annually as appropriate to the Probation Service in standard format covering a defined list of KPI’s. / Develop uniform database and agree reporting formats, frequency etc. with the Projects. / High / High / Reporting in place by Q4 - 2008
The level and frequency of external monitoring and evaluation of projects is inconsistent. / All projects which are solely funded and other projects in receipt of more than €100,000 should be independently evaluated and/or operationally reviewed at least once in a three year cycle. / One quarter of sole funded projects to be independently evaluated and /or operationally reviewed in each 12 month cycle beginning from 1 July, 2008. / Medium / High / Ongoing
The staff resources devoted to managing projects within the Probation Service do not reflect the size and importance of the funds involved or the services required. Organisation and administration of projects needs to be more structured. / Project support team to be established in the Probation Service to, inter alia, draw up guidelines for establishing and running projects, development of detailed service level agreements, assistance with legal queries, assessment of annual funding applications, capital expenditure assessments, reporting and operational evaluations, assistance to probation staff, governance etc. / Unit to be established with necessary skills by reallocating existing staff. / High / Medium / Q3 2008
The relative or absolute effectiveness of specific interventions is unknown. / Establish an independent accreditation process for programmes delivered by projects following consultation. Establish and support systems to monitor and independently evaluate such programmes. / Top 6 projects to have the programmes they deliver to Probation clients independently accredited. Rolled out to all sole funded projects subsequently. / High / High / Commence Q2– 2008
End Q2 2009
Many projects focus on inputs and activities. / Link funding to demonstrated achievements against Probation Service objectives measured in terms of outputs and outcomes. / Top 6 projects in funding terms to have funding linked to outputs by 2009. Extend to all sole funded projects in 2010. / High / High / Q2 2009
Projects suffer because of a lack of certainty regarding funding availability and levels. / Examine the feasibility of introducing longer term rolling funding commitments for key service providers that can demonstrate that they are aligned with the objectives of the Probation Service. / Identify funding mechanism and strategic providers by Q4 2008. / High / Low / Q4 2008
Tracking clients within projects and across the criminal justice system needs to be improved. / Intensify efforts to develop the case tracking system to allow clients to be traced through the Criminal Justice system. / Work already being carried out in this area to be expedited. / High / High / Substantive work to be underway by Q2 2009
Limited research on Recidivism in Ireland is taking place. / There is a need for more in-depth analysis of existing material and also the development of a baseline for future independent evaluations. / Commission research on aspects of recidivism in Ireland and the role and impact of the Probation Service.Provide funding for monitoring and independent evaluation of projects as part of core funding with results provided to the Probation Service. / High / Low / Commission Research Q3 2008
Recidivism rates for clients directly referred to Projects by the Probation Service should be collated by the Probation Service. / Probation Service to examine how best to track and report recidivism rates. / Client recidivism tracking format to be developed. and implemented. / High / Low/Medium / Q4 2008
1