Making Justice Work

Enabling Access to Justice Project:

Evaluation of the Making Advice Work programme

December2014

Contents

Foreword

Executive summary

Background

Purpose

Overview of methodology

Advice outcomes

Client outcomes

Justice outcomes

Research steering group

Data sources

Making Advice Work: Stream 1 evaluation

Making Advice Work: Stream 2 evaluation

Lessons for measuring the impact of advice and representation

Contribution of the programme to Money Advice Service and Scottish Government high level outcomes

Future monitoring of client outcomes

Lessons for funding and providing advice and representation services

Funding advice and representation services

Providing advice and representation services

Foreword

The Making Advice Work programme supports projects helping people in Scotland facing debt and other problems stemming from benefits changes and the ongoing impact of the economic downturn. The programme started in October 2013 and runs to end March 2015. It is jointly funded by the Scottish Government and the Money Advice Service, being administered by the Scottish Legal Aid Board.

We know that problems and disputes involving debt and benefits can have adverse consequences for people, affecting their confidence, well-being, financial situation and health. The client telephone survey work carried out for this evaluation provides important evidence on how information, advice and representation help people in these circumstances; demonstrating how the impact on individuals contributes to the strategic outcomes that are sought by the Money Advice Service and the Scottish Government.

The evaluation covers the first 11 months of the Making Advice Work programme, over which period projects have assisted nearly 18,000 people. Analysis of client survey results and project data demonstrates that projects are achieving positive impacts for those people accessing help:enabling them to resolve their current problem and equipping them to deal with future challenges.

The client survey forms a strand of the Enabling Access to Justice Project, which is part of the Making Justice Work programme that aims to modernise Scotland’s justice system. The client survey addresses a recommendation from the Landscape review of publicly funded legal assistance, which was for SLAB and stakeholders to seek to develop a suite of case-level outcomes appropriate to each substantive area of law, including common justice outcomes, that could be used by funders and organisations to measure success across different advice and representation services.

Alongside the Making Advice Work programme, the Scottish Government also provided funds to the Welfare Reform Mitigation Fund, which was administered by Citizens Advice Scotland. This fund is currently being evaluated with a focus on the wider impacts of advice and we look forward to the results, which will contribute to our overall understanding of the value of advice.

Already we are using the results of this reportto inform how projects seeking continued fundingfrom SLAB can implement lessons learnt regarding the provision of advice services. The methods used in the report and the questionnaires produced can also be applied more widely. For example, as major funders and providers of debt and benefits advice, local authorities may wish to carry out a similar survey exercise with clients of these services to assist with benchmarking and identifying good performance.

In order to encourage take up of this approach by more funders, we will be making the questionnaires and related materials available through the forthcoming good practice principles for funders of advice which will be hosted on SLAB’s website.

Jan Marshall, Deputy Director, Civil Law and Legal System Division, Scottish Government

Lindsay Montgomery CBE, Chief Executive, Scottish Legal Aid Board

Executive summary

  1. The purpose of the report is to assess the extent to which advice and representation provided by projects under Streams 1 and 2 of the Making Advice Work (MAW) programme help to achieve desired outcomes for individual clients.
  2. The desired outcomes include those set specifically for the MAW programme, as well as the Scottish Government’s national outcomes and the outcomes sought in the Money Advice Service’s framework.
  3. Overall, results from the data gathered have been positive, as compared to other research work[1] and the experience of those participating in the expert steering group. Furthermore, the telephone survey method of collecting information about client experiences and the impact of advice has been successful and provides a baseline set of findings for comparison in future. Running the survey again, on either a national or local basis, would help to build a fuller picture of performance and the potential for improvement.
  4. Each Stream of the programme has been assessed against its specific aims. Both Stream 1 and Stream 2 are performingwell against the indicators set for measuring success. A key overall measure of success is that 78% of the 419 problems reported in the client survey had been resolved fully or partially by the advice received from projects.
  5. In terms of the national outcomes set by the Scottish Government, the report sets out how the impact of advice on individuals clearly forms part of an assets-based approach to resolving problems that people face, empowering them to feel more in control of their current situation and equipping them to deal with future issues. This contributes to several national outcomes including those relating tojustice, health, reducing inequalities and improving opportunities for children and families.
  6. Similarly, the evaluation has provided good evidence that the projects funded as part of the MAW programme are achieving the positive outcomesset by the Money Advice Service for people facing debt problems. The results show that, following interaction with projects, clients understand the options for resolving debt problem, take action to address the problem and reduce outstanding debt, with benefits for their wellbeing and future resilience.
  7. Findings from the evaluation suggest that both funders and providers should be aware of the large proportion of clients who reported having mental health problems, despite this client group not being specifically targeted by projects. This suggests that mainstream advice should not be seen as a standalone service, but be linked to organisations that provide specialist support. This would help both to resolve the debt or benefits problem and to help tackle the root cause of the issue. This kind of approach is being trialled specifically under Stream 3 of the MAW programme, which will be evaluated separately.

Background

  1. The Making Advice Work (MAW) programme[2]was set up to support organisations helping people in Scotland facing debt and other problems stemming from benefits changes and the ongoing impact of the economic downturn. The £7.73M programme is jointly funded by the Scottish Government and the Money Advice Service. There are three Streams in the programme. This evaluation looks at Stream 1 and Stream 2, covering the 11 month period from 1st October 2013 to 31st August 2014. As we exclude from our analysis the three second tier advice[3] projects under Stream 1, this evaluation covers a total of 53 projects. Stream 3 will be subject to a separate evaluation.
  2. Stream 1 of the programme is designed to focus funding on projects that help connect people across a geographic area with assistance in resolving complex benefit and related debt problems, and projects that include targeted assistance to help people successfully make the transition to the new benefits system. These projects should tackle unmet need, connect to and add value to existing arrangements for advice, information and representation.
  3. Stream 2 of the programme focuses specifically on advice, information and representation for tenants of social landlords affected by changes to the benefit system, including helping to avoid homelessness and tackle housing debt problems.
  4. Stream 3 is a thematic stream and the main aim is to improve access to direct assistance to avoid/resolve problems formally or informally for particular groups of people. We invited applications under the thematic strand which would test options for achieving sustainable outcomes for them in holistic/multi-disciplinary ways – for example including partnerships between support agencies and specialist advice providers.
  5. In November 2013, data show that around 520,000working age individuals living in Scotland[4] were in receipt of one of the Department for Work and Pensions’ key benefits[5] and nearly 70,000 claimants of housing benefit were affected by the under-occupancy charge, or ‘bedroom tax’[6]. The projects funded under Streams 1 and 2 of the MAW programme recorded having assisted 17,900 people over the 11 month period from 1st October 2013 to 31st August 2014, which forms the period under assessment for this evaluation.
  6. Projects funded through the MAW programme are designed to complement and add value to existing publicly funded advice and representation services, such as those provided by local government, the third sector and solicitors. The application process and ongoing monitoring of projects seek to ensure this is happening in practice.

Purpose

  1. The purpose of the report is to measure the extent to which advice and representation help to achieve funders’ desired outcomes for individual clients. The objectives are to:
  2. Provide an evaluation of the Making Advice Work grant funding programme for Streams 1 and 2.
  3. Assess the contribution of the programme to Money Advice Service and Scottish Government high level outcomes.
  4. Test a basis for consistent monitoring of client outcomes in future.
  5. Inform future policy development of funding and providing advice and representation services.
  6. Addressing this set of aim and objectives helps to demonstrate whether the Making Advice Work (MAW) programme has achieved the aims set by Scottish Ministers and by the Money Advice Service[7]. This work also demonstrates how the funding has contributed to the achievement of outcomes sought in the Money Advice Service’s framework and the Scottish Government’s national outcomes.

Overview of methodology

  1. This report focuses on the Stream level, rather than looking at the performance and impact of individual projects. Projects are continuously monitored by SLAB’s policy and development team[8].
  2. Theevaluation mainly employed quantitative methods, using data collected through online survey, telephone interview and project databases. Thismethod providesmeasureable outcomes for advice, which assists with assessing the extent to which advice and representation supports different policy goals, split by advice, client and justice outcome[9] typology.

Advice outcomes

  1. Advice outcomes tend to be practical gains that an agency achieves for a client as a direct result of its work. Advice outcomes include things like:
  2. Increased income due to a client receiving a benefit
  3. Retention of housing – so the client is not homeless
  4. We captured most of the advice outcomes through the monitoring database that projects routinely complete.[10] More information about the database is set out in paragraph 22.

Client outcomes

  1. Client outcomes can be defined as those which relate to the wider aspects of the client’s life such as their physical and mental health and general well-being. They may include things like:
  2. increased confidence
  3. greater ability to help yourself
  4. We measured these through a telephone survey with people who had been assisted by projects. More information about the telephone survey is set out in paragraph 18 and detailed responses have been published alongside this report.[11]

Justice outcomes

  1. We would define justice outcomes as those which relate to the way in which problems are dealt with. They may include things like:
  2. trust in the adviser
  3. the end result being fair
  4. being satisfied with the way in which the problem was resolved
  5. We measured these through a telephone survey with people who had been assisted by projects. More information about the telephone survey is set out in paragraph 18 and detailed responses have been published alongside this report.[12]

Research steering group

  1. An expert group[13]was convened to assist SLAB by:
  2. Helping to revise the Money Advice Service’s evaluation tool for debt advice to fit with welfare benefits problems
  3. Providing oversight of research method for evaluation
  4. Assisting with interpreting results
  5. Commenting on draft reports
  6. The group met four times over the period from June to November 2014. Their advice and guidance was extremely valuable in drawing together this report.

Data sources: client telephone survey

  1. A telephone survey was conductedwith clients who had given permission to be contacted in connection with evaluation. The telephone survey was conducted by an independent research company, Progressive Partnership[14],who were awarded the contract after a competitive tender process.
  2. We received contact details for 3,021 clients and achieved 664 interviews, giving a good response rate of 22%. Respondents came from 45 out of the 56 projects involved in these Streams. In the report we refer to this as the ‘client survey’. Where a percentage figure is given, we have set out the actual number of responses that refers to in the format (n=number). For example, where 58% of respondents indicated that they had been diagnosed with a mental health problem, the figure (n=328) shows that the number of people giving that response in the client survey was 328.

Table 1: Sample overview for project client survey

  1. Fieldwork was undertaken between 24th September and 22nd October 2014. The average lengths of interview were:
  • Debt: 23 minutes
  • Benefits: 21 minutes
  1. We have not carried out any significance testing of the survey results as the sample is self-selecting and does not lend itself to this kind of statistical analysis. In all cases where we refer to the client survey, weset out the number of respondents underlying a percentage figure, so that readers can see where small sample sizes are being relied upon.

Data sources: project database and monitoring returns

  1. Each project provides SLAB with detailed information every three months covering progress against their work plan and outcomes, including feedback on the impact of welfare reform. This information allows SLAB to continually monitor and evaluate how each individual project is performing, as well as giving valuable insight into policy areas that can be fed into the Scottish Government’s welfare reform scrutiny group. SLAB has published case studies that illustrate how projects are helping people and there are plans to publish statistical information online as well.[15]
  2. Each project also submits an excel workbook (which we refer to as ‘the database’ in the report) every three months to the grant funding team. The database includes anonymised details of each matter dealt with by the project, such as the type of dispute and the outcome of the case, and characteristics of the clients who sought assistance[16]. The databases have been combined into a single workbook that covers all projects and Streams.
  3. Database outputs rely upon accurate and consistent recording across projects in the programme. Although every effort is made to ensure this consistency by SLAB’s policy development team, different approaches to recording can be expected to remain. Some areas of work appear to have been more problematic for projects to record accurately; where this is the case, we have highlighted the additional caveat in the report.
  4. A selection of applications and monitoring returns from projects has also been reviewed.

Data sources: online survey of social landlords

  1. Additionally, an online survey of social landlords was prepared by the Board’s Research Unit and the fieldwork took place in October 2014. Forty three responses were received from across the 50 social landlords invited to participate. This has been published separately alongside this report[17].

Making Advice Work: Stream 1 evaluation

  1. Stream 1 of the programme is designed to focus funding on projects that help connect people across a geographic area with assistance in resolving complex benefit and related debt problems, and projects that include targeted assistance to help people successfully make the transition to the new benefits system. These projects were set up to tackle unmet need and connect to and add value to existing arrangements for advice, information and representation.
  2. A total of 31 projects are funded under this Stream, of which 3 provide second tier support to advice and support agencies. Projects under this Stream were expected to start from 1st October 2013.
  3. Overall, in the period covered by this report projects have assisted around 11,700 people with 12,700 debt, benefit and housing matters.
  4. While the number of clients helped is slightly below what projects expected to achieve by this stage in the programme, this is mainly as a result of most projects being up and running slightly later than October due to recruitment processes, but also perhaps reflecting slower than anticipated roll out of welfare changes.
  5. Table 2 shows the number of interviews achieved by type of advice (benefits advice or debt advice) received by respondents, those who identified experiencing a specific problem noted in the survey and the number of specific problems experienced by those respondents[18].

Table 2: Overview of client survey responses for Stream 1

  1. We can see from Table 2 that a higher proportion of people who got advice on a debt problem experienced one of the issues specified in the survey (75%, n=76), as compared to those receiving benefits advice (49%, n=145). This lower proportion for benefits may reflect the omission from the questionnaire of a specific option on assistance with completing an application for benefits.
  2. The extent to which respondents reported that the advice received resolved the problem experienced is set out in Table 3.

Table 3: Resolution of specific problems from client survey for Stream 1