Evaluation Final Report

Evaluation to demonstrate the impact to date of the two rounds of NHS Kensington & Chelsea investment in Voluntary and Community Sector (VCS) services:

Final Report

March 2010

Consultant – Wendy Sugarman

Project Support Officer – Claire Jarrold

WSA CommunityConsultants Ltd

Tel: 020 8376 0985 Mobile: 07932 730477

E-mail:

Contents

Page no.
1 / Brief Summary / 3
2 / Conclusion and Recommendations / 5
3 / Introduction / 9
4 / Context / 10
5 / Evaluation Approach / 11
6 / Access / 14
7 / Reach / 18
8 / Community Engagement / 22
9 / Partnership / 25
10 / Quality / 29

Appendices

Page no.
A / Methodology / 33
B / List of Projects / 35
C / Focus Group and Interview Schedules / 37

Acknowledgements

We would like to thank the service users and service providers who were either interviewed directly by ourselves or via the project evaluations.

We would like to acknowledge and thank our research supervisor, Dr Gillian Granville from Gillian Granville Associates.
1Brief Summary

This evaluation does give a fair indication of the positive impact around health inequalities that the NHS Kensington and Chelsea investment in the voluntary and community sector has achieved. It is not possible to say the extent of the impact, but the key findings broken down in the body of the report give clear examples of where impact has been made and some of the effects of the investment.

The evaluation has been analysed under the key outcomes around health inequalities identified by the evaluation steering group at the beginning of this process. A key point to note under each of the main outcomes is as follows:

  • Access: There is some good evidence of increased access to NHS and statutory services, and even stronger evidence that a broad spectrum of service users have increased their knowledge of health issues and related services, giving the option to access services in the future.
  • Reach: Between them, the 19 projects estimated that they had reached just over 2000 beneficiaries in Quarter 3 (October - December 2009 inclusive) alone, with most beneficiaries based in the Borough of Kensington and Chelsea. However the projects have in some cases counted their reach differently and therefore it is hard to do much detailed analysis around this.
  • Community Engagement:There were few examples given by service providers of where feedback from a service user had influenced service planning. There were examples of some initiatives involving peer educators or volunteers, but this area could be better reflected and developed.
  • Partnership:Signposting is a particularly strong area, with most organisations including some level of this, and several projects making it a key focus. Challenges around this include the barrier of multiple assessment forms and referral mechanisms not always being in place.
  • Quality: All the projects showed progress with their intended outcomes and some of the projects exceeded their set targets. However, it is clear that for consistent quality, ongoing learning and adjustments are necessary for projects to develop and improve effectiveness, and that stronger structure and support in this area could be beneficial. There is some indication that more joined-up working could increase efficacy.

Several recommendations came out of this evaluation but the six key recommendations are detailed below:

  1. Build evaluation capacity of the sector: The evaluation identifies a number of inconsistencies in data including the way that projects complete beneficiary monitoring; the degree of information returned from different voluntary sector providers; and the variance in evaluation forms projects are using with their service users, making it hard to compare across services with different impact / satisfaction scales and qualitative questions. This in itself leads to a clear recommendation around building the evaluation capacity and capability of the sector to enable key data to be captured using both qualitative and quantitative methods. This would ensure a consistent approach to evaluation which would better inform the sector as well as commissioners.
  1. Provision of a member of staff to undertake development work around key recommendations, preferably ‘joint’ between KCSC and NHS Kensington and Chelsea: A key issue that needs to be addressed is that of joined-up working within the sector and also with NHS Service Providers. Many of the recommendations around quality, partnership, access and community engagement addressed within the main recommendations section need strategic support and central capacity, which should be jointly owned by the sector and NHS Kensington and Chelsea. Hence, our recommendation for a joint post to enable and support this.
  1. Accessible Information about the voluntary sector commissioned projects: The need for quick and easily accessible, high quality information about the voluntary sector commissioned projects came from the sector, service users and NHS providers. Ideas for both an online resource and leaflets mapping projects were put forward. Dissemination of leaflets should include GP surgeries, clinics, health and public services. The online resource was particularly discussed in relation to engaging GPs, whom many of the VCS services had found an especially challenging group to build partnerships with. A direct referral system would also help to enable a quick and easy route for GPs to refer patients.
  1. Commissioner Intervention around cross-sector referral: Following on from the last point, addressing challenges around referral at commissioner level would benefit the investment and the ability of the voluntary sector to build relationships and protocols with a range of NHS Service Providers. Some of the key issues for services in this area are multiple-assessment processes, willingness to refer, and understanding of voluntary sector service provision in the NHS services.
  1. Skills Audit and Training: Building on previous skills audit work KCSC has undertaken, implement a full skills audit and training needs analysis of volunteers and paid workers within the 19 projects, in order to work out a training and support programme for the sector in relation to work around health inequalities and NHS Kensington and Chelsea targets.
  1. Voluntary sector to be more joined up particularly in approaching the statutory sector: A joined-up approach is key particularly when the voluntary sector is finding it difficult to engage and work with statutory NHS service providers. The sector needs to ensure that statutory sector providers are not being approached by several different voluntary sector providers and that as far as possible there is information sharing around partnership within the sector to enable this, and some kind of protocol in place around the process for engaging and working with NHS service providers. Similarly the referral process between the voluntary sector projects themselves would benefit from being developed so that clients accessing specific services could be better signposted to other voluntary sector services where necessary.
    2Conclusions and Recommendations

2.1Access

Conclusions:

  • There is some good evidence of increased access to NHS and statutory services, and even stronger evidence that a broad spectrum of service users have increased their knowledge of health issues and related services, giving the option to access services in future.
  • Although data has not been specifically collected on whether or not positive experiences of project participants using NHS services have increased, there is demonstrated greater positivity about the concept of using services and some reported increased confidence of service users in both accessing services and engaging with NHS staff.
  • However some issues around or barriers to access have been identified that are specific to relations between the voluntary sector and the NHS. The voluntary sector is acknowledged by NHS service providers interviewed to be able to offer a more continuous service and enable long-term more holistic care/services, and build good trust relationships with clients, but the voluntary sector projects report that they struggle to get the referrals from NHS staff / teams.
  • As well as significant challenges in engaging with (and obtaining referrals from) NHS services (especially GPs), some projects also report struggling to obtain relevant information about NHS services and staff, and achieving (potentially) mutually beneficial relationships that utilise the skills of each and reduce resources. In turn however, some of the NHS staff interviewed also had concerns around access of voluntary sector information.
  • Both voluntary sector and NHS staff did however report some successes in joined-up approaches around access, as well as recognising the benefits of one another’s services.

Recommendations:

  • Develop an online resource with information about voluntary sector projects funded under this investment and a referral form that can be completed and submitted straight away
  • Develop a joined-up resource with all the services, covering how to access them and who they are for, with a map for service users that can be left in GP surgeries, clinics, pharmacies, Children’s Centres and other health-based and public services
  • Address the issues at a commissioning level of referral between the voluntary sector and NHS service providers and build in mechanisms to address this and ensure better links
  • Fully brief the voluntary sector providers with regard to all referral information required for the range of relevant NHS services, identify current gaps and provide information
  • Identify strategies to forge better working relationships between key NHS services and the voluntary and community sector. Ensure this is joined up so that NHS service providers are not being approached by several different voluntary sector providers but through contact with one will be given information about all 19 projects
  • Voluntary sector to consider centralising some of the information individual groups hold on NHS providers to support more joined-up working between the sectors
  • Consider expanding the role of the voluntary sector as a broker where there is breakdown in trust between the NHS service and the service user

2.2Reach

Conclusions:

  • Between them, the 19 projects estimated that they had reached just over 2000 beneficiaries in Quarter 3 (October - December 2009 inclusive) alone, with most beneficiaries based in the Borough of Kensington and Chelsea.
  • In analysis a large degree of variance was noted in the ways in which projects listed beneficiaries so the above figure is an estimate.
  • Ethnic data was collected from 733 individual service users, of whom just over half (59%) identified as non-White. More than 30 other individual or mixed ethnic groups were identified by service users. Gender data was collected from 1618 service users, of whom around one-third (34%) identified as male, around two-thirds (65.5%) as female, and 0.5% as transgender.
  • In the evidence submitted for the quarter three monitoring there were few examples of outreach work although we know that this is one of the areas considered a strength by service providers.
  • Partnerships, both between statutory and voluntary sectors, and between voluntary sector organisations, are shown to be a key way of extending Reach.
  • Resolving barriers such as illiteracy and language are clearly sometimes important factors in engagement. Several voluntary sector projects translated information, used (volunteer) translators or offered translation services, and supported service users to access services in their own language.
  • Service users, through the focus groups, identified the time limitation of some of the services and the effect that the end of a service has on service users as a concern.

Recommendations:

  • Develop clear guidance on data monitoring to support the uniformity of recording beneficiary information
  • Undertake more joined-up working around language issues and explore the possibility of a language bank for the voluntary sector that all groups can tap into
  • Consider investing in / extending use of peer educators / volunteer ambassadors within projects
  • Do further work to show outreach levels and good practice in outreach
  • Build on current partnerships to extend reach, support all projects to map key stakeholders around reach
  • Encourage joined-up working across the sector to increase cross-referral and information-sharing, particularly where client participation is coming to an end and they may benefit from another service
  • Address referral issues between the statutory NHS services and the voluntary sector. Identify barriers and how to reduce them and increase referral rates

2.3Community Engagement

Conclusions:

  • Projects are using widely differing evaluation methods, standards and terminology. There was little consistency in terminology used across projects or clear indication of what comparative scales meant. Whilst it is difficult to achieve statistically significant results where participant numbers are relatively low, the inconsistency around terminology and approach means that the feedback that is obtained may be less valuable or truly comparative in some instances.
  • Where NHS service providers were interviewed, those interviewees seemed largely unaware of voluntary sector feedback mechanisms and some voluntary sector services more generally.
  • One of the indicators around this outcome is how service users influence their own voluntary sector service; the evidence around this through this evaluation was patchy. Within the service providers’ focus group only two participants could give a concrete example of where feedback from a service user had influenced service planning. There were further examples from the quarter 3 monitoring of 3 or 4 initiatives involving peer educators or volunteers but our feeling is that this area could be better reflected and developed.

Recommendations:

  • Support self-evaluation of projects through a joint member of staff working for KCSC and NHS Kensington and Chelsea. Support to projects to be offered on a one-to-one basis and through action learning, training or workshops
  • To support the above and the wider evaluation of this work, develop standardisation in evaluation tools; look at ways for projects to share strong but simple evaluation methodologies, including qualitative and quantitative data
  • Develop some case studies for the sector to share good practice as to how service users have fed back their ideas and evaluation feedback to service providers and this has influenced changes in the service. Disseminate these case studies through a skills development workshop

2.4 Partnership

Conclusions:

  • Partnership working is evidently a strength where successfully developed and of particular validity given the limited resources that are available to most voluntary sector projects. There can be significant advantages of it in extending, enhancing and promoting services.
  • Signposting is a particularly strong area, with most organisations including some level of this, and several projects making it a key focus. Challenges around this include the barrier of multiple assessment forms and (simple, clear) referral mechanisms not always being in place.
  • There is some evidence of shared learning and increased understanding and skills either within or between voluntary organisations or across sectors as a result of partnership working. However there were relatively few specific examples of how and where this is happening. It appears to be inconsistent practice, and an area that could be developed.
  • There are also challenges and gaps evident around more basic information-sharing. This includes the concern that information on voluntary sector services is not always readily available for GPs and other NHS service providers, and that the voluntary sector sometimes struggles to identify relevant people and services to approach.

Recommendations:

  • See recommendations under access in terms of information sharing between the voluntary sector and NHS services
  • Look at ways to reduce administration and assessment forms where clients are multiple service users referred across services and / or sectors
  • Consider communal and consistent system of presentation of voluntary sector services to GPs and support staff, for example a database, to give GPs and other NHS staff an easy referral service to alert clients to voluntary sector services
  • Look at how models of passing on statistics / feedback to health professionals and commissioners can be replicated and further developed to inform commissioning

2.5Quality

Conclusions:

  • Trained staff and volunteers are of course crucial to strong projects and high quality results. Many examples are given of both direct service delivery and instances where NHS services have benefited as a result of VCS intervention.
  • This includes introductions for NHS staff to community groups, referrals and encouraging and facilitating service users to access services, helping to resolve individual concerns around services and so on.
  • It is clear that for consistent quality, ongoing learning and adjustments are also necessary for projects to develop and improve effectiveness, and that stronger structure and support in this area could be beneficial. There is some indication that more joined-up working could increase efficacy.
  • Through projects, individuals are clearly recognising changes that can be made, and that they wish to make, in their own lives; further support beyond the immediate scope of some projects may be needed to ensure progresshowever, and this is an area where joined-up working may also show dividends.

Recommendations:

  • Have a central member of staff to work jointly for the NHS and KCSC in implementing a number of recommendations around information, evaluation, partnership and quality
  • Undertake a skills audit of those working and volunteering across the 19 projects with a view to joined-up training and development opportunities
  • Encourage individual projects to look at how clients can feed back and receive support beyond direct interventions.
  • Look at how to share organisational learning and successful organisational learning models across the voluntary and community sector

3Introduction

In 2008, NHS Kensington and Chelsea embarked on a programme of new investment in community services through a commissioning programme aimed at voluntary and community organisations (VCOs). Two open bidding rounds were held, one in the Spring of 2008 and one in the Autumn of 2008, which resulted in 20 projects receiving commissioned contracts to deliver services to Kensington and Chelsea residents, to a total value of approximately £2M per year, with contracts extending to March 2011 for all except one project in the first round and to March 2012 for the second round. This evaluation involves 19 of the 20 funded projects.