Evaluation for Tender: Programme Evaluation of Breaking the Cycle of Violence
(A joint programme funded by BBC Children in Need and the Premier League)
Summary
BBC Children in Need (BBC CIN) and the Premier League (PL) have come together to work towards a £6 million joint funding programme aimed at reducing youth violence in the communities in which Premier League football clubs operate. They are working with a range of partners, including the Premier League Charitable Fund (PLCF) and the club community organisations (CCOs) of Premier League football clubs, to develop and implement this programme. A developmental evaluation was undertaken of a 12 month pilot phase by the programme’s Learning Partner, the Institute for Voluntary Action Research (IVAR).
How the programme will achieve change
BBC CIN and PL commissioned an evidence review of the nature and scope of youth violence in the UK and of interventions aimed at violence reduction here and around the world. The review also looked at evidence concerning the role of sport, and football organisations in particular, in delivering these interventions. The review provided a range of evidence which has and continues to support the development of the programme.
Targeted programmes which include both general and specific theories of change and associated intervention components, demonstrate the best outcomes. For children and young people who have experienced/are experiencing multiple childhood adversities, programmes which support attachments, build resilience, recognise risk and challenge negative attitudes and group norms which support youth violence seem to be most effective…
Overall, we recommend an open-access provision accompanied with a targeted component for children and young people at higher risk or already engaged in violence. The open-access component would provide a general prevention format whilst also enabling pro-social peer influence to be facilitated; a central aspect of effective interventions. (Premier League & BBC Children in Need Youth Violence Scoping Review)
This evidence has been used to build a programme Theory of Change[1], which aims to reduce youth violence in the communities in which clubs operate by achieving an improvement in protective factors, and a decrease in risk factors for individual children and young people engaged in the programme. It aims to do this through three key routes:
1) Funding CCOs to provide targeted interventions for more at risk groups, working in partnership with local agencies (majority of funding in Phase II focused on this space) ;
2) Some funding to CCOs to undertake preventative work, delivered through their existing open-access provision;
3) Utilising the respective brands of BBC CIN and PL (and individual clubs) to deliver public messaging designed to challenge the attitudes which contribute to violence.
The Pilot: January 2016 – July 2017
The pilot phase of the programme focused on the first of these three routes. It provided funding of up to £50,000 each to four CCOs to establish projects focused on targeted interventions over a period of 12 months for young people at risk of or involved in youth violence.[2] The learning focus of the developmental evaluation undertaken by IVAR was on the following questions:
· What is the role and contribution of CCOs in breaking the cycle of youth violence?
· What does it take to make this happen?
Learning from the pilots has identified that CCOs can develop and deliver targeted interventions which achieve sustained engagement with young people and promote attitudinal and behavioural change. It has also begun to indicate where their strengths lie and where additional support and guidance is needed to ensure they can deliver safely and effectively.
Phase II: January 2018 – December 2019
The next phase of development will continue to explore and refine our understanding of the role and contribution of the various partners in the programme to breaking the cycle of youth violence, with BBC CIN & PL investing up to £1.7m in order to:
· Expand the targeted strand of the programme, building on the pilot learning and allowing CCOs to deliver their own specific responses to youth violence in their communities, but with a tighter focus on working with young people specifically at risk of perpetrating youth violence: Engaging up to 10 CCOs (4 pilots and 6 new CCOs)
· Design and deliver a preventative strand accessed through CCOs existing open-access provision and focused on ensuring that the attitudes which underpin violence are challenged, and positive attitudes are promoted: Engaging up to 14 CCOs (4 pilots and 10 new CCOs)
· Undertake consultation and research to support the development and piloting of a communications / campaign strategy designed to challenge the attitudes which contribute to violence.
The activity and learning from Phase II will evolve our understanding of the evidence-base which informed the pilot and the learning extracted over its delivery. We will use this knowledge to inform the shape and focus of the programme’s third phase, which will invest approximately £4 million over three years. To generate this knowledge, we will commission a new programme evaluation that, alongside continued developmental learning supported by IVAR[3], will shift the learning focus to identifying the:
1) Effectiveness of the funded work: are projects improving protective factors and reducing risk factors, and is a reduction in violence occurring or likely to occur? If not, why not?
In order to support a decision by or before May 2019 as to whether BBC CiN and PL will enter a third phase of delivery.
2) Features of provision: what are the consistent elements of CCO provision that contribute to positive changes to risk and protective factors where they’re occurring?
To support the continued development of the programme Theory of Change and strategy, including refined criteria for grants made in the third phase and a programme of training and support for all CCOs.
Timeline
The timeline below is designed to ensure that the decision on a third phase can be made at a point when substantial additional learning from CCO delivery will be available.
Phase II Programme Evaluation
The programme evaluation will adopt the learning principles established at the pilot stage. These are:
· Learning is embedded in the way the programme works.
· Programme and project delivery is evidenced-based.
· Adaptation is built in to the ways all stakeholders work.
· Evaluation findings will usefully inform the full programme.
The Phase II programme evaluation will be framed by the programme aims and three routes set-out above. We need to track learning about the focus, effectiveness and impact of both the programme and the individual CCOs delivering within it.
Programme Learning
Our primary aim is to improve protective factors and decrease risk factors around violence for individual young people engaged in the programme. The programme evaluation will be expected to assess the targeted, preventative and wider activation strands in their broader senses, including how they interconnect and enhance (or not) the overall programme aims.
CCOs Learning
For the most part, the programme evaluation will concentrate on evaluating the plans, activities and effectiveness of the CCOs in delivering their targeted projects with young people who are already engaged in or are at significant risk of engaging in youth violence. There will be lighter-touch assessment of the nature and effectiveness of preventative activities CCOs are delivering elsewhere in their overall provision.
The knowledge from the Phase II evaluation will offer vital insight that will inform our decisions and planning for a third phase of this programme. We expect the evaluation will help us identify:
· Characteristics of the young people reached by the programme, for example, what is their experience of violence; what combination of risk and protective factors do they present?
· Types and levels of changes in risk and protective factors, and other personal outcomes, the programme enables in these young people’s lives.
· Elements from across the 10 targeted CCOs theories of change that are most important in creating change (or acting as barriers), including the targeting of particular risk and protective factors and the underlying effect models that inform the interventions (e.g., a social norms approach, attachment theory etc.).
· Insight about particular intervention models that emerge from strong individual or groups of CCO projects. Where relevant, insight about these intervention models (or aspects of them) will be taken forward further in Phase III.
· Common features of provision needed to deliver effective and impactful targeted projects. For example, the nature of local partnerships, experience and training of staff, targeting of young people worked with and referral routes. The evaluation will highlight good practice in order to inform future provision.
· Indicative learning about how well the preventative approaches in open access settings change attitudes towards youth violence, boost other key protective factors or help identify at risk young people.
· Where CCO delivery is unique or distinct from other community based youth work. We are especially interested in the role of football and sport, the Premier League and club brands and connections with existing community organisations that enhance the CCOs ability to make a difference in young people’s lives.
Programme Evaluator Responsibilities
· To independently engage with and collect evaluation data from all CCOs delivering projects as part of the programme. This involves:
o Working closely with approximately six CCOs (e.g., three existing pilots and three new CCOs) to extract detailed learning about their delivery.
o Having regular contact with all ten CCOs delivering targeted projects to track the effectiveness of their plans and outcomes achieved (or not) for young people.
o More lightly connecting with and learning from all 14 CCOs that are providing preventative strands (consisting of 10 CCOs delivering targeted and prevention activities and up to four ‘prevention only’ CCOs).
· While tailoring of each CCOs monitoring and evaluation practice is expected (allowing for the variety of interventions and their levels of development), the evaluator will need to identify a set of common and measurable outcomes for the CCOs, and establish monitoring reports that will provide impact data for each CCO and the overall programme. To help with this, we expect to draw on and develop methods and measures already in use by the pilots.
· The Programme Evaluator will need to liaise closely with the PLCF in order to access the CCOs and manage the quality of information coming from reporting. This will require working with the PLCF to shape how the Views monitoring system is used to effectively capture outcomes data.
· Work closely with our Learning Partner (IVAR) to ensure all evaluative learning is considered by the programme boards and CCOs, and that all stakeholders learn, adapt and develop as knowledge becomes available. It will be essential for the Learning Partner and Programme Evaluator to form an effective and open working relationship, with particular focus on their distinct roles in supporting the programme’s learning ambitions. This includes ensuring that a smooth and mature transition of learning leadership for the programme occurs over the course of Phase II. The Learning Partner role will cease at the end of Phase II but further programme evaluation support will be required if we enter a third phase.
· Ensure that young people’s direct views and experiences of projects are part of the evaluation. If, when and how young people are engaged will require careful planning and ethical consideration, in order to ensure their safety and wellbeing.
· Update and maintain a record of the evidence relating to youth violence building on the scoping review, and work with the Learning Partner to ensure this information is part of ongoing programme discussions and decision-making.
· Alongside PLCF and the Learning Partner, create and plan a programme of learning events for the CCOs and programme stakeholders. We anticipate that there will be at least four learning events during Phase II.
· Contribute regularly to the programme’s operational and strategic boards. The boards will continue to learn and adapt based on the knowledge coming from the developmental and programme evaluations and any changes in other external evidence. The nature of reporting will be agreed at the time, but are likely to consist of verbal updates and short briefings.
· Deliver at least three key evaluation or learning reports over Phase II at particular milestones to be agreed with the boards. Most important is the evaluation report on the impact and effectiveness of the programme due around April 2019 to support a decision on the future of the programme. There will be a strong dialogue with the funders about final messaging leading up to this.
Criteria, Budget and Selection Timeline
As a priority, we are looking for an organisation with expertise in research and evaluation into youth violence and a track record in negotiating the ethical issues that arise from work with young people experiencing these issues and programmes that support them. We understand that there will likely be involvement of a range of personnel to support the evaluation; however, we will be looking to a senior individual to lead and take a substantial role in the work, including representing the programme evaluation at the programme boards.
We ask that a short proposal (up to 4 pages, not including appendices) be submitted outlining your understanding of this evaluation, the Programme Evaluator role and what expertise you bring that will help fulfil our ambitions. Your proposal should include:
· Your interpretation of the programme’s evaluation needs, and what you see as the key questions to be addressed.
· How your approach will align with the programme learning principles, especially those of reflection and adaptation, and provide the learning necessary for Phase III.
· The research and data collection methods you will likely use within the evaluation.
· The ways in which you anticipate working with the Learning Partner to maintain effective cycles of learning, communication and responsiveness throughout Phase II.
· An outline of how you will use your expertise and resources to keep the programme stakeholders up-to-date on evidence and developments in addressing youth violence. We expect this to be more than a task assigned to a junior researcher.
· Details of all relevant experience of consultants involved including the level of involvement anticipated for each member of the team.