North Ayrshire SNAP - Evaluation of impact and outcomes, one year on.
Xanthe Wylie
Educational Psychologist
Ceri McMillan
Research Assistant Education & Youth Employment.
Introduction
North Ayrshire (NA) is one of 5 Local Authorities with the highest concentrations of multiple deprivation in Scotland. Where there are issues of inequality and exclusion, the health and wellbeing of children and families can be at risk, with parents potentially under stress and finding it difficult to implement effective behaviour management techniques, and some children displaying angry and aggressive behaviours. In NA the 2015 cost of Supervision Orders (SOs) for children looked after at home was £2.75 million and there continues to be an increase in the numbers of children being educated in Nurture Bases (see Appendix 1). Whilst in the past year, referrals to Social Emotional and Behavioural Need (SEBN) schools have decreased significantly, costs in 2015 reached over £2.3 million.
Since Summer 2013, Health and Social Care and Education partners have been working together to plan and effectively implement a 13 week cognitive behavioural programme called Stop Now And Plan (SNAP) programme in NA. The introduction of SNAP was motivated by the Early Intervention agenda and had two key aims, firstly, to reduce the numbers of primary school-aged children made subject to statutory orders and formal protection measures, and secondly, to reduce the use of alternative day placements by maintaining children in mainstream school.
In July 2014, SNAP began running with children and families from the Irvine and 3 Towns Areas. The first groups were positively evaluated with evidence of effective early intervention, behavioural change within families, children and parents developing positive mental health and strong partnership working. SNAP CANADA assessed NA fidelity and implementation as excellent in March 15, inviting NA to discuss SNAP implementation at the International Law and Mental Health Conference in Vienna in July 2015. The Integrated Children’s Service Plan 2016 sets out plans to expand SNAP into other NA localities to improve children’s outcomes
Since July 14, 4 cohorts of children and families have worked through the SNAP programme. This report evaluates the impact of the SNAP on the first 3 cohorts and whether the significant changes reported post SNAP are sustained 6 months and 1 year post SNAP.
The results contained in this study indicate that SNAP
· positively engages and supports high-risk children and their families at an early stage
· has a positive impact on problem behaviours within family systems and schools up to 6 months post intervention
· is delivered by a skilled team which is building strong partnerships across Health & Social Care and Education agencies
Background
SNAP is a Canadian cognitive behavioural therapy programme designed to help angry and aggressive children between the ages of 8-11, and their parents learn how to effectively manage their emotions. The programme runs for 13 weeks, with separate child and parent groups running on a weekly basis. Boys groups run separately from girl groups. Children learn cognitive and behavioural skills and are given structured practice experiences to apply these skills to specific circumstances. Topics include recognising body cues, joining in, playing fair, and managing peer pressure. Parents discuss with other parents their use of parenting strategies and are supported to manage their own issues and anxieties when parenting their children.
Between June 14 and July 15, 76 per cent (n=16) of Irvine and 3 Town schools referred into SNAP (see appendix 2). Following a screening process, 3 cohorts of children (n=51) between the ages of 8 – 11 and their parents or carers (n=62) entered into the NA SNAP programme. The mean completion rate was 76 per cent (n=39) for children’s groups and 70 per cent (n=42) for parent groups (see appendix 3.)
Evaluation
The purpose of the evaluation is to evaluate the impact of SNAP and its sustainability at 6 and 12 months post intervention. The evaluation was carried out using quantitative and qualitative measures. The following data was gathered and analysed in order to assess the effectiveness of SNAP:
· Child Behavioural Checklists (CBCL) completed by parents and Teacher Report Forms (TRF) pre and post SNAP programme.
· CBCL and TRF 6 months after SNAP for cohorts 1 & 2
· CBCL and TRF 12 months after SNAP for cohort 1
· Analysis of total problem scales on TRFs and CBCLs
· Analysis of total adaptive scales on TRFs
· Analysis of total competence scales on CBCL
· Analysis of Supervision Order information
· Analysis of nurture base and SEBN placement data
· Attendance records for young people in secondary school
· Fortnightly semi-structured interviews with SNAP workers, notes coded and reduced into effects matrix.
· Videos of child and parent groups, notes coded and reduced into effects matrix.
· Discussion with Head Teachers (HTs) at cluster meetings
· Feed-back from semi-structured interviews with secondary school Senior Managers and Pastoral care teachers
The children’s perceptions of SNAP are key to the evaluation. Structured interviews were carried out Interviews were undertaken with Cohort 1, 1 year on from SNAP, using a fixed questionnaire format, which involved open questions, prompts and some scaling questions using a Likert type scale. Analysis covered the following:
· use of SNAP
· perception of self-control
· perception of relationships with peers and adults
· engagement with school
Major Findings
Quantitative
Whilst a small number of children were on SOs before attending SNAP, there has been a small reduction in SO numbers post SNAP and no children have been placed on SOs during or after SNAP.
Table 1 –TOTAL NUMBER OF CHILDREN ON COMPULSORY SUPERVISION ORDERS 2015
LOOKED AFTER AT HOME (NA) / IRVINE AND 3 TOWNS (8-12YEARS) / SNAP CHILDREN PRE SNAP / SNAP CHILDREN POST SNAP230 / 45 / 5 / 3
The majority of children remain in mainstream schooling (See Table 2). Young people who have moved from primary to secondary into mainstream have made positive transitions, with 25 per cent (n=2) requiring Joint Support Team interventions and average attendance rates of 90.4 %. Only 1 young person has been unable to sustain mainstream schooling, moving to an SEBN school in April 2016, 17 months after attending SNAP (see Table 3.)
Table 2 - SCHOOL PLACEMENTS
NUMBER OF CHILDREN / MAINSTREAM PRIMARY / MAINSTREAM +NA NURTURE BASE / MAINSTREAM SECONDARY / SEBN SCHOOLS39 / 64% (25) / 15% (6 , 2 post SNAP) / 18 % (7) / 3% (1)
Table 3 - SECONDARY SCHOOL DATA
NUMBER OF YOUNG PEOPLE IN SECONDARY EDUCATION / % IN MAINSTREAM / AVERAGE ATTENDANCE / % REQUIRING JOINT SUPPORT TEAM INTERVENTION / % SEBN8 / 88% (7) / 90.4 % / 25% (2) / 12% (1)
Analysis of CBCL and TRF data (see appendix 4 for detail):
Following SNAP, total problems (withdrawn, somatic complaints, social, thought and attention problems, rule breaking and aggressive behaviour, internalizing, externalizing) decrease significantly ( p = 0.000)with a moderate effect size of 0.51. Positive impact is sustained and increased 6 months post SNAP with an effect size of 0.71 (see Tables 4 & 5).
Impact is not sustained 1 year post SNAP.
TABLE 4: TOTAL PROBLEMS PRE AND POST SNAP (CBCL and TRF)
TABLE 5: TOTAL PROBLEMS PRE SNAP AND 6 MONTHS POST SNAP (CBCL AND TRF)
Post SNAP total adaptive behaviours (academic, working hard, behaving, learning) improve significantly ( p = 0.01) with a moderate effect size of 0.51 (see Table 6) and positive impact is ongoing 6 months after SNAP with a slightly reduced effect size of 0.46 (see Table 7).
There is no impact on adaptive behaviours 1 year post SNAP.
TABLE 6: TOTAL ADAPTIVE BEHAVIOURS PRE AND POST SNAP (TRF)
TABLE 7: TOTAL ADAPTIVE BEHAVIOURS PRE SNAP AND 6 MONTHS POST SNAP (TRF)
Qualitative (see appendices 5, 6 and 7 for detail)
Pre SNAP it was evident that there were significant levels of verbal aggression and long-standing issues of non compliance in family homes, with relationships under stress and children struggling to maintain positive relationships in school with peers and adults.
X really struggles to get on with peers, therefore mum does not let him out to play in the local area….mum admits to giving X what he wants to save any hassle (Pre SNAP assessment information Child X , 10 years old, Cohort 2)
Where children were pushing boundaries and there was an apparent lack of consistency in effecting consequences, relationships were often under stress, with adults struggling to show affection and children reciprocating with aggression
There’s too much shouting in the house (Child D to SNAP worker pre SNAP cohort 2)
Many children had faced adversity in early childhood including domestic violence, parents with addiction issues, mental health issues, family break up and bereavement. These difficult early years experiences were impacting upon the wellbeing of the family.
Mum held a knife to her throat in Jan 2010….dad has a history of violence and Y isn’t allowed unsupervised visits with dad, doesn’t see his dad at present…..Y has threatened to throw himself out the window but has never done it. He has been physically aggressive to mum and gran by throwing things at them (Child Y, 10 years old, Pre SNAP information)
THEME 1 – BEHAVIOURAL CHANGE
Parents implementing positive change in daily routine
Working with parents to ensure immediate, positive responses to positive behaviours are given, and immediate sanctions for problem behaviours are received, resulted in significant changes in daily routine
K is going to school himself and sitting with the family for dinner (Snap Worker re K, Cohort 2)
Increased engagement in school
Prior to SNAP, many children were having issues with motivation, attention, and were often causing high level disruption. Intensive focus on identifying body cues, stop strategies, hard and cool thoughts and choice of positive plans resulted in positive change for some
G recently took a lesson on bullying in PSE class, he engaged with the class and had an enthusiasm for teaching (Pastoral Care Teacher St Matthews of G – Cohort 2)
‘It’s like night and day….he listens better in class and it’s much easier to get him back (when he’s losing focus) (Class Teacher of R – Cohort 3, Feb 2016, 5 months post SNAP)
THEME 2 – POSITIVE MENTAL HEALTH
Evidence of emotional regulation for children and parents
By systematically and regularly getting children and parents to regulate angry feelings by getting them to stop, think about the consequences of their behaviour and plan positive alternatives before acing impulsively, participants developed the ability to communicate better
At home it is calmer, there is more talking and less verbal aggression (SNAP worker re N)
Dad is calmer and dealing with things better, not a raging bull anymore (Group 3 parent)
One year post SNAP, of the children made available by schools for interview (n=6), 67 per cent (n=4) were able to remember what SNAP stood for and have used it again.
I use the stops to calm down – take a drink of water or count to 10. (Child E, Greenwood Academy S1)
Change in Mindset
Pre SNAP many parents perceived their children as being the problem, that behaviours were fixed and did not recognise the impact of positive parenting and nurture on behaviours
If only the parents knew what the group would do for them. All you need to do is listen to your children, spend time with them and stop shouting. I learned this early on in SNAP (Mother – cohort 2)
In instances where parents have previously taken their child’s version of events at face value, parents have shifted in their thinking
I’ll back you up where appropriate but you have to take responsibility’ (Mother about school – cohort 3)
There is evidence of a sustained shift in mindset. Child E’s primary school PT reported that 6 months after SNAP, at the end of Primary 7, E wrote a letter to his P6 teacher reflecting on his difficult behaviour during P6 and apologising. It appears that the impact of exploration and roleplay around apologizing (SNAP session 6) is sustained one year on
I get into trouble for wee things and then I apologise and teachers say thanks for apologising.(Child E, Greenwood S1 – Cohort 1 )
Increased confidence
Believing that behaviours are within child arguably promotes a resigned acceptance that one has no agency is unable to change behaviours. At the start, parents were coming to SNAP groups stressed, feeling that they were unable to control their children
Can’t thank SNAP enough, parent’s confidence…they carry through what they say they will do, even when K can’t manage, they manage him (Grand-mother of K, Cohort 2 – Dec 15, 3 months post SNAP)
Gran is stronger against her daughters and it as if the roles have shifted. On the last night gran had a new haircut and new clothes..she’s put her SNAP certificate on top of the TV (SNAP team leader – Grand-mother of S, Cohort 2, girls group)
Relationship Building
By enabling parents to implement effective behavioural management strategies in a consistent manner, and enabling the development of self regulation, relationships have become more positive for some families