Evaluating Family Interventions for childhood obesity using the National Obesity Observatory Standard Evaluation Framework

Dr Penney Upton, Miss Charlotte Taylor, Mrs Seonaid Beddows and Professor Dominic Upton

University of Worcester

Introduction

In response to the limited evidence of the effectiveness of weight management interventions and the need for methodologically sound evaluation, the National Obesity Observatory (NOO) has developed a Standard Evaluation Framework (SEF) which can be used to structure the evaluation of weight management programmes. As part of the continuing quality improvement process of the SEF, this case study discusses the application of this framework during an evaluation study carried out by the University of Worcester on behalf of the Department of Health West Midlands.

This evaluation represents the first application of the SEF at a regional rather than a programme level and included seven family based intervention programmes in place in the West Midlands between 1stJuly 2007 and 1st July 2009:

  1. Fun for Life: Walsall
  2. Fitter Families: Stoke on Trent
  3. Goals: Sandwell
  4. MEND: Birmingham; Coventry; Dudley; Herefordshire; North Staffordshire;Sandwell; Shropshire; Stoke on Trent; Walsall; Warwickshire; Wolverhampton; Worcestershire;
  5. One Body One Life: Coventry
  6. Watch It!: Birmingham
  7. YW8?: Telford and Wrekin

Evaluation strategy

Assessment was made of the range of data collected by each of the intervention teams in order to establish the extent to which interventions followed the best practice model provided by the SEF. Information concerning the feasibility (including barriers to data collection) of collecting desirable as well as essential data was gathered in order to establish which measures should be used routinely in practice. This part of the analysis was also used to inform recommendations for Key Performance Indicators.

Outcomes of audit of routine data collected by SEF

Routine data collected by each intervention were audited against SEF essential and desirable criteria. None of the programmes included in the evaluation collected 100% of the information highlighted as either essential or desirable by the SEF and variation between programmes was evident:

-The number of essential criteria collected ranged from 62%-95%;

-A similar pattern was evident for desirable criteria, with the percentage of completion ranging from 31% to77%;

-Easily accessible fields such as intervention name, contact details, intervention dates, timescales and locations were provided by all interventions in addition to basic data including participant age, weight and height;

-Quality assurance mechanisms and quality impact assessment were predominately omitted from data collection with the exception of 2 interventions;

-Information regarding participant satisfaction was not provided to the evaluation team;

-Measurement of physical activity levels is essential according to SEF guidelines and this data was collected by all except one intervention team. The measures used varied by intervention but mainly asked about the number of days in the past week during which moderate activity had been undertaken for 30-60 minutes;

-Dietary measures are also recommended as essential by SEF. This data was collected by 5 interventions. Actual measures varied, however one common question concerned the number of portions of fruit and vegetables consumed each day.

Efficacy of using the SEF for evaluation

In summary, the SEF enabled an effective comparison across child weight management interventions, in addition to the identification of positive areas for development. If used appropriately, the SEF has the potential to serve as a thorough data collection tool and increase the evidence base of effective paediatric weight management interventions and methodologically sound evaluation. For example, it is envisaged that the framework might be used as a development tool for staff involved in the evaluation process, emphasising the need for careful design at each stage, highlighting the importance of accurate and consistent measurement and the sharing of best practice. It may also enable comparison of different types of intervention.

Challenges to using the SEF

However, despite the potential of the SEF as an evaluation tool, a number of challenges to its application were also identified:

-Data collection and recording strategies varied between interventions. In addition, the quality of the information provided was not consistent. For example, one programme provided a thorough list of primary and secondary aims, with references, whereas others provided generic aims, e.g. “to reduce the risk of obesity...”

-The majority of the interventions had not previously used the SEF as an audit tool. As a result, the information provided did not fall neatly into the categories specified thus making the evaluation process difficult. Furthermore, it was clear that not all programme leads had in-depth knowledge and/or awareness of the SEF criteria. This presented significant challenges in extracting relevant information. In addition, whilst the use of essential and desirable categories provides a basic division, the relative importance of the criteria within the desirable category is unclear.

-Only height and weight, measures of dietary intake and measures of physical activity levels are classified by the SEF as essential criteria. If the aim is to provide a holistic evaluation of child weight management interventions then it may be suggested that additional facilitators and barriers to lifestyle change such as self esteem, should be classified as essential as opposed to desirable criteria. Similarly, if the evidence supporting the long-term efficacy of paediatric obesity interventions is to be improved, follow up data on key measures over a greater period than one year should be classified as an essential criterion.

How could these challenges be overcome?

In order to overcome the challenges identified, it is recommended that:

-Weight management programmes for children and families should follow SEF recommendations regarding what outcome measures to collect. For example, given the difficulty of gauging the impact of weight change on a child’s weight status, SEF recommend the use of BMI, rather than weight as a measure of physical change. The use of additional measures of adiposity such as waist circumference is also recommended by SEF. Furthermore, there needs to be some standardisation with regards to how this information is recorded.

-Behaviour change related to food intake and exercise should also be measured in a systematic and standardised way and this information should be fed back to clients as part of the change process.

-The SEF is marketed and promoted to relevant parties in an effective manner to increase knowledge and awareness.

-Consideration should be given to reclassifying additional facilitators and barriers to lifestyle change, such as self esteem, as essential as opposed to desirable criteria.

In order to establish the long term effectiveness of such programmes, follow up data on key measures over a greater period than one year should also be classified as an essential criterion in weight management programme evaluation.

1