Evidence Sub Group &
Economic Evidence Sub Group
Health Improvement Review
Assessment of MEND
The following report includes the assessments of the evidence base produced by the Evidence subgroup (section 1- Evidence of Effectiveness) and the Economic Evidence sub-group (section 2 – Evidence of Cost effectiveness).
Section 1:Evidence of Effectiveness
1.Introduction
A core component of the Health Improvement Review (HIR) has been the assessment of the evidence-base for initiatives included in the HIR. This report describes the methodology for, and findings of, the assessment of the Welsh MEND programme.
2.Methodology
Assessment of the initiative employed a dual approach:
- Assessment of the potential effectiveness of the initiative by review of research on the effectiveness of similar initiatives or of component interventions (some initiatives involve more than one intervention).
- Assessment of the actual effectiveness or impact of the initiative by review of any available evaluation reports for the initiative in Wales.
2.1Review of potential effectiveness
The methodology adopted for this review followed systematic review principles of transparency, a priori setting of the research question, search strategy, inclusion/exclusion criteria, critical appraisal and standardised data extraction.
2.1.1A ‘question’ was developed for each initiative following the PICO format[1]. For MEND, the question was: Is MEND effective in reducing BMI in overweight and obese children compared with other, or no, interventions?
2.1.2Due to the time constraints of the Health Improvement Review, a ‘best available evidence’ approach was taken for the reviews of research on potential effectiveness of initiatives.Key words and search terms were derived from the question and a pragmatic search strategy designed using specified health databases and search-engines. For initiatives where recent high quality secondary analyses of the primary literature were found, searches were narrower and terminated at an earlier stage. Searches for questions that yielded little high quality data initially were broadened by date or by search terms in an attempt to capture related work. The time-constraints did not enable hand-searching or contacts with experts in the field (external to Public Health Wales) to search for missed or unpublished data, however, an iterative process of related article searches were run on key papers for some questions to try and capture information that the initial search strategy had not identified. All reasonable efforts were made to locate the most relevant and highest quality evidence in the short-time frame allocated.
The search terms used for MEND were: MEND and obes* and the following databases were searched:
TRIP Database
NHS Evidence
Campbell Collaboration
EPPI Centre
PubMEd
Google Scholar
Health Evidence Canada
2.1.3Retrieved articles were recorded in the ‘Evidence Mapping Table’ for the initiative (Table 1) and were screened for inclusion by two reviewers independently (disagreements resolved by discussion), on the basis of direct relevance to the initiative or component interventions and type of article, thus single studies were not included if higher level evidence was available:
Primary group of sources:NICE guidance, Single systematic reviews from Cochrane, Campbell Libraries, the EPPI-Centre
Secondary group (include if no primary group evidence items available):RCT or evaluation of robust design looking at appropriate outcomes
Other study designs to be included if no primary or secondary sources are available, the
quality of these to be judged separately/recorded on a case-by-case basis.
2.1.4Information was extracted from each included article into a standardised template – the Evidence Mapping Table, for each initiative.
2.1.5Each included article was assessed in terms of ‘reliability’, strength and direction, using the following ‘Evidence Grading Scheme’:
++ Directly relevant evidence that the intervention evaluated is beneficial/useful/effective - the evidence comes from a reliable source[2] and is guidance based on RCTs, SRs or robust evaluations of appropriate outcomes or is a well conducted systematic review.
+ Directly relevant evidence that the intervention evaluated is beneficial/ useful/effective -the evidence comes from a reliable source and is a robust/large RCT or robust evaluation of appropriate outcomes.
+/- Conflicting evidence (from reliable sources) about the usefulness/efficacy of the intervention being evaluated.
0 Directly relevant evidence on effectiveness of an intervention the same as, or similar to, the initiative, of acceptable reliability, is lacking.
-- Directly relevant evidence that the intervention being evaluated is not
beneficial/useful or is ineffective - the evidence comes from a reliable source and is guidance based on RCTs, SRs or robust evaluations of appropriate outcomes or is a well conducted systematic review.
- Directly relevant evidence that the intervention evaluated is not beneficial/useful or is ineffective -the evidence comes from a reliable source and is a robust/large RCT or robust evaluation of appropriate outcomes.
QlWell conducted studies using robust qualitative research methods which cast lighton how/why intervention might be effective/ineffective or have important implications for interpretation of findings or other included studies.
The ‘evidence grades’ for each included article were recorded in the Evidence Mapping
Table for the initiative (Table 1).
2.1.6A subjective judgment of the overall balance of evidence grades given to included articles was then made by one reviewer, to give a ‘Summary Evidence Grade’ for the Initiative:
++There is consistent, strong relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be effective.
+There is some relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be effective.
+/-Relevant evidence (from reliable sources) about the likely effectiveness of the intervention/approach employed in the initiative is conflicting.
0Directly relevant evidence (from reliable sources) about the likely effectiveness of the intervention/approach employed in the initiative is lacking.
- -There is consistent, strong relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be ineffective.
-There is some relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be ineffective.
Checks were made for consistency of application of these ‘Summary Evidence Grades’
through comparison and discussion amongst the reviewer team.
2.2) Assessment of initiative
An ‘Initiative Assessment Log’ was then completed (Table 2). Information from any evaluation or other reports about the initiative in Wales was considered for relevance at this stage and pertinent information summarised into the log along with the Summary Evidence Grade and other information. A final‘Initiative Grade’ was then applied by one reviewer using set criteria (see Annex 1). This therefore takes into account both the evidence of potential effectiveness and evidence of actual effectiveness in Wales, where available. This Initiative Grade’ will feed directly into the Programme Budgeting and Marginal Analysis which forms the decision-making framework for the Health Improvement Review. Initiative grades were checked for consistency by comparison and discussion amongst the review team.
Initiative Evidence Assessment Report MEND finalPage 1
Public Health Wales Observatory / Health Improvement ReviewEvidence Sub Group &
Economic Evidence Sub Group
3.Findings of the assessment
Table 1: Evidence Mapping Table for MEND
Study Details / Study design / Outcome measures / Main findings / Results / Evidence Grading / Include?Reason for exclusion
- Sacher PM et al, 2010
Target group:
Children aged 7-13 and their families
(Children aged 8-12 years in RCT) / RCT
(n=116)
MEND versus waiting list for MEND (Control)
Attempts made to overcome lack of measurement blinding
Non validated questionnaire used to assess activity level.
Baseline characteristic similar apart from gender, more girls in intervention group. / Waist circumference, BMI, body composition, physical activity level, sedentary activities, cardiovascular fitness, and self-esteem were assessed at baseline, 6 & 12 months / Participation in the MEND Program was effective in reducing adiposity in children and effects were sustained 9 months after the intensive part of the intervention. Importantly, the program is one of the few paediatric obesity
interventions which conforms to expert recommendations and
is deliverable in a primary care setting. High-attendance rates suggest that families found this intensive community-based intervention acceptable. These results suggest that the MEND Program is a promising intervention to help
address the rising obesity problem in children. Further larger RCT is ongoing to measure the effectiveness of the program when delivered on a larger scale using methods that will address the limitations of the current trial. / Subjects were 60% white and 40% non-manual s-e group. Participants in the intervention group had a reduced waist circumference z-score (-0.37; P < 0.0001) and BMI z-score (-0.24; P < 0.0001) at 6 months when compared to the controls. Significant between-group differences were also observed in cardiovascular fitness, physical activity, sedentary behaviours, and self-esteem. Mean attendance for the MEND Program was 86%. At 12 months, children in the intervention group had reduced their waist and BMI z-scores by 0.47 (P < 0.0001) and 0.23 (P < 0.0001), respectively, and benefits in cardiovascular fitness, physical activity levels, and self-esteem were sustained.
Note: Groups only compared at 6 months. Data collected at 12 months for intervention group only. There was some attrition - 6 cases from the intervention group dropped out before the trial started. At 6 months, 17 from the intervention and 11 from the control groups did not attend the follow-up, leaving 37 children (62%) in the intervention group and 45 (80%) in the control group to be assessed at 6 months. The analysis was carried out without accounting for uncertainty due to missing data, or imputation analysis, in the sensitivity analysis. / +/-
Note:
Small study with methodol-ogical limitations / Yes
2. Techakehakij W, 2011
MEND
Target group:
Children aged 7-13 and their families / Ph D Thesis, cost-effectiveness analysis of MEND / Cost per QALY / Detailed critical appraisal of the MEND programme.
Suggested that there is selection bias...MEND participants being from less deprived backgrounds.
Analysis is for England - estimated ICER of £1,668.2 per QALY.
The thesis points out: In the roll-out phase of MEND eligibility criteria changed from BMI ≥ 98th centile (as for the RCT) to BMI ≥ 91st centile, therefore coverage of MEND has been extended to overweight children without supporting evidence from an RCT. The rollout also differs from the RCT in that the qualifications of the staff were ‘diminished’ in order to make the programme cost-effective. The thesis compared health indicators before and after this phase of the intervention - health indicators of children at the end of the programme are significantly improved compared to their baselines. However, it is unclear as to whether the improvement in health during the roll-out phase actually resulted from the programme effect, due to the lack of a control group. Also regression to the mean bias may apply. In addition, the duration of follow-up is considered too short. Programme effectiveness derived from a 10-week follow-up cannot be used to predict long-term outcomes, as a considerable number of long-term uncertainties are involved. Owing to these weaknesses, inferences drawn from the short-term data are limited in their applicability, particularly for use in the policy decision making process. / The data used in this cost-effectiveness analysis came from the MEND rollout phase (January 2007 to December 2009) and included 6,828 participants, with an average follow-up duration of 10 weeks (this is a relatively short period of time re: outcomes and was due to large amounts of missing data from the roll-out phase). The study concluded that children with particular characteristics experience a higher degree of the effectiveness of MEND 7-13 in reducing BMIs. These characteristics are: male, White ethnicity, high baseline BMI, and frequent attendance of the programme. For Asian children, the only significant predictor is having parents who own their own houses; no evidence of association is shown in Black children; and total attendance and gender are significant predictors in the ‘Other’ group. The results also showed that the programme centre influences the change in BMI to some extent in all children. / +/- / Yes
3. York Health Economics Consortium and NEF Consulting 2010
MEND
Target group:
Children aged 7-13 and their families / Economic evaluation which combines cost-effectiveness evaluation with an assessment using ‘Social Return on Investment’ principles.
The analyses in this report are informed by a Randomised Control Trial (RCT) conducted in 2005-07 (26 above), data from the subsequent roll-out to 16,000 children and new stakeholder engagement with children, parents and programme staff conducted specifically for the analysis. / Average cost per enrolled child
Combined value of health and well-being outcomes
Incremental cost-effectiveness ratio (ICER) / The study concludes that MEND 7-13 is an effective intervention to reduce the number of obese children, and that it represents a cost-effective use of healthcare resources.
There are methodological weaknesses in the data used in this analysis (see refs 1 & 2) above and thus the assumptions made for the analysis about the expected change in BMI and other outcomes are questionable. / The incremental cost-effectiveness ratio (ICER) of the programme is £1,671 per QALY gained, considerably below the NICE threshold for cost-effectiveness of £20,000-£30,000 per QALY gained and compares favourably with other obesity interventions.
Health and well-being combined value; £4,021.42-£5,534.12 per enrolled child.
Average cost per child: £415,77 / +/- / Yes
4. Sacher PM et al,2010 / Paper notes Aus RCT due to start in 2010 and US RCT due to start in 2011.
Aus RCT believed to be in preschoolers- not relevant to age specified. No record found of US trial. / No
No comparison group
No follow-up beyond 10 weeks
5. Singhal, 2009 / RCT
N=300
Age 7-13 / This is just a web page – record of the trial in the clinical tria.ls database 2009...no results / No- no data yet
Completion 2013
References:
- Sacher PM et al, 2010. Randomized controlled trial of the MEND program: a family-based community intervention for childhood obesity. Obesity18 (Suppl 1): S62-S68.
Available at: - Techakehakij W, 2011. Cost effectiveness of child obesity interventions. York: University of York.
Available at: - York Health Economics Consortium & NEF Consulting,2010. The social and economic value of the MEND 7-13 Programme. York: York Health Economics Consortium.
Available at: - Sacher PM, 2010. From clinical trial to large-scale community implementation: evaluation of the MEND multicomponent, family-based, child weight management programme in overweight and obese 7-13 year old children in the United Kingdom . Obesity Reviews 11(S1):88
- Singhal, A. & Institute of Child Health, 2009. Trial of the MEND Childhood Obesity Treatment. Clinical Trials, US National Institutes of Health.
Table 2: Initiative Assessment Log
Initiative / MENDPriority area / Obesity
Intervention summary / The MEND programme runs as part of the implementation of the Food and Fitness for Children and Young People action plan. MEND is a community programme for children between 7-13 and their families, which uses a ‘train the trainer’ approach. The highly prescriptive multi-disciplinary programme, delivered as a group-based educational intervention, places equal emphasis on healthy eating, physical activity and behavioural change, aimed at empowering the child, building self confidence and personal development. The programme is delivered over 2 x 2hr sessions per week for 10 weeks. Children are referred by health professionals or parental self referral. A grant for an additional year was provided following the pilot, whilst a tender exercise was undertaken for a contract to continue the delivery of a Children's Obesity referral scheme across Wales. (Ministerial Briefing P3/4/5). The aim of the contract is to appoint an organisation with the capacity, skills and knowledge required to: Provide a standardised, evidence based programme for children who are overweight or obese and their parent(s)/carer(s); Deliver appropriate training to enable the programme to be delivered across Wales; Provide marketing and recruitment tools for delivery partners to use and provide advice and support on recruitment; Monitor and report on a set of indicators agreed with the client, including appropriate inputs, outputs and outcomes; Deliver a programme which: supports children who are overweight or obese to maintain weight; empowers families and children to sustain lifestyle changes and move towards maintenance of a healthier weight; improves health related outcomes.
Life-course group / Children aged 7-13 and their families
Outcome category / Individual level behaviour change – direct outcome
Evidence-base / Has the intervention implemented by this initiative been subject to an ATTRACT evidence enquiry? Yes
Refer to Evidence Mapping Table below for details.
Summary evidence grade: +/-
Relevant evidence (from reliable sources) about the likely effectiveness of the intervention/approach employed in the initiative is conflicting.
Evaluation? / Data on outcomes from MEND in Wales have been reported by MEND central for 2 periods between Sept 2008 and March 2012, relating to the initial two contracts held by MEND with the Welsh Government. Both reports show statistically significant (beneficial) changes in the outcome measures being monitored by the programme. The level of change is similar to that reported for the national (English) roll-out of MEND. The report does not comment on the clinical significance of the magnitude of the measured changes. The data presented are means and confidence intervals for each outcome measure. It is not stated whether each data item is normally distributed and therefore that the statistical tests used are appropriate. The possibility of ‘regression to the mean’ is not discussed in the report and therefore cannot be ruled out. Data are for monitoring purposes rather than evaluation therefore there is no comparator. It would be useful if the reports had also presented data showing how changes in each outcome measure are distributed in order to elucidate the extent to which there is variation in outcome and also, to see data broken down by gender and other socio-economic variables, where data are available.
Pilot? / No
Population impact / Limited: Intervention is limited to a small population group, children and their families who are enrolled on the programme.
Initiative Grade:
A III / There is some, limited, evidence to suggest that the MEND programme has the potential to be effective in relation to its primary objectives, however the current evidence-base is not strong. The available evidence suggests that effectiveness may not evenly distributed across children from different genders/socio-economic/ethnic groups and that local factors can have a significant impact on effectiveness. The data from MEND Wales are not presented for these different sub-groups nor are data presented for outcome measure distribution and thus individual-level effectiveness cannot be judged fully. There is likely to be considerable variation in local factors in Wales, which may impact upon implementation and may limit effectiveness for some areas/groups.
Initiative Evidence Assessment Report MEND finalPage 1