Concept Note: Impact Evaluation of Nadiees Perfecto,

a Program to Improve Parenting Skills in Chile

Emanuela Galasso (DECPI), Pedro Carneiro (UCL), Miguel Cordero (MOH-Chile)

Background

Parenting practices are key determinants of the physical and socio-emotional health of children, and parental education interventions are potentially an important strategy employed to improve parenting practices. Rapid social and lifestyle changes, increased female labor force participation, increased hours at work for both parents, and the gradual disappearance of the extended family as a traditional source of informal support, are increasing the pressures on parenting practices of families living in the so-called emerging economies.

Parents play a critical role in fostering the development of children, and this particularly important in the early years. Healthy development in a child’s earliest years sets the stage for a strong development throughout the life span (not only in terms of health outcomes, but also cognitive and noncognitive skills, as suggested in Currie, 2009). There is evidence that parenting and the home environment accounts for a significant fraction of the preschool cognitive achievements (Schady, Paxson 2007) and school test-score gaps among different socioeconomic groups (Todd, Wolpin 2007).Parents, however, do not always have access to adequate help and information to support them in making decisions that can assist in fostering healthy child development. This problem is especially important among the poor.

A recent LANCET comprehensive review of strategies to promote child development (Engle et al 2007) lists a few parenting and parent-child interventions to conclude that parenting programs combined with home visits and/or that involve direct opportunities for skill building and practices with the children are more effective than providing information to parents in enhancing child development outcomes.Evidence from Bangladesh (Aboud 2007,Hamadani et al 2006) suggests that both types of programs improve maternal knowledge, but only those combined with direct demonstrations and interaction with children showed significant gains child development scores. Extending our evidence on the relative effectiveness of such programs in other settings and with varying modalities of service delivery (community based vs individualized interaction) is however of critical policy importance, as they are associated with substantially different costs and potential for scalability in low and middle income settings.

Furthermore, parenting interventions are often proposed as a tool for improving child outcomes, and an example is Nobody's Perfect, the Canadian program on which Nadiees Perfecto(NEP hereafter) is based on. There is very little evidence based on rigorous impact evaluations on this topic, and it is important to document how and under what circumstances maternal knowledge translates into improved behavioral change in parenting practices and how these benefits are in turn passed on to their child.

“Nobody’s Perfect” the most widely disseminated parenting program in Canada, is one example of this research-practice gap.Canada has championed the development and provision of this type of intervention. An existing combined qualitative and quantitative evaluation (Chislettand Kennet 2007;Skrypnek and Charchun 2009) found that Nobody’s Perfect contributes to improvement in a number of parental outcomes that are potentially associated with superior child outcomes. The evaluation shows a decrease in negative or punitive practices, and improved parental ability to cope with parenting stressors, problem solving ability and perceptions of social support.[1]Yet, the evaluation fails to rigorously examine whether these changes in parenting practices translate into gains in terms child development outcomes. There is now a significant opportunity to address this problem as the Chilean government is committed to launch a major evaluation of Nobody’s Perfect(or Nadie Es Perfecto). The government of Chile has selected this Canadian parenting intervention within their new nationwide early childhood policy, Chile CreceContigo and it is planning to gradually roll it out within the country.

The aim of this rigorous evaluation is to understand whether parenting programs are successful in enhancing child development. We plan to allow variation in the intensity of the intervention to test how the relative cost-effectiveness of different delivery mechanisms and feed back into the design of the program during its scaling up phase. In doing so, we aim to go be beyond the results of the impact evaluation and uncover the mechanisms by which a parenting program may or may not affect child development.

Description of the Intervention

Nobody’s Perfect is a parenting education program that has been developed Public Health Agency of Canada. It was introduced in the early 1980s in a few locations, and since 1987 it has been offered in the whole country. It targets mainlyparents with children aged 0 to 5 who live in poor and isolated conditions. It is expected that the program leads to:

  1. an increase in participants’ knowledge and understanding of their children’s health, safety and behavior;
  2. a positive change in the parenting skill of participants in relation to their children’s health, safety and behavior;
  3. an improvement in participants’ confidence and self-image as parents;
  4. an improvement in participants’ coping skills as parents; and ,
  5. and increase in self-help and mutual support among parents

The innovative approach builds on parents’ existing knowledge and capacities through group discussion and problem-solving learning activities. Facilitators create opportunities for change through building trusting relationships with parents and creating groups characterized by mutual support.

Approaches to parenting that use teaching and reasoning in disciplining children rather than punitive approaches are associated with to superior child outcomes (Dooley & Stewart, 2007; Fletcher, Walls, Cook, Madison, & Bridges, 2008; Gershoff, 2002; Lansford et al., 2005).

The basic Nobody’s Perfect parenting programis composed of6 to 8 group-based parent education sessions. The group sessions are lead by a trained facilitator. The program employs adult education strategies to enhance participation and learning. This entailed encouraging participation, building on strengths, and limited use of didactics. Participating parents identify preferred parenting topics which are then emphasized in the sessions by the facilitator. The basic topics will include: (i) positive attending to your child, (ii) use of non-physical disciplinary strategies, (iii) importance of monitoring for safety, (iv) child nutrition, (v) approaches to enhance attachment, (vi) management of basic health concerns, (vii) strategies of self care (for the parent), and (viii) creating a stimulating/learning environment. Each of the set themes has (a) clear learning objectives (key messages), (b) in-session hands-on activities, (c) set learning materials for the facilitators, and (d) home-work activities.

Group discussions will be supplemented by parent education bookletsthat are provided to participants. Five of the booklets are a component piece of the Nobody’s Perfect parenting program that provide basic points about parenting of young children with an easy reading level and brightly illustrated. The booklets are also available on the Chile CreceContigo website.

In health centers that will not receive the program, the Child Health National Program considers non-structured parent education sessions in primary care around different topics: respiratory disease prevention, oral health, home accidents prevention, basic care for new born, and others. These topics are covered by a nurse or other primary care professionals.These sessions are of universal coverage (100% of health centers and nearly 60% of the children in the target group (DEIS data, 2009). Nobody's Perfect will be added to a selected set of centers over and above this unstructured parental education. Therefore the comparison we will do is between clinics where the Nobody's Perfect (or its intensive version) is offered on top of already existing activities (not only unstructured parental education, but also some direct stimulation of children offered already in half the clinics), and those where Nobody's Perfect is not offered and only the already existing set of activities is available.

  1. Primary Research Questions

The main research questions are the following:

  • Does a structured parenting program promote improvement in knowledge and behavioral change in parenting practices?
  • Does the intervention improve the mental health of the mother? Does it translate in reduced parental stress?
  • Does the improvement in parenting practices translate into detectable changes in child development outcomes, ranging from health, to cognitive and socio-emotional development?
  • What is the value added of increased intensityto the basic parenting program on knowledge, parenting and child development outcomes? (The exact nature of ‘intensity’ is discussed at more length in the description of the intervention).
  1. Outcome indicators

We will focus our study on children ages 0 to 5.Messages regarding health, early stimulation and enhancing attachment are likely to be critical for the younger age group while sessions relating to behavioral issues, discipline will be more likely to apply to children above one/one and half years, once children are walking and start exploring the world. We extended the focus on the 0-2 to be able to measure the impact of the intervention on a wider range of behaviors and outcomes.

The direct testing of children will conform to the guidelines on protocols and informed consent by the national IRB committee (Comite de Etica de Investicaciones con Humanos, Servicios de Salud).

  • Child Development outcomes
  • Young children (0-3.5 years): We plan to use the Bayley Scales of Infant Development administered by a trained rater. The test identifies the child development competencies and deficits across five major developmental domains: cognitive, language, socio-emotional, and fine and gross motor development and adaptive behavior.The Bayley scale also includes an examiner assessment aimed at measuring executive function. An adaptation of this scale has been widely used in Chile.A team member of this evaluation from the Psychiatric Department of the University of Chile has recently participated in the training and field work supervision of a large clinical trial using the Bayley’s (Rojas et al 2007). The evaluation team will also draw from the administrative data on the history of scores on the Escala de Evaluacion de DesarrolloPsicomotor (EEDP) that has been normed for Chile and used within as a screening device within the health visits (more on this below);
  • Older children (3.5-6 years): We will focus on a subset of domains of child development that have been linked to school readiness and subsequent performance in school:
  • Vocabularywill be measured using the Test de Vocabulario en Imagines Peabody (TVIP), the Spanish version of the Peabody Picture Vocabulary Test (PPVT).
  • Attention and speed of processing: the proposed instrument is the Leiter-R.
  • Socio-emotional development: the proposed instrument is the Achenbach Child Behavior Checklist
  • Executive function: the choice of the instrument is currently under discussion, allowing for experimentation during the pretest (Backward Digit Span and Stroop test).
  • Parenting and home inputs: To assess family milieu and well-being as well as the quality of the mother-infant relationship we will use an abbreviated version of the Home Observation for Measurement of the Environment (HOME), infant version, containing 6 sub-scales and 45 items (Caldwell and Bradley 1984). This is an instrument designed to assess the quality and quantity of support for cognitive, social and emotional development that is available to a child through the home environment. It has been widely used in studies in the UK, US.

It has been translated and used in Chile (Bulnes et al 1979, Tippie 2003). A Parent-Child Interaction Scale included in the CanadianNational Longitudinal Survey of Children and Youth (NLSCY) will also be pre-tested (Johnson and Mash 1989).

  • Maternal mental health and psychosocial well-being: The team will also rely on the expertise of the local psychologists at the U. of Chile to adapt measures of maternal self-efficacy and Parenting Sense of Competence Scale (Ohan et al 2000).

Maternal mental health will be measured using:

  • the Center for Epidemiologic Studies Depression Scale (CESD) which captures symptoms of clinical depression,
  • a measure of psychosocial distress used in Mexico and Chile in the context of the evaluation of large scale conditional cash transfer programs.

Another key dimension arising from qualitative reports in Canada and preliminary structured interviews in Chile relates to the parental perceived sense of social support arising from the group interaction. Parental social support, problem-solving, and ability to cope with stress have all been found to impact quality of parenting (e.g., Cochran & Niego, 1995; Crnic & Low, 2002).

  • Administrative data: The survey data will also be merged to detailed administrative data on each child-mother pair. CreceContigo has a unique feature of a longitudinal follow-up of each child (apoyo al desarrollobiopsicosocial), with key administrative data on maternal pregnancy, birth outcomes and health visits linked for each child in the Integrated Social Information System (on line platform information system used for social protection programs in Chile). We will be therefore able to access retrospective history on risk factors and health outcomes for each child by collecting information on the personal national identification number. An informed consent will be administered to parents to be able access the health records of their children in the health clinics. In particular, the following key retrospective outcomes will be obtained from the clinical folder for each child:
  • Maternal outcomes:
  • EPsA: Evaluacionpsicosocialabreviada – screening test applied during prenatal checkups to detect multiple psychosocial risks[2]
  • EPDS: Edinburgh Postnatal Depression Scale– early screening for postnatal depression at 2 and 6 months of age of the child.
  • Child outcomes:
  • Anthropometrics data: birthweight, head circumference, weight, and length of the infant examined using standard measures at birth, 8 weeks, and at the final follow-up assessment. These indices are measured regularly as part of a national program monitoring the growth and development of infants.
  • Protocolo de Neurodesarrollo: screening for neurodevelopment between 0 to 2 months.
  • Massie/Campbell Scale of Mother-Infant Attachment applied at 4 and 12 months.
  • EEDP: Escala de Evaluación de DesarrolloPsicomotor (0-24 months old) applied at 8 and 18 months [3]and TEPSI: Test de DesarrolloPsicomotor (2-5 years old)administered at 36 months of age. Both are screeningmeasures of language, social, coordination and gross motor delays developed in Chile and normedfor the Chilean population.There is also a reduced version of the EEDP (pautabreve) administered at 1, 12, 21 and 24 months.

In addition, basic socio-economic characteristics of the family can be obtained from the Ficha de Proteccion Social, the main targeting instrument of social programs in Chile, which captures socio-demographic information, employment, income and incidence disability and health shocks. More in-depth socio-economic characteristics will be measured with a light household survey instrument, with special attention to time use and expenditures in children.

Finally, we will attempt to collect data on the characteristics of health facilities and of program facilitators, using a combination of administrative and surveydata. We plan to add a facilitator questionnaire, collecting demographic and socio-economic background information and proxies for their experience and ability for engage groups.

Detailed cost information about the program will be collected in conjunction with the Ministry of Health. A critical component of the evaluation will be a cost-benefit analysis of the intervention.

  1. Methodology

This program is to be implemented nationally in Chile but at present it does not exist yet anywhere. The target population of clinics includes basic family health care, rural health clinics, urban health clinics and health establishments with minimal service complexity: this corresponds to about 600 clinics in Chile, covering 342 municipalities.

The evaluation will follow an experimental design, using the health clinic as the unit of randomization. The gradual roll-out of the program is necessary because of implementation constraints[4], and it is not an imposition of the evaluation (rather, it is an opportunity we take advantage of). The fact that the set of clinics chosen for the early and late stages of the roll-out is done using a lottery is an imposition of the evaluation.

Therefore, in an initial stage, a set of health clinics (nationally representative) will be randomly chosen from the universe of health clinics where the program will be first implemented. The first group covers about 2/3 of the health clinics in Chile.

Within the first set of clinics, two equal size randomized arms will be implemented:

Group A: Basic NEP – parenting only, as currently designed

Group B:NEP with increased intensity: two extensions are under discussion with the program team and are of interest in light of its learning potential for scaling up:

  1. NEP basic program with two added group sessions that include with play and direct interaction with children;
  2. Basic NEP that holds constant the number of sessions but includes observation of video-materials.

Group C:The remaining clinics (about 1/3) in group two will not be able to receive the program for a year and will form the control group.

In larger health centers which have more than one facilitator, we will also experiment with variation within centers, by introducing encouragement protocols for certain age groups, such as those under the age of two.

Children (and their respective household) attending these clinics will be assessed at baseline (before the program is implemented) and one year after implementation. Follow-up funding for a medium term assessment three years after implementation will be sought at a later stage.

Power size calculations

With 100 health clinics in treatment A, 100 clinics in B, and 100 health clinics in the control group, if we sample 5 mother/children pairs per clinic, assuming an intra-cluster correlation of 0.2, it is possible to detect an effect across pairs of alternatives of 0.2376 standard deviations in the Bayley's test using a 5% level of significance (and power of 80%). Underthese power calculations suggest that we should survey at least 500 households in the each of the treatment groups and 500 households in the control group, which accounts for 1,500 households in total.

One potential problem with this calculation is that participation in the program is voluntary, and therefore, it is likely that even though a program is offered in a clinic, less than 100% of the mothers take it up. Based on the take-up of other education programs offered in Chilean clinics on a voluntary basis, we believe we can predict a take-up rate of 50% to 80%, which would imply increasing this sample size by 25% to 100% (1 to 5 mother child pairs per center).