PEACH Evaluation Report /
United Way Social Innovation Fund Grant /
NKFM Evaluation Team: Ken Resnicow, Lauren Nichols, Art Franke and Crystal D’Agostino /
10/31/2013 /
This report summarizes the evaluation activities and findings from the first year of the NKFM SIF Grant. /
Report Table of Contents / Page
A. Introduction / 2
B. Program Delivery / 2
- Description of the Program
- Description of Program Participants
- Quality of/Satisfaction with Program Delivery
C. Counterfactual Condition / 5
- Description of Counterfactual Condition
- Baseline Equivalence Analysis
- Differential Attrition Analysis
C. Evaluation Data Collection and Measurement / 6
- Types of Data Collected
- Description of Data Collection Methods
- Measure Validation Results
- Completion Rate
- Program Outcomes/Impacts
D. Analyses / 8
- Aims
- Types of Analyses Used
- Power Analysis Findings
- Response Rate
- Missing Data Analysis
E. Findings / 9
- Implementation Findings
- Outcomes Findings
- Preliminary Impact Findings
F. Lessons Learned/Next Steps / 13
G. Appendix / 15
- Introduction
In the first year of Social Innovation Fund implementation and evaluation, we provided programming and collected data from a large number of children and parents in areas with high rates of poverty, need for services and opportunity for improvement in the area of kindergarten readiness as identified by United Way for Southeast Michigan. We were also able to develop positive relationships with many administrators and teachers, train them in our programs and teach them about the need for the programming. Additionally, National Kidney Foundation of Michigan (NKFM) staff weretrained in our internal evaluation protocol. After our first year, we have collected promising preliminary data about the effects of Regie’s Rainbow AdventureTM (RRA) on children and parent’s health and kindergarten readiness, and have learned a considerable amount about how to improve our protocol moving forward. In the following years, we will build off our existing community partnerships and relationships, our staff knowledge, and our experiences of conducting evaluation in the field and be able to collect the information in a more accurate and efficient manner.
B. Program Delivery
- Description of the Programs
The three Project for EArly Childhood Health (PEACH) programs were implemented in early childhood education sites during the program year 2012-2013: Regie’s Rainbow Adventure (RRA), Healthy Families Start with You (HFSY) and (NAP SACC) Nutrition And Physical activity Self-Assessment for Child Care. These programs are briefly summarized below.
Regie’s Rainbow Adventure™ is a seven session program with children ages 3-5 enrolled in Head Start programs and other early childhood education settings. Each session lasts approximately 20 minutes and is conducted once a week. The overall goal is to promote healthy nutrition by educating preschool children about the importance of increasing physical activity, decreasing screen time and eating a wide variety of fruits and vegetables based on the colors of the rainbow. RRA is implemented by early childhood educators who are trained by NKFM staff to read a series of seven storybooks each focusing on a different color of the rainbow to the children. In addition, teachers provide classroom and take-home activities, and offer children an opportunity to sample fruits and vegetables. RRA also reaches parents through shopping lists, bags of fresh produce and recipes brought home by children.
Healthy Families Start with You (HFSY) is a lay health educator based program involving parents through individualized coaching and targeted group nutrition and physical activity education sessions. After receiving a standardized training from NKFM, staff within the site conduct two health chats with participating parents where parents complete a health assessment form, set health related goals, and discuss nutrition and health messages. Group educational classes are also offered at participating sites. Through educating parents about nutrition and physical activity, HFSY aims to improve parents’ behaviors so that they can ultimately act as positive role models for their children.
NAP SACC (Nutrition And Physical activity Self-Assessment for Child Care) is an evidence-based program that aims to assess and improve the child care environment to promote healthy eating and physical activity. The program uses a tool to identify the strengths and limitations of the child care facility in its ability to promote nutrition and physical activity and includes components for goal setting and education. In addition, NKFM provides resources to achieve the goals set by providers so that if a provider set a goal to provide water to children while playing outside, NKFM may purchase a large water cooler for the site. The program consists of a pre-assessment meeting at which point a provider completes the assessment and creates goals, a post-assessment meeting and technical assistance in between.
The chart below reflects the programs implementation timeline according to early childhood education sites:
- Description of Program Participants
In the first year of programming, 2,239 children were reachedin two of the 10 identified service regions, Northwest Detroit and River Rouge. However, 999 participants received SIF programming at Hartford Head Start that served as a pilot, which did not receive all evaluation materials, only the Scantron surveys and implementation checklist. There were 903children participating in the program in the intervention sites (New St. Paul Head Start, Rising Advocates for Young Children, The Guidance Center River Rouge location), and 274 children in the comparison sites (The Guidance Center Ecorse Location, Children’s Center Head Start Academy).
Demographic information collected from the RRA parent/guardian fruit and vegetable surveys describe the population reached by the programs in this first year. Mothers were the most common person to complete the survey (82%), followed by fathers (10%) with an average age of 31. The most frequently reported annual income was less than $20,000 (77%) with 17% of the families reporting an income of $20,000 to $35,000. The sample was predominantly Black or African American (95%) with 2.6% of respondents reporting that they were white. Lastly, in terms of self-reported education, 80% of respondents reported receiving some high school to some college education, 11% reported receiving a bachelor’s degree or above, and 10% reported receiving ‘less than high school’ level education.
- Quality of/satisfaction with program delivery
Participant quality and satisfaction with program delivery was collected through teacher and parent Scantron surveys and teacher key informant interviews. The teacher Scantron surveys (n=155) provided information about satisfaction with specific components of the program, and suggested areas of improvement. In general, teachers were very satisfied with the quality of the Regie stories, materials, activities and riddle so that the majority of teachers reported that the stories and materials were of ‘good’ or ‘great’ quality. Teachers also stated that they particularly enjoyed the food tasting. Several described their excitement with being able to expose children to fruits and vegetables they had never had before and the resulting discussion about the flavor and texture. Similarly, over half of teachers reported that the food activities and riddle were ‘great’ in terms of usefulness. Lastly, over half of teachers agreed or strongly agreed that the program was easy to implement and easy to follow (94% and 93%, respectively).
Teachers’ suggestions for improvement including adding a video, music or increasing the amount of information about the importance of physical activity in the curriculum. Additional suggestions for improvement were collected in the key informant interviews. Specifically, more than half the providers noted a lack of parental involvement, and many providers suggested including a tangible item as part of the program to further engage the students.
Parent satisfaction with program delivery was also collected by asking the open ended question, “Is there anything else you would like to share with us?” on the Scantron survey. In response, parents reported that they particularly enjoyed receiving information on nutrition facts, recipes and activities they can do with their children. The data collected on program satisfaction will be used internally to improve programming and fill in gaps with community events and educational presentations.
C. Counterfactual Condition
- Description of the Counterfactual
In order to isolate the program effect on the participants, the evaluators assigned sites as comparison and intervention groups to analyze the counterfactual. Due to logistical constraints and the desire to foster relationships with community partners for future work, it was not feasible to randomly assign sites to serve as comparison groups for the program. Therefore, to increase the internal validity of the evaluation (the selection threat) and to ensure that the comparison group and intervention groups were similar, propensity score matching was used. The prediction of the propensity score for a center was based on the following center characteristics as statistics were available: 1) average household income, 2) average MEAP scores, 3) racial/ethnic composition, 4) center type, 5) center size, 6) availability of transportation, 7) access to local food stores, 8) neighborhood safety, 9) access to parks. For each center in the intervention groups, a comparison center was selected using the nearest neighbor matching method without replacement.
Aggregate characteristics used for establishing similarity among groups included household income, MEAP scores, racial/ethnic composition, child care center type and size, and transportation among other potential factors. Environmental factors such as access to local food stores, neighborhood safety, and parks were also considered. Matching criteria were selected based on available population level indicators that have been shown to influence outcome variables of nutrition and physical activity and kindergarten readiness. In some cases, sites within the same region were able to serve as comparison sites, so that one Head Start in River Rouge served as the intervention group, and the other served as the comparison group, for example, which further increased the internal validity of the data. The intervention sites included: New St. Paul Head Start, Rising Advocates for Young Children and The Guidance Center River Rouge Location. Children’s Center Head Start Academy served as the comparison group for New St. Paul Head Start and The Guidance Center Ecorse location served as the comparison group for The Guidance Center River Rouge. The Rising group did not have a comparison group as this was a group of home based and small daycare providers; a group that was comparable was unable to be found.
- Baseline Equivalence Analysis
Baseline equivalence analysis was run to compare the intervention and comparison groups in terms of the person completing the survey, annual household income, age of respondent, education, racial background, baseline fruit and vegetable intake and baseline physical activity. There were not statistically significant differences between the comparison and intervention group in the respondent’s age, education, racial background or relationship to the child. However, the comparison and intervention group were significantly different in income and total minutes of physical activity reported. The small differences found between the two groups are common for cluster designs. Controlling for income did not impact the statistical significance of the findings below.
- Differential Attrition Analysis
Data was not collected on participants who did not complete the survey so that differential attrition analysis was unable to be calculated.
C. Evaluation Methods
- Data collection and measurement:
Types of Data Collected - Several data measures were used in order to triangulate and therefore increase the internal validity of the data. In addition, various types of measures were needed to collect data on all the outcomes (see logic model in appendix). Data was collected from parents, teachers, and in some cases administrators. Table 1 below lists and describes all measures used across all programs.
Table 1. Description of Data Collection Instruments UsedName of instrument / Respondent / Pre/Post / Type of Data Collected
Regie’s Rainbow AdventureTM
Fruit & vegetable questionnaire / Parent / Pre & Post / Nutrition
Physical activity questionnaire / Parent / Pre & post / Physical activity
Parent Scantron survey / Parent / Post / Satisfaction with program, behavior change/knowledge of child
Teacher Scantron survey / Teacher / Post / Satisfaction with program, behavior change/knowledge of students
Key Informant Interviews / Teacher / Post / Process related data, program outcomes and preliminary data about childhood readiness
Implementation checklist / Teacher / Post / Program fidelity
NAPSACC
Assessment tool / Teachers/ administrators / Pre & post / Nutrition and physical activity practices within the childcare environment
Description of Data Collection Procedures -All surveys are paper surveys and were delivered to teachers or administrators which are then given to teachers and parents. Teachers either distribute the parent surveys to parents directly when children are being dropped off or picked up, or place the surveys in the child’s ‘take home’ folder to be delivered. Teachers then complete their surveys and compile all surveys. At the end of the program, the NKFM program coordinator picks up the surveys and delivers them back to the NKFM data entry team. Key informant interviews are conducted in the child care center with teachers after program implementation.
Measure Validation Results -The majority of evaluation measures are drawn from previously conducted research or created by an experienced evaluation team. The physical activity parent survey is adapted from a validated questionnaire developed by Harro (Harro, 1997). This measure was validated against heart rate monitoring and motion accelerometer data with 4-8 year olds. Body mass index is also being used as an outcome. This is the recommended screening measure from the CDC for overweight and obesity. The pre and post-assessment measure for NAPSACC has also been used in many published research studies with one study finding that it is an accurate and stable measure of the child care environment with high reliability and validity (Benjamin, Neelon, Ball, Bangdiwala, Ammerman, & Ward, 2007). The fruit and vegetable survey, however, was created by the evaluation team at Michigan Public Health Institute after determining that a previously selected instrument was not valid as a parent report, or for the age range of the children participating in the evaluation. Lastly, the Scantron surveys have been part of regular NKFM programming for over five years, and the implementation checklist corresponds exactly to the RRA curriculum to monitor fidelity. Both the Scantron survey and implementation checklist are predominantly used for internal program improvement.
Completion Rate -The completion rate for RRA varied by the type of survey, and the follow-up completion rate was lower than the baseline completion rate. We received 155 teacher and 1216 parent post implementation Scantron surveys. In terms of implementation checklists, 37 were completed by teachers and returned following program implementation. At baseline, 659 fruit and vegetable parent surveys were returned compared to 356 follow-up surveys. Therefore, 54% of people who completed the baseline fruit and vegetable survey also completed the follow-up fruit and vegetable survey. In terms of the physical activity parent surveys, 481 baseline, and 293 follow up surveys were completed so that 61% of parents who completed the baseline physical activity survey also completed the follow-up survey. Key informant interviews were also conducted with seven teachers and five teaching assistants in River Rouge and Northwestern Detroit.
In terms of the other two PEACH programs, NAPSACC was provided to seven sites with 5 sites completing the pre and post-assessment and HFSY was provided to 13 parent/guardians. The small number of participants reached through HFSY resulted in inadequate power to analyze the effect of this program but the results of this program will be included in future reports.
Program Outcomes/Impacts- Data was collected on all the outcomes and impacts included in the logic model. However, a summary of the most important findings will be included below. The following outcomes and impacts were measured as part of the PEACH programsin the first program year:
- Initial outcomes expected include:
- knowledge -parents/children know the benefits of good nutrition and physical activity
- skills -parents/children have the skills to make healthier choices
- access -child care centers have policies supporting nutritious food and encouraging physical activity
- Intermediate outcomes expected include:
- parental behavior-healthier food choices, more physical activity
- child behavior -healthier food choices, more physical activity
- The expected long-term outcome is improved school readiness among participating children.
Process related data was also collected to inform future programming, and potentially improve data collection tools for future years of the evaluation. Specifically, teachers completed an implementation checklist after each weekly session of the program indicating whether or not they had completed every component for that session such as reading the story and offer the optional physical activity exercise. Teachers also completed Scantron surveys that asked about program satisfaction, program materials, and suggestions for program improvements. Lastly, post implementation key informant interviews were conducted with teachers asking more in depth questions about the program, any adaptations they made to the curriculum and aspects they particularly enjoyed.
D. Evaluation Analysis
- Aims
Using the data from the first year, the evaluation team analyzed: