Appendix A
Overall Evaluation Questions for IA-DMM
Primary and Secondary Questions
Level 1: Family Participants
Primary Evaluation Question(s):
Ø How confident do family participants feel at the now of the IA-DMM compared to when they started?
Ø How satisfied were the family participants in the use of the IA-DMM and what, if any, barriers did they identify?
Secondary Evaluation Questions:
Ø How does technology impact family members’ confidence and competence in implementing FGRBI?
Ø What did family participants want to “get out” of this intervention or what did family participants “get” from this new way of delivering services?
Measure(s):
1. Online/Phone survey: All family participants will be invited to complete the Parenting Confidence Scale (Dunst & Trivette, 2009) through an online or over the phone format. The instrument will use a retrospective post-then pre survey and will be conducted at the end of their cohort phase (September, 2014; January, 2015; September, 2015; January 2016; September, 2016; January, 2017; September, 2017). Attachment C.1 provides a draft of this instrument.
2. Phone Interview: A sample (10%) of the family participants will complete a short phone exit interview that collects their consumer satisfaction with the IA-DMM. Again, these will be done at exit time – see Measure 1 for tentative timelines. Attachment C.2 provides potential exit interview questions.
Level 2: Home Interventionists:
Primary Evaluation Question(s):
Ø How do service providers assess the “readiness” of their agency/school district for organizational change? (ROLE Instrument) How do service providers assess their own readiness for change (differences between cohorts 1-2 and cohorts 7-8)? (Attachment E)
Ø How do service providers shift in their abilities to implement FGRB Interventions?
Ø Time1 (initial) Time 2 (six month) time 3 (one year anniversary) Proposed structure.
Secondary Evaluation Questions:
Ø How does the IA-DMM change practices on home visits?
Ø What do providers have as the “most significant change” they hope for with this PD project? What was the most significant change as reported by providers?
Ø What, if any, shifts do we see in self-report assessment on knowledge, skills, and/or attitudes (KSA) given PD sessions on DMM. (Attachment D)
Level 3: Implementation Teams
How do the implementation teams assess the “readiness” of their agency for organizational change? (ROLE Instrument) (Attachment E)
How do partnerships on the implementation teams change over the grant period? (Collaboration Survey - Attachment F)
What, if any, changes happen to the membership of the implementation teams over time?
What types of network/ collaboration changes occur over time? (mapping)
How and when do they begin to identify "others" who need to be part of their implementation team?
What are the barriers that impact service providers in supporting this model; what policies might impact usage of this model.
How did systems change to accommodate the IA-DMM?
Level 4: State Level
How does this professional development by DE impact relationships with the larger collaboration of EC entities at the state level?
How did EC systems change to accommodate this initiative across the state?
How are state IHEs impacted by the IA-DMM? (Modified Attachment D)
What lessons learned can we share with other states wanting to develop focused PD?
IA-DMM Draft Evaluation Questions 9-25-13Page 1