Every Kid in Our Communities…..
Leeds and Grenville
Final Report
Planning for Evaluation Grant
(EPG 1320)
Creating an
Evaluation Framework for Integration
Submitted by Every Kid in our Communities of Leeds-Grenville—Lead Agency: Children’s Mental Health of Leeds-Grenville
Project Lead: Margaret Fancy Coordinator Every Kid in our Communities of Leeds-Grenville
Table of Contents
Executive Summary……………………………………………………………………… Page 2
Introduction and Literature Review…………………………………………………… Page 4
Methodology ……………………………………………………………………………… Page 8
Pilot Results ……………………………………………………………………………… Page 11
Conclusion and Recommendations/Next Steps …………………………………… Page 15
References…………………………………………………………………………………. Page 17
Knowledge Exchange…………………………………………………………………….. Page 19
Executive Summary
Organization Name: Every Kid in our Communities of Leeds and Grenville – Lead Agency: Children’s Mental Health Leeds-Grenville
Program Title: Creating an Evaluation Framework for Integration
Project Lead: Margaret Fancy, Coordinator Every Kid in our Communities of Leeds-Grenville
Every Kid in our Communities is a coalition of over 30 agencies, organizations and individuals working together to improve health and wellbeing for children, youth and families in Leeds-Grenville. The project supports the development of an evaluation framework to measure the impact of collaboration on agencies involved and ultimately on children and families served.
The Purpose:
- To build increased competence for collaboration through the identification of key components of effective collaboration and development of tools to measure levels of collaboration within and across member agencies
The Program
Every Kid in our Communities is a coalition of over 30 agencies, organizations, government and individuals whose members come together to improve health and wellbeing of children, youth and families in Leeds-Grenville ( using the framework of Developmental Assets (
Executive directors and senior managers meet monthly to plan, coordinate and advocate for collaborative evidence-based actions. Workgroupshave been formed to carry out actions identified. (A complete list of workgroups and members (
The coalition has been in existence since 2003; since 2007 Every Kid has acted as theMCYS (South east Region) children’s planning table for Leeds-Grenville.
The coalition has had limited capacity, skills or tools to evaluate the effectiveness of working together other than through anecdote or archived meeting notes. The development of an evaluation framework will help members identify the components of effective integration, and inform future training for staff and community related to collaboration skills.
The Plan
Planned activities for the evaluation product included the development of an evaluation logic model to provide a framework for evaluation questions and activities.
An extensive literature review would determine the components of effective collaboration and identify indicators that could be used to measure effectiveness of the Every Kid coalition.
A data collection tool will be developed to track EK activities (attendance, who, what etc). An evaluation framework will include tools to regularly measure knowledge exchange and the process of collaborative activities. Annual collaboration measures with all community partners, all working groups etc. will be identified through the literature review.
The Product
Planning for evaluation of collaboration provided the opportunity for ongoing capacity-building and knowledge exchange across agencies.
An evaluation workgroup comprised of representatives from member agencies (health, children’s mental health, sexual assault and response, CAS) as well as the data analysis coordinator for Leeds-Grenville and the coordinator for Every Kid, developed an evaluation logic model and conducted a literature review. Regular updates with opportunities for feedback were provided at monthly meetings of agency leaders and at workgroups. Relevant literature was highlighted and links provided.
A tool for consistent collection of data related to number and sector(s) involved in all EK activities was developed. A “Retrospective pre-test” will be developed to measure knowledge exchange. A “Peer/Group Evaluation” tool will be developed tomeasure the collaboration process after completion of a project.
A survey to determine member knowledge about the components of successful collaboration and type of participation and satisfaction with Every Kid was piloted with a group of agency leaders.
Measures for annual measures with all community partners were identified through the literature review.
Knowledge exchange activities include posting of literature review on EK website, presentations to all workgroups and ongoing review of evaluation results by members.
Amount awarded: $20.000.00
Final Report Submitted: October 31, 2011
Region: Southeast Region MCYS
C. Introduction and Literature Review
Every Kid in our Communities (Every Kid) is a coalition of over 30 agencies, organizations and individuals whose members come together to improve health and wellbeing of children, youth and families in Leeds-Grenville ( using the framework of Developmental Assets ( Presently Every Kid is primarily about collaboration including social service agencies funded by the province (MCYS, EDU, MOHLTC, MCSS), with some NGO’s, municipal government, funders, private business and individuals contributing. (for a complete list of members go to
The coalition has been in existence since 2003; since 2007 Every Kid has acted as the MCYS (South east Region) children’s planning table for Leeds-Grenville.
Through the multiple representation of service providers members of Every Kid strive to improve outcomes for kids through greater awareness, communication, cooperation, collaboration and integration of services and support. Six goals guide the work of the coalition. (
Executive Directors and Senior Managers (Lead Table) meet monthly to plan, coordinate and advocate for collaborative evidence-based actions to support children, youth and families. Workgroups (composed primarily of managers and frontline staff) which include Triple P, Transportation, Best Start Network and Making Play Possible have been formed to carry out the collaborative actions identified.(For a complete list of current workgroups see ( ) A research and evaluation committee has recently been formed to ensure actions are data driven and measured. Support and leadership exists through a coordinator paid in part by the South East Region of MCYS.
While working together began from the greater perspective of “coordinating existing services” it has now moved increasingly toward a collaborative strategy where the need and intent is to change fundamentally the way services are designed and delivered – an integration of particular evidence based services such as a parent support program or rural transportation system.
To date no formalized measures have been developed to evaluate the effectiveness of the Every Kid coalition and its workgroups. The Service Integration Index , provided by MCYS for Best Start Networks has been used as a benchmark but not in a formalized way. Tracking of number and sectors involved in Every Kid activities has been done through archived meeting notes and anecdote.
The process of evaluating the coalition will allow Every Kid to increase competence of its members for collaboration through a shared understanding of the components of effective collaboration within and across agencies and the development of a common language around the evaluation of working together. Currently terms such as partnership, coalition, collaboration and integration are used interchangeably, making it difficult to evaluate steps in the integration process. Evaluation measures will improve our ability to assess how well the integrative process is working and provide a means of identifying specific areas to focus on in order to make the process work better. It will inform future training for staff related to collaboration skills.
The framework will also allow agencies and Every Kid member tables to better respond to provincial initiatives such as Student Support Leadership, Best Start, Child and Family Centres that indicate an expectation of integration as do documents such as “A Shared Responsibility” (MCYS).
The Evaluation Logic Model (Appendix 1) identifies the components of the evaluation framework: Communication (Knowledge Exchange, Collaboration (Internal) and Community (External).
Four evaluation questions were developed:
What Every Kid initiatives have members been involved in?
How has involvement in Every Kid enabled members to strengthen
partnerships with other organizations serving our community?
How is the work of Every Kid improving the health and wellbeing of children, youth and families in Leeds and Grenville?
What are the components of effective collaborations?
Over time, standardized tools will be identified to measure the impact that integration of services has on the mental, physical and intellectual wellbeing health of children and youth but as is validated by the research of Gina Browne (2007) and others measurement work related to networks must precede any attempt to measure network effectiveness in terms of client outcomes.
Literature Review
The Literature Review focused on two aspects of the evaluation project:
- Identification of key components of collaboration
- Tools to measure collaboration
As all members of the Evaluation Workgroup were involved in the literature review it provided opportunity for knowledge exchange and capacity-building. A full review of articles reviewed will be available on our website (
The literature review helped identify key components for effective collaboration. These may be summarized under the following general headings:
1)Membership Capacity – involves the knowledge, skills and perspectives required
2)Organizational Capacity – structures required to organize members in a productive manner
3)Program capacity – capacity to identify community needs
1)Membership Capacity – includes the following components:
Shared goal/vision – In interviews with 33 stakeholders (program chiefs) from social service agencies and state departments in Ohio, L. Johnson et al found that in successful collaborations time is spent in coming to agreement about a shared goal/vision. Gajda and Koliba (2007) showed similar findings in their evaluation of school improvement initiatives, as did Hodges et al (1998) in evaluating collaborations related to improving mental health outcomes for children. Hodges points out that goals must be relevant to each of the participating agencies
Trust and Respect – The establishment of a team whose members trust one another
has been found to be an important indicator of success in collaboration (L. Johnson, G.
Browne et al 2007; Hodges) Foster-Fishman et al speak to the importance of positive
attitudes about the other stakeholders (e.g. viewing them as capable, legitimate, needed
+ valuing their diversity) “Networks that seek, adhere to, and measure a well-articulated
vision and forge trustfuland committed relationships among network participants are
morelikely to operate at numerous levels and impact on multiple and
diverse stakeholders (Hill, 2002).”
Common Language -- A common language is seen as the first step in the creation of a
shared sense of community that will surround the collaborative process. Gajda and
Koliba speak to the importance of collaboration literacy, while Hodges et al point to the
importance of understanding the language of various mandates. Browne et al (2007)
indicated that the lack of a common language around terms related to integration is a
barrier to collaboration effectiveness.
Communication – open and frequent communication (Johnson et al, Popp et al 2005)
between members within and across agencies as well as a process for conflict
resolution and decision making were highlighted in the literature
(London 1995; Browne et al 2007, Hodges)
Respect for Diversity -- Equality in decision making and voicing of opinions and
perspectives must be maintained inthe collaborative process. (Corbett, Weiss,
Hodges) Diversity allows for a broader range of ideas and solutions. Corbett also
speaks the importance of valuing the involvement and perspectives of smaller
agencies.
2)Organizational Capacity includes:
Leaders with skills – Leadership both within agencies and across networks is identified
as a critical component of effective collaboration. Johnson et al speak to the need to
have the support and involvement of decision makers. London, Hodges and Corbett et al
2005 describe the importance of moving from the traditional style of leadership to one
that “convenes, energizes, facilitates and sustains the process” (London). Leaders need
to give permission for collaboration, provide time and resources and encourage a culture
of learning. Weiss et al designed a tool to measure leadership effectiveness for
collaboration.
Formalized procedures/structures – procedures for governance, conflict resolution,
service agreements, accountability, work plans, timelines have been identified as critical
to thesuccess of collaboration (Weiss et al, 2002; Provan and Milward 2001, 2006;
Corbett et al 2006, Browne et al 2007). A formalized structure minimizes “turf issues”
and maximizes outcomes (Hodges, Weiss, Johnson)
Well-developed internal communication system –that promotes information sharing and
problem discussion and resolution on a frequent basis between and among staff and
members. (Johnson, Hodges, Provan and Milward, Corbett)
Human, Financial and In Kind Resources—Allocation of sufficient resources to
accomplish goals including time for agency staff to participate, administrative and
technical support (Weiss, 2001; Johnson, Provan and Milward). Weiss et al also
describe the ability to apply for and manage grants, endorsements, advocacy, etc.
A Continuous Learning Orientation -- consistently seeks and responds to (internal &
external) feedback. This is crucial to adapt to changing contexts, overcome barriers as
they arise, promote accountability. (London, Gajda & Koliba, Weiss)
Structure of Network – extent (number of sectors), scope (number of agency or service
types) Browne et al 2004 have done extensive research in this area.
Evaluation – the complexity of networks calls for mechanisms to be in place that support
ongoing formative and summative evaluation of the process and product of network
involvement. (Hodges, Browne (2007)) All members of the collaboration need to be
engaged in knowledge exchange and professional learning activities related to the
language and measures of evaluation.
3) Program Capacity
Program capacity is the ability to identify community needs (through comprehensive
needs assessments and data analysis), to design clear, focused, realistic, efficient, unique
and innovative solutions that have real and meaningful impacts within their communities.
(London, Foster-Fishman et al)
The literature review confirmed the complexity and interrelatedness of the components of
collaboration and confirmed the need for a number of measures to evaluate effectiveness.
Tools for Measuring Collaboration
Existing tools for measuring collaboration were reviewed. They include:
The Partnership Self-Assessment Tool (Weiss et al 2002) measures the Quality of Integration in terms of Synergy, Leadership, Administration and Management, and Levels of Satisfaction.
The Integration Measure Browne et al, 2004 measures the scope, extent, depth and congruence of integration.
The Community of Practice: Collaboration Assessment Rubric CoPCAR (Gajda and Koliba) presents quality of interpersonal collaboration across a continuum of 1 to 6. The scale is ordinal in nature; the categories represent an inherent order (weaker to stronger), and the numbers assigned to the categories do not indicate an equal magnitude of distance between them. “Professional learning community” is denoted as the strongest and “networking” is denoted as the weakest functioning form of interpersonal collaboration.
A number of continuums which describe of ordered stages of collaboration; the type and degree of connectedness between programs and services are described in the literature. (MCYS Service Integration Index 2006, Continuum for Collaboration Community of Practice: Collaboration Assessment Rubric, Gajda and Koliba 2007; Integration Continuum MCYS 2011)
D. Methodology (see Outcome and Indicators Chart Appendix 2)
The evaluation framework for this project was designed by members of the Evaluation Workgroup which included:
Allan Hogan (Director Family and Children Services)
Laurie Bourne McKeigan (Director Assault Response and Care Centre)
Katie Jackson (Leeds-Grenville Lanark Health Unit)
Jessica Deschamp Baird (Data Analysis Coordinator, United Counties of Leeds-Grenville)
Lorena Crosbie (Manager, Children’s Mental Health Leeds Grenville)
Keith McPhee (co-chair Every Kid)
Margaret Fancy (coordinator Every Kid)
The evaluation framework was shared at Lead Table and workgroup meetings of Every Kid for feedback and revisions. The development of an evaluation logic model was a long and ultimately valuable process for our workgroup and Lead Table members. The complexities of evaluating collaboration are well- documented in the research; the workgroup wanted to ensure that there was agreement on the components that were to be measured. The research, consultation and knowledge exchange that were part of the development of the logic model directly informed the development of evaluation questions.
The evaluation questions map directly onto the 3 main goals and components listed in the logic model.
The first piece of the evaluation design was to develop a process for data collection in terms of the number of professional learning activities, number of members involved in committees/workgroups/ resource development; and sectors represented.
There was recognition for the need to collect this type of data on an ongoing basis (monthly meetings of workgroups/committees, at professional development activities, at community presentations)
Initially the committee looked at historical data collection because we did not have a standardized tool in place. We selected three activities: development and training related to “Checkered Flags”; a presentation on EDI, and Community Profiles Presentations with Municipal Councils.
The need for a “go forward” standard data collection tool was recognized, given the number and scope of Every Kid activities. The tool developed uses Survey Monkey to help track EKIOC activities. The Evaluation Logic Model was used to form the questions, particularly those related to Outputs andShort Term outcomes.
This tool would take approximately 2 minutes to complete and would be best completed by the minute taker of the meeting as they would have the information. This would eliminate the need to do ‘archival’ data collection in the future. Preference would be to complete it on-line; however a paper copy can be created for the minute taker, but it still would need to be entered into Survey Monkey at some time. Frequency of collection would be monthly or as activities occur. Initial survey data will be accessed by the data analysis coordinator who will aggregate the data to Windows of focus and other working groups of Every Kid, then grouped at the program level for analysis. The tool was reviewed, piloted and approved by three workgroup committee chairs. (Appendix 3)