Programme Effectiveness

Programme Effectiveness

Self-Review Tool

October

2012

This is a draft version of the tool.Comments on this tool are welcome and can be sent to .

Introduction

All programmes ending their first programme cycle are required to conduct a self-review using World Vision’sProgramme Effectiveness (PE) Standards[1]. This self-review is completed as part of the programme’s overall evaluation or review.It strongly informs the programme’s redesign.

As part of their self-review, programmes complete an action plan to transition towards achievement of these standards in the next programme cycle(s), as appropriate to their contexts.The pace of this transition should be gradual, reflecting and honouring existing commitments to children, families, communities, and partners.

Overview of the Tool

This document is a tool to help programmes complete the Programme Effectiveness self-review and action plan.This tool is not designed to give a score of any kind.It simply helps programme teams and the national and support office staff who support them to reflect on a programme’s current practices and to plan the changes necessary to achieve the PE standards over time.

In this tool, the 13 PE Standards are organised within the four main aspects of World Vision’s Development Programme Approach:

Contributing to child well-being
Working effectively with communities and partners
Equipping local level staff
Basic programme parameters.

Each standard includes sample criteria to clarify its meaning.These criteria can be adapted by a national office (NO) to reflect the NO’s contextualisation of WV’s Development Programme Approach.

Programmes reflect on how well they are doing on each of the criteria by using the following ratings:

Emerging (E):If a programme is just beginning to meet this criterion
Growing (G):If a programme is making good progress in meeting this criterion
Maturing (M):If a programme is consistently meeting this criterion

After rating themselves and noting the evidence for the rating, programmes use the tool tostate the changes needed to move toward Maturing:meeting the criterion consistently.If the rating is already Maturing, they have the option of including changes to maintain or further improve their practice.

The tool also provides the space for programmes to confirm the person(s) responsible, the timeframe, and the resources needed to make these changes for each criterion.

Process for Completing a Programme Effectiveness Self-Review

It is important that the programme team, appropriate cluster or national office staff, and staff of the support office(s) supporting the programme are involved in the programme effectiveness self-review and action planning.This improves quality and accuracy by ensuring multiple perspectives are included, and prepares all these staff to support the actions agreed.

The following process is recommended to complete a programme self-review:

The programme team completes an initial draft using the self-review tool.

This is often done in a workshop of one or more days facilitated by a member of the national office programme effectiveness or ministry quality team.
It may be appropriate to consult with community groups and partners to receive insight on key issues.This may be done along with other parts of the evaluation, or as a focused process.

The programme team shares their completed programme self-review (including the action plan) with the designated reviewer(s) in the national office – typically the team’s supervisor and/or programme effectiveness/ministry quality staff at cluster or national level.
The national office reviewer(s) dialogue with the programme team to provide feedback.
When the programme and national office reach agreement on the self-review with action plan, the national office sends these to the support office(s) supporting the programme.
The support office reviewer(s) provides feedback on the self-review with action plan, preferably through a face-to-face meeting or a virtual discussion.
The national office and programme work together to finalise the self-review with action plan based on feedback from the supportoffice.

Programme Effectiveness Self-Review Tool:Draft Version, October 20121

Programme Effectiveness Self-Review Tool

Standards for contributing to child well-being

Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #1: The programme contributes to the sustained well-being of children, especially the most vulnerable.
1.1 / The programme and all projects focuses on contribution to the sustained well-being of children, especially the most vulnerable. / The programme is beginning to increase its focus on addressing child well-being. / The programme can demonstrate a clear contribution towards improvement in child well-being in all projects.
Standard #2: The programme vision and priorities are developed with and owned by the community and local partners.
2.1 / The programme’s vision and priorities for child well-being are developed through the use of participatory approaches that include children and adults, especially the most vulnerable, and key partners. / The programme staff are beginning to develop an understanding of the child well-being outcomes, The work within communities focuses on broad development goals. Programming priorities are usually decided by adults only. / Programme staff have experience in using participatory methods to engage children and adults. Results are used to develop vision and identify local priorities for child-wellbeing.
The programme actively engages, builds capacity or mobilises local partners to promote child well-being and include the most vulnerable
2.2 / The programme intentionally identifies and includes the most vulnerable children. / The programme does not have a clear focus on the most vulnerable children. / Programme DME can demonstrate how and to what extent the lives of the most vulnerable are impacted by the programme. The programme staff and community stakeholders are actively sharing lessons and promising practices around the inclusion of the most vulnerable.
Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #3: The programme integrates lines of ministry, sectors and themes, reflecting the national office strategy and the local vision and context.
3.1 / The programme demonstrates a flexible, context-appropriate integration of the ministry lines (transformational development, disaster management and advocacy), sectors and themes contributing to the community vision and priorities for child well-being. / The programme team has initiated discussions on how to integrate the ministry lines, sectors and themes in the programme. / The programme team demonstrate the ability to integrate ministry lines, sectors and themes in context-appropriate ways.
The programme has many examples of effective integration.
3.2 / The programme contributes to WV national office strategic priorities, including child well-being targets, appropriately in context. / The programme cannot show any clear link to the National Office strategy and targets. / The programme can clearly show how most outcomes are contributing to the National Office strategy and targets.
Standard #4: Programme design, monitoring, evaluation and reporting clearly reflect the programme’s contributions to child well-being outcomes.
4.1 / Programme goal, outcomes and indicators clearly indicate WV’s intent to improve child well-being. / Discussions have begun among staff on how the programme can become more focused on child well-being. The logical link between the programme outcomes and the child well-being outcomes is weak. / The programme goal, and all outcomes and indicators are clearly linked to child well-being outcomes.
The programme uses indicators for child well-being as an integral part of the monitoring and evaluation system.
4.2 / Baseline measurements are completed within the first year of implementation, with the participation of communities, partners and other stakeholders, including children. The results are used to improve programme design and to inform regular monitoring and reporting on child well-being. / A baseline survey was not undertaken. / A baseline has been completed on time, with full participation of communities, partners and including children. The results have been clearly used to improve the programme design, and are used to inform regular reporting and learning.
4.3 / Regular six month and annual management reports (including financial reports) clearly show progress towards improved child well-being.
Monitoring is participatory and actively involves children.
Monitoring and reporting are designed to generate learning and to improve the effectiveness of implementation. / Programme reports are developed regularly and an effort is made to show progress towards achievement of outcomes.
Reporting is not focused on child well-being.
Monitoring and reporting focuses on achievement of activities and outcomes, and does not generate useful learning. / The programme regularly and clearly reports its progress towards achieving child well-being outcomes.
Monitoring and reporting generates useful learning and there is evidence of improvements to implementation as a result of this learning.
4.4 / Evaluations are designed, implemented and analysed with the participation of communities and partners, including children.
Evaluations are designed to create opportunities for communities and partners to strengthen their capacities, strengthen partnerships and to reflect on sustainability of programme outcomes. / Evaluations are conducted for some projects, but not all projects.
Evaluation results are only used to show achievements of programme or project outcomes and are not used by local stakeholders for decision making. / All programmes and projects are regularly evaluated.
Communities and partners, including children, are actively involved in the design, implementation and analysis of evaluations.
Evaluations are used to generate learning, improve accountability and improve programme effectiveness
Evaluation findings are shared throughout the organisation and with other organisations to influence policy and practice.
Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #5: In programmes with sponsorship, Sponsorship Minimum Programming Standards are applied through programming that contributes to the sustained well-being of children, including registered children, and enables meaningful engagement with sponsors.
5.1 / Registered children (RC) are among primary beneficiaries of the programme. They participate in and contribute to changes while the programme applies shared direct benefits. / RCs receive exclusive benefits but this practice is beginning to change.
The concept of shared direct benefits is understood by staff but has not been communicated to RCs and their families.
RCs participate in project activities, but in limited way.
Many RCs are not among the primary beneficiaries of programme outcomes. / The concept of shared benefits is understood by staff, RCs, their families and communities, and is consistently applied.
Evidence from monitoring indicates that RCs actively participate in the design, implementation, monitoring and evaluation of programmes.
All RCs are among primary beneficiaries of programme outcomes.
5.2 / RC monitoring is an integral part of the overall development process in the programme / RCs are monitored for individual welfare, Sponsorship processes only related to RC management operations and are not integrated into the development processes. / RC monitoring takes place as an integral part of the programme and overall development process.
Every monitoring activity with children is fun and meaningful for them. They serve to raise children’s issues that parents and community might be concerned about.
5.3 / Sponsorship Minimum Programming Standards are fulfilled in the programme. / Only a few of the SMPS are applied consistently / All SMPS are consistently applied across all projects.
5.4 / Communication between children and sponsors is effective and meaningful. / Communication to sponsors is carried out as a requirement, and provides basic data on each RC’s health and education through the annual progress report. / RC communications with sponsors clearly shows the impact of the programme on their lives. Creative, fun and developmental methodsare used to generate stories of change. The process of communication with sponsors is designed to be efficient, responsive to sponsors’ needs and serves to illuminate them on issues that children and communities face.

Standards for working effectively with communities and partners

Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #6: World Vision’s preferred local role is to serve as a catalyst and builder of capacity of local partners and partnerships for child well-being. In areas where children face critical well-being needs demanding immediate action, World Vision works with local authorities to plan its direct operational role in addressing these issues while strengthening the capacity of partners to assume the role over time.
6.1 / The WV team is helping communities and key partners work collaboratively toward improving and sustaining child well-being. / The programme does not work collaboratively with other local stakeholders. / The programme actively works with the community to identify all relevant stakeholders and works collaboratively towards child well-being.
6.2 / WV is playing a role appropriate to the partnering context – as a catalyst, broker, mobiliser, and/or builder of capacity. / The programme is not aware working effectively with partners. / Programme staff have a thorough understanding of the local context and are working effectively with partners in a way that is appropriate to the context.
6.3 / As appropriate in context, WV is collaborating with government; churches, faith-based organisations (FBOs), non-governmental organisations (NGOs),community-based organisations (CBOs), community groups, and other civil society groups; and local businesses. / The programme does not regularly collaborate with local groups. / The programme has worked with communities to identify the local groups that are working towards child well-being, and are working effectively with them.
6.4 / In situations where it is safe and contextually appropriate, children’s and youth organisations are intentionally included in multi-stakeholder collaborations and supported to engage effectively. / The programme does not work with children and youth organisations. / The programme knows and works with children and youth organisations, supporting them to engage actively in multi-stakeholder partnerships
6.5 / When playing a direct operational role, WV seeks to model effective, accountable implementation and works to strengthen the capacity of partners to assume the role over time. / The programme is directly implementing in response to urgent child well-being needs, but does not consult or plan with local authorities / When responding operationally to urgent needs, the programme consults with, plans with and builds the capacity of local authorities and partners.
Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #7: Programme staff support communities and local partners in advocacy with government and other authorities.
7.1 / Project outcomes, outputs and implementation strategies incorporate advocacy approaches adapted for the context. / Programme staff are becoming aware of the importance of using advocacy to transform systems and structures that are harmful to children. Advocacy approaches like citizen empowerment, social accountability and polity influence are relatively are not used. / Project outcomes, outputs and implementation strategies express local level advocacy approaches as a central theme. These approaches are carefully adapted to fit the local context, and have a measurable impact on improved child well-being.
7.2 / The programme supports and strengthens local groups and citizens to hold government service providers accountable for their commitments to improve child well-being / The programme team is becoming aware of the importance to support communities to engage with government and service providers to improve child well-being. / The programme supports communities in their actions to self-organize, raise their voice, participate with and influence government and service providers to improve child well-being.
7.3 / The programme collects evidence to support advocacy activities at local, district/provincial, and national levels. / The programme is doing some local advocacy work, but the information is not being used to inform district or national advocacy. / Local advocacy is a core programme strategy and information is consistently used to support advocacy at higher levels.

Standards for equipping local level staff

Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #8: Key programme staff are committed to incarnational living among the people they serve.
8.1 / The national office and the programme apply their own clear definition of what it means to “live among people”
This is understood clearly by all staff, and helps to develop a deep understanding of the challenges and potential of the communities and partners. / Programme staff have begun to discuss what it means to “ live among the people”, but have not yet developed a clear, shared understanding. / All programme staff live in the communities as per the NO definition. The effect of staff living in the community is evident in terms of improved relationships, participation and ownership, and in increased programme impact.
8.2 / The programme team follows Jesus’ example of love, justice, humility and servant leadership in their engagement with community members and partners. / There is no evidence that programme staff relate to communities and partners in a way that demonstrates these characteristics. / Communities and partners recognise these characteristics in the way programme staff relate to them.
8.3 / Prayer and discernment are integral parts of the programme team’s work and are done in a sensitive, context-appropriate manner. / Time is given to prayer and discernment only in times of crisis. It is not done in a way that includes staff of other faith communities. / Prayer and discernment is a visible and valued part of team life. It is done in a way that openly includes staff from other faith communities.
Standard #9: Programme staff have the competencies required to fulfil their roles.
9.1 / The competencies required for individual team members and the team as a whole are discussed, adapted, and understood by all team members. / The team members have discussed the core competencies, but have only a limited understanding of how to apply them. / Core competencies are clearly understood, adapted and agreed upon for each team member and for the team as a whole.
9.2 / Plans are designed and implemented to develop the capacities of each team member and the team as a whole, based on the competencies. / The team members have limited information on how core competencies will be used in the building of team members’ capacity. / The team members participate fully in the development of capacity building plans for themselves and their teams. They are fully aware of the capacity building plan and they derive measurable benefit from the plan.
9.3 / Individual and team competencies and capacity building activities are regularly monitored and continuously improved. / Individuals have vague competency development objectives.
Development of staff competencies is not measured in a meaningful way. / Individuals and the team as a whole have well defined competency development objectives. They have regular and structured times to reflect on their progress, receive feedback. celebrate achievements and identify further areas for competency development.
Criteria / Rating E/G/M / Emerging / Growing / Maturing
Standard #10: Programme staff have access to the technical resources and support they need.
10.1 / The programme, communities and partners are able to access the technical resources they require to implement the programme effectively. / The programme, communities and partners have inadequate technical support to ensure the programme is implemented effectively. Resources are mainly internal to WV. / Programme, community and partners are able to access the technical resources they require to ensure the programme is implemented effectively.
10.2 / Requests for technical support are planned well in advance, and technical resources are delivered on time from WV or other technical service providers. / Technical support needs are planned at short notice and technical resources are usually not delivered on time.. / Technical support is planned and requested in good time, taking into account the constraints of the local context. Technical resources are usually delivered on time.
Standard #11: Programme staff engage in regular, intentional reflection and learning that leads to improved practice.
11.1 / Programme staff have structured opportunities for reflection and learning as a regular part of the management cycle.
Learning derived from these sessions is routinely used to improve programme effectiveness. / Learning and reflection takes place in ad hoc manner. It is not regularly planned, and results are not documented. / Regular learning and reflection sessions form a valuable part of the management cycle for this programme. Sessions are planned in advance. The sessions generate valuable lessons which are documented, shared and which lead to tangible improvements in programme effectiveness.
11.2 / Staff regularly include partners and other stakeholders in reflection and learning sessions.
Lessons are shared with partners and other stakeholders to improve development practice. / Learning and reflection sessions usually only involve WV staff.
Lessons learned are not shared systematically with partners and other stakeholders. / It is normal practice for this programme to include partners and other stakeholders in regular reflection and learning sessions.
Lessons learned are routinely and systematically shared with partners and other stakeholders to improve development practice.

Standards for basic programme parameters