SignHealth Uganda (SU)
Signal and Partners Sign Health Uganda
Uganda Deaf Awareness and Communication
(U-DAC) project in Uganda
End of Project Evaluation
Report produced by Independent Consultants: Symon P Wandiembe (PhD), Saint Kizito Omala (PhD), and Martin Ariapa (MSc) of School of Statistics and Planning, Makerere University, Kampala Uganda
July 2016
Table of Contents
List of Acronyms and Abbreviations
Executive Summary
1.0 Introduction and Background
1.1 Introduction
1.2 Purpose of the evaluation
1.3 Project Summary
1.4 Evaluation criteria and objectives
1.4.1 Primary assessment criteria
1.4.2 Learning questions
1.4.3 Evaluation questions
2.0Methodology/Approach
2.1 Overview of the Evaluation Methodology
2.2 Study sample sizes and Data collection methods
2.3 Ethical Considerations and Quality Assurance
3.0 Evaluation Findings
3.1 Findings Summary
3.2 Evaluation Question 1: To what degree has this project outcome been achieved?
3.2.1 [Outcome 1]: Community actively support & advocate for HIC for inclusive schooling
3.2.2 [Outcome 2]: Increased enrolment and retention of DC&YP in primary education
3.2.3 [Outcome 3]: Improved quality of education, relevance and school experience for DC&YP in mainstream schools
3.2.4 [Outcome 4]: Improved involvement of parent/family support groups in the promotion of DC&YPs education
3.3 Evaluation Question 2: Who has benefited from this and in what ways?
3.3.1 Group of beneficiaries in the U-DAC project
3.3.2 Integration of gender and general disability in the project design
3.3.3 Integration of children’s voice in the project implementation
3.4 Question 3: Are the changes due to the project relevant to people’s needs?
Question 4: Are the changes likely to be sustainable in the long term and can they be replicated in other districts in Uganda and elsewhere?
Question 5: Have there been changes to policies, practice and attitudes of decision and policy makers to benefit the project’s target groups?
5.Lessons learnt
5.1Lessons learnt about the target populations
5.1.1What are the factors affecting the retention and academic performance of DC&YP in inclusive schools?
5.1.2What are the gender issues relating to DC&YP?
5.1.3How do parents understand their role in education of children with hearing impairment?
5.1.4What is the effect of parents’ literacy on their support to education of deaf children?
5.1.5How do the deaf perceive/experience learning under inclusive setting with their hearing counterparts?
5.1.6How does the community perceive inclusive education for DC&YP?
5.2Lessons learnt about the project implementation process
6.Conclusion and recommendations
6.1Conclusion
6.2Recommendations
References
Appendices
Acknowledgements
This evaluation and report was commissioned by Signal UK and the field work organized in Uganda by Sign Health Uganda. We would like to acknowledge Sigh Health Uganda staff’s warm hospitality and for making the process happen.
We are most grateful to the evaluation informants for their willingness to provide information for the study, particularly for setting time aside to meet with us. Many caregivers and community leaders had to make time to travel long distances to meet with us and we are very grateful for that.
List of Acronyms and Abbreviations
CBOsCommunity Based Organizations
CORSUComprehensive Rehabilitation Services in Uganda
DC&YPDeaf Children and Young People
DEODistrict Education Officer
ENTEar Nose and Throat
FGDsFocus Group Discussions
FOHOFoundation of Hope
HICHearing Impaired Children
IDIsIn-depth Interviews
IGAsIncome Generating Activities
KIIsKey Informant Interviews
NGOsNon-Governmental Organizations
PSGsParent Support Groups
SNESpecial Needs Education
SUSign Health Uganda
TORsTerms of Reference
U-DACUganda Deaf Awareness and Communication
UKUnited Kingdom
USDCUganda Society for Disabled Children
VHTsVillage Health Teams
Executive Summary
Introduction
This is an Evaluation Report of Uganda Deaf Awareness and Communication (U-DAC) project that was implemented in the Greater Masaka districts of Kalungu, Masaka, Lwengo, Rakai and Bukomansimbi between April 2013 and February 2016. The project was funded by Comic Relief and implemented by Signhealth Uganda (SU) in partnership with Signal, UK. The main aim of the project was to increase access to education, improve retention and academic achievement for deaf and hearing impaired children (HIC) and challenge negative cultural stereotypes regarding deafness in Masaka districts.
The purpose of the evaluation was to undertake an independent and impartial review of Signal’s U-DAC project, and the extent to which it delivered its intended outcomes to the target beneficiaries. The evaluation explored the lessons learned from this project; how this learning can be used and shared; the relevance and appropriate targeting of project activities; the extent to which attitudinal change has been affected; and the long-term sustainability of project activities.
Overview of the Evaluation Methodology
The evaluation was performance based evaluation and used mixed-methods approach, utilizing primarily mostly qualitative data collection and evaluation methods. The evaluation involved extensive desk review and analysis of existing quantitative project data and documentation, and primary collection and analysis of qualitative data. Data collection consisted of a survey questionnaire among a random sample of 70 hearing impaired children (HIC); Key Informant Interviews (KIIs) with8 caregivers, 13 teachers, 8 local government representatives, 2 project staff and 5 community leaders (exposed to U-DAC project);Focus Group Discussions (FGDs) among 8 groups of caregivers and community members;In-depth Interviews (IDIs) with 8 HIC;and Observation of 6 HIC in the school playground.
Evaluation findings
The U-DAC project began in 2013 and directly reached 1,015 (592 girls; 423 boys) HIC in 20 primary schools and the surrounding communities across the 5 districts of Masaka region. The project reached the caregivers or parents of the HIC, 124 primary school teachers, school management committees and the community leaders including Village health team members and religious leaders. Project team worked closely with the district authorities in selection of target communities. These were all relevant target groups that enabled the project to achieve its outcomes.
The project activities were mainly pilot in nature, but effectively and efficiently achieved the desired outcomes. To a large extent the project was successful in increasing awareness about hearing impairment, and changing the attitudes of the caregivers, community leaders and teachers about educational achievements of the HIC in the mainstream schools. There are strong improvements in the HIC enrolment in mainstream schools,in their academic performance and in their school learning environment. Although, the project objectives and activities were pre-determined, many stakeholders considered them very relevant to the context and plight of the HIC. The project has led to the district education and community development authorities to start developing district level strategic plans to support the identification of HIC and to streamline their support in the mainstream schools.
The project demonstrated that the prevalence of HIC in mainstream schools is as high as 4% and the ear problems of most HIC are temporary and can be unblocked by health-workers with use of correct and fairly inexpensive treatment. It also showed that with positive attitudes and appropriate support from the teachers and caregivers, these children can progress normally through the mainstream schooling system.
An overview of progress towards study objectives are noted for each evaluation question below.
Question 1: To what degree has this project outcome been achieved? Were there any unexpected outcomes? Could anything have been done differently to enhance the project’s outcomes?
To a large extent the project was successful in increasing awareness about hearing impairment, and changing the attitudes of the caregivers, community leaders and teachers about educational achievements of the HIC in the mainstream schools (Project Outcome 1). There are strong improvements in the HIC enrolment in mainstream schools, their academic performance and retention in schools (Project Outcome 2). The project used an effective model that combined the child’s right to education with community and school mobilization to rally the community and teachers for action and engagement in support for the HIC. Teachers and community members actively participated in the identification of the HIC and referredthem for treatment (Project Outcome 4).
The teachers, including project focal teachers and those trained by focal teachers, reported to have gained skills in identifying, handling and supporting the HIC. These skills have helped them to improve on the quality of education for the HIC (Project Outcome 3). Many teachers and parents gave testimonies of how their children’s academic performances have improved since they were treated.Further, of the 70HIC interviewed, 90% reported an improved school environment and indicated that compared to before the treatment this improvement is more than five-fold. They now have “real” friends who are willing to consult with them and play with them. Over 85% HIC reported enjoying school now more than before the treatment. Nonetheless, without appropriate support from the teachers some of the HIC will continue to feel isolated even after recovery of hearing ability.
An additional activity that should be executed by the future projects is the trainingof the health-workers at HC III and IIs in ENT service delivery to treat the HIC newly identified in the communities.
Overall, the project executed all the planned activities and achieved all the planned outcomes including achieving some unexpected outcomes such as some adults who were treated and recovered from hearing impairments. Further, the project focal teachers in schools are now being consulted by caregivers of children with disabilities from communities far from their schools.The supported schools are looked at as centers of excellence for the HIC. The levels of achievement are summarized in Fig A.
Figure A: Summary of achievement scores for the different outcomes
Key: 3 = fully achieved; 2 = mostly achieved,1 = partly achieved, 0 = not at all/not done
Question 2: Who has benefited from this and in what ways? Have questions of gender and general disability been considered throughout the design and implementation? Has the voice of the child been heard?
Beneficiaries: The U-DAC project not only targeted the HIC but also the school teachers and caregivers, the community leaders, and concerned district authorities. These were relevant targets that enabled the project to achieve its outcomes. These groups benefited in various ways with (a) HIC reporting improvement or recovery of hearing, improved self-esteem, reduced stigma and social interactions with fellow pupils, and improved academic performance; (b) Caregivers of HICs reported being able to communicate with their children, positive attitude change towardspotential academic achievements of HIC in mainstream schools, improved skills in setting up and implementing IGAs and improved household economic status, andopportunities to interact with teachers, community leaders and policy makers; (c) Teachers gained knowledge in identifying and helping the HIC, improved communication with HIC, positive attitudes toward potential education achievements of HIC in mainstream schools, knowledge of some basic language; (d) community leaders gained knowledge about hearing impairment and deafness, and identifying the HIC and referrals for treatment, and also positive attitude change towardspotential education achievements of HIC in mainstream schools, while (e) district leaders received project reports demonstrating the high prevalence of hearing impairment and how HIC can be helped.
Integration of gender and general disability in the project design: In general, the questions of gender and general disability were well considered in the design and implementation. However, the project design adaptation to arising gender issues of female HIC such as need for life skills coaching for adolescent girls were not well integrated into the project.
Integration of children’s voice in the project implementation: Through regular interactions with the HIC, the project was able to integrate some of their suggestions in the project implementation. For example, when the HIC reported to the project that they are still facing discrimination from their peers without hearing impairments and suggested sensitization meetings with them, the project did so with amazing outputs.
Question 3: Are the changes due to the project relevant to people’s needs? Did the project achieve its objectives in relation to the funding requested? Was the project implemented in the most efficient way compared to alternatives?
The scores or ratings for the relevance, effectiveness and efficiency are summarized in Figure B. The project design flexibility discussed in Question 2 is included here for completeness.
Relevance: The project relevance had a score of 3 (highly relevant). As noted before, although, the project objectives and activities were pre-determined, many stakeholders considered them very relevant to the context and plight of the HIC. The project was well appreciated by various stakeholders as unique and no other organization or government has ever attempted to support the HIC before. The project has led to the district education authorities to start developing district level strategic plans to support the identification of HIC and to streamline their support in the mainstream schools.
Efficiency: In consultation with community leaders, caregivers of HIC and teachers, the project implemented various activities. All the activities that attracted appreciable costs were necessary and were implemented as planned. They were directly related to objectives and outcomes for which the project was funded for. The addition of provision of hearing assessment and treatment through clinic and community outreaches was perhaps the most important and crucial activity in supporting the HIC.
The findings of this evaluation indicate improvements or complete recovery of hearing, improvements in the learning environment of the HIC and their academic performance, and positive change in attitudes of teachers, caregivers and the community toward potential education achievements of HIC in mainstream schools. The evaluation concludes that there is substantial evidence that the negative conditions that existed in the schools and community for the HIC at the start of this project have been well addressed.
Figure B: Ratings for project relevance, effectiveness and efficiency
- Achievement levels with respect to the funding received [Efficiency]: With respect to the funding received, the project executed all the agreed on funded project activities and delivered well over target on all the objectives. The number of HIC, teachers, community leaders and the caregivers that the project had targeted were surpassed in the course of project implementation as in Figure C.
Figure C: Project beneficiaries’ targets and actual numbers served
Question 4: Are the changes likely to be sustainable in the long term and can they be replicated in other districts in Uganda and elsewhere? How has the project affected the development of Signhealth Uganda as implementing partner?
Sustainability: Overall, the prospect for sustainability of the project outcomes is good.Some of the changes brought about by the project such as improvement in the hearing ability of the HIC and their improved academic performance are immutable, hence are sustainable. However, the improvements in knowledge and attitudes of the teachers, caregivers, the community and community leaders require some essential and well planned activities to sustain them.
The most challenging change to sustain is the provision of hearing assessment and treatment services during the outreaches. It is unlikely that Masaka Referral Hospital can organize clinic outreaches in rural areas without any logistics facilitation. Further, most of the parents from remote communities might not be able to bring their children for treatment in Masaka Referral Hospital.
Replication of the U-DAC project outcomes in other districts: Most of the activities and the related changes observed in the U-DAC project areas are needed in other districts in Uganda. It is possible that many other districts have higher prevalence of HIC in mainstream schools than the districts in the greater Masaka region, and thus the relevance of the project like UDAC. Using the same design as the U-DAC project, most of the changes noted here can be achieved elsewhere.
Sign Health Uganda as implementing partner: The implementation of the U-DAC project has shown that Signhealth Uganda as an organization is capable of reaching grassroots and achieving agreed on project results by managing community complexities, expectations and delicate politics, and also in forming strong partnerships with other NGOs and CBOs.
Question 5: Have there been changes to policies, practice and attitudes of decision policy makers to benefit the project’s target groups? Has this contributed to the achievements of broader national international targets in Uganda? Have outcomes been influenced by external context and factors?
Changes in policies, practices and attitudes of decision and policy makers: It has been noted previously that many of the district authorities have bought in the idea of identifying HIC, their treatment and the required support in the mainstream schools. The district education offices in Lwengo, Kalungu and Masaka are in a process of developing district-wide strategic plans to support the education of the HIC. In Lwengo and Kalungu, the districts authorities reported to have started sharing U-DAC project reports during their meetings with all the head teachers of primary schools in their districts. Thus, whereas, there are no changes to the policy yet, there are positive changes to practices and attitudes of the decision and policy makers that will benefit the HIC.
Influence of external context and factors on the project outcomes: There were a number of external factors that influenced the observed project outcomes as noted by the project staff. These included: (a) High acceptance and willingness from community leaders and district authorities; (b) Existence and role of the Greater Masaka Disability Network has helped some HIC to get scholastic materials which in turn has assisted in the retention rates of HIC in school; (c) Limited hearing assessment and treatment services for referral of HIC in rural settings and the project was forced to provide hearing assessment and treatment through clinic and community outreaches; (d) Household level poverty and limited support of the HIC at the family level; and (f) Signal UK Cooperation and Flexibility: The level of collegiality, cooperation and flexibility exercised by Signal UK in support of the numerous innovations and revisions of the project strategies by Sign Health Uganda such as allowing inclusion of hearing assessment and treatmentservices on the project list of activities contributed to the project success.