Ministry of Health
Data Quality Assessment Manual
Tools and Guidelines for Implementation
ACKNOWLEGEMENTS
The Data Quality Assessment Manual – Tools and Guidelines for Implementation have been developed by the Ministry of Health–Resource Centre in collaboration with the Ministry of Health technical programmes, districts, and various HMISImplementing Partners. The MOH-Resource Centre is grateful to all stakeholders including local governments, organizations and individuals whohave given assistance and supportedthe planning, development and finalization of the DQAManual.The following individuals in particular contributed at different stages in the development of the Data Quality Assessment Manual:
Technical Team members who participated in the development of the DQA Tools:
Name / Title/Organization / Name / Title/OrganizationDr. Eddie Mukooyo / Assistant Commissioner Resource Center / Aguti Evelyn Francis / Training officer/Coordinator –PREFA
Mr Ally Kibwika / Assistant Commissioner- Human Resource Development / Lyazi Micheal Ivan / M&E officer PREFA
Ms. Caroline Kyozira / Principal Biostatistician / Mary Namuyomba / Kayunga District HMIS –FP
Natseri Nasan / Data Manager/WHO / John Kissa / Statistician /RC
Allan Nsubuga / Statistician /RC / Yawe Moses / Nakaseke District Biostatistician
Joel Kisubi / Malaria Advisor PMI/USAID / Dennis Businge / M&E Director/STAR EC
Bright Asiimwe Wandera / M&E Manager - STOP Malaria Project / Kayanja Edward / Luweero District HMIS-FP
Kayongo Emmanuel / M&E officer STOP Malaria Project / Areke James Stephen / Soroti District Biostatistician
Flavia Kazibwe / Data officer STOP Malaria Project / Etoma Chalres / Masindi District Biostatistician
Andrew Anguko / M&E specialist– Malaria consortium / Kusiima Wilson / Buliisa District HMIS FP
Dr. Ebony Quinto / M&E specialist- National Malaria Control Program / Iteset Ruth Atiro / Kaberamaido District HMIS FP
Dr. Suzie Nasar / Malaria Advisor PMI/CDC / Asiimwe Patricia / Hoima District HMIS FP
Ssetuba Abasolom / Strategic information officer STAR– South West / Angida Teddy / Senior Statistician – Mulago National Referral Hospital
Kasozi Sam / HIS Analyst –CDC / Opolot Francis / Kumi District HMIS –FP
Anna Collator Awor / Statistician– CDC / Thomas Emeetai / Data Manager/STRIDES
Dr Moses Walakira / Director SI/STAR SW / Nakamya Phellister / M&E Specialist – UCCM Secretariat
ACROYNMS AND ABBREVIATIONS
ANC / Antenatal CareART / Antiretroviral Therapy
ARV / Antiretroviral
BCC / Behavior Change Communication
DHO / District Health Office
DOTS / Directly Observed Therapy
DQA / Data Quality Assessment
HC / Health Centre
HMIS / Health Management Information System
HSSIP / Health Sector Strategic and Investment Plan
ITN / Insecticide-Treated Nets
M&E / Monitoring and Evaluation
MIP / Malaria Cases In Pregnancy
MOH / Ministry Of Health
MOH-RC / Ministry of Health – Resource Centre
NGO / Non Governmental Organizations
PMTCT / Prevention of Mother to Child Transmission
RC / Resource Centre
RDQA / Routine Data Quality Assessment
VCT / Voluntary HIV Counseling and Testing
FOREWARD
National health programs and other partner institutions are working towards achieving goals as stipulated in the Health Sector Strategic and Investment Plan - Monitoring and Evaluation (HSSIP M&E) plan 2010/2011 – 2014/15. In that regard, the Ministry of Health recognizes the importance of improving the National Health Management Information System (HMIS) for the successful planning, delivery and implementation of the minimum health services as indicated in the Health Sector Strategic & Investment Plan (HSSP).
To achieve the above, the HMIS needs to provide quality strategic information for decision makers who will combine this information with other strategic information from other agreed upon sources to make evidence-based decisions.
In a bid to improve the quality of data generated from the national health system, numerous efforts have been instigated by Ministry of Health – Resource Centre in close collaboration with MoH health departments, donors and implementing partners to strengthen the national HMIS.
In consideration of the fact that there has always been an increasing demand for quality data at all levels by donor and policy makers, the Ministry of Health through the Resource Centre has developed harmonized standard DQA tools and guidelines for data quality assessments at national, district and health facility levels.
The Data Quality Assessment Tools and Guidelines developed should be utilized appropriately at the various levels of Health Care delivery to instill a culture of collection and utilization of quality data for better decision making.
It is my appeal that the various health institutions and departments across the country use the standard and harmonized processesin this manual during the implementation of data quality assessment at the Health Care Service delivery levels.
Dr. Aceng Jane Ruth
Director General Health Services
TABLE OF CONTENTS
ACKNOWLEGEMENTS
ACROYNMS AND ABBREVIATIONS
FOREWARD
EXECUTIVE SUMMARY
INTRODUCTION
1.1 BACKGROUND
1.2 OBJECTIVES
1.3 CONCEPTUAL FRAMEWORK
1.4 METHODOLOGY
1.5 SELECTION OF SITES
1.6 OUTPUTS
1.7 ETHICAL CONSIDERATIONS
1.8 IMPLEMENTATION
PHASE 1: PREPARATION AND INITIATION
STEP 1: SELECT INDICATOR/DATA ELEMENT(S) AND REPORTING PERIOD
STEP 2: NOTIFY DISTRICTS, REQUEST DOCUMENTATION AND OBTAIN AUTHOURIZATION
STEP 3: SELECT SITES TO BE ASSESSED
STEP 4: PREPARE FOR ON-SITE ASSESSMENT VISITS
STEP 5: REVIEW DOCUMENTATION
PHASE 2: NATIONAL/RESOURCE CENTER LEVEL
STEP 6: ASSESS DATA MANAGEMENT SYSTEMS (AT THE NATIONAL/RESOURCE CENTER LEVEL)
STEP 7: TRACE AND VERIFY RESULTS FROM DISTRICTS (AT THE NATIONAL/RESOURCE CENTER LEVEL)
PHASE 3: DISTRICT LEVEL
STEP 8: ASSESS DATA MANAGEMENT SYSTEMS (DISTRICT/SUB-DISTRICT LEVEL)
STEP 9: TRACE AND VERIFY RESULTS FROM SITE REPORTS (AT THE INTERMEDIATE AGGREGATION LEVELS)
PHASE 4: HEALTH FACILITIES
STEP 10: ASSESS DATA MANAGEMENT SYSTEMS
(AT HEALTH FACILITY LEVEL)
STEP 11: TRACE AND VERIFY RESULTS FROM SOURCE DOCUMENTS (AT SERVICE DELIVERY LEVELS)
PHASE 5: NATIONAL/RESOURCE CENTER LEVEL
STEP 12: CONSOLIDATE ASSESSMENT OF DATA MANAGEMENT SYSTEMS
STEP 13: DRAFT PRELIMINARY FINDINGS AND RECOMMENDATION NOTES
STEP 14: CONDUCT A CLOSEOUT MEETING
PHASE 6: COMPLETION
STEP 15: DRAFT ASSESSMENT REPORT
STEP 16: COLLECT AND REVIEW FEEDBACK FROM NATIONAL/RESOURCE CENTER LEVEL
STEP 17: FINALIZE ASSESSMENT REPORT
ANNEX 1: DQA Protocols
Protocol 1: Systems Assessment Protocol
Protocol 2: Data Verification Protocol
ANNEX 2: Templates for the Assessment Team
Annex 2, Step 2 – Template 1. Information Sheet required at the National/Resource Center level during the DQA
Annex 2, Step 2 – Template 2. Information Sheet for the Intermediate Aggregation (District/Health sub-district) Levels selected for the DQA
Annex 2, Step 2 – Template 3. Information Sheet for all Service Delivery Sites Selected for the DQA
Annex 2, Step 4 – Template 1
Annex 2, Step 4 – Template 4. Checklist for Audit Team Preparation for Audit Site Visits
REFERENCES
1
Ministry of Health - Resource Centre April 2012
EXECUTIVE SUMMARY
Accurate, timely and accessible health care data play a vital role in the planning, development and maintenance of health care services. Quality improvement and the timely dissemination of quality data are essential if health authorities wish to maintain health care at an optimal level.In recent years, data quality has become an important issue, not only because of its importance in promoting high standards of patient care, but also because of its impact on government budgets for the maintenance of health services. Authorities at all levels of health care, including hospitals, community health centres, outlying clinics and aid posts, as well as ministries or departments of health, should be concerned about poordata quality and the impact it has on the quality of health care.
In a bid to improve the quality of data generated from the national health system, numerous efforts have been instigated by MoH-RC, donors and implementing partners to strengthen the national HMIS. Additional, there has been increasing demand for quality data at all levels by donor and policy makers triggering the need for harmonized standard DQA guidelines and tools for data quality assessments at national, district and health facility levels.
The guidelines for implementing DQA will ensure that processes undertaken at the various levels of Health Care Service delivery are standardized. The assessment will aim at:
- Verifying the quality of reported data for key indicators at selected different levels of the Health Care Service delivery.
And
- Assessing the overall ability of data management systems to collect and report quality data within the Health Sector.
The DQA guidelines shall provide guidance on processes, protocols, and templates addressing how to:
Determine the scope of the data quality assessment.
Engage the different levels of health service delivery and prepare for the assessment exercise.
Assess the design and implementation of the data management and reporting systems at the level visited.
Trace and verify (recount) selected data element and hence indicator results.
Develop and present the Assessment Team’s findings and recommendations.
The guidelines stipulate the steps that should be taken while conducting DQA at National, District/Health Sub-district and Health Facility level. The steps include all necessary preparations required to facilitate the DQA exercise at all levels including obtaining authorization, communication and liaison with the sites at which DQA will be conducted for their notification, Assessment of data management and reporting systems, trace and verifying of counts of the selected indicators and finally preparation of draft assessment report that includes proposed recommendations and follow-up areas from the assessment exercise.
The implementation of the DQA will be supported by two protocols (see ANNEX 1):
Protocol 1: System Assessment Protocol.
Protocol 2: Data Verification Protocol.
These protocols will be administered appropriately at each level of the data-collection and reporting system (i.e., National/Resource Centre, Districts or sub-districts and Health facilities)
The Data Quality Assessment guideline is not designed to assess the quality of services provided but ratherensures improvements in health service delivery quality as a result of the availability of better quality data related to health sector performance and decision making.
INTRODUCTION
1.1 BACKGROUND
National programs and other partner institutions are working towards achieving goalsas stipulated in the Health Sector Strategic and Investment Plan - Monitoring and Evaluation (HSSIP M&E) plan 2010/2011 – 2014/15. In order to attain the successof these goals, the HMIS needs to providequality strategic information for decision makers who will combine this information with other strategic information from other agreed upon sources to make evidence-based decisions. This is relevant to both national and sub-national (district, sub-district and facility) levels.
In accordance to the HSSIP M&E, all reports submitted to the Resource Centre will be reviewed for accuracy and clarification sought where necessary. Even where there is no need for clarification, acknowledgement of receipt of reports will be provided before the due date for the subsequent report.
There are plans to implement Data quality assurance processes which include periodic Data Quality Assessments (DQA) of recorded data by supervisors; regular training of staff, and provision of routine feedback to staff at all levels on completeness, reliability and validity of data; and data quality assessment and adjustment which will be carried out periodically.
The objective of Data Quality Assessments and hence data validation is to ensure that the data used by the stakeholders to make decisions is sound and accurate. Specific efforts will be made to undertake data quality assessments including: application of the computed validation/data accuracy index into district annual reports; specific support for outliers; routine (quarterly) data checks on a sample of districts among others.
Periodic data validation, training and provision of feedback on validity of health facility data will be carried out by the Ministry of Health-Resource Center. DQA isto be carried out at all levels of health care service delivery. The various processes in the DQA will take care of: - data collection methods, aggregationand analysis by the technical staff of the RC at the districts and by the District Biostatistician/HMIS FP within the districts at the various sub-districts and health units (Hospitals, HCIV, HCIII, and HCII).
The DQA implementation guidelines and tools have been developed for application at all levels of health care service delivery. The Assistant Commissioner in charge of the RC will be responsible for the overall implementation of the DQA processes on behalf of the health sector. The RC will carry out regular DQAs, training of staff in conducting DQAs, and provision of routine feedback on completeness, reliability and validity of data generated through the HMIS. This should be done on a quarterly basis.
TheDQA implementation guidelines and tools focus exclusively on:-
(1) Verifying the quality of reported data
(2) assessing the underlying data management and reporting systems in line with the National Health Information Strategy (National HIS)
1.2 OBJECTIVES
The objective of the guideline for implementing DQAis to ensure that processes undertaken at the various levels of Health Care Service delivery are standardized. The DQA will aim at:
- Verifying the quality of reported data for key indicators at selected different levels of the Health Care Service delivery.
- Assess the overall ability of data management systems to collect and report quality data within the Health Sector.
It should be noted that the findings of the DQA can also be very useful for improving and strengthening the overallHealth Sectordata management and reporting systems.
This DQA guidelinefor implementation provides processes, protocols, and templates addressing how to:
Determine the scope of the data quality assessment.
The DQA guideline begins with suggested criteria, technical area to be addressed, and indicators or data elements to be reviewed. In most cases, the technical area on which the DQA will be conducted will be determined from the initial analysis of the hard copies and database by the Resource Center. The Resource Centre will have the overall responsibility in ensuring that all data collected through the HMIS is subject to DQA at a routine basis and that the systems to strengthen the quality of data are continuously revised and updated in collaboration with key stakeholders.
Engage the different levels of health service deliveryand prepare for the assessmentexercise.
The DQA guideline emphasizes the importance of notifying the health service delivery levels (Resource Centre, District, Health Sub district, Health facility)of the impending data quality assessment as well as obtaining relevant authorizations as spelt out in the HMIS Standard Operating Procedure Manual.
Assess the design and implementation of the data management and reporting systems at the level visited.
The DQA guideline provides steps and a protocol to identify potential risks to data quality created by the existence and non-existence data management and reporting system.
Trace and verify (recount) selected data element and hence indicator results.
The DQA guidelineprovides protocol(s) with special instructions, based on the indicator/data element and type of Health Service Delivery Site (e.g. health facility or community-based). These protocols will direct the AssessmentTeam as it verifies data for the selected indicator/data element from source documents and compares the results to the reported results.
Develop and present the Assessment Team’s findings and recommendations.
The DQAguideline provides instructions on how and when to present the DQA findings and recommendations bynational level officials to the beneficiary level where the DQA was conducted and how to plan for follow-up activities to ensure that agreed-upon steps to improve systems and data quality are completed.
Note: While the Data Quality Assessment guideline is not designed to assess the quality of services provided, its use could facilitate improvements in health service delivery quality as a result of the availability of better quality data related to health sector performance and decision making.
1
Ministry of Health - Resource Centre April 2012
1.3 CONCEPTUAL FRAMEWORK
The conceptual framework for the DQA and RDQA is illustrated in the Figure 1(below). Generally the quality of reported data is dependent on the underlying data management and reporting systems; stronger systems should produce better quality data. In other words, for good quality data to be produced a data management system, key functional components need to be in place at all levels of the system — the points of Health Care Service delivery (Health facilities), the intermediate level(s) where the data are aggregated (e.g. sub-districts, districts, regions), and the Resource Center at the highest level to which data are reported. The DQA and RDQA tools are therefore designed to:
(1) Verify the quality of the data,
(2) Assess the system that produces that data, and
(3) Develop action plans to improve both.
Introduction – Figure 1: Conceptual Framework for the (R)DQA: Data Management and Reporting Systems, Functional Areas, and Data Quality.
1.4 METHODOLOGY
The DQA and RDQA are grounded in the components of data quality in which the health sector and other health programs need accurate, reliable, precise, complete and timely data reports that the leadership arm and other policy and decision makers can use to effectively direct available resources and to evaluate progress toward established goals (see Introduction Table 1 below). Furthermore, the data must have integrity to be considered credible and should be produced ensuring standards agreed upon by the sector