Public Health Action VOL
InternaƟonal Union Against Tuberculosis and Lung Disease Health soluƟons for the poor
3 NO 1 PUBLISHED 21 MARCH 2013
Assessing and improving data quality from community health workers: a successful intervention in Neno, Malawi A. J. Admon,1 J. Bazile,2,3 H. Makungwa,3 M. A. Chingoli,3 L. R. Hirschhorn,2,4 M. Peckarsky,2 J. Rigodon,2,3 M. Herce,2,3 F. Chingoli,5 P. N. Malani,1,6 B. L. Hedt-Gauthier4 http://dx.doi.org/10.5588/pha.12.0071
Setting: A community health worker (CHW) program was established in Neno District, Malawi, in 2007 by Partners In Health in support of Ministry of Health activities. Routinely generated CHW data provide critical information for program monitoring and evaluation. Informal assessments of the CHW reports indicated poor quality, limiting the usefulness of the data. Objectives: 1) To establish the quality of aggregated measures contained in CHW reports; 2) to develop interventions to address poor data quality; and 3) to evaluate changes in data quality following the intervention. Design: We developed a lot quality assurance samplingbased data quality assessment tool to identify sites with high or low reporting quality. Following the first assessment, we identified challenges and best practices and followed the interventions with two subsequent assessments. Results: At baseline, four of five areas were classified as low data quality. After 8 months, all five areas had achieved high data quality, and the reports generated from our electronic database became consistent and plausible. Conclusion: Program changes included improving the usability of the reporting forms, shifting aggregation responsibility to designated assistants and providing aggregation support tools. Local quality assessments and targeted interventions resulted in immediate improvements in data quality.
C
ommunity health worker (CHW) programs have been employed by Partners In Health (PIH) since the mid-1980s, providing health education and linking populations with poor health care access to local health care resources.1 Abwenzi Pa Za Umoyo (APZU), the PIH sister organization in Malawi, established a CHW program to support primary care service delivery in rural, resource-poor Neno District. This program includes nearly 700 community health workers (referred to as village health workers by APZU), each receiving specialized training to serve as health educators, to accompany patients with human immunodeficiency virus (HIV) and/or tuberculosis (TB) to clinic visits, and to link patients to the formal health system. Along with these activities, about one third of CHWs (n = 240) are also tasked with active case finding for priority diseases and monthly data collection through the Household Chart (HHC) Program. For the HHC Program, CHWs visit each of their approximately 40 households monthly to collect demographic and health data on nearly 30 indicators. The
CHWs generate household reports; Health Surveillance Assistants (HSAs) then aggregate across all household reports to generate reports for each CHW in their area. These CHW reports are then entered into an electronic database from which monthly reports are generated for PIH and Ministry of Health (MoH) management. As with other health programs, CHW data could play a critical role in supporting implementation by assisting in identifying and addressing programmatic gaps.2–4 However, there were numerous concerns over the quality of the data due to the large volume and complex aggregation pathways. An informal inspection revealed several instances in which chart data were aggregated incorrectly, leading to significant inaccuracies in the final reports presented to MoH/PIH. In this article, we describe our system for the formal assessment of the quality of CHW reports; we provide detail on the development and implementation of interventions to address poor data quality; and we conclude with a discussion on the impact of improved data quality on the ability to successfully manage the program and suggestions of extending lessons learned to other programs.
STUDY POPULATION, DESIGN AND METHODS Although there have been considerable improvements in health in Malawi over the last 10 years, the country still suffers from high rates of under-five and maternal mortality, at respectively 112 per 1000 and 675 per 100 000 live births.5 Formed in 2003, Neno District is home to approximately 110 000 individuals who live in four ‘Traditional Authority’ areas. The district is served by a district hospital, one community hospital, 11 government health centers and 78 smaller health posts. Each health center has a catchment area that draws from three to 11 health posts, and each health post serves several villages. This study focuses on 21 of the 78 health posts from three health centers, grouped into five clusters. Each cluster contained three to five health posts from a single health center. Each cluster received a classification of high or low data quality using the data quality assessment tool described below.
Data quality assessment tools Lot quality assurance sampling (LQAS) is a classification procedure that classifies the performance of a ‘lot’ by evaluating a representative sample.6–8 To implement,
AFFILIATIONS 1 Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, USA 2 Partners In Health, Boston, Massachusetts, USA 3 Abwenzi Pa Za Umoyo, Neno, Malawi 4 Department of Global Health and Social Medicine, Harvard Medical School, Cambridge, Massachusetts, USA 5 Ministry of Health, Neno District, Malawi 6 Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan, USA CORRESPONDENCE Bethany Hedt-Gauthier Department of Global Health and Social Medicine Harvard Medical School Cambridge, MA 02115, USA Tel: (+1) 617 432 7260 Fax: (+1) 617 432 2565 e-mail: [email protected] ACKNOWLEDGEMENTS The authors thank the village health workers, health surveillance assistants and community health worker (CHW) site supervisors for their diligent work in Neno District and their commitment to improving the household chart program. The authors also extend their sincere gratitude to the Neno District Health Office and the Ministry of Health for supporting the CHW and household chart (HHC) programs. Finally, they also thank B Chabwera for his work in supporting the HHC program. This project was supported in part by funding from Global REACH, University of Michigan Medical School, Ann Arbor, MI, USA. BHG received support from the Department of Global Health and Social Medicine Research Core at Harvard Medical School, Cambridge, MA, USA. Conflict of interest: none declared. KEY WORDS lot quality assurance sampling; supervision; quality improvement
Received 2 October 2012 Accepted 1 December 2012
PHA 2013; 3(1): 56–59 © 2013 The Union
Public Health Action n units are randomly sampled and the number with the trait of interest (i.e., report reliability) is compared to a predetermined decision rule, d. If d or more of the units have the trait of interest, the area is classified as high with respect to that metric. If fewer than d units have the trait of interest, then the area is classified as low. As the end result is a two-level classification and not an estimate with high levels of precision, LQAS can often be implemented with relatively small sample sizes to meet program goals, translating into lower costs and more rapid feedback loops than other assessment methods. Furthermore, the classification of an area as ‘high performing’ or ‘low performing’ on a particular metric can immediately be linked to programmatic improvement. LQAS has been used in over 800 health applications,9 including for data quality assessments in other types of health programs10–12 and monitoring CHW service delivery.13 Due to the precedence in successful use of LQAS for program monitoring, the ability to classify areas with relatively small sample sizes, the direct link of classifications to action and the ability to integrate LQAS into ongoing supervision activities, APZU opted to use this method to formally assess the quality of CHW reports. To determine the sample size and decision rules for LQAS, four parameters must be defined: the definition of high data quality, the definition of low data quality and the allowable misclassification risks at each of these two thresholds. Based on extensive discussions on the necessary levels of data quality to be reliable for program management, we defined high data quality as 90% or better agreement between CHW reports and the corresponding household reports. We defined low data quality as 70% or worse agreement between CHW and household reports, and limited misclassification risks to 100%. The post-intervention phase, August–March 2012, shows results that are consistent and within plausible ranges.
Public Health Action
Improving CHW program data in Malawi
TABLE 2 Percentage of reviewed reports with poor data quality, pre- and post-intervention, by indicator, Neno District, Malawi Reports with poor data quality Preintervention (July 2011) % Indicator 1: number of households Indicator 2: number of individuals tested for HIV Indicator 3: number of individuals aged >15 years tested for HIV Indicator 4: number of children aged
InternaƟonal Union Against Tuberculosis and Lung Disease Health soluƟons for the poor
3 NO 1 PUBLISHED 21 MARCH 2013
Assessing and improving data quality from community health workers: a successful intervention in Neno, Malawi A. J. Admon,1 J. Bazile,2,3 H. Makungwa,3 M. A. Chingoli,3 L. R. Hirschhorn,2,4 M. Peckarsky,2 J. Rigodon,2,3 M. Herce,2,3 F. Chingoli,5 P. N. Malani,1,6 B. L. Hedt-Gauthier4 http://dx.doi.org/10.5588/pha.12.0071
Setting: A community health worker (CHW) program was established in Neno District, Malawi, in 2007 by Partners In Health in support of Ministry of Health activities. Routinely generated CHW data provide critical information for program monitoring and evaluation. Informal assessments of the CHW reports indicated poor quality, limiting the usefulness of the data. Objectives: 1) To establish the quality of aggregated measures contained in CHW reports; 2) to develop interventions to address poor data quality; and 3) to evaluate changes in data quality following the intervention. Design: We developed a lot quality assurance samplingbased data quality assessment tool to identify sites with high or low reporting quality. Following the first assessment, we identified challenges and best practices and followed the interventions with two subsequent assessments. Results: At baseline, four of five areas were classified as low data quality. After 8 months, all five areas had achieved high data quality, and the reports generated from our electronic database became consistent and plausible. Conclusion: Program changes included improving the usability of the reporting forms, shifting aggregation responsibility to designated assistants and providing aggregation support tools. Local quality assessments and targeted interventions resulted in immediate improvements in data quality.
C
ommunity health worker (CHW) programs have been employed by Partners In Health (PIH) since the mid-1980s, providing health education and linking populations with poor health care access to local health care resources.1 Abwenzi Pa Za Umoyo (APZU), the PIH sister organization in Malawi, established a CHW program to support primary care service delivery in rural, resource-poor Neno District. This program includes nearly 700 community health workers (referred to as village health workers by APZU), each receiving specialized training to serve as health educators, to accompany patients with human immunodeficiency virus (HIV) and/or tuberculosis (TB) to clinic visits, and to link patients to the formal health system. Along with these activities, about one third of CHWs (n = 240) are also tasked with active case finding for priority diseases and monthly data collection through the Household Chart (HHC) Program. For the HHC Program, CHWs visit each of their approximately 40 households monthly to collect demographic and health data on nearly 30 indicators. The
CHWs generate household reports; Health Surveillance Assistants (HSAs) then aggregate across all household reports to generate reports for each CHW in their area. These CHW reports are then entered into an electronic database from which monthly reports are generated for PIH and Ministry of Health (MoH) management. As with other health programs, CHW data could play a critical role in supporting implementation by assisting in identifying and addressing programmatic gaps.2–4 However, there were numerous concerns over the quality of the data due to the large volume and complex aggregation pathways. An informal inspection revealed several instances in which chart data were aggregated incorrectly, leading to significant inaccuracies in the final reports presented to MoH/PIH. In this article, we describe our system for the formal assessment of the quality of CHW reports; we provide detail on the development and implementation of interventions to address poor data quality; and we conclude with a discussion on the impact of improved data quality on the ability to successfully manage the program and suggestions of extending lessons learned to other programs.
STUDY POPULATION, DESIGN AND METHODS Although there have been considerable improvements in health in Malawi over the last 10 years, the country still suffers from high rates of under-five and maternal mortality, at respectively 112 per 1000 and 675 per 100 000 live births.5 Formed in 2003, Neno District is home to approximately 110 000 individuals who live in four ‘Traditional Authority’ areas. The district is served by a district hospital, one community hospital, 11 government health centers and 78 smaller health posts. Each health center has a catchment area that draws from three to 11 health posts, and each health post serves several villages. This study focuses on 21 of the 78 health posts from three health centers, grouped into five clusters. Each cluster contained three to five health posts from a single health center. Each cluster received a classification of high or low data quality using the data quality assessment tool described below.
Data quality assessment tools Lot quality assurance sampling (LQAS) is a classification procedure that classifies the performance of a ‘lot’ by evaluating a representative sample.6–8 To implement,
AFFILIATIONS 1 Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, USA 2 Partners In Health, Boston, Massachusetts, USA 3 Abwenzi Pa Za Umoyo, Neno, Malawi 4 Department of Global Health and Social Medicine, Harvard Medical School, Cambridge, Massachusetts, USA 5 Ministry of Health, Neno District, Malawi 6 Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan, USA CORRESPONDENCE Bethany Hedt-Gauthier Department of Global Health and Social Medicine Harvard Medical School Cambridge, MA 02115, USA Tel: (+1) 617 432 7260 Fax: (+1) 617 432 2565 e-mail: [email protected] ACKNOWLEDGEMENTS The authors thank the village health workers, health surveillance assistants and community health worker (CHW) site supervisors for their diligent work in Neno District and their commitment to improving the household chart program. The authors also extend their sincere gratitude to the Neno District Health Office and the Ministry of Health for supporting the CHW and household chart (HHC) programs. Finally, they also thank B Chabwera for his work in supporting the HHC program. This project was supported in part by funding from Global REACH, University of Michigan Medical School, Ann Arbor, MI, USA. BHG received support from the Department of Global Health and Social Medicine Research Core at Harvard Medical School, Cambridge, MA, USA. Conflict of interest: none declared. KEY WORDS lot quality assurance sampling; supervision; quality improvement
Received 2 October 2012 Accepted 1 December 2012
PHA 2013; 3(1): 56–59 © 2013 The Union
Public Health Action n units are randomly sampled and the number with the trait of interest (i.e., report reliability) is compared to a predetermined decision rule, d. If d or more of the units have the trait of interest, the area is classified as high with respect to that metric. If fewer than d units have the trait of interest, then the area is classified as low. As the end result is a two-level classification and not an estimate with high levels of precision, LQAS can often be implemented with relatively small sample sizes to meet program goals, translating into lower costs and more rapid feedback loops than other assessment methods. Furthermore, the classification of an area as ‘high performing’ or ‘low performing’ on a particular metric can immediately be linked to programmatic improvement. LQAS has been used in over 800 health applications,9 including for data quality assessments in other types of health programs10–12 and monitoring CHW service delivery.13 Due to the precedence in successful use of LQAS for program monitoring, the ability to classify areas with relatively small sample sizes, the direct link of classifications to action and the ability to integrate LQAS into ongoing supervision activities, APZU opted to use this method to formally assess the quality of CHW reports. To determine the sample size and decision rules for LQAS, four parameters must be defined: the definition of high data quality, the definition of low data quality and the allowable misclassification risks at each of these two thresholds. Based on extensive discussions on the necessary levels of data quality to be reliable for program management, we defined high data quality as 90% or better agreement between CHW reports and the corresponding household reports. We defined low data quality as 70% or worse agreement between CHW and household reports, and limited misclassification risks to 100%. The post-intervention phase, August–March 2012, shows results that are consistent and within plausible ranges.
Public Health Action
Improving CHW program data in Malawi
TABLE 2 Percentage of reviewed reports with poor data quality, pre- and post-intervention, by indicator, Neno District, Malawi Reports with poor data quality Preintervention (July 2011) % Indicator 1: number of households Indicator 2: number of individuals tested for HIV Indicator 3: number of individuals aged >15 years tested for HIV Indicator 4: number of children aged
