EDM Forum
EDM Forum Community eGEMs (Generating Evidence & Methods to improve patient outcomes)
Publish
12-15-2016
Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data Katharine H. McVeigh PhD, MPH NYC Department of Health and Mental Hygiene, [email protected]
Remle Newton-Dame MPH NYC Department of Health and Mental Hygiene (at the time of the study), [email protected]
Pui Ying Chan MPH NYC Department of Health and Mental Hygiene, [email protected]
Lorna E. Thorpe PhD NYU School of Medicine, Department of Population Health, [email protected] See next pages for additional authors
Follow this and additional works at: http://repository.edm-forum.org/egems Part of the Epidemiology Commons Recommended Citation McVeigh, Katharine H. PhD, MPH; Newton-Dame, Remle MPH; Chan, Pui Ying MPH; Thorpe, Lorna E. PhD; Schreibstein, Lauren MS; Tatem, Kathleen S.; Chernov, Claudia MPH; Lurie-Moroni, Elizabeth; and Perlman, Sharon E. MPH (2016) "Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data," eGEMs (Generating Evidence & Methods to improve patient outcomes): Vol. 4: Iss. 1, Article 27. DOI: http://dx.doi.org/10.13063/2327-9214.1267 Available at: http://repository.edm-forum.org/egems/vol4/iss1/27
This Methods Empirical Research is brought to you for free and open access by the the Publish at EDM Forum Community. It has been peer-reviewed and accepted for publication in eGEMs (Generating Evidence & Methods to improve patient outcomes). The Electronic Data Methods (EDM) Forum is supported by the Agency for Healthcare Research and Quality (AHRQ), Grant 1U18HS022789-01. eGEMs publications do not reflect the official views of AHRQ or the United States Department of Health and Human Services.
Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data Abstract
Introduction: Electronic health records (EHRs) offer potential for population health surveillance but EHRbased surveillance measures require validation prior to use. We assessed the validity of obesity, smoking, depression, and influenza vaccination indicators from a new EHR surveillance system, the New York City (NYC) Macroscope. This report is the second in a 3-part series describing the development and validation of the NYC Macroscope. The first report describes in detail the infrastructure underlying the NYC Macroscope; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. This second report, which addresses concerns related to sampling bias and data quality, describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods described in this report to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia. Methods: NYC Macroscope prevalence estimates, overall and stratified by sex and age group, were compared to reference survey estimates for adult New Yorkers who reported visiting a doctor in the past year. Agreement was evaluated against 5 a priori criteria. Sensitivity and specificity were assessed by examining individual EHR records in a subsample of 48 survey participants. Results: Among adult New Yorkers in care, the NYC Macroscope prevalence estimate for smoking (15.2%) fell between estimates from NYC HANES (17.7 %) and CHS (14.9%) and met all 5 a priori criteria. The NYC Macroscope obesity prevalence estimate (27.8%) also fell between the NYC HANES (31.3%) and CHS (24.7%) estimates, but met only 3 a priori criteria. Sensitivity and specificity exceeded 0.90 for both the smoking and obesity indicators. The NYC Macroscope estimates of depression and influenza vaccination prevalence were more than 10 percentage points lower than the estimates from either reference survey. While specificity was > 0.90 for both of these indicators, sensitivity was < 0.70. Discussion: Through this work we have demonstrated that EHR data from a convenience sample of providers can produce acceptable estimates of smoking and obesity prevalence among adult New Yorkers in care; gained a better understanding of the challenges involved in estimating depression prevalence from EHRs; and identified areas for additional research regarding estimation of influenza vaccination prevalence. We have also shared lessons learned about how EHR indicators should be constructed and offer methodologic suggestions for validating them. Conclusions: This work adds to a rapidly emerging body of literature about how to define, collect and interpret EHR-based surveillance measures and may help guide other jurisdictions. Acknowledgements
The authors would like to thank Thomas Farley, Carolyn Greene, Jesse Singer, Elisabeth Snell, Laura Jacobson, Amy Freeman, Jesica Rodriguez-Lopez, Rhoda Schlamm, Kevin Konty, Stephen Immerwahr, Shadi Chamany, Sarah Shih, Tiffany Harris, James Hadler, and Charon Gwynn for their contributions to this work. This work This empirical research is available at EDM Forum Community: http://repository.edm-forum.org/egems/vol4/iss1/27
has been made possible by the financial support of the de Beaumont Foundation, the Robert Wood Johnson Foundation including its National Coordinating Center for Public Health Services and Systems Research, the Robin Hood Foundation, the NY State Health Foundation, the Doris Duke Charitable Foundation, and the U.S. Centers for Disease Control and Prevention (U28EH000939). The contents of this paper are solely the responsibility of the authors and do not represent the official views of the funders. Keywords
Population Health, Electronic Health Records, Surveillance, Validity, Chronic Disease Disciplines
Epidemiology | Medicine and Health Sciences | Public Health Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License. Authors
Katharine H McVeigh, NYC Department of Health and Mental Hygiene; Remle Newton-Dame, NYC Department of Health and Mental Hygiene (at the time of the study); Pui Ying Chan, NYC Department of Health and Mental Hygiene; Lorna E Thorpe, NYU School of Medicine, Department of Population Health; Lauren Schreibstein, NYC Department of Health and Mental Hygiene; Kathleen S Tatem, NYC Department of Health and Mental Hygiene; Claudia Chernov, NYC Department of Health and Mental Hygiene; Elizabeth LurieMoroni, NYC Department of Health and Mental Hygiene; Sharon E Perlman, NYC Department of Health and Mental Hygiene.
This empirical research is available at EDM Forum Community: http://repository.edm-forum.org/egems/vol4/iss1/27
McVeigh et al.: NYC Macroscope Validation Study
eGEMs Gener
&DQ(OHFWURQLF+HDOWK5HFRUGV%H8VHGIRU 3RSXODWLRQ+HDOWK6XUYHLOODQFH"9DOLGDWLQJ 3RSXODWLRQ+HDOWK0HWULFV$JDLQVW(VWDEOLVKHG 6XUYH\'DWD .DWKDULQH+0F9HLJK3K'03+ i5HPOH1HZWRQ'DPH03+ ii3XL
EDM Forum Community eGEMs (Generating Evidence & Methods to improve patient outcomes)
Publish
12-15-2016
Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data Katharine H. McVeigh PhD, MPH NYC Department of Health and Mental Hygiene, [email protected]
Remle Newton-Dame MPH NYC Department of Health and Mental Hygiene (at the time of the study), [email protected]
Pui Ying Chan MPH NYC Department of Health and Mental Hygiene, [email protected]
Lorna E. Thorpe PhD NYU School of Medicine, Department of Population Health, [email protected] See next pages for additional authors
Follow this and additional works at: http://repository.edm-forum.org/egems Part of the Epidemiology Commons Recommended Citation McVeigh, Katharine H. PhD, MPH; Newton-Dame, Remle MPH; Chan, Pui Ying MPH; Thorpe, Lorna E. PhD; Schreibstein, Lauren MS; Tatem, Kathleen S.; Chernov, Claudia MPH; Lurie-Moroni, Elizabeth; and Perlman, Sharon E. MPH (2016) "Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data," eGEMs (Generating Evidence & Methods to improve patient outcomes): Vol. 4: Iss. 1, Article 27. DOI: http://dx.doi.org/10.13063/2327-9214.1267 Available at: http://repository.edm-forum.org/egems/vol4/iss1/27
This Methods Empirical Research is brought to you for free and open access by the the Publish at EDM Forum Community. It has been peer-reviewed and accepted for publication in eGEMs (Generating Evidence & Methods to improve patient outcomes). The Electronic Data Methods (EDM) Forum is supported by the Agency for Healthcare Research and Quality (AHRQ), Grant 1U18HS022789-01. eGEMs publications do not reflect the official views of AHRQ or the United States Department of Health and Human Services.
Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data Abstract
Introduction: Electronic health records (EHRs) offer potential for population health surveillance but EHRbased surveillance measures require validation prior to use. We assessed the validity of obesity, smoking, depression, and influenza vaccination indicators from a new EHR surveillance system, the New York City (NYC) Macroscope. This report is the second in a 3-part series describing the development and validation of the NYC Macroscope. The first report describes in detail the infrastructure underlying the NYC Macroscope; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. This second report, which addresses concerns related to sampling bias and data quality, describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods described in this report to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia. Methods: NYC Macroscope prevalence estimates, overall and stratified by sex and age group, were compared to reference survey estimates for adult New Yorkers who reported visiting a doctor in the past year. Agreement was evaluated against 5 a priori criteria. Sensitivity and specificity were assessed by examining individual EHR records in a subsample of 48 survey participants. Results: Among adult New Yorkers in care, the NYC Macroscope prevalence estimate for smoking (15.2%) fell between estimates from NYC HANES (17.7 %) and CHS (14.9%) and met all 5 a priori criteria. The NYC Macroscope obesity prevalence estimate (27.8%) also fell between the NYC HANES (31.3%) and CHS (24.7%) estimates, but met only 3 a priori criteria. Sensitivity and specificity exceeded 0.90 for both the smoking and obesity indicators. The NYC Macroscope estimates of depression and influenza vaccination prevalence were more than 10 percentage points lower than the estimates from either reference survey. While specificity was > 0.90 for both of these indicators, sensitivity was < 0.70. Discussion: Through this work we have demonstrated that EHR data from a convenience sample of providers can produce acceptable estimates of smoking and obesity prevalence among adult New Yorkers in care; gained a better understanding of the challenges involved in estimating depression prevalence from EHRs; and identified areas for additional research regarding estimation of influenza vaccination prevalence. We have also shared lessons learned about how EHR indicators should be constructed and offer methodologic suggestions for validating them. Conclusions: This work adds to a rapidly emerging body of literature about how to define, collect and interpret EHR-based surveillance measures and may help guide other jurisdictions. Acknowledgements
The authors would like to thank Thomas Farley, Carolyn Greene, Jesse Singer, Elisabeth Snell, Laura Jacobson, Amy Freeman, Jesica Rodriguez-Lopez, Rhoda Schlamm, Kevin Konty, Stephen Immerwahr, Shadi Chamany, Sarah Shih, Tiffany Harris, James Hadler, and Charon Gwynn for their contributions to this work. This work This empirical research is available at EDM Forum Community: http://repository.edm-forum.org/egems/vol4/iss1/27
has been made possible by the financial support of the de Beaumont Foundation, the Robert Wood Johnson Foundation including its National Coordinating Center for Public Health Services and Systems Research, the Robin Hood Foundation, the NY State Health Foundation, the Doris Duke Charitable Foundation, and the U.S. Centers for Disease Control and Prevention (U28EH000939). The contents of this paper are solely the responsibility of the authors and do not represent the official views of the funders. Keywords
Population Health, Electronic Health Records, Surveillance, Validity, Chronic Disease Disciplines
Epidemiology | Medicine and Health Sciences | Public Health Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License. Authors
Katharine H McVeigh, NYC Department of Health and Mental Hygiene; Remle Newton-Dame, NYC Department of Health and Mental Hygiene (at the time of the study); Pui Ying Chan, NYC Department of Health and Mental Hygiene; Lorna E Thorpe, NYU School of Medicine, Department of Population Health; Lauren Schreibstein, NYC Department of Health and Mental Hygiene; Kathleen S Tatem, NYC Department of Health and Mental Hygiene; Claudia Chernov, NYC Department of Health and Mental Hygiene; Elizabeth LurieMoroni, NYC Department of Health and Mental Hygiene; Sharon E Perlman, NYC Department of Health and Mental Hygiene.
This empirical research is available at EDM Forum Community: http://repository.edm-forum.org/egems/vol4/iss1/27
McVeigh et al.: NYC Macroscope Validation Study
eGEMs Gener
&DQ(OHFWURQLF+HDOWK5HFRUGV%H8VHGIRU 3RSXODWLRQ+HDOWK6XUYHLOODQFH"9DOLGDWLQJ 3RSXODWLRQ+HDOWK0HWULFV$JDLQVW(VWDEOLVKHG 6XUYH\'DWD .DWKDULQH+0F9HLJK3K'03+ i5HPOH1HZWRQ'DPH03+ ii3XL
