Journal of Pediatric Surgery 49 (2014) 845–847
Contents lists available at ScienceDirect
Journal of Pediatric Surgery journal homepage: www.elsevier.com/locate/jpedsurg
Correspondence
References
Pitfalls abound when using administrative health databases
[1] Kelley-Quon LI, Tseng C-H, Janzen C, et al. Congenital malformations associated with assisted reproductive technology: A California statewide analysis. J Pediatr Surg 2013;48:1218–24. [2] Zhang Z, Macaluso M, Cohen B, et al. Accuracy of assisted reproductive technology information on the Massachusetts birth certificate, 1997–2000. Fertil Steril 2010;94:1657–61. [3] Cohen B, Bernson D, Sappenfield W, et al. Accuracy of Assisted Reproductive Technology information on birth certificates: Florida and Massachusetts, 2004– 2006. Paediatr Perinat Epidemiol 2014 [in press]. [4] Kirby RS, Salihu HM. Back to the future? A critical commentary on the 2003 U.S. National Standard Certificate of Live Birth. Birth 2006;33:238–44. [5] Kirby RS. Invited commentary: Using vital statistics databases for perinatal epidemiology: Does the quality go in before the name goes on? Am J Epidemiol 2001;154:889–90. [6] Major Birth Defects Data from Population-based Birth Defects Surveillance Programs in the United States, 2006–2010. Birth Defects Res A 2013;97:S1–S171. [7] Kirby RS. The Quality of Data Reported on Birth Certificates [Letter]. Am J Public Health 1997;87:301. [8] Jurek AM, Greenland S. Adjusting for multiple-misclassified variables in a study using birth certificates. Ann Epidemiol 2013;23:515–20.
To the Editor, Kelley-Quon et al. [1] are to be commended for their interest in outcomes of pregnancies conceived by assisted reproductive technology (ART). Unfortunately, their choice of administrative health datasets involves issues both with missingness and potentially differential misclassification bias concerning both the key exposure (ART) and the outcome (birth defects) in this analysis. Several studies have examined reporting of ART in U.S. vital statistics [2,3], using data on completed cycles reported in the National ART Surveillance System (NASS) databases linked to birth certificates. None of these studies have identified epidemiologically acceptable sensitivity or predictive value positive for the birth certificate questions on use of ART. Thus, it is likely that most pregnancies conceived by ART were not correctly identified as such in the linked birth and hospital discharge database. Vital statistics data must be used with caution, and researchers should not assume an adequate level of data quality, especially for newly added data elements [4,5]. The prevalence of major congenital malformations reported by Kelley-Quon et al. [1] is also curiously higher than typically found by U.S. or international population-based birth defects surveillance programs, and especially higher than for data based primarily on hospital discharges [BDRA 2013]. Typical overall prevalence of major birth defects is in the range of 2.5–3.5% with follow-up through the first year of life in databases like that used in this study [6]. The finding of 6.3–6.6% in the comparison groups (Table 3) strongly suggests overreporting of birth defects in this database, and especially for cardiac defects. It is unclear whether birth defect cases were also identified from data elements on the California birth certificate; it is well established that these data are of insufficient quality to be used in epidemiologic investigation [7]. Limiting the study to counties under surveillance by the California Birth Defects Monitoring Program over a longer time period and identifying cases of specific defects from the surveillance database would improve reliability and validity of the outcome data for this analysis. Sensitivity analyses should be conducted in situations where there is a high likelihood of misclassification of key exposure and outcomes variables [8]. This would assist readers in interpreting the meaning of the findings reported by Kelley-Quon et al. [1]. However, given the issues with reliability of the primary exposure variable, ART, and the outcome, birth defects, sensitivity analyses may not be sufficient to shed light on the true nature of the relationship. Meanwhile, the authors’ conclusion that ART increases the risk for birth defects is likely not supported by a judicious review of the evidence presented here. Russell S. Kirby Department of Community and Family Health College of Public Health University of South Florida Tampa, FL 33612 E-mail address: [email protected] http://dx.doi.org/10.1016/j.jpedsurg.2013.12.028 0022-3468/© 2014 Elsevier Inc. All rights reserved.
Reply to Letter to the Editor To the Editor, We would like to thank Dr. Kirby for interest in our recent study [1] describing the association of assisted reproductive technologies (ART) with birth defects and appreciate his many contributions to the field. Dr. Kirby has highlighted some important considerations regarding epidemiological research on birth defects that is obviously pertinent to our manuscript. The authors concur with Dr. Kirby's contention regarding statistically insensitive classification of ART on the birth certificate due primarily to under-reporting. Our study intentionally used a propensity matched case–control series design to counteract the expected low sensitivity of the exposure variable and utilize its high specificity (estimated to be 99.7% from comparison of Massachusetts birth certificate records to a N 80% deterministic linkage to the Centers for Disease Control (CDC) National ART Surveillance System (NASS) [2]). By creating a propensity score matched control group and creating a high (10:1) control to case mix ratio, the study design minimized the impact of falsely negative reported ART subjects in the control group and becomes less dependent upon a high sensitivity for the exposure variable as it would with an observational cohort study [3]. Propensity score matching of factors that were highly associated with ART also minimized the impact of similar covariates that are associated with concurrent development of birth defects. And as such, the study’s control group reflects subjects at a selectively higher risk for birth defects rather than the California population as a whole. The utility of a sensitivity analysis for the ART variable under this study design would be of questionable value and excessively conservative [4–6]. Dr. Kirby suggests the selection of the codes used for categorizing birth defects [7] and/or the use of the California Birth-linked cohort dataset (containing linked maternal–infant inpatient discharge records
Contents lists available at ScienceDirect
Journal of Pediatric Surgery journal homepage: www.elsevier.com/locate/jpedsurg
Correspondence
References
Pitfalls abound when using administrative health databases
[1] Kelley-Quon LI, Tseng C-H, Janzen C, et al. Congenital malformations associated with assisted reproductive technology: A California statewide analysis. J Pediatr Surg 2013;48:1218–24. [2] Zhang Z, Macaluso M, Cohen B, et al. Accuracy of assisted reproductive technology information on the Massachusetts birth certificate, 1997–2000. Fertil Steril 2010;94:1657–61. [3] Cohen B, Bernson D, Sappenfield W, et al. Accuracy of Assisted Reproductive Technology information on birth certificates: Florida and Massachusetts, 2004– 2006. Paediatr Perinat Epidemiol 2014 [in press]. [4] Kirby RS, Salihu HM. Back to the future? A critical commentary on the 2003 U.S. National Standard Certificate of Live Birth. Birth 2006;33:238–44. [5] Kirby RS. Invited commentary: Using vital statistics databases for perinatal epidemiology: Does the quality go in before the name goes on? Am J Epidemiol 2001;154:889–90. [6] Major Birth Defects Data from Population-based Birth Defects Surveillance Programs in the United States, 2006–2010. Birth Defects Res A 2013;97:S1–S171. [7] Kirby RS. The Quality of Data Reported on Birth Certificates [Letter]. Am J Public Health 1997;87:301. [8] Jurek AM, Greenland S. Adjusting for multiple-misclassified variables in a study using birth certificates. Ann Epidemiol 2013;23:515–20.
To the Editor, Kelley-Quon et al. [1] are to be commended for their interest in outcomes of pregnancies conceived by assisted reproductive technology (ART). Unfortunately, their choice of administrative health datasets involves issues both with missingness and potentially differential misclassification bias concerning both the key exposure (ART) and the outcome (birth defects) in this analysis. Several studies have examined reporting of ART in U.S. vital statistics [2,3], using data on completed cycles reported in the National ART Surveillance System (NASS) databases linked to birth certificates. None of these studies have identified epidemiologically acceptable sensitivity or predictive value positive for the birth certificate questions on use of ART. Thus, it is likely that most pregnancies conceived by ART were not correctly identified as such in the linked birth and hospital discharge database. Vital statistics data must be used with caution, and researchers should not assume an adequate level of data quality, especially for newly added data elements [4,5]. The prevalence of major congenital malformations reported by Kelley-Quon et al. [1] is also curiously higher than typically found by U.S. or international population-based birth defects surveillance programs, and especially higher than for data based primarily on hospital discharges [BDRA 2013]. Typical overall prevalence of major birth defects is in the range of 2.5–3.5% with follow-up through the first year of life in databases like that used in this study [6]. The finding of 6.3–6.6% in the comparison groups (Table 3) strongly suggests overreporting of birth defects in this database, and especially for cardiac defects. It is unclear whether birth defect cases were also identified from data elements on the California birth certificate; it is well established that these data are of insufficient quality to be used in epidemiologic investigation [7]. Limiting the study to counties under surveillance by the California Birth Defects Monitoring Program over a longer time period and identifying cases of specific defects from the surveillance database would improve reliability and validity of the outcome data for this analysis. Sensitivity analyses should be conducted in situations where there is a high likelihood of misclassification of key exposure and outcomes variables [8]. This would assist readers in interpreting the meaning of the findings reported by Kelley-Quon et al. [1]. However, given the issues with reliability of the primary exposure variable, ART, and the outcome, birth defects, sensitivity analyses may not be sufficient to shed light on the true nature of the relationship. Meanwhile, the authors’ conclusion that ART increases the risk for birth defects is likely not supported by a judicious review of the evidence presented here. Russell S. Kirby Department of Community and Family Health College of Public Health University of South Florida Tampa, FL 33612 E-mail address: [email protected] http://dx.doi.org/10.1016/j.jpedsurg.2013.12.028 0022-3468/© 2014 Elsevier Inc. All rights reserved.
Reply to Letter to the Editor To the Editor, We would like to thank Dr. Kirby for interest in our recent study [1] describing the association of assisted reproductive technologies (ART) with birth defects and appreciate his many contributions to the field. Dr. Kirby has highlighted some important considerations regarding epidemiological research on birth defects that is obviously pertinent to our manuscript. The authors concur with Dr. Kirby's contention regarding statistically insensitive classification of ART on the birth certificate due primarily to under-reporting. Our study intentionally used a propensity matched case–control series design to counteract the expected low sensitivity of the exposure variable and utilize its high specificity (estimated to be 99.7% from comparison of Massachusetts birth certificate records to a N 80% deterministic linkage to the Centers for Disease Control (CDC) National ART Surveillance System (NASS) [2]). By creating a propensity score matched control group and creating a high (10:1) control to case mix ratio, the study design minimized the impact of falsely negative reported ART subjects in the control group and becomes less dependent upon a high sensitivity for the exposure variable as it would with an observational cohort study [3]. Propensity score matching of factors that were highly associated with ART also minimized the impact of similar covariates that are associated with concurrent development of birth defects. And as such, the study’s control group reflects subjects at a selectively higher risk for birth defects rather than the California population as a whole. The utility of a sensitivity analysis for the ART variable under this study design would be of questionable value and excessively conservative [4–6]. Dr. Kirby suggests the selection of the codes used for categorizing birth defects [7] and/or the use of the California Birth-linked cohort dataset (containing linked maternal–infant inpatient discharge records
