bs_bs_banner
doi: 10.1111/ppe.12147
353
Commentary
Importance of Bias Analysis in Epidemiologic Research Richard F. MacLehose,a Martha M. Werlerb a
Division of Epidemiology and Community Health, University of Minnesota, School of Public Health, Minneapolis, MN b
Department of Epidemiology, Boston University School of Public Health, Boston, MA
Despite public health warnings against marijuana use during pregnancy,1 exposures do occur among women who don’t heed such warnings or whose pregnancies aren’t recognised until well into gestation.2 Clear evidence on the risks and safety of marijuana use in pregnancy is lacking, due in part to limitations in observational studies. van Gelder and colleagues deserve praise for addressing one such limitation in their study on maternal cannabis use in relation to birth defects in offspring, published in this issue of Paediatric and Perinatal Epidemiology.3 Exposure misclassification abounds in perinatal epidemiology, and marijuana use is no exception. The authors employed Bayesian bias analyses to adjust for the possible influence of exposure misclassification, specifically maternal under-reporting of cannabis use, on observed measures of association. Studies of risk factors for birth defects are especially amenable to sensitivity analyses that address misclassification of exposures for a few reasons. First, structural birth defects are rare, making the case– control design the most feasible from the perspective of operational efficiency and, thus, the most popular to implement. The retrospective measurement of exposure status (often employed in case–control studies) is vulnerable to misclassification, especially when measurement relies on maternal recall of events that may have occurred months or years in the past. Inaccurate reporting of exposure status can also be influenced by societal factors. It is widely presumed that illicit drugs can cause birth defects. Public health warnings, whether grounded in strong empirical evidence or arising from scant data, inform public impressions. Reports in mainstream media may have an even greater impact on viewpoints, and often strive to catch audiences’ attention rather than provide balanced information. Regardless of the influences on Correspondence: Richard F. MacLehose, Division Epidemiology and Community Health, University of Minnesota, School of Public Health, 1300 South 2nd Street, Suite 300, Minneapolis, MN 55454, USA. E-mail: [email protected]
© 2014 John Wiley & Sons Ltd Paediatric and Perinatal Epidemiology, 2014, 28, 353–355
any one person’s viewpoints of an exposure, study subjects may be more vulnerable to falsely denying exposure when that exposure is thought to be unhealthy. A final reason why assessment of the impact of exposure misclassification is necessary in studies of birth defects stems from the general presumption among clinicians and researchers alike that women bias their reports of exposures, depending on whether their offspring was affected with a birth defect or not. Reports of positive associations between any maternally reported exposure and a birth defect are often followed by criticism that they could be due to recall bias. The van Gelder et al. study collected data from mothers up to 2 years following delivery when memories may have faded; cannabis use during pregnancy carries a negative connotation; and previously reported case–control studies of cannabis use and birth defects grappled with the issue of recall bias.3 Thus, adjustment for exposure misclassification is warranted given their research question. The authors approached their bias analyses using two general sets of models. In the first set of models, the authors assumed that they knew the amount of exposure misclassification with certainty and specified sensitivity values that corresponded to three scenarios: no misclassification, non-differential misclassification, or differential misclassification. In these scenarios, the authors specified various magnitudes of misclassification and assumed there was no uncertainty in these values. In this case, the interval estimates obtained from the analysis may be of less interest (since they do not incorporate any uncertainty in the sensitivity values) than the adjusted odds ratio (OR). However, since so little is known about the actual amount of misclassification, it can be very useful to examine what the adjusted OR would be for a range of sensitivity values. This can be an important descriptive first step leading up to a bias analysis that incorporates uncertainty in the misclassification. Indeed, the authors find that for the values of sensitivity and specificity that they chose to examine, the
354
R. F. MacLehose and M. M. Werler
misclassification-adjusted ORs were extremely similar to one another and often quite similar to the estimates obtained from assuming no misclassification. Chu et al.4 have extended this idea to graphically present misclassification-adjusted ORs over a wide range of possible sensitivities and specificities; this approach could be valuable in studies such as this one. However, while bias analyses that assume the amount of misclassification is known with certainty can be an important step in a bias analysis, they can also pose substantial risks. Unless the extent of the misclassification is specified perfectly, results may be biased. Worse, the amount of bias does not depend entirely on how close the specified misclassification is to the true amount of misclassification; an extremely good guess at the amount of misclassification can still result in an extremely biased estimate under certain circumstances.5 For this reason, the authors wisely conducted a bias analysis (their scenario 4) in which they assumed the amount of misclassification was not known with certainty. This analysis not only adjusts the OR for exposure misclassification but also provides credible interval estimates that incorporate both uncertainties in the sensitivity values as well as the random error traditionally accounted for in confidence intervals. These credible (often alternatively called uncertainty) intervals provide a range of effect estimates and may prove useful in giving direction to future research as well. In a previous example of exposure misclassification in birth defects research, a very precise OR that did not incorporate uncertainty in the amount of misclassification became relatively imprecise once uncertainty in misclassification was incorporated.6 In this instance, the precision of the estimate indicated that more studies were needed to estimate the sensitivity and specificity of misclassification more precisely. In the study by van Gelder et al., however, the opposite may be true. There is relatively little reduction in the precision of the ORs once misclassification is incorporated. The relative imprecision of the estimates in the current study seems to be more a function of the sparseness of cases among the exposed than the uncertainty in the misclassification. Unfortunately, for van Gelder et al., there were no previous studies on the misclassification rate of cannabis use in the periconceptional period among mothers whose children had birth defects. The authors dealt with this limitation as best they could but it also implies that the credible intervals they produced may not reflect
the true amount of uncertainty in their estimates. Validation studies among this group of women, while costly, could be of use in birth defects research. Further, it would be helpful for future bias analyses if validation studies report their misclassification rates within strata (such as among mothers of children with birth defects) as well as overall. The ability of bias analyses to produce meaningful adjusted estimates relies on the ability of the investigator to correctly specify the structure of the bias. The differential misclassification models used by van Gelder et al. assumed mothers of cases would be better reporters (have a higher sensitivity) than mothers of controls. This assumption follows the reasoning of traditional recall bias, in which mothers of cases have made greater efforts to review their pregnancy in order to determine how the birth defect could have occurred and thus are more accurate reporters than mothers of controls. For an exposure with a negative connotation like marijuana, however, mothers of cases may be more likely to deny exposure than mothers of controls. Analysing the results from analyses that allowed higher sensitivity among controls than cases could be completed with relative ease and would be an important extension of the study. Another missed opportunity in this study was further exploration of cases and controls with missing information on other illicit drug exposures. This subgroup of women may be most vulnerable to cannabis under-reporting than those stating no such exposure. Incorporation of this subgroup into the overall analysis with lower sensitivity assumption could, again, be completed with relative ease and would be a valuable addition to the assessment of misclassification bias. Quantitative bias analysis is now well established in the epidemiologic literature with numerous articles and books describing the application of these methods;4–11 however, actual application of these methods remains limited. One reason for the scarcity of these methods is likely because of the perceived difficulty in implementing them. While van Gelder and colleagues adopted a common Bayesian method for bias analysis, this is not the only possible approach. Indeed, the Bayesian method may be somewhat more complicated because of the need to learn specialised software such as WinBUGs and specialised statistical techniques such as Markov chain Monte Carlo (MCMC) algorithms. Probabilistic bias analysis (PBA) offers, in some situations, a substantially easier way to implement bias analyses. PBA approaches can © 2014 John Wiley & Sons Ltd Paediatric and Perinatal Epidemiology, 2014, 28, 353–355
Commentary be excellent approximations to Bayesian approaches and while they do require iterative sampling, they do not require MCMC.11 As such, authors interested in implementing bias analyses may find these methods preferable. We do remain concerned that authors may avoid bias analysis methods out of fear that positive associations could vanish once biases and associated uncertainties in the magnitude of those biases are incorporated. Given the importance of the research (and vast amounts of public money being spent on this research), we strongly encourage authors to follow the lead of van Gelder and colleagues in implementing bias analyses to provide point estimates that are as meaningful as possible and interval estimates that incorporate more of the uncertainty contained in our research.
2
3
4
5
6
About the authors Richard MacLehose is an associate professor in epidemiology and community health at the University of Minnesota. His research interests include Bayesian statistics, epidemiologic methods, applied biostatistics, and reproductive and environmental health. Martha Werler is chair and professor of epidemiology at the Boston University School of Public Health. Her research focuses on risk factors for birth defects and outcomes among children affected with craniofacial malformations.
7
8 9
10
11
References 1 Alvarado S. Baby blog: Clearing the smoke about marijuana use during pregnancy. 2014. http://
© 2014 John Wiley & Sons Ltd Paediatric and Perinatal Epidemiology, 2014, 28, 353–355
355
www.mothertobabyca.org/high-times-for-marijuana -clearing-the-smoke-about-marijuana-use-during -pregnancy/ [last accessed August 5, 2014.]. Van Gelder MM, Reefhuis J, Caton AR, Werler MM, Druschel CM, Roeleveld N. Maternal periconceptional illicit drug use and the risk of congenital malformations. Epidemiology (Cambridge, Mass.) 2010; 20:60–66. Van Gelder MMHJ, Donders RT, Devine O, Roeleveld N, Reefhuis J. Using Bayesian models to assess the effects of underreporting of cannabis use on the association with birth defects, National Birth Defects Prevention Study, 1997–2005. Paediatric and Perinatal Epidemiology 2014; 28:424–433. Chu H, Wang Z, Cole SR, Greenland S. Sensitivity analysis of misclassification: a graphical and a Bayesian approach. Annals of Epidemiology 2006; 16:834–841. Gustafson P, Le ND, Saskin R. Case–control analysis with partial knowledge of exposure misclassification probabilities. Biometrics 2001; 57:598–609. MacLehose RF, Olshan AF, Herring AH, Honein MA, Shaw GM, Romitti PA. Bayesian methods for correcting misclassification: an example from birth defects epidemiology. Epidemiology (Cambridge, Mass.) 2009; 20:27–35. Lash TL, Fink AK, Fox MP. Applying Quantitative Bias Analysis to Epidemiologic Data, 1st edn. New York: Springer, 2009. Rothman KJ, Greenland S, Lash TL. Modern Epidemiology, 3rd edn. Philadelphia: Lippincott Williams & Wilkins, 2008. Greenland S. Basic methods for sensitivity analysis of biases. International Journal of Epidemiology 1996; 25:1107–1116. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless L, Greenland S. Good practices for quantitative bias analysis. International Journal of Epidemiology 2014; doi: 10.1093/ije/dyu149. MacLehose RF, Gustafson P. Is probabilistic bias analysis approximately Bayesian? Epidemiology (Cambridge, Mass.) 2009; 23:151–158.
doi: 10.1111/ppe.12147
353
Commentary
Importance of Bias Analysis in Epidemiologic Research Richard F. MacLehose,a Martha M. Werlerb a
Division of Epidemiology and Community Health, University of Minnesota, School of Public Health, Minneapolis, MN b
Department of Epidemiology, Boston University School of Public Health, Boston, MA
Despite public health warnings against marijuana use during pregnancy,1 exposures do occur among women who don’t heed such warnings or whose pregnancies aren’t recognised until well into gestation.2 Clear evidence on the risks and safety of marijuana use in pregnancy is lacking, due in part to limitations in observational studies. van Gelder and colleagues deserve praise for addressing one such limitation in their study on maternal cannabis use in relation to birth defects in offspring, published in this issue of Paediatric and Perinatal Epidemiology.3 Exposure misclassification abounds in perinatal epidemiology, and marijuana use is no exception. The authors employed Bayesian bias analyses to adjust for the possible influence of exposure misclassification, specifically maternal under-reporting of cannabis use, on observed measures of association. Studies of risk factors for birth defects are especially amenable to sensitivity analyses that address misclassification of exposures for a few reasons. First, structural birth defects are rare, making the case– control design the most feasible from the perspective of operational efficiency and, thus, the most popular to implement. The retrospective measurement of exposure status (often employed in case–control studies) is vulnerable to misclassification, especially when measurement relies on maternal recall of events that may have occurred months or years in the past. Inaccurate reporting of exposure status can also be influenced by societal factors. It is widely presumed that illicit drugs can cause birth defects. Public health warnings, whether grounded in strong empirical evidence or arising from scant data, inform public impressions. Reports in mainstream media may have an even greater impact on viewpoints, and often strive to catch audiences’ attention rather than provide balanced information. Regardless of the influences on Correspondence: Richard F. MacLehose, Division Epidemiology and Community Health, University of Minnesota, School of Public Health, 1300 South 2nd Street, Suite 300, Minneapolis, MN 55454, USA. E-mail: [email protected]
© 2014 John Wiley & Sons Ltd Paediatric and Perinatal Epidemiology, 2014, 28, 353–355
any one person’s viewpoints of an exposure, study subjects may be more vulnerable to falsely denying exposure when that exposure is thought to be unhealthy. A final reason why assessment of the impact of exposure misclassification is necessary in studies of birth defects stems from the general presumption among clinicians and researchers alike that women bias their reports of exposures, depending on whether their offspring was affected with a birth defect or not. Reports of positive associations between any maternally reported exposure and a birth defect are often followed by criticism that they could be due to recall bias. The van Gelder et al. study collected data from mothers up to 2 years following delivery when memories may have faded; cannabis use during pregnancy carries a negative connotation; and previously reported case–control studies of cannabis use and birth defects grappled with the issue of recall bias.3 Thus, adjustment for exposure misclassification is warranted given their research question. The authors approached their bias analyses using two general sets of models. In the first set of models, the authors assumed that they knew the amount of exposure misclassification with certainty and specified sensitivity values that corresponded to three scenarios: no misclassification, non-differential misclassification, or differential misclassification. In these scenarios, the authors specified various magnitudes of misclassification and assumed there was no uncertainty in these values. In this case, the interval estimates obtained from the analysis may be of less interest (since they do not incorporate any uncertainty in the sensitivity values) than the adjusted odds ratio (OR). However, since so little is known about the actual amount of misclassification, it can be very useful to examine what the adjusted OR would be for a range of sensitivity values. This can be an important descriptive first step leading up to a bias analysis that incorporates uncertainty in the misclassification. Indeed, the authors find that for the values of sensitivity and specificity that they chose to examine, the
354
R. F. MacLehose and M. M. Werler
misclassification-adjusted ORs were extremely similar to one another and often quite similar to the estimates obtained from assuming no misclassification. Chu et al.4 have extended this idea to graphically present misclassification-adjusted ORs over a wide range of possible sensitivities and specificities; this approach could be valuable in studies such as this one. However, while bias analyses that assume the amount of misclassification is known with certainty can be an important step in a bias analysis, they can also pose substantial risks. Unless the extent of the misclassification is specified perfectly, results may be biased. Worse, the amount of bias does not depend entirely on how close the specified misclassification is to the true amount of misclassification; an extremely good guess at the amount of misclassification can still result in an extremely biased estimate under certain circumstances.5 For this reason, the authors wisely conducted a bias analysis (their scenario 4) in which they assumed the amount of misclassification was not known with certainty. This analysis not only adjusts the OR for exposure misclassification but also provides credible interval estimates that incorporate both uncertainties in the sensitivity values as well as the random error traditionally accounted for in confidence intervals. These credible (often alternatively called uncertainty) intervals provide a range of effect estimates and may prove useful in giving direction to future research as well. In a previous example of exposure misclassification in birth defects research, a very precise OR that did not incorporate uncertainty in the amount of misclassification became relatively imprecise once uncertainty in misclassification was incorporated.6 In this instance, the precision of the estimate indicated that more studies were needed to estimate the sensitivity and specificity of misclassification more precisely. In the study by van Gelder et al., however, the opposite may be true. There is relatively little reduction in the precision of the ORs once misclassification is incorporated. The relative imprecision of the estimates in the current study seems to be more a function of the sparseness of cases among the exposed than the uncertainty in the misclassification. Unfortunately, for van Gelder et al., there were no previous studies on the misclassification rate of cannabis use in the periconceptional period among mothers whose children had birth defects. The authors dealt with this limitation as best they could but it also implies that the credible intervals they produced may not reflect
the true amount of uncertainty in their estimates. Validation studies among this group of women, while costly, could be of use in birth defects research. Further, it would be helpful for future bias analyses if validation studies report their misclassification rates within strata (such as among mothers of children with birth defects) as well as overall. The ability of bias analyses to produce meaningful adjusted estimates relies on the ability of the investigator to correctly specify the structure of the bias. The differential misclassification models used by van Gelder et al. assumed mothers of cases would be better reporters (have a higher sensitivity) than mothers of controls. This assumption follows the reasoning of traditional recall bias, in which mothers of cases have made greater efforts to review their pregnancy in order to determine how the birth defect could have occurred and thus are more accurate reporters than mothers of controls. For an exposure with a negative connotation like marijuana, however, mothers of cases may be more likely to deny exposure than mothers of controls. Analysing the results from analyses that allowed higher sensitivity among controls than cases could be completed with relative ease and would be an important extension of the study. Another missed opportunity in this study was further exploration of cases and controls with missing information on other illicit drug exposures. This subgroup of women may be most vulnerable to cannabis under-reporting than those stating no such exposure. Incorporation of this subgroup into the overall analysis with lower sensitivity assumption could, again, be completed with relative ease and would be a valuable addition to the assessment of misclassification bias. Quantitative bias analysis is now well established in the epidemiologic literature with numerous articles and books describing the application of these methods;4–11 however, actual application of these methods remains limited. One reason for the scarcity of these methods is likely because of the perceived difficulty in implementing them. While van Gelder and colleagues adopted a common Bayesian method for bias analysis, this is not the only possible approach. Indeed, the Bayesian method may be somewhat more complicated because of the need to learn specialised software such as WinBUGs and specialised statistical techniques such as Markov chain Monte Carlo (MCMC) algorithms. Probabilistic bias analysis (PBA) offers, in some situations, a substantially easier way to implement bias analyses. PBA approaches can © 2014 John Wiley & Sons Ltd Paediatric and Perinatal Epidemiology, 2014, 28, 353–355
Commentary be excellent approximations to Bayesian approaches and while they do require iterative sampling, they do not require MCMC.11 As such, authors interested in implementing bias analyses may find these methods preferable. We do remain concerned that authors may avoid bias analysis methods out of fear that positive associations could vanish once biases and associated uncertainties in the magnitude of those biases are incorporated. Given the importance of the research (and vast amounts of public money being spent on this research), we strongly encourage authors to follow the lead of van Gelder and colleagues in implementing bias analyses to provide point estimates that are as meaningful as possible and interval estimates that incorporate more of the uncertainty contained in our research.
2
3
4
5
6
About the authors Richard MacLehose is an associate professor in epidemiology and community health at the University of Minnesota. His research interests include Bayesian statistics, epidemiologic methods, applied biostatistics, and reproductive and environmental health. Martha Werler is chair and professor of epidemiology at the Boston University School of Public Health. Her research focuses on risk factors for birth defects and outcomes among children affected with craniofacial malformations.
7
8 9
10
11
References 1 Alvarado S. Baby blog: Clearing the smoke about marijuana use during pregnancy. 2014. http://
© 2014 John Wiley & Sons Ltd Paediatric and Perinatal Epidemiology, 2014, 28, 353–355
355
www.mothertobabyca.org/high-times-for-marijuana -clearing-the-smoke-about-marijuana-use-during -pregnancy/ [last accessed August 5, 2014.]. Van Gelder MM, Reefhuis J, Caton AR, Werler MM, Druschel CM, Roeleveld N. Maternal periconceptional illicit drug use and the risk of congenital malformations. Epidemiology (Cambridge, Mass.) 2010; 20:60–66. Van Gelder MMHJ, Donders RT, Devine O, Roeleveld N, Reefhuis J. Using Bayesian models to assess the effects of underreporting of cannabis use on the association with birth defects, National Birth Defects Prevention Study, 1997–2005. Paediatric and Perinatal Epidemiology 2014; 28:424–433. Chu H, Wang Z, Cole SR, Greenland S. Sensitivity analysis of misclassification: a graphical and a Bayesian approach. Annals of Epidemiology 2006; 16:834–841. Gustafson P, Le ND, Saskin R. Case–control analysis with partial knowledge of exposure misclassification probabilities. Biometrics 2001; 57:598–609. MacLehose RF, Olshan AF, Herring AH, Honein MA, Shaw GM, Romitti PA. Bayesian methods for correcting misclassification: an example from birth defects epidemiology. Epidemiology (Cambridge, Mass.) 2009; 20:27–35. Lash TL, Fink AK, Fox MP. Applying Quantitative Bias Analysis to Epidemiologic Data, 1st edn. New York: Springer, 2009. Rothman KJ, Greenland S, Lash TL. Modern Epidemiology, 3rd edn. Philadelphia: Lippincott Williams & Wilkins, 2008. Greenland S. Basic methods for sensitivity analysis of biases. International Journal of Epidemiology 1996; 25:1107–1116. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless L, Greenland S. Good practices for quantitative bias analysis. International Journal of Epidemiology 2014; doi: 10.1093/ije/dyu149. MacLehose RF, Gustafson P. Is probabilistic bias analysis approximately Bayesian? Epidemiology (Cambridge, Mass.) 2009; 23:151–158.
