American Journal of Epidemiology Advance Access published December 5, 2014 American Journal of Epidemiology © The Author 2014. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. All rights reserved. For permissions, please e-mail: [email protected].
DOI: 10.1093/aje/kwu273
Response to Invited Commentary Marshall and Galea Respond to “Data Theory in Epidemiology”
* Correspondence to Dr. Brandon D. L. Marshall, Department of Epidemiology, Brown University School of Public Health, 121 South Main Street, Box G-S-121-2, Providence, RI 02912 (e-mail: [email protected]).
Initially submitted September 6, 2014; accepted for publication September 9, 2014.
Abbreviation: ABM, agent-based model.
We concur with Diez Roux that systems modeling will not replace experimental or observational research—in fact, “best practice” ABMs are wholly reliant on it. We have learned from our past work (6, 7) that the utility of agent-based modeling is significantly improved when relevant data are readily accessible. Ideally, modeling and primary data collection should explicitly inform each other and proceed in tandem. Successful examples of iterative, multimethod approaches to epidemiologic research are currently limited and should be supported. Although elucidation of the dynamics and evolution of complex health systems are worthy avenues of investigation in their own right, epidemiology has identified, and should continue to identify, tangible disease causes that are both modifiable and amenable to public health intervention. As Hernán and Diez Roux discuss, agent-based modeling allows us to move away from questions of causal isolation with downstream foci to a more systems-based and life course– oriented epidemiology. We add to the challenges noted by each author that there is a clear risk of building ABMs so complex that meaningful causal inference (and subsequent public health action) is obfuscated. ABMs informed by all relevant theory and all available data tend to be cumbersome and of limited scientific utility (8). The process of simplification—noted by Diez Roux to be a prerequisite for empirical science—is therefore equally essential in complex systems science. In observational studies, we select confounders based on prior knowledge (informed by data) and conceptual models (e.g., directed acyclic graphs). So too should the construction, calibration, and analysis of ABMs be informed by prior knowledge, valid data, and grounded theory. The charge for epidemiologists, then, is to identify public health questions that are sufficiently complex to warrant a systems science approach and about which relevant data and adequate understanding of the system’s subcomponents exist to construct and calibrate valid models.
Over the past decade, the use of agent-based models (ABMs) and the development of causal inference methods have proceeded rather independently in epidemiology. In our article (1), we aimed to provide an initial foundation to bridge the two. We argued for a more rigorous causal inference framework in epidemiologic applications of ABMs and sought to generate discussion about how best to accomplish this. We therefore wish to thank Diez Roux (2) and Hernán (3) for extending this discussion and note here broad points of unanimity and motivating next steps. Centrally, we support Hernán’s call for research and consensus on acceptable methodological standards for agentbased modeling in epidemiology. Our paper was a first shot across the conceptual bow. However, there is much work that remains to be done in identifying specific agent-based modeling “best practices” that are relevant to epidemiology. These include, but are not limited to, methods for ABM construction and calibration (including the incorporation of data from different populations and various sources); model validation procedures (including reproducibility of observed real-world phenomena); and acceptable analytic practices (including sufficient sensitivity analyses to evaluate model robustness). Such standards have been developed in other disciplines (4, 5) but remain in their infancy throughout epidemiology. It is our hope that the debate from which our work was motivated continues to inspire the development and adoption of rigorous standards for agent-based modeling in our field. We also agree with Hernán that agent-based modeling requires a shift away from causal inference pursuits that are exclusively data driven. However, we do not advocate for the application of ABMs that are entirely suppositional and void of empirical foundation. In addition to theory and prior knowledge, the construction and calibration of valid simulation models rest on the availability of relevant and unbiased data. 1
Downloaded from http://aje.oxfordjournals.org/ at University of Iowa Libraries/Serials Acquisitions on December 11, 2014
Brandon D. L. Marshall* and Sandro Galea
2 Marshall and Galea
ACKNOWLEDGMENTS
REFERENCES 1. Marshall BDL, Galea S. Formalizing the role of agent-based modeling in causal inference and epidemiology. Am J Epidemiol. 2014;000(00):0000–0000.
Downloaded from http://aje.oxfordjournals.org/ at University of Iowa Libraries/Serials Acquisitions on December 11, 2014
Author affiliations: Department of Epidemiology, School of Public Health, Brown University, Providence, Rhode Island (Brandon D. L. Marshall); and Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, New York (Sandro Galea). This work was supported by a developmental grant from the Lifespan/Tufts/Brown Center for AIDS Research (grant P30-AI042853). B.D.L.M was also funded by a Richard B. Salomon Faculty Research Award from Brown University. Conflict of interest: none declared.
2. Diez Roux AV. Invited commentary: the virtual epidemiologist— promise and peril. Am J Epidemiol. 2014;000(00):0000–0000. 3. Hernán MA. Invited commentary: agent-based models for causal inference—reweighting data and theory in epidemiology. Am J Epidemiol. 2014;000(00):0000–0000. 4. Heath B, Hill R, Ciarallo F. A survey of agent-based modeling practices (January 1998 to July 2008). JASSS - J Artif Soc S. 2009;12(4):A143–A177. 5. Richiardi M, Leombruni R, Saam N, et al. A common protocol for agent-based social simulation. JASSS - J Artif Soc S. 2006;9(1). 6. Marshall BDL, Friedman SR, Monteiro JF, et al. Prevention and treatment produced large decreases in HIV incidence in a model of people who inject drugs. Health Aff (Millwood). 2014;33(3): 401–409. 7. Marshall BDL, Paczkowski MM, Seemann L, et al. A complex systems approach to evaluate HIV prevention in metropolitan areas: preliminary implications for combination intervention strategies. PLoS One. 2012;7(9):e44833. 8. Grimm V, Revilla E, Berger U, et al. Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science. 2005;310(5750):987–991.
DOI: 10.1093/aje/kwu273
Response to Invited Commentary Marshall and Galea Respond to “Data Theory in Epidemiology”
* Correspondence to Dr. Brandon D. L. Marshall, Department of Epidemiology, Brown University School of Public Health, 121 South Main Street, Box G-S-121-2, Providence, RI 02912 (e-mail: [email protected]).
Initially submitted September 6, 2014; accepted for publication September 9, 2014.
Abbreviation: ABM, agent-based model.
We concur with Diez Roux that systems modeling will not replace experimental or observational research—in fact, “best practice” ABMs are wholly reliant on it. We have learned from our past work (6, 7) that the utility of agent-based modeling is significantly improved when relevant data are readily accessible. Ideally, modeling and primary data collection should explicitly inform each other and proceed in tandem. Successful examples of iterative, multimethod approaches to epidemiologic research are currently limited and should be supported. Although elucidation of the dynamics and evolution of complex health systems are worthy avenues of investigation in their own right, epidemiology has identified, and should continue to identify, tangible disease causes that are both modifiable and amenable to public health intervention. As Hernán and Diez Roux discuss, agent-based modeling allows us to move away from questions of causal isolation with downstream foci to a more systems-based and life course– oriented epidemiology. We add to the challenges noted by each author that there is a clear risk of building ABMs so complex that meaningful causal inference (and subsequent public health action) is obfuscated. ABMs informed by all relevant theory and all available data tend to be cumbersome and of limited scientific utility (8). The process of simplification—noted by Diez Roux to be a prerequisite for empirical science—is therefore equally essential in complex systems science. In observational studies, we select confounders based on prior knowledge (informed by data) and conceptual models (e.g., directed acyclic graphs). So too should the construction, calibration, and analysis of ABMs be informed by prior knowledge, valid data, and grounded theory. The charge for epidemiologists, then, is to identify public health questions that are sufficiently complex to warrant a systems science approach and about which relevant data and adequate understanding of the system’s subcomponents exist to construct and calibrate valid models.
Over the past decade, the use of agent-based models (ABMs) and the development of causal inference methods have proceeded rather independently in epidemiology. In our article (1), we aimed to provide an initial foundation to bridge the two. We argued for a more rigorous causal inference framework in epidemiologic applications of ABMs and sought to generate discussion about how best to accomplish this. We therefore wish to thank Diez Roux (2) and Hernán (3) for extending this discussion and note here broad points of unanimity and motivating next steps. Centrally, we support Hernán’s call for research and consensus on acceptable methodological standards for agentbased modeling in epidemiology. Our paper was a first shot across the conceptual bow. However, there is much work that remains to be done in identifying specific agent-based modeling “best practices” that are relevant to epidemiology. These include, but are not limited to, methods for ABM construction and calibration (including the incorporation of data from different populations and various sources); model validation procedures (including reproducibility of observed real-world phenomena); and acceptable analytic practices (including sufficient sensitivity analyses to evaluate model robustness). Such standards have been developed in other disciplines (4, 5) but remain in their infancy throughout epidemiology. It is our hope that the debate from which our work was motivated continues to inspire the development and adoption of rigorous standards for agent-based modeling in our field. We also agree with Hernán that agent-based modeling requires a shift away from causal inference pursuits that are exclusively data driven. However, we do not advocate for the application of ABMs that are entirely suppositional and void of empirical foundation. In addition to theory and prior knowledge, the construction and calibration of valid simulation models rest on the availability of relevant and unbiased data. 1
Downloaded from http://aje.oxfordjournals.org/ at University of Iowa Libraries/Serials Acquisitions on December 11, 2014
Brandon D. L. Marshall* and Sandro Galea
2 Marshall and Galea
ACKNOWLEDGMENTS
REFERENCES 1. Marshall BDL, Galea S. Formalizing the role of agent-based modeling in causal inference and epidemiology. Am J Epidemiol. 2014;000(00):0000–0000.
Downloaded from http://aje.oxfordjournals.org/ at University of Iowa Libraries/Serials Acquisitions on December 11, 2014
Author affiliations: Department of Epidemiology, School of Public Health, Brown University, Providence, Rhode Island (Brandon D. L. Marshall); and Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, New York (Sandro Galea). This work was supported by a developmental grant from the Lifespan/Tufts/Brown Center for AIDS Research (grant P30-AI042853). B.D.L.M was also funded by a Richard B. Salomon Faculty Research Award from Brown University. Conflict of interest: none declared.
2. Diez Roux AV. Invited commentary: the virtual epidemiologist— promise and peril. Am J Epidemiol. 2014;000(00):0000–0000. 3. Hernán MA. Invited commentary: agent-based models for causal inference—reweighting data and theory in epidemiology. Am J Epidemiol. 2014;000(00):0000–0000. 4. Heath B, Hill R, Ciarallo F. A survey of agent-based modeling practices (January 1998 to July 2008). JASSS - J Artif Soc S. 2009;12(4):A143–A177. 5. Richiardi M, Leombruni R, Saam N, et al. A common protocol for agent-based social simulation. JASSS - J Artif Soc S. 2006;9(1). 6. Marshall BDL, Friedman SR, Monteiro JF, et al. Prevention and treatment produced large decreases in HIV incidence in a model of people who inject drugs. Health Aff (Millwood). 2014;33(3): 401–409. 7. Marshall BDL, Paczkowski MM, Seemann L, et al. A complex systems approach to evaluate HIV prevention in metropolitan areas: preliminary implications for combination intervention strategies. PLoS One. 2012;7(9):e44833. 8. Grimm V, Revilla E, Berger U, et al. Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science. 2005;310(5750):987–991.
