Opinion
Editorials represent the opinions of the authors and JAMA and not those of the American Medical Association.
EDITORIAL
Comparative Effectiveness Research and Outcomes of Diabetes Treatment Monika M. Safford, MD
Randomized clinical trials (RCTs) are the gold standard for advancing the science of medicine. However, many important clinical questions probably will never be answered by RCTs simply because many trials are extraordinarily expensive and RCTs might not always be apRelated article page 2288 propriate for addressing some research questions. In fact, most clinical trials do not enroll typical patients; trial participants are volunteers who agree to be studied with limited compensation and often do so primarily to benefit other patients. Clinical trials are designed to answer questions about whether something works (efficacy), but usually are poorly suited to answer questions about how well something works on usual patients seen in clinical practice (effectiveness). Numerous studies have shown that the effect sizes reported in clinical trials are rarely achieved in practice, raising concerns that more generalizable results are needed to better inform real-world clinical decisions.1-5 Well-designed observational studies using data from large populations can address some of the questions clinical trials cannot. The report in this issue of JAMA by Roumie and colleagues6 is an example of a state-of-the-art observational comparative effectiveness (CE) study. Using national data from the Veterans Health Administration, Medicare, and National Death Index, the authors compared the risk of a composite outcome of acute myocardial infarction, stroke, or death in veterans with type 2 diabetes in whom metformin did not control their condition and who added either a sulfonylurea or insulin to their treatment regimen. The analytic methods included not only the now-familiar propensity score approaches and new user designs, but also the use of marginal structural Cox proportional hazards models, inverse probability weights, and estimates of the magnitude of imbalance in an unmeasured confounder that would be required to alter the results. After a 5:1 propensity score match, 12 180 users of metformin + sulfonylurea and 2436 users of metformin + insulin were included in analyses. The primary finding was that an increased risk of the composite outcome was observed in patients who added insulin as second-line therapy, with an adjusted hazard ratio of 1.30 (95% CI, 1.07-1.58). The authors provide a careful and comprehensive analysis of the study question. The signal for potential harm is certainly concerning, suggesting that confirmatory studies addressing this issue should be conducted relatively quickly using other databases (such as Medicare, Kaiser, or Group Health Cooperative of Puget Sound). However, the findings reported by Roumie et al could cause uncertainty for many clinicians and patients. First and fore-
most, although insulin resistance has been linked to cancer death7 (a major driver of the results in the study by Roumie et al), it is not clear whether exogenous insulin or unmeasured confounders such as the degree of insulin resistance could be responsible for the findings. The relative risks associated with insulin resistance for some cancers such as colon cancer are reportedly higher than the relative hazard of 1.25 cited in the unmeasured confounder imbalance analysis presented by Roumie et al. Even for the 1.25 hazard, the possibility of imbalance is real; if the prevalence of advanced insulin resistance is 20% in the sulfonylurea group, it need only be 28% (40% imbalance) in the insulin group to explain the findings. Unmeasured confounding, particularly confounding by indication, is an important threat to the validity of observational studies, even those with analytic approaches as refined as that of Roumie et al. Additional potential unmeasured confounders that could influence these results include the aggressiveness of disease progression, which varies substantially from patient to patient; the duration of diabetes; and the long-term history of other medications patients have tried. The concern about unmeasured confounding is especially noteworthy because one of the clinical trials cited by the authors had no excess cancer mortality or cardiovascular mortality among insulin users,8 and the follow-up of that trial was longer than the follow-up in the report by Roumie et al. Given these caveats, many clinicians will probably refrain from making practice changes based on this study. Second, while the designs and interpretation of RCTs are familiar to clinicians, the complex methods used to conduct CE studies are unfamiliar to many clinicians who need to interpret the findings of these studies and determine the relevance for clinical practice. Comparative effectiveness research is raising an interesting issue: the methods are narrow but the implications are potentially broad. Nearly every branch of science has specialized methods that are understood in depth by relatively few people, and these same people are often also the audience for the results. This is not the case for CE research. Many readers will not have a detailed understanding of methods such as inverse probability weighting, used in the analysis by Roumie et al, or other sophisticated CE research methods such as high-dimensional propensity scores9 or instrumental variables.10 On the other hand, these results potentially reach deeply into clinical practice. Within a short time of entering practice, every primary care physician confronts the conundrum of which therapeutic agent to prescribe next when metformin alone is no longer achieving acceptable glycemic control in patients with type 2 diabetes. Like
jama.com
JAMA June 11, 2014 Volume 311, Number 22
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a New York University User on 05/21/2015
2275
Opinion Editorial
much CE research, the results reported by Roumie et al provide information that helps inform a common, currently unanswered, and critically important clinical question that deserves a reliable answer. The clinicians who need this answer may be left wondering how trustworthy the results are because they are unable to judge the methods. Because it is unlikely that most physicians in practice will have a working knowledge of the sophisticated methods used in CE research, such as those used in study by Roumie et al, many readers may increasingly rely on peer review and therefore the opinion of just a few experts rather than their own informed opinion when interpreting CE research results. The role of journals and peer reviewers in their evaluation of studies that use complex statistical methods is likely to be increasingly important not only in ensuring validity, but also in preARTICLE INFORMATION Author Affiliation: Department of Medicine, University of Alabama at Birmingham. Corresponding Author: Monika M. Safford, MD, Department of Medicine, University of Alabama at Birmingham, 1717 11th Ave S, MT643, Birmingham, AL 35294-4410 ([email protected]). Conflict of Interest Disclosures: The author has completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. REFERENCES 1. Henggeler SW. Decreasing effect sizes for effectiveness studies- implications for the transport of evidence-based treatments: comment on curtis, ronan, and borduin (2004). J Fam Psychol. 2004;18 (3):420-423.
2276
senting findings clearly and with appropriate caveats. Doing so could help prevent readers from adopting the results of CE research and immediately changing clinical practice, which could be worriedly premature for findings from observational studies. Comparative effectiveness research is creating new challenges as it generates much needed new evidence. The very methods that make studies like that of Roumie et al novel also create barriers to interpretation that may make it more difficult to apply their results. Some of the creativity being brought to bear on advancing methods of analysis may also be needed to advance methods of communicating both methods and results to practicing clinicians, and perhaps more importantly, to the patients who are facing decisions that may (or may not) have profound implications for their health and well-being.
2. Nallamothu BK, Hayward RA, Bates ER. Beyond the randomized clinical trial: the role of effectiveness studies in evaluating cardiovascular therapies. Circulation. 2008;118(12):1294-1303.
among patients with diabetes. JAMA. doi:10.1001 /jama.2014.4312.
3. Dans AL, Dans LF, Guyatt GH, Richardson S; Evidence-Based Medicine Working Group. Users’ guides to the medical literature, XIV: how to decide on the applicability of clinical trial results to your patient. JAMA. 1998;279(7):545-549.
8. Gerstein HC, Bosch J, Dagenais GR, et al; ORIGIN Trial Investigators. Basal insulin and cardiovascular and other outcomes in dysglycemia. N Engl J Med. 2012;367(4):319-328.
4. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82-93. 5. Flather M, Delahunty N, Collinson J. Generalizing results of randomized trials to clinical practice: reliability and cautions. Clin Trials. 2006;3(6):508512. 6. Roumie CL, Greevy RA, Grijalva CG, et al. Association between intensification of metformin treatment with insulin vs sulfonylureas and cardiovascular events and all-cause mortality
7. Boyd DB. Insulin and cancer. Integr Cancer Ther. 2003;2(4):315-329. doi:10.1177/1534735403259152.
9. Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology. 2009;20(4):512-522. 10. Stukel TA, Fisher ES, Wennberg DE, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007;297(3):278-285.
JAMA June 11, 2014 Volume 311, Number 22
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a New York University User on 05/21/2015
jama.com
Editorials represent the opinions of the authors and JAMA and not those of the American Medical Association.
EDITORIAL
Comparative Effectiveness Research and Outcomes of Diabetes Treatment Monika M. Safford, MD
Randomized clinical trials (RCTs) are the gold standard for advancing the science of medicine. However, many important clinical questions probably will never be answered by RCTs simply because many trials are extraordinarily expensive and RCTs might not always be apRelated article page 2288 propriate for addressing some research questions. In fact, most clinical trials do not enroll typical patients; trial participants are volunteers who agree to be studied with limited compensation and often do so primarily to benefit other patients. Clinical trials are designed to answer questions about whether something works (efficacy), but usually are poorly suited to answer questions about how well something works on usual patients seen in clinical practice (effectiveness). Numerous studies have shown that the effect sizes reported in clinical trials are rarely achieved in practice, raising concerns that more generalizable results are needed to better inform real-world clinical decisions.1-5 Well-designed observational studies using data from large populations can address some of the questions clinical trials cannot. The report in this issue of JAMA by Roumie and colleagues6 is an example of a state-of-the-art observational comparative effectiveness (CE) study. Using national data from the Veterans Health Administration, Medicare, and National Death Index, the authors compared the risk of a composite outcome of acute myocardial infarction, stroke, or death in veterans with type 2 diabetes in whom metformin did not control their condition and who added either a sulfonylurea or insulin to their treatment regimen. The analytic methods included not only the now-familiar propensity score approaches and new user designs, but also the use of marginal structural Cox proportional hazards models, inverse probability weights, and estimates of the magnitude of imbalance in an unmeasured confounder that would be required to alter the results. After a 5:1 propensity score match, 12 180 users of metformin + sulfonylurea and 2436 users of metformin + insulin were included in analyses. The primary finding was that an increased risk of the composite outcome was observed in patients who added insulin as second-line therapy, with an adjusted hazard ratio of 1.30 (95% CI, 1.07-1.58). The authors provide a careful and comprehensive analysis of the study question. The signal for potential harm is certainly concerning, suggesting that confirmatory studies addressing this issue should be conducted relatively quickly using other databases (such as Medicare, Kaiser, or Group Health Cooperative of Puget Sound). However, the findings reported by Roumie et al could cause uncertainty for many clinicians and patients. First and fore-
most, although insulin resistance has been linked to cancer death7 (a major driver of the results in the study by Roumie et al), it is not clear whether exogenous insulin or unmeasured confounders such as the degree of insulin resistance could be responsible for the findings. The relative risks associated with insulin resistance for some cancers such as colon cancer are reportedly higher than the relative hazard of 1.25 cited in the unmeasured confounder imbalance analysis presented by Roumie et al. Even for the 1.25 hazard, the possibility of imbalance is real; if the prevalence of advanced insulin resistance is 20% in the sulfonylurea group, it need only be 28% (40% imbalance) in the insulin group to explain the findings. Unmeasured confounding, particularly confounding by indication, is an important threat to the validity of observational studies, even those with analytic approaches as refined as that of Roumie et al. Additional potential unmeasured confounders that could influence these results include the aggressiveness of disease progression, which varies substantially from patient to patient; the duration of diabetes; and the long-term history of other medications patients have tried. The concern about unmeasured confounding is especially noteworthy because one of the clinical trials cited by the authors had no excess cancer mortality or cardiovascular mortality among insulin users,8 and the follow-up of that trial was longer than the follow-up in the report by Roumie et al. Given these caveats, many clinicians will probably refrain from making practice changes based on this study. Second, while the designs and interpretation of RCTs are familiar to clinicians, the complex methods used to conduct CE studies are unfamiliar to many clinicians who need to interpret the findings of these studies and determine the relevance for clinical practice. Comparative effectiveness research is raising an interesting issue: the methods are narrow but the implications are potentially broad. Nearly every branch of science has specialized methods that are understood in depth by relatively few people, and these same people are often also the audience for the results. This is not the case for CE research. Many readers will not have a detailed understanding of methods such as inverse probability weighting, used in the analysis by Roumie et al, or other sophisticated CE research methods such as high-dimensional propensity scores9 or instrumental variables.10 On the other hand, these results potentially reach deeply into clinical practice. Within a short time of entering practice, every primary care physician confronts the conundrum of which therapeutic agent to prescribe next when metformin alone is no longer achieving acceptable glycemic control in patients with type 2 diabetes. Like
jama.com
JAMA June 11, 2014 Volume 311, Number 22
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a New York University User on 05/21/2015
2275
Opinion Editorial
much CE research, the results reported by Roumie et al provide information that helps inform a common, currently unanswered, and critically important clinical question that deserves a reliable answer. The clinicians who need this answer may be left wondering how trustworthy the results are because they are unable to judge the methods. Because it is unlikely that most physicians in practice will have a working knowledge of the sophisticated methods used in CE research, such as those used in study by Roumie et al, many readers may increasingly rely on peer review and therefore the opinion of just a few experts rather than their own informed opinion when interpreting CE research results. The role of journals and peer reviewers in their evaluation of studies that use complex statistical methods is likely to be increasingly important not only in ensuring validity, but also in preARTICLE INFORMATION Author Affiliation: Department of Medicine, University of Alabama at Birmingham. Corresponding Author: Monika M. Safford, MD, Department of Medicine, University of Alabama at Birmingham, 1717 11th Ave S, MT643, Birmingham, AL 35294-4410 ([email protected]). Conflict of Interest Disclosures: The author has completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. REFERENCES 1. Henggeler SW. Decreasing effect sizes for effectiveness studies- implications for the transport of evidence-based treatments: comment on curtis, ronan, and borduin (2004). J Fam Psychol. 2004;18 (3):420-423.
2276
senting findings clearly and with appropriate caveats. Doing so could help prevent readers from adopting the results of CE research and immediately changing clinical practice, which could be worriedly premature for findings from observational studies. Comparative effectiveness research is creating new challenges as it generates much needed new evidence. The very methods that make studies like that of Roumie et al novel also create barriers to interpretation that may make it more difficult to apply their results. Some of the creativity being brought to bear on advancing methods of analysis may also be needed to advance methods of communicating both methods and results to practicing clinicians, and perhaps more importantly, to the patients who are facing decisions that may (or may not) have profound implications for their health and well-being.
2. Nallamothu BK, Hayward RA, Bates ER. Beyond the randomized clinical trial: the role of effectiveness studies in evaluating cardiovascular therapies. Circulation. 2008;118(12):1294-1303.
among patients with diabetes. JAMA. doi:10.1001 /jama.2014.4312.
3. Dans AL, Dans LF, Guyatt GH, Richardson S; Evidence-Based Medicine Working Group. Users’ guides to the medical literature, XIV: how to decide on the applicability of clinical trial results to your patient. JAMA. 1998;279(7):545-549.
8. Gerstein HC, Bosch J, Dagenais GR, et al; ORIGIN Trial Investigators. Basal insulin and cardiovascular and other outcomes in dysglycemia. N Engl J Med. 2012;367(4):319-328.
4. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82-93. 5. Flather M, Delahunty N, Collinson J. Generalizing results of randomized trials to clinical practice: reliability and cautions. Clin Trials. 2006;3(6):508512. 6. Roumie CL, Greevy RA, Grijalva CG, et al. Association between intensification of metformin treatment with insulin vs sulfonylureas and cardiovascular events and all-cause mortality
7. Boyd DB. Insulin and cancer. Integr Cancer Ther. 2003;2(4):315-329. doi:10.1177/1534735403259152.
9. Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology. 2009;20(4):512-522. 10. Stukel TA, Fisher ES, Wennberg DE, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007;297(3):278-285.
JAMA June 11, 2014 Volume 311, Number 22
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a New York University User on 05/21/2015
jama.com
