Letters
should have been discarded in favor of a model that allowed for study-to-study variability. Reporting only pooled results when the I2 statistic is large is a recipe for misleading results. With modern-day sample sizes being relatively small, a large observed I2 statistic is more likely to reflect model failure than the failings of the statistic itself. Third, Mills and colleagues suggest that a large I2 statistic may mistakenly lead to rejection of the pooled model when true. However, this is easily avoided by using an appropriately specified random-effects model that allows for unexplained study-to-study variability, as was done in our analysis. The more complex model will reduce to the pooled model when the pooled model is true or properly account for unexplained heterogeneity otherwise. The price for this robustness is that the complex model can sometimes be inefficient when the pooled model is true. On a technical point, our model averages on the log-odds scale, not on the probability or odds ratio scale as Mills and colleagues suggest. It makes little sense to compare variances between a proportion and an odds ratio, and their suggestion to hold sample sizes “equal” is puzzling (one is a 1-sample statistic and the other a 2-sample statistic). Focus on the (relative) magnitude of I2 statistics is misplaced; the issue is whether there is sufficient departure from zero to abandon the pooled model. Jeffrey D. Blume, PhD Stephen A. Deppen, PhD Eric L. Grogan, MD, MPH
Matthew Schoenfeld, MD Raj Machhar, MD Anna Maw, MD Author Affiliations: New York Presbyterian/Weill Cornell Medical Center, New York, New York. Corresponding Author: Anna Maw, MD, Department of Medicine, New York Presbyterian/Weill Cornell Medical Center, 525 E 68th St, New York, NY 10021 ([email protected]). Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. 1. Holland TL, Arnold C, Fowler VG Jr. Clinical management of Staphylococcus aureus bacteremia: a review. JAMA. 2014;312(13):1330-1341.
Author Affiliations: Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee (Blume); Department of Thoracic Surgery, Vanderbilt University Medical Center, Nashville, Tennessee (Deppen, Grogan). Corresponding Author: Jeffrey D. Blume, PhD, Department of Biostatistics, Vanderbilt University Medical Center, 2525 West End Ave, Nashville, TN 37203 ([email protected]). Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. 1. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264-269, W64.
Diagnosing Endocarditis in Patients With Staphylococcus aureus Bacteremia To the Editor Dr Holland and colleagues1 performed a systematic review of the clinical management of patients with Staphylococcus aureus bacteremia and concluded that for those with certain low-risk characteristics, transesophageal echocardiography can be safely deferred. Our objection to this assertion is 2-fold. First, we agree with the authors’ assessment that the available evidence is low quality. In particular, the relevant studies used a reference standard that was less sensitive than it could have been, in that there were low rates of transesophageal echocardiography performed. Thus, some cases of endocarditis could have been missed and falsely labeled as having no endocarditis. Second, the consequences of missed diagnoses (mortality rates for S aureus endocarditis are as high as 50%2) are 420
much greater than the costs and risks of transesophageal echocardiography.3,4 Therefore, a recommendation to forgo transesophageal echocardiography for these low-risk patients seems imprudent. In addition, even if the available evidence were high quality with a perfect reference standard, the negative predictive value of the clinical prediction rule was too imprecise (ie, the confidence interval was too broad) to reliably exclude such a dangerous disease with confidence. Until there is a large, highquality study at low risk of bias that uses an optimally sensitive reference standard (ie, requires transesophageal echocardiography), we believe it is inappropriate to recommend forgoing transesophageal echocardiography. Otherwise, patients may be undertreated for this common and lethal disease.
2. Chang FY, MacDonald BB, Peacock JE Jr, et al. A prospective multicenter study of Staphylococcus aureus bacteremia: incidence of endocarditis, risk factors for mortality, and clinical impact of methicillin resistance. Medicine (Baltimore). 2003;82(5):322-332. 3. Khandheria BK, Seward JB, Tajik AJ. Transesophageal echocardiography. Mayo Clin Proc. 1994;69(9):856-863. 4. Rosen AB, Fowler VG Jr, Corey GR, et al. Cost-effectiveness of transesophageal echocardiography to determine the duration of therapy for intravascular catheter-associated Staphylococcus aureus bacteremia. Ann Intern Med. 1999;130(10):810-820.
In Reply Dr Schoenfeld and colleagues emphasize the low quality of the available evidence regarding the use of transesophageal echocardiography in patients with S aureus bacteremia and the possibility that cases of endocarditis may have been missed in the relevant studies. We agree with this point and would welcome definitive high-quality studies to address this possibility. A mandatory transesophageal echocardiography design seems unlikely to succeed on a practical basis; for example, in one study using that approach, only 59% of patients had it performed.1 A trial comparing a strategy of universal vs targeted transesophageal echocardiography, with attention to clinical outcomes, would be a pivotal contribution to the literature. In the absence of such high-quality data, we believe that for a carefully selected minority of patients meeting low-risk criteria, transthoracic echocardiography is adequate. Thomas L. Holland, MD Vance G. Fowler Jr, MD, MHS
JAMA January 27, 2015 Volume 313, Number 4 (Reprinted)
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Carleton University User on 05/11/2015
jama.com
Letters
Author Affiliations: Division of Infectious Diseases and International Health, Duke University School of Medicine, Durham, North Carolina.
Wales, Sydney, Australia (Haire); Department of Anthropology, University of California, Irvine (Peterson).
Corresponding Author: Vance G. Fowler Jr, MD, MHS, Division of Infectious Diseases and International Health, Department of Medicine, Duke University School of Medicine, PO Box 102359, Durham, NC 27710 ([email protected]).
Corresponding Author: Morenike Oluwatoyin Folayan, MBChD, MBA, FWACS, Obafemi Awolowo University, Ile-Ife, Nigeria ([email protected]).
Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Holland reported being a consultant for The Medicines Co, Affinium Pharmaceuticals Inc, and Forest/Cerexa. Dr Fowler reported receiving grants from the National Institutes of Health, MedImmune, Forest/Cerexa, Pfizer, Merck, Advanced Liquid Logics, Theravance, Novartis, and Cubist; serving as chair of the scientific advisory board for V710 Staphylococcus aureus vaccine for Merck; being a consultant for Pfizer, Novartis, Galderma, Novadigm, Durata, Debiopharm, Genentech, Achaogen, Affinium, Medicines Co, Cerexa, Tetraophase, Trius, MedImmune, Bayer, Theravance, and Cubist; having a patent from NCGR; receiving royalties from UpToDate; and receiving payment for educational presentations from Cubist, Cerexa, Durata, and Theravance. 1. Holden E, Bashir A, Das I, et al. Staphylococcus aureus bacteraemia in a UK tertiary referral centre: a “transoesophageal echocardiogram for all” policy. J Antimicrob Chemother. 2014;69(7):1960-1965.
Ethical Testing of Experimental Ebola Treatments To the Editor In his Viewpoint, Dr Joffe1 sought to justify the need for randomization of experimental drugs used for the management of the Ebola crisis due to the need for scientific information. He explicitly argued against the compassionate use of experimental interventions. Instead, we argue for compassionate use of experimental interventions for Ebola management in ways that do not legitimize inequitable access to resources, especially in countries where human rights violations routinely occur. We prefer access to experimental drugs through a first-come, firstserved approach for allocating scarce resources.2 We reject the notion to conduct randomized clinical trials (RCTs) during the current Ebola epidemic in West Africa on scientific and ethical grounds. First, there is insufficient stock of experimental therapies to conduct RCTs of sufficient power to test the level of efficacy. Second, observational data obtained through compassionate access programs can provide preliminary data that can be useful in designing subsequent RCTs needed for drug approval. The experience with ZMapp so far provides useful data for future trials.3 The current Ebola epidemic requires measures that can increase survival. The decision for compassionate use of experimental drugs by the World Health Organization4 is laudable, especially for countries where inadequate health systems do not provide optimum supportive therapy that could otherwise increase the rate of survival. We do not know whether the current experimental therapies are effective, but having a control group is not ethically justifiable until such time as a trial can be designed with sufficient statistical power to answer that research question. In the meantime, compassionate access can provide some preliminary data as well as give hope to patients. Morenike Oluwatoyin Folayan, MBChD, MBA, FWACS Bridget Haire, PhD Kristin Peterson, PhD Author Affiliations: Obafemi Awolowo University, Ile-Ife, Nigeria (Folayan); School of Public Health and Community Medicine, University of New South
Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. 1. Joffe S. Evaluating novel therapies during the Ebola epidemic. JAMA. 2014; 312(13):1299-1300. 2. Persad G, Wertheimer A, Emanuel EJ. Principles for allocation of scarce medical interventions. Lancet. 2009;373(9661):423-431. 3. Ross W. Ebola crisis: doctors in Liberia “recovering after taking ZMapp.” http: //www.bbc.com/news/world-africa-28860204. Accessed September 1, 2014. 4. Maurice J. WHO meeting chooses untried interventions to defeat Ebola. Lancet. 2014;384(9948):e45-e46.
To the Editor Dr Joffe1 argued for the importance of randomized study designs to ensure the acquisition of scientific knowledge required for the response to the Ebola epidemic to progress, as well as to ensure ethical treatment of patients. Many calls for RCTs have culminated with the establishment, by the Wellcome Trust, of an international consortium to conduct clinical trials of experimental therapeutics for Ebola virus disease in West Africa.2 Joffe’s focus on randomization and controls appears to suggest standard RCT study designs. However, adaptive clinical trials should be viewed as the most efficient way to assess efficacy and safety, and hence the more ethical path on which to proceed. Unlike standard RCT designs, adaptive designs allow for prospective adaptations to their statistical procedures after initiation, without undermining their validity and integrity.3 Although several designs may offer efficient decision making, approaches with adaptive randomization and stopping rules, such as the play-the-winner design, provide distinct advantages. Allowing the randomization schedule to be modified to unequal probabilities of treatment based on observed probabilities of success (adaptive randomization) will reduce the number of patients exposed to less effective treatments while acquiring scientific knowledge. For example, if after the first week, the probability of survival is twice as high in the treatment group, then patients will subsequently be randomized at a 2:1 ratio in favor of treatment. Ebola treatments lend themselves well to this design because treatment duration is relatively short. Conventional RCT stopping rules allow for premature trial termination due to an overwhelming signal of effectiveness, while adaptive stopping rules allow for flexible and more ethical trial termination. An adaptive stopping rule in Ebola RCTs could be set as a maximum acceptable observed difference in mortality rates between treatments and the associated statistical certainty. The World Health Organization Ebola Response Team estimated a 70.8% case fatality rate among patients with known clinical outcomes, and the current death toll is higher than all previous outbreaks combined.4 In response, health authorities have deemed ethical the use of experimental treatments to help curb the Ebola outbreak,5 despite not following the conventional (and drawn-out) RCT process.
jama.com
(Reprinted) JAMA January 27, 2015 Volume 313, Number 4
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Carleton University User on 05/11/2015
421
should have been discarded in favor of a model that allowed for study-to-study variability. Reporting only pooled results when the I2 statistic is large is a recipe for misleading results. With modern-day sample sizes being relatively small, a large observed I2 statistic is more likely to reflect model failure than the failings of the statistic itself. Third, Mills and colleagues suggest that a large I2 statistic may mistakenly lead to rejection of the pooled model when true. However, this is easily avoided by using an appropriately specified random-effects model that allows for unexplained study-to-study variability, as was done in our analysis. The more complex model will reduce to the pooled model when the pooled model is true or properly account for unexplained heterogeneity otherwise. The price for this robustness is that the complex model can sometimes be inefficient when the pooled model is true. On a technical point, our model averages on the log-odds scale, not on the probability or odds ratio scale as Mills and colleagues suggest. It makes little sense to compare variances between a proportion and an odds ratio, and their suggestion to hold sample sizes “equal” is puzzling (one is a 1-sample statistic and the other a 2-sample statistic). Focus on the (relative) magnitude of I2 statistics is misplaced; the issue is whether there is sufficient departure from zero to abandon the pooled model. Jeffrey D. Blume, PhD Stephen A. Deppen, PhD Eric L. Grogan, MD, MPH
Matthew Schoenfeld, MD Raj Machhar, MD Anna Maw, MD Author Affiliations: New York Presbyterian/Weill Cornell Medical Center, New York, New York. Corresponding Author: Anna Maw, MD, Department of Medicine, New York Presbyterian/Weill Cornell Medical Center, 525 E 68th St, New York, NY 10021 ([email protected]). Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. 1. Holland TL, Arnold C, Fowler VG Jr. Clinical management of Staphylococcus aureus bacteremia: a review. JAMA. 2014;312(13):1330-1341.
Author Affiliations: Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee (Blume); Department of Thoracic Surgery, Vanderbilt University Medical Center, Nashville, Tennessee (Deppen, Grogan). Corresponding Author: Jeffrey D. Blume, PhD, Department of Biostatistics, Vanderbilt University Medical Center, 2525 West End Ave, Nashville, TN 37203 ([email protected]). Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. 1. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264-269, W64.
Diagnosing Endocarditis in Patients With Staphylococcus aureus Bacteremia To the Editor Dr Holland and colleagues1 performed a systematic review of the clinical management of patients with Staphylococcus aureus bacteremia and concluded that for those with certain low-risk characteristics, transesophageal echocardiography can be safely deferred. Our objection to this assertion is 2-fold. First, we agree with the authors’ assessment that the available evidence is low quality. In particular, the relevant studies used a reference standard that was less sensitive than it could have been, in that there were low rates of transesophageal echocardiography performed. Thus, some cases of endocarditis could have been missed and falsely labeled as having no endocarditis. Second, the consequences of missed diagnoses (mortality rates for S aureus endocarditis are as high as 50%2) are 420
much greater than the costs and risks of transesophageal echocardiography.3,4 Therefore, a recommendation to forgo transesophageal echocardiography for these low-risk patients seems imprudent. In addition, even if the available evidence were high quality with a perfect reference standard, the negative predictive value of the clinical prediction rule was too imprecise (ie, the confidence interval was too broad) to reliably exclude such a dangerous disease with confidence. Until there is a large, highquality study at low risk of bias that uses an optimally sensitive reference standard (ie, requires transesophageal echocardiography), we believe it is inappropriate to recommend forgoing transesophageal echocardiography. Otherwise, patients may be undertreated for this common and lethal disease.
2. Chang FY, MacDonald BB, Peacock JE Jr, et al. A prospective multicenter study of Staphylococcus aureus bacteremia: incidence of endocarditis, risk factors for mortality, and clinical impact of methicillin resistance. Medicine (Baltimore). 2003;82(5):322-332. 3. Khandheria BK, Seward JB, Tajik AJ. Transesophageal echocardiography. Mayo Clin Proc. 1994;69(9):856-863. 4. Rosen AB, Fowler VG Jr, Corey GR, et al. Cost-effectiveness of transesophageal echocardiography to determine the duration of therapy for intravascular catheter-associated Staphylococcus aureus bacteremia. Ann Intern Med. 1999;130(10):810-820.
In Reply Dr Schoenfeld and colleagues emphasize the low quality of the available evidence regarding the use of transesophageal echocardiography in patients with S aureus bacteremia and the possibility that cases of endocarditis may have been missed in the relevant studies. We agree with this point and would welcome definitive high-quality studies to address this possibility. A mandatory transesophageal echocardiography design seems unlikely to succeed on a practical basis; for example, in one study using that approach, only 59% of patients had it performed.1 A trial comparing a strategy of universal vs targeted transesophageal echocardiography, with attention to clinical outcomes, would be a pivotal contribution to the literature. In the absence of such high-quality data, we believe that for a carefully selected minority of patients meeting low-risk criteria, transthoracic echocardiography is adequate. Thomas L. Holland, MD Vance G. Fowler Jr, MD, MHS
JAMA January 27, 2015 Volume 313, Number 4 (Reprinted)
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Carleton University User on 05/11/2015
jama.com
Letters
Author Affiliations: Division of Infectious Diseases and International Health, Duke University School of Medicine, Durham, North Carolina.
Wales, Sydney, Australia (Haire); Department of Anthropology, University of California, Irvine (Peterson).
Corresponding Author: Vance G. Fowler Jr, MD, MHS, Division of Infectious Diseases and International Health, Department of Medicine, Duke University School of Medicine, PO Box 102359, Durham, NC 27710 ([email protected]).
Corresponding Author: Morenike Oluwatoyin Folayan, MBChD, MBA, FWACS, Obafemi Awolowo University, Ile-Ife, Nigeria ([email protected]).
Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Holland reported being a consultant for The Medicines Co, Affinium Pharmaceuticals Inc, and Forest/Cerexa. Dr Fowler reported receiving grants from the National Institutes of Health, MedImmune, Forest/Cerexa, Pfizer, Merck, Advanced Liquid Logics, Theravance, Novartis, and Cubist; serving as chair of the scientific advisory board for V710 Staphylococcus aureus vaccine for Merck; being a consultant for Pfizer, Novartis, Galderma, Novadigm, Durata, Debiopharm, Genentech, Achaogen, Affinium, Medicines Co, Cerexa, Tetraophase, Trius, MedImmune, Bayer, Theravance, and Cubist; having a patent from NCGR; receiving royalties from UpToDate; and receiving payment for educational presentations from Cubist, Cerexa, Durata, and Theravance. 1. Holden E, Bashir A, Das I, et al. Staphylococcus aureus bacteraemia in a UK tertiary referral centre: a “transoesophageal echocardiogram for all” policy. J Antimicrob Chemother. 2014;69(7):1960-1965.
Ethical Testing of Experimental Ebola Treatments To the Editor In his Viewpoint, Dr Joffe1 sought to justify the need for randomization of experimental drugs used for the management of the Ebola crisis due to the need for scientific information. He explicitly argued against the compassionate use of experimental interventions. Instead, we argue for compassionate use of experimental interventions for Ebola management in ways that do not legitimize inequitable access to resources, especially in countries where human rights violations routinely occur. We prefer access to experimental drugs through a first-come, firstserved approach for allocating scarce resources.2 We reject the notion to conduct randomized clinical trials (RCTs) during the current Ebola epidemic in West Africa on scientific and ethical grounds. First, there is insufficient stock of experimental therapies to conduct RCTs of sufficient power to test the level of efficacy. Second, observational data obtained through compassionate access programs can provide preliminary data that can be useful in designing subsequent RCTs needed for drug approval. The experience with ZMapp so far provides useful data for future trials.3 The current Ebola epidemic requires measures that can increase survival. The decision for compassionate use of experimental drugs by the World Health Organization4 is laudable, especially for countries where inadequate health systems do not provide optimum supportive therapy that could otherwise increase the rate of survival. We do not know whether the current experimental therapies are effective, but having a control group is not ethically justifiable until such time as a trial can be designed with sufficient statistical power to answer that research question. In the meantime, compassionate access can provide some preliminary data as well as give hope to patients. Morenike Oluwatoyin Folayan, MBChD, MBA, FWACS Bridget Haire, PhD Kristin Peterson, PhD Author Affiliations: Obafemi Awolowo University, Ile-Ife, Nigeria (Folayan); School of Public Health and Community Medicine, University of New South
Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. 1. Joffe S. Evaluating novel therapies during the Ebola epidemic. JAMA. 2014; 312(13):1299-1300. 2. Persad G, Wertheimer A, Emanuel EJ. Principles for allocation of scarce medical interventions. Lancet. 2009;373(9661):423-431. 3. Ross W. Ebola crisis: doctors in Liberia “recovering after taking ZMapp.” http: //www.bbc.com/news/world-africa-28860204. Accessed September 1, 2014. 4. Maurice J. WHO meeting chooses untried interventions to defeat Ebola. Lancet. 2014;384(9948):e45-e46.
To the Editor Dr Joffe1 argued for the importance of randomized study designs to ensure the acquisition of scientific knowledge required for the response to the Ebola epidemic to progress, as well as to ensure ethical treatment of patients. Many calls for RCTs have culminated with the establishment, by the Wellcome Trust, of an international consortium to conduct clinical trials of experimental therapeutics for Ebola virus disease in West Africa.2 Joffe’s focus on randomization and controls appears to suggest standard RCT study designs. However, adaptive clinical trials should be viewed as the most efficient way to assess efficacy and safety, and hence the more ethical path on which to proceed. Unlike standard RCT designs, adaptive designs allow for prospective adaptations to their statistical procedures after initiation, without undermining their validity and integrity.3 Although several designs may offer efficient decision making, approaches with adaptive randomization and stopping rules, such as the play-the-winner design, provide distinct advantages. Allowing the randomization schedule to be modified to unequal probabilities of treatment based on observed probabilities of success (adaptive randomization) will reduce the number of patients exposed to less effective treatments while acquiring scientific knowledge. For example, if after the first week, the probability of survival is twice as high in the treatment group, then patients will subsequently be randomized at a 2:1 ratio in favor of treatment. Ebola treatments lend themselves well to this design because treatment duration is relatively short. Conventional RCT stopping rules allow for premature trial termination due to an overwhelming signal of effectiveness, while adaptive stopping rules allow for flexible and more ethical trial termination. An adaptive stopping rule in Ebola RCTs could be set as a maximum acceptable observed difference in mortality rates between treatments and the associated statistical certainty. The World Health Organization Ebola Response Team estimated a 70.8% case fatality rate among patients with known clinical outcomes, and the current death toll is higher than all previous outbreaks combined.4 In response, health authorities have deemed ethical the use of experimental treatments to help curb the Ebola outbreak,5 despite not following the conventional (and drawn-out) RCT process.
jama.com
(Reprinted) JAMA January 27, 2015 Volume 313, Number 4
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Carleton University User on 05/11/2015
421
