EDITORIAL
European Heart Journal (2015) 36, 340–341 doi:10.1093/eurheartj/ehu311
Reanalysis or redefinition of the hypothesis? Jody D. Ciolino 1 and Rickey E. Carter 2* 1
Department of Preventive Medicine, Northwestern University, Chicago, IL, USA; and 2Department of Health Sciences Research, Mayo Clinic, Rochester, MN, USA
Online publish-ahead-of-print 20 May 2014
This editorial refers to ‘Applying novel methods to assess clinical outcomes: insights from the TRILOGY ACS trial’†, by J. A. Bakal et al., on page 385
The opinions expressed in this article are not necessarily those of the Editors of the European Heart Journal or of the European Society of Cardiology. † doi:10.1093/eurheartj/ehu262
* Corresponding author. Department of Health Sciences Research, Mayo Clinic, 200 First St SW, Rochester, MN 55905, USA. Tel: +1 507 284 4156; Fax: +1 507 284 9542, Email: [email protected] Published on behalf of the European Society of Cardiology. All rights reserved. & The Author 2014. For permissions please email: [email protected].
Downloaded from by guest on February 9, 2015
Though a seemingly simple notion, defining a clear research question or hypothesis at the conception of a clinical study can be a daunting task for investigators. Through the TRILOGY ACS study,1 Bakal and colleagues2 illustrate that subtle changes in the primary outcome definition for a clinical trial may result in substantial changes in the analytical methods, altering the overall interpretation of the study’s result. Perhaps as an unintended consequence, their article also emphasizes a pitfall of post-hoc analyses. This editorial is intended to provide some insight into the statistical considerations in their article. In doing so, it aggregates their ideas to provide recommendations for appropriate analytical techniques driven by specific research questions. Most disease progressions are associated with the realization of multiple complications. Cardiovascular disease, for example, may have elevated blood pressure, atherosclerosis, altered perfusion, ischaemia, changes in activities of daily living, etc. From a clinical perspective, a myocardial infarction resulting in death may be more important than reduced cardiopulmonary function that only limits a patient’s activities of daily living. That said, these extreme examples are intrinsically linked in a common disease process and represent different aspects of disease progression. These inter-relationships, when simultaneously modelled, often introduce the need for more sophisticated statistical analyses. These analyses can sometimes obfuscate the findings of the study. Thus, investigators often seek to balance clinical relevance with interpretability of the primary outcome measure. Composite outcome measures help with this process. The TRILOGY ACS study, like many contemporary clinical trials,3,4 utilized a composite primary outcome measure of time-to(first)occurrence of any of the following: death, myocardial infarction, or stroke.1 The appropriate statistical analysis based on this outcome compares the time-to-first-event across intervention arms without differentiating event severities (e.g. a cardiovascular death from a mild, non-disabling stroke). This analytical method addresses a research question that assumes that the severity of any
such event in relation to others is unimportant, and the true question pertains to ‘event-free survival’. Composite endpoints may be useful not only when the investigator is interested in event-free survival, but also when investigators cannot choose between equally important event definitions, the events or components all carry equal weight such that they may be exchangeable, and there is concern of type I error rate inflation due to multiple testing of multiple endpoints.4,5 By pooling the occurrence of any events, a trial typically gains power by increasing the number of events observed that may contribute to analyses.3 On the other hand, power may further increase if analyses account for any recurrent events (i.e. those after the first event where the first event was not death). For this scenario, the authors recognize the utility of the ‘Anderson–Gill’ method.6 Another challenge we face with the use of composite endpoints in the traditional manner lies in the difficulty in interpretation when components exhibit effects in different directions and/or magnitudes across intervention arms. Further, the research question may pertain not only to time-to-event, but also to overall event counts across arms. Bakal and colleagues discuss the Anderson–Gill method as a potential alternative to the traditional survival analysis techniques since it uses all events (e.g. death, myocardial infarction, and stroke) from the TRILOGY ACS study to contribute to analyses (table 2 in Bakal et al. 2). Nevertheless, the authors caution that this method also carries drawbacks in that it treats all events as exchangeable with one another (i.e. death and a mild stroke may be given equal weight), and the methods may yield an overall biased result. In fact, the reanalysis of the TRILOGY ACS data using this method results in a statistically significant difference between intervention arms while the other methods did not. The reader is left to speculate on the possibility of a type I error. The authors have proposed an alternative approach in the weighted composite endpoints (WCEs)7 in order to combat the shortcomings noted with the Anderson–Gill and traditional methods. The WCE, as the name implies, does not adhere to the traditional composite definition and each component of the composite outcome carries a weight. Analyses for this type of outcome address the research question pertaining to time-to-event and
341
Editorial
efficiency of a clinical trial, address a specific research question, and are defined a priori. In an attempt to apply the results from Bakal et al., we suggest that investigators first focus on the true research question and the nature of the hypotheses: are they interested in differences in (i) time-to-first-event (regardless of the severity of the event) across arms; (ii) the time-to-event while incorporating overall event counts (again, regardless of severity) across arms; (iii) the event rate across arms based on matched pairs while incorporating event severity; or (iv) time-to-event while incorporating overall event counts and relative severity of events across arms. In the cases of (i), (ii), (iii), and (iv), we recommend the traditional time-to-first-event analyses, Anderson–Gill method, the win ratio method, and the WCE method, respectively. The research question, though sometimes difficult to specify unambiguously, should ultimately drive the choice of primary outcome which will in turn dictate the appropriate analyses for a given clinical trial. The results from Bakal et al. suggest that any method must be pre-specified and interpreted with caution. Conflict of interest: none declared.
References 1. Roe M, Armstrong P, Fox K, White HD, Prabhakaran D, Goodman SG, Cornel JH, Bhatt DL, Clemmensen P, Martinez F, Ardissino D, Nicolau JC, Boden WE, Gurbel PA, Ruzyllo W, Dalby AJ, McGuire DK, Leiva-Pons JL, Parkhomenko A, Gottlieb S, Topacio GO, Hamm C, Pavlides G, Goudev AR, Oto A, Tseng CD, Merkely B, Gasparovic V, Corbalan R, Cinteza˘ M, McLendon RC, Winters KJ, Brown EB, Lokhnygina Y, Aylward PE, Huber K, Hochman JS, Ohman EM; TRILOGY ACS Investigators. Prasugrel versus clopidogrel for acute coronary syndromes without revascularization. N Engl J Med 2012;367:1297 –1309. 2. Bakal JA, Roe MT, Ohman EM, Goodman SG, Fox KA, Zheng Y, Westerhout CM, Hochman JS, Lokhnygina Y, Brown EB, Armstrong PW. Applying novel methods to assess clinical outcomes: insights from the TRILOGY ACS trial. Eur Heart J 2015;36: 385– 392. 3. Freemantle N, Calvert M, Wood J, Eastaugh J, Griffin C. Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA 2003;289: 2554–2559. 4. Ferreira-Gonza´lez I, Permanyer-Miralda G, Busse JW, Bryant DM, Montori VM, Alonso-Coello P, Walter SD, Guyatt GH. Methodologic discussions for using and interpreting composite endpoints are limited, but still identify major concerns. J Clin Epidemiol 2007;60:651 –657. 5. Lewis JA. Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Stat Med 1999;18:1903 – 1942. 6. Anderson P, Gill R. Cox’s regression model for counting processes: a large sample study. Ann Stat 1982;1982:1100 – 1120. 7. Bakal J, Westerhout C, Armstrong P. Impact of weighted composite compared to traditional composite endpoints for the design of randomized controlled trials. Stat Methods Med Res 2012.
Downloaded from by guest on February 9, 2015
overall event counts while simultaneously taking into consideration the event severities in relation to one another. The WCE method requires that investigators select and provide weights for clinical events and/or measurements that are essential to understanding how an intervention affects the disease progression a priori. All participants begin the study with a ‘score’ of 100%, and any observed event results in a decrease in score with magnitude depending on the severity of the event. The most severe event, death, results in a score of 0% for the participant. The less severe events will diminish a participant’s score, but at a smaller scale such that his/her information may still contribute to the statistical analysis. WCE analyses replace the traditional survival curves with a decaying process that allows for multiple events. The selection of the weights requires careful consideration. Through judicious choice of weights (and variables included in the composite outcome), one may be able to reach any desired conclusion that is more reflective of his or her working hypothesis. Bakal and colleagues emphasize that investigators must agree upon and document the weighting schedule (including the events comprising the composite endpoint) prior to study implementation. As with any element surrounding the statistical analyses of study data, the International Conference on Harmonization5 guidelines recommend avoidance of post-hoc determination of the elements of the composite endpoint. Bakal and colleagues discuss yet another endpoint, the win ratio, which addresses the research question regarding event rates between arms while taking into consideration differing event severities. As the authors acknowledge, the win ratio method ‘approaches [study] data differently’ from the other methods mentioned, and it involves a matching mechanism which may lend itself to conflicting results and interpretations. Thus, the reader is cautioned with this approach. This brings us back to the interpretation of Bakal’s reanalysis. Should the results of the TRILOGY ACS be reconsidered? Bakal and colleagues show that if one considers a recurrent event survival model (Andersen–Gill model), the observed difference between two intervention arms reaches statistical significance. The WCE results essentially fall between this recurrent event analysis (that did not involve any relative weighting mechanism) and the traditional time-to-first-event analysis. Of more relevance is perhaps whether the analyses properly addressed the research question(s). The article of Bakal and colleagues is important as it demonstrates the need for objective primary endpoints that maximize the statistical
European Heart Journal (2015) 36, 340–341 doi:10.1093/eurheartj/ehu311
Reanalysis or redefinition of the hypothesis? Jody D. Ciolino 1 and Rickey E. Carter 2* 1
Department of Preventive Medicine, Northwestern University, Chicago, IL, USA; and 2Department of Health Sciences Research, Mayo Clinic, Rochester, MN, USA
Online publish-ahead-of-print 20 May 2014
This editorial refers to ‘Applying novel methods to assess clinical outcomes: insights from the TRILOGY ACS trial’†, by J. A. Bakal et al., on page 385
The opinions expressed in this article are not necessarily those of the Editors of the European Heart Journal or of the European Society of Cardiology. † doi:10.1093/eurheartj/ehu262
* Corresponding author. Department of Health Sciences Research, Mayo Clinic, 200 First St SW, Rochester, MN 55905, USA. Tel: +1 507 284 4156; Fax: +1 507 284 9542, Email: [email protected] Published on behalf of the European Society of Cardiology. All rights reserved. & The Author 2014. For permissions please email: [email protected].
Downloaded from by guest on February 9, 2015
Though a seemingly simple notion, defining a clear research question or hypothesis at the conception of a clinical study can be a daunting task for investigators. Through the TRILOGY ACS study,1 Bakal and colleagues2 illustrate that subtle changes in the primary outcome definition for a clinical trial may result in substantial changes in the analytical methods, altering the overall interpretation of the study’s result. Perhaps as an unintended consequence, their article also emphasizes a pitfall of post-hoc analyses. This editorial is intended to provide some insight into the statistical considerations in their article. In doing so, it aggregates their ideas to provide recommendations for appropriate analytical techniques driven by specific research questions. Most disease progressions are associated with the realization of multiple complications. Cardiovascular disease, for example, may have elevated blood pressure, atherosclerosis, altered perfusion, ischaemia, changes in activities of daily living, etc. From a clinical perspective, a myocardial infarction resulting in death may be more important than reduced cardiopulmonary function that only limits a patient’s activities of daily living. That said, these extreme examples are intrinsically linked in a common disease process and represent different aspects of disease progression. These inter-relationships, when simultaneously modelled, often introduce the need for more sophisticated statistical analyses. These analyses can sometimes obfuscate the findings of the study. Thus, investigators often seek to balance clinical relevance with interpretability of the primary outcome measure. Composite outcome measures help with this process. The TRILOGY ACS study, like many contemporary clinical trials,3,4 utilized a composite primary outcome measure of time-to(first)occurrence of any of the following: death, myocardial infarction, or stroke.1 The appropriate statistical analysis based on this outcome compares the time-to-first-event across intervention arms without differentiating event severities (e.g. a cardiovascular death from a mild, non-disabling stroke). This analytical method addresses a research question that assumes that the severity of any
such event in relation to others is unimportant, and the true question pertains to ‘event-free survival’. Composite endpoints may be useful not only when the investigator is interested in event-free survival, but also when investigators cannot choose between equally important event definitions, the events or components all carry equal weight such that they may be exchangeable, and there is concern of type I error rate inflation due to multiple testing of multiple endpoints.4,5 By pooling the occurrence of any events, a trial typically gains power by increasing the number of events observed that may contribute to analyses.3 On the other hand, power may further increase if analyses account for any recurrent events (i.e. those after the first event where the first event was not death). For this scenario, the authors recognize the utility of the ‘Anderson–Gill’ method.6 Another challenge we face with the use of composite endpoints in the traditional manner lies in the difficulty in interpretation when components exhibit effects in different directions and/or magnitudes across intervention arms. Further, the research question may pertain not only to time-to-event, but also to overall event counts across arms. Bakal and colleagues discuss the Anderson–Gill method as a potential alternative to the traditional survival analysis techniques since it uses all events (e.g. death, myocardial infarction, and stroke) from the TRILOGY ACS study to contribute to analyses (table 2 in Bakal et al. 2). Nevertheless, the authors caution that this method also carries drawbacks in that it treats all events as exchangeable with one another (i.e. death and a mild stroke may be given equal weight), and the methods may yield an overall biased result. In fact, the reanalysis of the TRILOGY ACS data using this method results in a statistically significant difference between intervention arms while the other methods did not. The reader is left to speculate on the possibility of a type I error. The authors have proposed an alternative approach in the weighted composite endpoints (WCEs)7 in order to combat the shortcomings noted with the Anderson–Gill and traditional methods. The WCE, as the name implies, does not adhere to the traditional composite definition and each component of the composite outcome carries a weight. Analyses for this type of outcome address the research question pertaining to time-to-event and
341
Editorial
efficiency of a clinical trial, address a specific research question, and are defined a priori. In an attempt to apply the results from Bakal et al., we suggest that investigators first focus on the true research question and the nature of the hypotheses: are they interested in differences in (i) time-to-first-event (regardless of the severity of the event) across arms; (ii) the time-to-event while incorporating overall event counts (again, regardless of severity) across arms; (iii) the event rate across arms based on matched pairs while incorporating event severity; or (iv) time-to-event while incorporating overall event counts and relative severity of events across arms. In the cases of (i), (ii), (iii), and (iv), we recommend the traditional time-to-first-event analyses, Anderson–Gill method, the win ratio method, and the WCE method, respectively. The research question, though sometimes difficult to specify unambiguously, should ultimately drive the choice of primary outcome which will in turn dictate the appropriate analyses for a given clinical trial. The results from Bakal et al. suggest that any method must be pre-specified and interpreted with caution. Conflict of interest: none declared.
References 1. Roe M, Armstrong P, Fox K, White HD, Prabhakaran D, Goodman SG, Cornel JH, Bhatt DL, Clemmensen P, Martinez F, Ardissino D, Nicolau JC, Boden WE, Gurbel PA, Ruzyllo W, Dalby AJ, McGuire DK, Leiva-Pons JL, Parkhomenko A, Gottlieb S, Topacio GO, Hamm C, Pavlides G, Goudev AR, Oto A, Tseng CD, Merkely B, Gasparovic V, Corbalan R, Cinteza˘ M, McLendon RC, Winters KJ, Brown EB, Lokhnygina Y, Aylward PE, Huber K, Hochman JS, Ohman EM; TRILOGY ACS Investigators. Prasugrel versus clopidogrel for acute coronary syndromes without revascularization. N Engl J Med 2012;367:1297 –1309. 2. Bakal JA, Roe MT, Ohman EM, Goodman SG, Fox KA, Zheng Y, Westerhout CM, Hochman JS, Lokhnygina Y, Brown EB, Armstrong PW. Applying novel methods to assess clinical outcomes: insights from the TRILOGY ACS trial. Eur Heart J 2015;36: 385– 392. 3. Freemantle N, Calvert M, Wood J, Eastaugh J, Griffin C. Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA 2003;289: 2554–2559. 4. Ferreira-Gonza´lez I, Permanyer-Miralda G, Busse JW, Bryant DM, Montori VM, Alonso-Coello P, Walter SD, Guyatt GH. Methodologic discussions for using and interpreting composite endpoints are limited, but still identify major concerns. J Clin Epidemiol 2007;60:651 –657. 5. Lewis JA. Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Stat Med 1999;18:1903 – 1942. 6. Anderson P, Gill R. Cox’s regression model for counting processes: a large sample study. Ann Stat 1982;1982:1100 – 1120. 7. Bakal J, Westerhout C, Armstrong P. Impact of weighted composite compared to traditional composite endpoints for the design of randomized controlled trials. Stat Methods Med Res 2012.
Downloaded from by guest on February 9, 2015
overall event counts while simultaneously taking into consideration the event severities in relation to one another. The WCE method requires that investigators select and provide weights for clinical events and/or measurements that are essential to understanding how an intervention affects the disease progression a priori. All participants begin the study with a ‘score’ of 100%, and any observed event results in a decrease in score with magnitude depending on the severity of the event. The most severe event, death, results in a score of 0% for the participant. The less severe events will diminish a participant’s score, but at a smaller scale such that his/her information may still contribute to the statistical analysis. WCE analyses replace the traditional survival curves with a decaying process that allows for multiple events. The selection of the weights requires careful consideration. Through judicious choice of weights (and variables included in the composite outcome), one may be able to reach any desired conclusion that is more reflective of his or her working hypothesis. Bakal and colleagues emphasize that investigators must agree upon and document the weighting schedule (including the events comprising the composite endpoint) prior to study implementation. As with any element surrounding the statistical analyses of study data, the International Conference on Harmonization5 guidelines recommend avoidance of post-hoc determination of the elements of the composite endpoint. Bakal and colleagues discuss yet another endpoint, the win ratio, which addresses the research question regarding event rates between arms while taking into consideration differing event severities. As the authors acknowledge, the win ratio method ‘approaches [study] data differently’ from the other methods mentioned, and it involves a matching mechanism which may lend itself to conflicting results and interpretations. Thus, the reader is cautioned with this approach. This brings us back to the interpretation of Bakal’s reanalysis. Should the results of the TRILOGY ACS be reconsidered? Bakal and colleagues show that if one considers a recurrent event survival model (Andersen–Gill model), the observed difference between two intervention arms reaches statistical significance. The WCE results essentially fall between this recurrent event analysis (that did not involve any relative weighting mechanism) and the traditional time-to-first-event analysis. Of more relevance is perhaps whether the analyses properly addressed the research question(s). The article of Bakal and colleagues is important as it demonstrates the need for objective primary endpoints that maximize the statistical
