Ò
PAIN 155 (2014) 461–466
www.elsevier.com/locate/pain
Research papers
Reporting of primary analyses and multiplicity adjustment in recent analgesic clinical trials: ACTTION systematic review and recommendations Jennifer S. Gewandter a,⇑, Shannon M. Smith a, Andrew McKeown a, Laurie B. Burke b, Sharon H. Hertz b, Matthew Hunsinger c, Nathaniel P. Katz d,e, Allison H. Lin b, Michael P. McDermott f, Bob A. Rappaport b, Mark R. Williams a, Dennis C. Turk g, Robert H. Dworkin a a
Department of Anesthesiology, University of Rochester School of Medicine and Dentistry, Rochester, NY, USA United States Food and Drug Administration, Silver Spring, MD, USA c School of Professional Psychology, Pacific University, Hillsboro, OR, USA d Analgesic Solutions, Natick, MA, USA e Tufts University, Boston, MA, USA f Department of Biostatistics and Computational Biology, University of Rochester School of Medicine and Dentistry, Rochester, NY, USA g Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, WA, USA b
Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article.
a r t i c l e
i n f o
Article history: Received 10 September 2013 Accepted 15 November 2013
Keywords: Primary analysis reporting Multiplicity Systematic review
a b s t r a c t Performing multiple analyses in clinical trials can inflate the probability of a type I error, or the chance of falsely concluding a significant effect of the treatment. Strategies to minimize type I error probability include prespecification of primary analyses and statistical adjustment for multiple comparisons, when applicable. The objective of this study was to assess the quality of primary analysis reporting and frequency of multiplicity adjustment in 3 major pain journals (ie, European Journal of Pain, Journal of Pain, and PAINÒ). A total of 161 randomized controlled trials investigating noninvasive pharmacological treatments or interventional treatments for pain, published between 2006 and 2012, were included. Only 52% of trials identified a primary analysis, and only 10% of trials reported prespecification of that analysis. Among the 33 articles that identified a primary analysis with multiple testing, 15 (45%) adjusted for multiplicity; of those 15, only 2 (13%) reported prespecification of the adjustment methodology. Trials in clinical pain conditions and industry-sponsored trials identified a primary analysis more often than trials in experimental pain models and non-industry-sponsored trials, respectively. The results of this systematic review demonstrate deficiencies in the reporting and possibly the execution of primary analyses in published analgesic trials. These deficiencies can be rectified by changes in, or better enforcement of, journal policies pertaining to requirements for the reporting of analyses of clinical trial data. Ó 2013 International Association for the Study of Pain. Published by Elsevier B.V. All rights reserved.
1. Introduction Double-blind, randomized controlled trials (RCTs) are considered the gold standard for determining treatment efficacy [6,17]. When publishing the results of RCTs, it is essential for researchers to present the details of their statistical analyses clearly and in a sufficiently comprehensive manner so that readers can evaluate the appropriateness of the analyses performed and the validity of the conclusions drawn. ⇑ Corresponding author. Address: Department of Anesthesiology, University of Rochester School of Medicine and Dentistry, 601 Elmwood Ave, Box 604, Rochester, NY 14642, USA. Tel.: +1 585 276 5661; fax: +1 585 244 7271. E-mail address: [email protected] (J.S. Gewandter).
It is well established that performing multiple statistical tests can inflate the chance of falsely concluding a significant treatment effect [7,13,18,20]. The problem of multiplicity can arise due to multiple comparisons among different treatment groups, analyses of treatment effects in multiple subgroups, use of multiple outcome variables or the same outcome variable defined at multiple time points, interim analyses, or using multiple methods for statistical analysis to address the same scientific question [19]. If the primary analysis includes more than one statistical test, statisticians [2,21], CONSORT guidelines [1], regulatory agencies [3,27], and others [26] recommend the use of various methods to adjust for multiple testing to minimize chance findings. These methods include but are not limited to defining co-primary outcome variables, hierarchical testing procedures [4,5,28], Bonferroni
0304-3959/$36.00 Ó 2013 International Association for the Study of Pain. Published by Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.pain.2013.11.009
462
Ò
J.S. Gewandter et al. / PAIN 155 (2014) 461–466
correction [22] and related stepwise procedures (eg, Holm [10], Hochberg [9], and Hommel [11]), and Tukey’s honestly significant difference test [25]. Comprehensive reviews of these methods are available elsewhere [22,26]. To prevent bias due to choosing the primary analysis and method to adjust for multiplicity after examining the data (ie, ex post facto), the primary outcome variable and analysis (including the outcome variable, statistical test, and the specific time point or interval for which the outcome analysis will be conducted) used to conclude whether the trial provides evidence of a treatment effect must be identified prior to data unblinding. Previous studies have found deficiencies in the clarity of primary analysis reporting in RCTs in medical research [8,12,14,15]. To our knowledge, however, no studies have systematically investigated the reporting of adjustment for multiple comparisons in pain trials. The objective of the present study was to evaluate the frequency with which primary analyses and multiplicity adjustment, when applicable, were reported in RCTs published in 3 major pain journals. The overall goal of these analyses was to identify deficiencies in statistical reporting that can be improved to increase the interpretability of clinical trial results presented in pain journals. Although deficiencies in statistical practice cannot be inferred by shortcomings in reporting found in this study, we expect that any stricter expectations for authors to report appropriate statistical methods, such as prespecification and adjustment for multiplicity, that result from this work will, in turn, encourage authors to amend any imperfections in statistical practice that may exist. 2. Methods 2.1. Article selection Reports of RCTs of noninvasive pharmacologic treatments and invasive treatments (any analgesic device, therapy, or drug treatment that breaks the skin) published between 2006 and 2012 were selected from the European Journal of Pain, the Journal of Pain, and PAINÒ. Selected articles reporting trials that were randomized and double blind, compared at least 2 treatments (one of which could be placebo), and had one of the following primary or secondary outcome variables: a patient-reported outcome measuring pain intensity, relief, qualities (eg, burning pain, allodynia), or area; any composite patient-reported outcome including pain (eg, pain, numbness, and tingling); or amount of opioid or other analgesic sparing. Trials with outcomes measuring pain behaviors (eg, grimacing), function, or other symptoms such as sleep and fatigue, were not included. Smith et al. [23] describe retrieval of the included trials of noninvasive pharmacologic treatments published between 2006 and 2011. Articles investigating invasive treatments published between 2006 and 2012 and those investigating noninvasive pharmacological treatments published in 2012 were identified by M.R.W. and J.S.G. 2.2. Data extraction A coding manual was developed to investigate questions pertaining to the reporting of the primary outcome variable, primary statistical analysis plan, and adjustment for multiple testing in the primary analysis, when applicable. The coding manual was pretested and modified by coding 2 rounds of 3 and 2 rounds of 2 (total = 10) articles published in the 3 journals between 2000 and 2003. In each round, 3 authors (J.S.G., A.M., and M.H.) coded each article separately and discussed any discrepancies. The manual was modified for clarity in response to these discussions. Two authors (J.S.G. and A.M.) coded the 2006–2012 articles according to this manual, which instructed coders to search the Methods and
Results sections of the articles. The order of the articles was randomized for each coder. After the articles were coded independently, the coders reviewed the data for discrepancies. Discrepancies due to oversight were corrected and discrepancies due to alternative interpretations were discussed to arrive at a consensus. J.S.G. performed a post hoc review of the Discussion sections of the 18 articles that identified multiple primary analyses and did not mention any adjustment for multiplicity to determine if the authors noted multiple comparisons as a limitation of interpreting the analyses. She also performed a post hoc search of all articles to determine if the authors declared the trials exploratory in any section of the manuscripts. Journal, study design, and number of randomized participants were also extracted for each trial. In addition, information regarding trial type (experimental pain model vs clinical pain condition), trial size (P or
PAIN 155 (2014) 461–466
www.elsevier.com/locate/pain
Research papers
Reporting of primary analyses and multiplicity adjustment in recent analgesic clinical trials: ACTTION systematic review and recommendations Jennifer S. Gewandter a,⇑, Shannon M. Smith a, Andrew McKeown a, Laurie B. Burke b, Sharon H. Hertz b, Matthew Hunsinger c, Nathaniel P. Katz d,e, Allison H. Lin b, Michael P. McDermott f, Bob A. Rappaport b, Mark R. Williams a, Dennis C. Turk g, Robert H. Dworkin a a
Department of Anesthesiology, University of Rochester School of Medicine and Dentistry, Rochester, NY, USA United States Food and Drug Administration, Silver Spring, MD, USA c School of Professional Psychology, Pacific University, Hillsboro, OR, USA d Analgesic Solutions, Natick, MA, USA e Tufts University, Boston, MA, USA f Department of Biostatistics and Computational Biology, University of Rochester School of Medicine and Dentistry, Rochester, NY, USA g Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, WA, USA b
Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article.
a r t i c l e
i n f o
Article history: Received 10 September 2013 Accepted 15 November 2013
Keywords: Primary analysis reporting Multiplicity Systematic review
a b s t r a c t Performing multiple analyses in clinical trials can inflate the probability of a type I error, or the chance of falsely concluding a significant effect of the treatment. Strategies to minimize type I error probability include prespecification of primary analyses and statistical adjustment for multiple comparisons, when applicable. The objective of this study was to assess the quality of primary analysis reporting and frequency of multiplicity adjustment in 3 major pain journals (ie, European Journal of Pain, Journal of Pain, and PAINÒ). A total of 161 randomized controlled trials investigating noninvasive pharmacological treatments or interventional treatments for pain, published between 2006 and 2012, were included. Only 52% of trials identified a primary analysis, and only 10% of trials reported prespecification of that analysis. Among the 33 articles that identified a primary analysis with multiple testing, 15 (45%) adjusted for multiplicity; of those 15, only 2 (13%) reported prespecification of the adjustment methodology. Trials in clinical pain conditions and industry-sponsored trials identified a primary analysis more often than trials in experimental pain models and non-industry-sponsored trials, respectively. The results of this systematic review demonstrate deficiencies in the reporting and possibly the execution of primary analyses in published analgesic trials. These deficiencies can be rectified by changes in, or better enforcement of, journal policies pertaining to requirements for the reporting of analyses of clinical trial data. Ó 2013 International Association for the Study of Pain. Published by Elsevier B.V. All rights reserved.
1. Introduction Double-blind, randomized controlled trials (RCTs) are considered the gold standard for determining treatment efficacy [6,17]. When publishing the results of RCTs, it is essential for researchers to present the details of their statistical analyses clearly and in a sufficiently comprehensive manner so that readers can evaluate the appropriateness of the analyses performed and the validity of the conclusions drawn. ⇑ Corresponding author. Address: Department of Anesthesiology, University of Rochester School of Medicine and Dentistry, 601 Elmwood Ave, Box 604, Rochester, NY 14642, USA. Tel.: +1 585 276 5661; fax: +1 585 244 7271. E-mail address: [email protected] (J.S. Gewandter).
It is well established that performing multiple statistical tests can inflate the chance of falsely concluding a significant treatment effect [7,13,18,20]. The problem of multiplicity can arise due to multiple comparisons among different treatment groups, analyses of treatment effects in multiple subgroups, use of multiple outcome variables or the same outcome variable defined at multiple time points, interim analyses, or using multiple methods for statistical analysis to address the same scientific question [19]. If the primary analysis includes more than one statistical test, statisticians [2,21], CONSORT guidelines [1], regulatory agencies [3,27], and others [26] recommend the use of various methods to adjust for multiple testing to minimize chance findings. These methods include but are not limited to defining co-primary outcome variables, hierarchical testing procedures [4,5,28], Bonferroni
0304-3959/$36.00 Ó 2013 International Association for the Study of Pain. Published by Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.pain.2013.11.009
462
Ò
J.S. Gewandter et al. / PAIN 155 (2014) 461–466
correction [22] and related stepwise procedures (eg, Holm [10], Hochberg [9], and Hommel [11]), and Tukey’s honestly significant difference test [25]. Comprehensive reviews of these methods are available elsewhere [22,26]. To prevent bias due to choosing the primary analysis and method to adjust for multiplicity after examining the data (ie, ex post facto), the primary outcome variable and analysis (including the outcome variable, statistical test, and the specific time point or interval for which the outcome analysis will be conducted) used to conclude whether the trial provides evidence of a treatment effect must be identified prior to data unblinding. Previous studies have found deficiencies in the clarity of primary analysis reporting in RCTs in medical research [8,12,14,15]. To our knowledge, however, no studies have systematically investigated the reporting of adjustment for multiple comparisons in pain trials. The objective of the present study was to evaluate the frequency with which primary analyses and multiplicity adjustment, when applicable, were reported in RCTs published in 3 major pain journals. The overall goal of these analyses was to identify deficiencies in statistical reporting that can be improved to increase the interpretability of clinical trial results presented in pain journals. Although deficiencies in statistical practice cannot be inferred by shortcomings in reporting found in this study, we expect that any stricter expectations for authors to report appropriate statistical methods, such as prespecification and adjustment for multiplicity, that result from this work will, in turn, encourage authors to amend any imperfections in statistical practice that may exist. 2. Methods 2.1. Article selection Reports of RCTs of noninvasive pharmacologic treatments and invasive treatments (any analgesic device, therapy, or drug treatment that breaks the skin) published between 2006 and 2012 were selected from the European Journal of Pain, the Journal of Pain, and PAINÒ. Selected articles reporting trials that were randomized and double blind, compared at least 2 treatments (one of which could be placebo), and had one of the following primary or secondary outcome variables: a patient-reported outcome measuring pain intensity, relief, qualities (eg, burning pain, allodynia), or area; any composite patient-reported outcome including pain (eg, pain, numbness, and tingling); or amount of opioid or other analgesic sparing. Trials with outcomes measuring pain behaviors (eg, grimacing), function, or other symptoms such as sleep and fatigue, were not included. Smith et al. [23] describe retrieval of the included trials of noninvasive pharmacologic treatments published between 2006 and 2011. Articles investigating invasive treatments published between 2006 and 2012 and those investigating noninvasive pharmacological treatments published in 2012 were identified by M.R.W. and J.S.G. 2.2. Data extraction A coding manual was developed to investigate questions pertaining to the reporting of the primary outcome variable, primary statistical analysis plan, and adjustment for multiple testing in the primary analysis, when applicable. The coding manual was pretested and modified by coding 2 rounds of 3 and 2 rounds of 2 (total = 10) articles published in the 3 journals between 2000 and 2003. In each round, 3 authors (J.S.G., A.M., and M.H.) coded each article separately and discussed any discrepancies. The manual was modified for clarity in response to these discussions. Two authors (J.S.G. and A.M.) coded the 2006–2012 articles according to this manual, which instructed coders to search the Methods and
Results sections of the articles. The order of the articles was randomized for each coder. After the articles were coded independently, the coders reviewed the data for discrepancies. Discrepancies due to oversight were corrected and discrepancies due to alternative interpretations were discussed to arrive at a consensus. J.S.G. performed a post hoc review of the Discussion sections of the 18 articles that identified multiple primary analyses and did not mention any adjustment for multiplicity to determine if the authors noted multiple comparisons as a limitation of interpreting the analyses. She also performed a post hoc search of all articles to determine if the authors declared the trials exploratory in any section of the manuscripts. Journal, study design, and number of randomized participants were also extracted for each trial. In addition, information regarding trial type (experimental pain model vs clinical pain condition), trial size (P or
