The Journal of Pain, Vol 16, No 3 (March), 2015: pp 199-206 Available online at www.jpain.org and www.sciencedirect.com
Critical Review Reporting of Sample Size Calculations in Analgesic Clinical Trials: ACTTION Systematic Review Andrew McKeown,* Jennifer S. Gewandter,* Michael P. McDermott,y,z,{{ Joseph R. Pawlowski,* Joseph J. Poli,* Daniel Rothstein,* John T. Farrar,x Ian Gilron,{ Nathaniel P. Katz,jj,** Allison H. Lin,yy Bob A. Rappaport,yy Michael C. Rowbotham,zz Dennis C. Turk,xx Robert H. Dworkin,*,z,{{ and Shannon M. Smith* Departments of *Anesthesiology, yBiostatistics and Computational Biology, zNeurology, and {{Center for Human Experimental Therapeutics, University of Rochester School of Medicine and Dentistry, Rochester, New York. x University of Pennsylvania, Philadelphia, Pennsylvania. { Queen’s University, Kingston, Ontario, Canada. jj Analgesic Solutions, Natick, Massachusetts. **Department of Anesthesiology, Tufts University, Boston, Massachusetts. yy Center for Drug Evaluation and Research, United States Food and Drug Administration, Silver Spring, Maryland. zz California Pacific Medical Center, San Francisco, California. xx Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, Washington.
Abstract: Sample size calculations determine the number of participants required to have sufficiently high power to detect a given treatment effect. In this review, we examined the reporting quality of sample size calculations in 172 publications of double-blind randomized controlled trials of noninvasive pharmacologic or interventional (ie, invasive) pain treatments published in European Journal of Pain, Journal of Pain, and Pain from January 2006 through June 2013. Sixty-five percent of publications reported a sample size calculation but only 38% provided all elements required to replicate the calculated sample size. In publications reporting at least 1 element, 54% provided a justification for the treatment effect used to calculate sample size, and 24% of studies with continuous outcome variables justified the variability estimate. Publications of clinical pain condition trials reported a sample size calculation more frequently than experimental pain model trials (77% vs 33%, P < .001) but did not differ in the frequency of reporting all required elements. No significant differences in reporting of any or all elements were detected between publications of trials with industry and nonindustry sponsorship. Twenty-eight percent included a discrepancy between the reported number of planned and randomized participants. This study suggests that sample size calculation reporting in analgesic trial publications is usually incomplete. Investigators should provide detailed accounts of sample size calculations in publications of clinical trials of pain treatments, which is necessary for reporting transparency and communication of pre-trial design decisions. Perspective: In this systematic review of analgesic clinical trials, sample size calculations and the required elements (eg, treatment effect to be detected; power level) were incompletely reported.
The views expressed in this article are those of the authors and no official endorsement by the Food and Drug Administration (FDA) or the pharmaceutical and device companies that provided unrestricted grants to support the activities of the Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) public-private partnership should be inferred. Financial support for this project was provided by the ACTTION public-private partnership which has received research contracts, grants, or other revenue from the FDA (grant no. U01 FD004187), multiple pharmaceutical and device companies, and other sources.
Supplementary data accompanying this article are available online at www.jpain.org and www.sciencedirect.com. Address reprint requests to Shannon M. Smith, PhD, Department of Anesthesiology, University of Rochester School of Medicine and Dentistry, 601 Elmwood Ave, Box 604, Rochester, NY 14642. E-mail: shannon1_smith@ urmc.rochester.edu 1526-5900/$36.00 ª 2015 by the American Pain Society http://dx.doi.org/10.1016/j.jpain.2014.11.010
199
200
Reporting of Sample Size Calculations
The Journal of Pain
A lack of transparency regarding sample size calculations may raise questions about the appropriateness of the calculated sample size. ª 2015 by the American Pain Society Key words: Sample size, power, pain research.
S
ample size calculations use information about study design and estimates of nuisance parameters such as the variability of the outcome measure, the event rate in the control group, or the anticipated attrition rate to approximate the number of participants necessary to provide sufficiently high power to detect a given treatment effect. An adequately determined and reported sample size calculation should provide information about primary outcome variables17 and analyses, stating whether these were pre-specified by the investigators during the planning and design of the trial. Additionally, reported sample size calculations should describe any pre-specified adjustments made to the overall significance level to account for multiple primary comparisons and interim analyses. Failure to transparently report a sample size calculation may indicate other methodological problems with a trial,20,23 which prompted the Consolidated Standards of Reporting Trials (CONSORT) organization to recommend transparent reporting of all essential elements necessary to calculate sample size along with accompanying assumptions and justifications.19 Sample size is related to power (ie, the probability that a study will reject the null hypothesis of the absence of a treatment effect when one truly exists) and thus has potential ethical implications.1,13 For example, assuming an unrealistically large treatment effect in a sample size calculation yields a smaller sample size but may result in inadequate power to detect a smaller and perhaps more realistic treatment effect that might still be clinically important.1,13 This exposes participants to risks in a trial that might not be adequately designed to answer the primary question. Conversely, assuming a small but not clinically meaningful treatment effect in a sample size calculation will yield a large sample size, exposing more participants to risks than is necessary to answer the primary question. Transparent reporting of sample size calculations allows readers to independently evaluate these important considerations in the design of clinical trials. Previous reviews of the general medical literature3,16 found inadequate reporting of sample size calculations in published randomized clinical trials (RCTs). The objective of the current systematic review is to describe the extent to which sample size calculations are reported comprehensively in analgesic RCTs published in the pain literature (see Table 1 for a summary of the key elements of a sample size calculation). We also examined whether justifications were provided for the assumptions underlying the sample size calculations such as the magnitude of the treatment effect and the measure of variability of the outcome measure. Further, this review builds on previous research by including trials with crossover designs in addition to parallel group
designs, trials with more than 2 treatment groups, and trials that did not specify a primary outcome variable. Additionally, we compared the quality of sample size calculation reporting between industry- and nonindustry-sponsored trials as well as between trials in clinical pain conditions and those in experimental pain models.
Method Article Selection The selection process and inclusion criteria for articles have been previously reported.10,11 Briefly, the sample included primary publications of randomized, doubleblind clinical trials of pharmacologic and interventional (ie, invasive)6 pain treatments published from January 2006 through June 2013 in 3 prominent pain journals (ie, European Journal of Pain, Journal of Pain, and Pain).
Data Extraction A preliminary coding manual was developed by A.M. and revised with feedback and discussion with R.H.D., S.M.S., J.S.G., and M.P.M. Three authors (A.M., S.M.S., and J.S.G.) independently coded a set of 8 training articles. The training articles were chosen to be similar to the sample articles but did not meet the inclusion criteria for this review (eg, were published earlier than 2006 or were published in other journals). Discrepancies were discussed and the coding manual was further revised for clarity. For each article, we recorded trial characteristics including trial design, number of treatment arms or periods, and type of pain being studied (ie, acute, chronic, recurrent, experimental). From the articles’ Methods sections, we recorded whether the article reported a sample size calculation or power analysis, and if so, the statistical test on which it was based. All sample size calculation elements reported in the primary publication were recorded (ie, planned sample size [N], inflation of planned sample size due to anticipated attrition rate or multiple comparisons [adjustment percent or adjusted N], significance level [including 1- or 2-tailed], power, treatment effect to be detected [eg, for continuous outcomes, the difference between group means; for dichotomous outcomes, the proportions in each treatment group], estimated variability [for continuous outcomes]). If justifications were given for the assumptions of the treatment effect to be detected, the variability of continuous outcomes, or planned sample size inflation, those justifications were recorded. From the articles’ Results section or CONSORT diagram, the actual randomized sample size was recorded. If no explicit statement was made regarding the number of participants randomized
McKeown et al Table 1.
The Journal of Pain
201
Elements of a Sample Size Calculation ELEMENT
Significance levely Powery Treatment effect to be detected Treatment group difference*,y
Variability (for continuous outcome measures)*,y Planned sample size inflation
DESCRIPTION Probability of a type I error (ie, falsely rejecting the null hypothesis of no treatment effect); conventionally set at .05 Probability of rejecting the null hypothesis of no treatment effect given a specified magnitude of the true treatment effect20; conventionally set at .80 or .90 Assumed proportions in treatment and control groups for a binary outcome variable,17,19 or the difference in mean response between the treatment groups for a continuous outcome variable21 Assumed standard deviations in treatment and control groups or their pooled standard deviation17,19 Additional participants required to accommodate possible participant withdrawal or noncompliance
*Note: For sample size calculations involving continuous outcome measures, treatment group difference can be expressed as a ratio of the difference in group means to the pooled standard deviation, yielding a standardized effect size. However, solely reporting the standardized effect size does not provide full transparency regarding the relative contribution of these 2 components, and we recommend reporting both the difference in group means and the pooled standard deviation. yNote: CONSORT required elements of a sample size calculation.17
and no CONSORT diagram was provided, the number was taken from implicit statements (eg, number ‘‘enrolled’’ or ‘‘included’’). When articles reported a sample size calculation, we recorded whether the actual number of randomized participants differed by $10% from the planned number of participants for reasons other than stopping after an interim analysis, and if so, whether an explanation for the discrepancy was reported. The articles were organized into 2 randomized lists and each article was coded by 2 independent coders. One author (A.M.) coded all articles and 3 other authors (J.R.P., J.J.P., and D.R.) coded one-third each. Trials were classified as having industry sponsorship (ie, funded entirely or in part by industry) or as funded by nonindustry sources (ie, governmental, institutional, or failed to disclose financial support). Articles were further categorized as reporting either experimental pain models (eg, studies including experimentally-induced pain, healthy volunteers, or laboratory quantitative sensory testing) or clinical pain conditions. Trials were categorized as having
Critical Review Reporting of Sample Size Calculations in Analgesic Clinical Trials: ACTTION Systematic Review Andrew McKeown,* Jennifer S. Gewandter,* Michael P. McDermott,y,z,{{ Joseph R. Pawlowski,* Joseph J. Poli,* Daniel Rothstein,* John T. Farrar,x Ian Gilron,{ Nathaniel P. Katz,jj,** Allison H. Lin,yy Bob A. Rappaport,yy Michael C. Rowbotham,zz Dennis C. Turk,xx Robert H. Dworkin,*,z,{{ and Shannon M. Smith* Departments of *Anesthesiology, yBiostatistics and Computational Biology, zNeurology, and {{Center for Human Experimental Therapeutics, University of Rochester School of Medicine and Dentistry, Rochester, New York. x University of Pennsylvania, Philadelphia, Pennsylvania. { Queen’s University, Kingston, Ontario, Canada. jj Analgesic Solutions, Natick, Massachusetts. **Department of Anesthesiology, Tufts University, Boston, Massachusetts. yy Center for Drug Evaluation and Research, United States Food and Drug Administration, Silver Spring, Maryland. zz California Pacific Medical Center, San Francisco, California. xx Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, Washington.
Abstract: Sample size calculations determine the number of participants required to have sufficiently high power to detect a given treatment effect. In this review, we examined the reporting quality of sample size calculations in 172 publications of double-blind randomized controlled trials of noninvasive pharmacologic or interventional (ie, invasive) pain treatments published in European Journal of Pain, Journal of Pain, and Pain from January 2006 through June 2013. Sixty-five percent of publications reported a sample size calculation but only 38% provided all elements required to replicate the calculated sample size. In publications reporting at least 1 element, 54% provided a justification for the treatment effect used to calculate sample size, and 24% of studies with continuous outcome variables justified the variability estimate. Publications of clinical pain condition trials reported a sample size calculation more frequently than experimental pain model trials (77% vs 33%, P < .001) but did not differ in the frequency of reporting all required elements. No significant differences in reporting of any or all elements were detected between publications of trials with industry and nonindustry sponsorship. Twenty-eight percent included a discrepancy between the reported number of planned and randomized participants. This study suggests that sample size calculation reporting in analgesic trial publications is usually incomplete. Investigators should provide detailed accounts of sample size calculations in publications of clinical trials of pain treatments, which is necessary for reporting transparency and communication of pre-trial design decisions. Perspective: In this systematic review of analgesic clinical trials, sample size calculations and the required elements (eg, treatment effect to be detected; power level) were incompletely reported.
The views expressed in this article are those of the authors and no official endorsement by the Food and Drug Administration (FDA) or the pharmaceutical and device companies that provided unrestricted grants to support the activities of the Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) public-private partnership should be inferred. Financial support for this project was provided by the ACTTION public-private partnership which has received research contracts, grants, or other revenue from the FDA (grant no. U01 FD004187), multiple pharmaceutical and device companies, and other sources.
Supplementary data accompanying this article are available online at www.jpain.org and www.sciencedirect.com. Address reprint requests to Shannon M. Smith, PhD, Department of Anesthesiology, University of Rochester School of Medicine and Dentistry, 601 Elmwood Ave, Box 604, Rochester, NY 14642. E-mail: shannon1_smith@ urmc.rochester.edu 1526-5900/$36.00 ª 2015 by the American Pain Society http://dx.doi.org/10.1016/j.jpain.2014.11.010
199
200
Reporting of Sample Size Calculations
The Journal of Pain
A lack of transparency regarding sample size calculations may raise questions about the appropriateness of the calculated sample size. ª 2015 by the American Pain Society Key words: Sample size, power, pain research.
S
ample size calculations use information about study design and estimates of nuisance parameters such as the variability of the outcome measure, the event rate in the control group, or the anticipated attrition rate to approximate the number of participants necessary to provide sufficiently high power to detect a given treatment effect. An adequately determined and reported sample size calculation should provide information about primary outcome variables17 and analyses, stating whether these were pre-specified by the investigators during the planning and design of the trial. Additionally, reported sample size calculations should describe any pre-specified adjustments made to the overall significance level to account for multiple primary comparisons and interim analyses. Failure to transparently report a sample size calculation may indicate other methodological problems with a trial,20,23 which prompted the Consolidated Standards of Reporting Trials (CONSORT) organization to recommend transparent reporting of all essential elements necessary to calculate sample size along with accompanying assumptions and justifications.19 Sample size is related to power (ie, the probability that a study will reject the null hypothesis of the absence of a treatment effect when one truly exists) and thus has potential ethical implications.1,13 For example, assuming an unrealistically large treatment effect in a sample size calculation yields a smaller sample size but may result in inadequate power to detect a smaller and perhaps more realistic treatment effect that might still be clinically important.1,13 This exposes participants to risks in a trial that might not be adequately designed to answer the primary question. Conversely, assuming a small but not clinically meaningful treatment effect in a sample size calculation will yield a large sample size, exposing more participants to risks than is necessary to answer the primary question. Transparent reporting of sample size calculations allows readers to independently evaluate these important considerations in the design of clinical trials. Previous reviews of the general medical literature3,16 found inadequate reporting of sample size calculations in published randomized clinical trials (RCTs). The objective of the current systematic review is to describe the extent to which sample size calculations are reported comprehensively in analgesic RCTs published in the pain literature (see Table 1 for a summary of the key elements of a sample size calculation). We also examined whether justifications were provided for the assumptions underlying the sample size calculations such as the magnitude of the treatment effect and the measure of variability of the outcome measure. Further, this review builds on previous research by including trials with crossover designs in addition to parallel group
designs, trials with more than 2 treatment groups, and trials that did not specify a primary outcome variable. Additionally, we compared the quality of sample size calculation reporting between industry- and nonindustry-sponsored trials as well as between trials in clinical pain conditions and those in experimental pain models.
Method Article Selection The selection process and inclusion criteria for articles have been previously reported.10,11 Briefly, the sample included primary publications of randomized, doubleblind clinical trials of pharmacologic and interventional (ie, invasive)6 pain treatments published from January 2006 through June 2013 in 3 prominent pain journals (ie, European Journal of Pain, Journal of Pain, and Pain).
Data Extraction A preliminary coding manual was developed by A.M. and revised with feedback and discussion with R.H.D., S.M.S., J.S.G., and M.P.M. Three authors (A.M., S.M.S., and J.S.G.) independently coded a set of 8 training articles. The training articles were chosen to be similar to the sample articles but did not meet the inclusion criteria for this review (eg, were published earlier than 2006 or were published in other journals). Discrepancies were discussed and the coding manual was further revised for clarity. For each article, we recorded trial characteristics including trial design, number of treatment arms or periods, and type of pain being studied (ie, acute, chronic, recurrent, experimental). From the articles’ Methods sections, we recorded whether the article reported a sample size calculation or power analysis, and if so, the statistical test on which it was based. All sample size calculation elements reported in the primary publication were recorded (ie, planned sample size [N], inflation of planned sample size due to anticipated attrition rate or multiple comparisons [adjustment percent or adjusted N], significance level [including 1- or 2-tailed], power, treatment effect to be detected [eg, for continuous outcomes, the difference between group means; for dichotomous outcomes, the proportions in each treatment group], estimated variability [for continuous outcomes]). If justifications were given for the assumptions of the treatment effect to be detected, the variability of continuous outcomes, or planned sample size inflation, those justifications were recorded. From the articles’ Results section or CONSORT diagram, the actual randomized sample size was recorded. If no explicit statement was made regarding the number of participants randomized
McKeown et al Table 1.
The Journal of Pain
201
Elements of a Sample Size Calculation ELEMENT
Significance levely Powery Treatment effect to be detected Treatment group difference*,y
Variability (for continuous outcome measures)*,y Planned sample size inflation
DESCRIPTION Probability of a type I error (ie, falsely rejecting the null hypothesis of no treatment effect); conventionally set at .05 Probability of rejecting the null hypothesis of no treatment effect given a specified magnitude of the true treatment effect20; conventionally set at .80 or .90 Assumed proportions in treatment and control groups for a binary outcome variable,17,19 or the difference in mean response between the treatment groups for a continuous outcome variable21 Assumed standard deviations in treatment and control groups or their pooled standard deviation17,19 Additional participants required to accommodate possible participant withdrawal or noncompliance
*Note: For sample size calculations involving continuous outcome measures, treatment group difference can be expressed as a ratio of the difference in group means to the pooled standard deviation, yielding a standardized effect size. However, solely reporting the standardized effect size does not provide full transparency regarding the relative contribution of these 2 components, and we recommend reporting both the difference in group means and the pooled standard deviation. yNote: CONSORT required elements of a sample size calculation.17
and no CONSORT diagram was provided, the number was taken from implicit statements (eg, number ‘‘enrolled’’ or ‘‘included’’). When articles reported a sample size calculation, we recorded whether the actual number of randomized participants differed by $10% from the planned number of participants for reasons other than stopping after an interim analysis, and if so, whether an explanation for the discrepancy was reported. The articles were organized into 2 randomized lists and each article was coded by 2 independent coders. One author (A.M.) coded all articles and 3 other authors (J.R.P., J.J.P., and D.R.) coded one-third each. Trials were classified as having industry sponsorship (ie, funded entirely or in part by industry) or as funded by nonindustry sources (ie, governmental, institutional, or failed to disclose financial support). Articles were further categorized as reporting either experimental pain models (eg, studies including experimentally-induced pain, healthy volunteers, or laboratory quantitative sensory testing) or clinical pain conditions. Trials were categorized as having
