Journal of Clinical Pharmacy and Therapeutics, 2014
doi: 10.1111/jcpt.12172
Biomedical journals lack a consistent method to detect outcome reporting bias: a cross-sectional analysis L. N. Huan* Bsc(Pharm) ACPR, A. M. Tejani† Bsc(Pharm) PharmD and G. Egan* Bsc(Pharm) ACPR *Lower Mainland Pharmacy Services, Richmond General Hospital, Pharmacy, Richmond, BC, and †Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
Received 5 March 2014, Accepted 14 April 2014
Keywords: evidence-based medicine, evidence-based pharmacotherapy, evidence-based practice
important form of bias which is underappreciated despite its widespread prevalence and potential to undermine evidencebased medicine by skewing the availability of outcome data. Evidence has suggested that up to 40–62% of randomized controlled trials (RCTs) changed, introduced or omitted at least one primary outcome.2 This was coupled with a strong association (OR 22 to 47) for publication of outcomes which were statistically significant.2 The impact of ORB extends beyond RCTs, heavily influencing secondary research. The incomplete availability of outcomes for included trials has been shown to change the magnitude and direction of the determined effect.3,4 After adjusting for ORB in Cochrane Systematic Reviews, 8 of 42 (19%) previously statistically significant reviews became non-significant, whereas another 11 of 42 (26%) had a reduced treatment effect of 20% or more.3 Another study re-examined the results of 42 meta-analyses by including unpublished trial outcome data, finding 19 of 41 studies (46%) had a decrease in efficacy and another 19 of 41 studies (46%) had an increase in efficacy depending on the drug or outcome examined.4 The effect of ORB is not limited to efficacy endpoints but also includes safety outcomes. A prime example is the association of angiotensin receptor blockers with the incidence of cancer, a finding arising due to the presence of ORB in the included articles.5 One effort to increase the transparency of the results was to introduce a clinical trial registry to allow for greater public access to results and design.6 However, registration has been unable to eliminate deviations or changes from the submitted protocol.7,8 One study has identified that 19 of 23 (826%) included studies had discrepancies between the protocol and the publication despite trial registration.7 These discrepancies led to a favouring of statistically significant results or omission of non-significant primary outcomes.7 Another study analysing RCTs published in ICMJE journals found changes to the primary outcome (n = 21/ 123 RCTs, 17%) and key secondary outcomes (n = 19/123 RCTs, 15%) occurred frequently.8 In addition, many registration entries were missing fields such as key secondary outcomes (67/152 RCTs, 441%) and primary outcomes (n = 59/152 RCTs, 388%).8 However, it has been identified that trial investigators may not comprehend the importance and implications of not reporting all outcomes and changes to a protocol.1,8 Fortunately, ORB is gaining widespread attention, and in recent years, an assessment for ORB has even been incorporated into the Cochrane risk of bias assessment.9 Tutorials on how to assess and compensate for ORB when conducting meta-analyses or reviews have also been emerging.10,11 Despite this increased attention, as little as one-third of peer reviewers are examining the trial registration for discrepancies.12
SUMMARY What is known and objective: An increasing amount of recently published literature has implicated outcome reporting bias (ORB) as a major contributor to skewing data in both randomized controlled trials and systematic reviews; however, little is known about the current methods in place to detect ORB. This study aims to gain insight into the detection and management of ORB by biomedical journals. Methods: This was a cross-sectional analysis involving standardized questions via email or telephone with the top 30 biomedical journals (2012) ranked by impact factor. The Cochrane Database of Systematic Reviews was excluded leaving 29 journals in the sample. Results: Of 29 journals, 24 (83%) responded to our initial inquiry of which 14 (58%) answered our questions and 10 (42%) declined participation. Five (36%) of the responding journals indicated they had a specific method to detect ORB, whereas 9 (64%) did not have a specific method in place. The prevalence of ORB in the review process seemed to differ with 4 (29%) journals indicating ORB was found commonly, whereas 7 (50%) indicated ORB was uncommon or never detected by their journal previously. The majority (n = 10/14, 72%) of journals were unwilling to report or make discrepancies found in manuscripts available to the public. Although the minority, there were some journals (n = 4/14, 29%) which described thorough methods to detect ORB. What is new and conclusion: Many journals seemed to lack a method with which to detect ORB and its estimated prevalence was much lower than that reported in literature suggesting inadequate detection. There exists a potential for overestimation of treatment effects of interventions and unclear risks. Fortunately, there are journals within this sample which appear to utilize comprehensive methods for detection of ORB, but overall, the data suggest improvements at the biomedical journal level for detecting and minimizing the effect of this bias are needed. WHAT IS KNOWN AND OBJECTIVE Outcome reporting bias (ORB) refers to the publication of a subset of outcomes originally identified in the study protocol.1 This is an Correspondence: L. N. Huan, Lower Mainland Pharmacy Services, Richmond General Hospital, Pharmacy, 7000 Westminister Highway, Richmond, BC, Canada V6X 1A2. Tel.: 604 278 9711 (ext: 4819); fax: 604 244 5195; e-mail: [email protected]
© 2014 John Wiley & Sons Ltd
1
L. N. Huan et al.
Biomedical journals and outcome reporting bias
Thus, due to the prevalence and serious implications of ORB, coupled with a lack of awareness and effective processes in place to minimize its effects, we sought to discover a potential deficiency in our current pathway for bias assessment of literature. The focus of our research was biomedical journals as no literature currently examines how they are detecting and managing ORB. We aimed to analyse the current practices undertaken by the top 30 biomedical journals ranked by impact factor to identify and minimize the incidence of ORB and to propose a practice which if implemented can minimize the occurrence of ORB.
Data analysis We utilized a concurrent data collection method, and descriptive statistics were used to analyse question responses. In a post hoc analysis of the results, a mixed-methods approach was employed, grouping responses into categories (See Table 2) to search for trends, looking for potential deficiencies or adequacies in the review process. Categorization of the responses was performed independently by two investigators, and where discrepancies arose, discussion was held until consensus could be achieved for appropriate classification. Initial responses from biomedical journals were low, and there was concern by some editors about the anonymity of responses. Thus, journals were informed their responses would be kept anonymous in publication with journals listed randomly as journals 1 through 29 without an identifiable name.
METHODS Study design and participants This research study consisted of a prospective, cross-sectional analysis via email and telephone with biomedical journals. Biomedical journals eligible for inclusion in this study included the top 30 biomedical journals ranked by impact factor in the overall medicine category based on impactfactor.weebly.com/ medicine.html as of August 2012. Of these thirty journals, the Cochrane Database of Systematic Reviews was excluded as they are comprised of over fifty different entities, each of which would have to be interviewed separately for their processes. This left a total of 29 biomedical journals available for contact which was chosen as a convenience sample. No other inclusion or exclusion criteria were specified.
RESULTS Journal participants There were 29 biomedical journals eligible for inclusion in this study. Of these, 24 (83%) journals responded to our study inquiry, whereas 5 (17%) did not provide a response. Of the responding journals, 14 (58%) provided answers to our questions, whereas 10 (42%) declined participation in the study. To obtain these replies, it took on average 18 emails and 08 phone calls per journal, not including inquiries for clarification or forwarding to different journal personnel.
Procedures Email and telephone inquiries were conducted with editor-inchiefs, editors or administrators of the remaining top 29 biomedical journals ranked by impact factor. Contact information was procured online or via phone call. An email was sent to the corresponding representative who, upon agreement, would participate in the study. This involved the administration of a standard set of questions regarding the journal’s policies and procedures (See Table 1). We created and refined these questions, aiming to assess the biomedical journal’s identification and management of ORB. These questions could be administered either by email or telephone, and all inquiries were conducted either by the principal investigator or research assistant under guidance. If an incomplete response or misinterpretation of the question was received, another email was sent to elucidate discrepancies and procure missing information. If at any time there was no response received from the biomedical journal for more than 2 weeks, another email was sent. If no response was still received after 2 weeks, phone contact was attempted to a maximum of three times. If still no response was achieved, the biomedical journal was marked as ‘non-response’.
Survey responses Five (36%) of the 14 responding journals indicated they had a specific method to detect ORB, whereas the other 9 (64%) journals did not. The methods for detecting ORB by 4 of the 5 journals included a comparison of submission outcomes to the protocol or trial registration documents conducted by both editors and peer reviewers for the majority or all submissions. This was specified in their responses to target randomized controlled trials specifically. The responsibility for cross-referencing the manuscript to the protocol belonged to various people across the journals. A total of 6 (43%) journals delegated this responsibility to both peer reviewers and editors, whereas 4 (29%) relied solely on peer reviewers. The remaining 4 (29%) journals did not indicate either peer reviewers or editors as responsible for detecting ORB. No involved journal kept statistics on the frequency of detection of ORB. Varied estimations were received (numerical and/or descriptive) which were categorized broadly as per Table 2. Fifty percent (n = 7/14) of journals indicated ORB is found uncommonly or was never detected by their journal. The other 4/14 (29%) journals indicated ORB was found commonly, and 3/14
Table 1. Standard set of questions posed to the biomedical journals Question Question Question Question Question
1: 2: 3: 4: 5:
Does your journal have a specific method with which to detect outcome reporting bias? If so, what is this method? Does your journal ALWAYS compare submitted manuscripts to their registered protocols or do you rely on peer reviewers? How often is outcome reporting bias detected from submitted manuscripts? If outcome reporting bias is detected, what is done to address it? Are the discrepancies that your journal identifies reported or made public? Would you be willing to release this information upon request?
© 2014 John Wiley & Sons Ltd
Journal of Clinical Pharmacy and Therapeutics, 2014 2
L. N. Huan et al.
Biomedical journals and outcome reporting bias
Table 2. Categorization of responses from biomedical journals Question 1:
The journal indicates they have a specific method or describes a specific method for detecting outcome reporting bias or the journal indicates they do not have a specific method for detecting outcome reporting bias. Comparison of submissions to protocols is dependent on peer reviewers and editors, solely on peer reviewers, solely on editors or no parties are employed to look for outcome reporting bias. Outcome reporting bias occurs commonly (≥10%), uncommonly (
doi: 10.1111/jcpt.12172
Biomedical journals lack a consistent method to detect outcome reporting bias: a cross-sectional analysis L. N. Huan* Bsc(Pharm) ACPR, A. M. Tejani† Bsc(Pharm) PharmD and G. Egan* Bsc(Pharm) ACPR *Lower Mainland Pharmacy Services, Richmond General Hospital, Pharmacy, Richmond, BC, and †Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
Received 5 March 2014, Accepted 14 April 2014
Keywords: evidence-based medicine, evidence-based pharmacotherapy, evidence-based practice
important form of bias which is underappreciated despite its widespread prevalence and potential to undermine evidencebased medicine by skewing the availability of outcome data. Evidence has suggested that up to 40–62% of randomized controlled trials (RCTs) changed, introduced or omitted at least one primary outcome.2 This was coupled with a strong association (OR 22 to 47) for publication of outcomes which were statistically significant.2 The impact of ORB extends beyond RCTs, heavily influencing secondary research. The incomplete availability of outcomes for included trials has been shown to change the magnitude and direction of the determined effect.3,4 After adjusting for ORB in Cochrane Systematic Reviews, 8 of 42 (19%) previously statistically significant reviews became non-significant, whereas another 11 of 42 (26%) had a reduced treatment effect of 20% or more.3 Another study re-examined the results of 42 meta-analyses by including unpublished trial outcome data, finding 19 of 41 studies (46%) had a decrease in efficacy and another 19 of 41 studies (46%) had an increase in efficacy depending on the drug or outcome examined.4 The effect of ORB is not limited to efficacy endpoints but also includes safety outcomes. A prime example is the association of angiotensin receptor blockers with the incidence of cancer, a finding arising due to the presence of ORB in the included articles.5 One effort to increase the transparency of the results was to introduce a clinical trial registry to allow for greater public access to results and design.6 However, registration has been unable to eliminate deviations or changes from the submitted protocol.7,8 One study has identified that 19 of 23 (826%) included studies had discrepancies between the protocol and the publication despite trial registration.7 These discrepancies led to a favouring of statistically significant results or omission of non-significant primary outcomes.7 Another study analysing RCTs published in ICMJE journals found changes to the primary outcome (n = 21/ 123 RCTs, 17%) and key secondary outcomes (n = 19/123 RCTs, 15%) occurred frequently.8 In addition, many registration entries were missing fields such as key secondary outcomes (67/152 RCTs, 441%) and primary outcomes (n = 59/152 RCTs, 388%).8 However, it has been identified that trial investigators may not comprehend the importance and implications of not reporting all outcomes and changes to a protocol.1,8 Fortunately, ORB is gaining widespread attention, and in recent years, an assessment for ORB has even been incorporated into the Cochrane risk of bias assessment.9 Tutorials on how to assess and compensate for ORB when conducting meta-analyses or reviews have also been emerging.10,11 Despite this increased attention, as little as one-third of peer reviewers are examining the trial registration for discrepancies.12
SUMMARY What is known and objective: An increasing amount of recently published literature has implicated outcome reporting bias (ORB) as a major contributor to skewing data in both randomized controlled trials and systematic reviews; however, little is known about the current methods in place to detect ORB. This study aims to gain insight into the detection and management of ORB by biomedical journals. Methods: This was a cross-sectional analysis involving standardized questions via email or telephone with the top 30 biomedical journals (2012) ranked by impact factor. The Cochrane Database of Systematic Reviews was excluded leaving 29 journals in the sample. Results: Of 29 journals, 24 (83%) responded to our initial inquiry of which 14 (58%) answered our questions and 10 (42%) declined participation. Five (36%) of the responding journals indicated they had a specific method to detect ORB, whereas 9 (64%) did not have a specific method in place. The prevalence of ORB in the review process seemed to differ with 4 (29%) journals indicating ORB was found commonly, whereas 7 (50%) indicated ORB was uncommon or never detected by their journal previously. The majority (n = 10/14, 72%) of journals were unwilling to report or make discrepancies found in manuscripts available to the public. Although the minority, there were some journals (n = 4/14, 29%) which described thorough methods to detect ORB. What is new and conclusion: Many journals seemed to lack a method with which to detect ORB and its estimated prevalence was much lower than that reported in literature suggesting inadequate detection. There exists a potential for overestimation of treatment effects of interventions and unclear risks. Fortunately, there are journals within this sample which appear to utilize comprehensive methods for detection of ORB, but overall, the data suggest improvements at the biomedical journal level for detecting and minimizing the effect of this bias are needed. WHAT IS KNOWN AND OBJECTIVE Outcome reporting bias (ORB) refers to the publication of a subset of outcomes originally identified in the study protocol.1 This is an Correspondence: L. N. Huan, Lower Mainland Pharmacy Services, Richmond General Hospital, Pharmacy, 7000 Westminister Highway, Richmond, BC, Canada V6X 1A2. Tel.: 604 278 9711 (ext: 4819); fax: 604 244 5195; e-mail: [email protected]
© 2014 John Wiley & Sons Ltd
1
L. N. Huan et al.
Biomedical journals and outcome reporting bias
Thus, due to the prevalence and serious implications of ORB, coupled with a lack of awareness and effective processes in place to minimize its effects, we sought to discover a potential deficiency in our current pathway for bias assessment of literature. The focus of our research was biomedical journals as no literature currently examines how they are detecting and managing ORB. We aimed to analyse the current practices undertaken by the top 30 biomedical journals ranked by impact factor to identify and minimize the incidence of ORB and to propose a practice which if implemented can minimize the occurrence of ORB.
Data analysis We utilized a concurrent data collection method, and descriptive statistics were used to analyse question responses. In a post hoc analysis of the results, a mixed-methods approach was employed, grouping responses into categories (See Table 2) to search for trends, looking for potential deficiencies or adequacies in the review process. Categorization of the responses was performed independently by two investigators, and where discrepancies arose, discussion was held until consensus could be achieved for appropriate classification. Initial responses from biomedical journals were low, and there was concern by some editors about the anonymity of responses. Thus, journals were informed their responses would be kept anonymous in publication with journals listed randomly as journals 1 through 29 without an identifiable name.
METHODS Study design and participants This research study consisted of a prospective, cross-sectional analysis via email and telephone with biomedical journals. Biomedical journals eligible for inclusion in this study included the top 30 biomedical journals ranked by impact factor in the overall medicine category based on impactfactor.weebly.com/ medicine.html as of August 2012. Of these thirty journals, the Cochrane Database of Systematic Reviews was excluded as they are comprised of over fifty different entities, each of which would have to be interviewed separately for their processes. This left a total of 29 biomedical journals available for contact which was chosen as a convenience sample. No other inclusion or exclusion criteria were specified.
RESULTS Journal participants There were 29 biomedical journals eligible for inclusion in this study. Of these, 24 (83%) journals responded to our study inquiry, whereas 5 (17%) did not provide a response. Of the responding journals, 14 (58%) provided answers to our questions, whereas 10 (42%) declined participation in the study. To obtain these replies, it took on average 18 emails and 08 phone calls per journal, not including inquiries for clarification or forwarding to different journal personnel.
Procedures Email and telephone inquiries were conducted with editor-inchiefs, editors or administrators of the remaining top 29 biomedical journals ranked by impact factor. Contact information was procured online or via phone call. An email was sent to the corresponding representative who, upon agreement, would participate in the study. This involved the administration of a standard set of questions regarding the journal’s policies and procedures (See Table 1). We created and refined these questions, aiming to assess the biomedical journal’s identification and management of ORB. These questions could be administered either by email or telephone, and all inquiries were conducted either by the principal investigator or research assistant under guidance. If an incomplete response or misinterpretation of the question was received, another email was sent to elucidate discrepancies and procure missing information. If at any time there was no response received from the biomedical journal for more than 2 weeks, another email was sent. If no response was still received after 2 weeks, phone contact was attempted to a maximum of three times. If still no response was achieved, the biomedical journal was marked as ‘non-response’.
Survey responses Five (36%) of the 14 responding journals indicated they had a specific method to detect ORB, whereas the other 9 (64%) journals did not. The methods for detecting ORB by 4 of the 5 journals included a comparison of submission outcomes to the protocol or trial registration documents conducted by both editors and peer reviewers for the majority or all submissions. This was specified in their responses to target randomized controlled trials specifically. The responsibility for cross-referencing the manuscript to the protocol belonged to various people across the journals. A total of 6 (43%) journals delegated this responsibility to both peer reviewers and editors, whereas 4 (29%) relied solely on peer reviewers. The remaining 4 (29%) journals did not indicate either peer reviewers or editors as responsible for detecting ORB. No involved journal kept statistics on the frequency of detection of ORB. Varied estimations were received (numerical and/or descriptive) which were categorized broadly as per Table 2. Fifty percent (n = 7/14) of journals indicated ORB is found uncommonly or was never detected by their journal. The other 4/14 (29%) journals indicated ORB was found commonly, and 3/14
Table 1. Standard set of questions posed to the biomedical journals Question Question Question Question Question
1: 2: 3: 4: 5:
Does your journal have a specific method with which to detect outcome reporting bias? If so, what is this method? Does your journal ALWAYS compare submitted manuscripts to their registered protocols or do you rely on peer reviewers? How often is outcome reporting bias detected from submitted manuscripts? If outcome reporting bias is detected, what is done to address it? Are the discrepancies that your journal identifies reported or made public? Would you be willing to release this information upon request?
© 2014 John Wiley & Sons Ltd
Journal of Clinical Pharmacy and Therapeutics, 2014 2
L. N. Huan et al.
Biomedical journals and outcome reporting bias
Table 2. Categorization of responses from biomedical journals Question 1:
The journal indicates they have a specific method or describes a specific method for detecting outcome reporting bias or the journal indicates they do not have a specific method for detecting outcome reporting bias. Comparison of submissions to protocols is dependent on peer reviewers and editors, solely on peer reviewers, solely on editors or no parties are employed to look for outcome reporting bias. Outcome reporting bias occurs commonly (≥10%), uncommonly (
