ORIGINAL CONTRIBUTION
Are Our Publications Failing the Inspection?: A Review of the Publications in Rectal Cancer Surgery Between 2002 and 2012 Maria C. Mora Pinzon, M.D. • Dana M. Hayden, M.D. • Darlene Ariel, B.S. Kimberly A. Bartosiak, B.S. • Michael V. Chiodo, B.S. • Konstantine Kosmidis, H.S. Ann Evans, M.D. • Theodore J. Saclarides, M.D. Division of Colon and Rectum Surgery, Department of Surgery, Loyola University Medical Center, Maywood, Illinois
BACKGROUND: Quality of publications is considered a
subjective measurement, and more weight is placed on prospective studies, especially randomized clinical trials and meta-analyses. OBJECTIVE: This study describes the type of publications and evaluates the quality of randomized clinical trials and review articles using an objective measurement. DATA SOURCES: Medline (PubMed) is the data source for
this work. STUDY SELECTION: We used the terms “rectal neoplasms/ surgery” and the filters “10 years,” “humans,” and “English.” MAIN OUTCOME MEASURES: We measured compliance with checklist items. Randomized clinical trials were reviewed using the Consolidates Standards of Reporting Trials statement; systematic reviews/metaanalyses were reviewed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. RESULTS: A total of 3603 articles were identified: 20.8% were case report/series, 20.5% were retrospective cohorts, 14.0% were reviews or meta-analyses, 16.4% were prospective cohorts, 14.0% were other types of articles (comments, letters, or editorials), 5.5% were clinical
trials (phase I/II), 4.2% were randomized clinical trials, and 4.4% were cross-sectional studies. We reviewed 108 randomized clinical trials; the maximum score possible was 74.0, the average score was 44.6 (range, 20.0-64.0), 4 (3.7%) were graded as “excellent,” 21 (19.4%) were “good,” 44 (40.7%) were “deficient,” and 39 (36.1%) were graded as “fail.” The predictors of higher scores for randomized clinical trials were year of publication after 2007 (p = 0.00), higher impact factor (p = 0.03), and declared funding (p = 0.01). Twenty-nine meta-analyses were reviewed; the average score was 19.64 (range, 12.025.0); 5 articles (17.2%) were graded as “excellent,” 12 (41.4%) were “good,” 10 (34.5%) were “deficient,” and 2 (6.9%) were “fail.” LIMITATIONS: Only 1 electronic database was used, so we lacked a validated score. In addition, the search terms did not include “colorectal.” CONCLUSIONS: A total of 20.8% of the articles published were case reports and 25.0% of the articles were prospective or clinical trials. Although randomized clinical trials and systematic reviews provide the highest level of evidence, publications with missing data limit replication of the study and affect the generalizability of results to other populations. To improve the quality of our publications, authors, reviewers, and journal editors should consider the endorsement of standardize checklists.
Financial Disclosure: None reported. Podium presentation at the meeting of The American Society of Colon and Rectal Surgeons, Phoenix, AZ, April 27 to May 1, 2013.
key words: CONSORT; PRISMA; Quality; Rectal cancer; Review.
Correspondence: Theodore J. Saclarides, M.D., 2160 S First Ave, Maywood, IL 60153. E-mail: [email protected]
R
Dis Colon Rectum 2014; 57: 983–992 DOI: 10.1097/DCR.0000000000000169 © The ASCRS 2014 Diseases of the Colon & Rectum Volume 57: 8 (2014)
ecently we have seen an increase in the number of publications that are regularly available in the surgical literature, thus increasing the amount of articles that a surgeon must read to keep up with technology and new discoveries. In general, we tend to prefer 983
984
randomized clinical trial (RCT) and review articles over retrospective or prospective cohorts, but there are few mechanisms available to determine which articles provide useful information or comply with the scientific method. To improve the reporting of RCTs in 1996, the first Consolidates Standards of Reporting Trials (CONSORT) guidelines were published1 and subsequently updated in 20012 and 2010.3,4 The aim of these guidelines was to increase the transparency of reporting and to improve the level of evidence provided by such articles. Since the introduction of CONSORT guidelines, an improvement of publications has been identified.5–7 Previous studies have reviewed the quality of reporting in surgical trials using the CONSORT statement or Jadad score, and deficiencies in reporting were common across specialties.7–12 Similar results have been described in the review of systematic reviews and meta-analysis,13,14 which led to the development of the Quality of Reporting of Meta-Analysis Statement in 1999, which was updated in 2009 as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement.15,16 Although many studies have evaluated the quality of reporting in meta-analyses,17–19 to our knowledge, this is the first that focuses on rectal cancer surgery. The purpose of our study was to describe the type of publications in rectal cancer surgery over the past 10 years and to evaluate the compliance with reporting guidelines (CONSORT and PRISMA statements) of highly evidenced publications (RCTs and systematic reviews/meta-analyses) in rectal cancer surgery. This systematic review will help to identify deficiencies inherent to rectal cancer reports, and its results can be used to improve the reports of other topics in colorectal surgery.
MATERIALS AND METHODS Our study is a systematic review of the articles published on rectal cancer surgery over the past 10 years. The study protocol was designed and performed by the authors with departmental funding, and no registration number was used. We conducted an electronic search of PubMed using the term “rectal neoplasms/surgery” (medical subject headings). Filters were “10 years,” “humans,” and “English.” The search was performed on May 26, 2012. The resulting list was downloaded to an electronic Microsoft Excel database containing the following variables: year of publication, journal name, article title, first author, affiliation, country of first author, publication type, number of participants, and country of publication. In addition, the following information was added to the database for all RCT and reviews: impact factor of the journal, number of references, number of citations, and number of institutions involved.
MORA PINZON ET AL: PUBLICATIONS IN RECTAL CANCER SURGERY
The authors decided to use PubMed with broad search categories because it is the largest and most commonly used database; therefore, a broad search would capture the most commonly requested articles in the specified topic. Any search strategy is at risk of missing articles. In our case, those can be considered randomly missed that would not affect the purpose of the study, which is to provide a general idea of the type and quality of articles published. The use of a librarian was not considered in the elaboration of the study, because the search terms were broad and did not refer to any specific pathology or treatment. All of the articles were classified according to publication type, which was obtained from the downloaded file; the titles and abstracts of those classified as “journal article” or “multi-institutional” were reviewed by 1 investigator to determine the type of study, and further classification into 1 of the following categories was performed: case reports/ series, retrospective cohorts, prospective cohorts, crosssectional, clinical trials, RCTs, population study, review or meta-analysis, and others. If there was insufficient information in the title and abstract to determine the category, the full article was reviewed. Those publications classified as RCTs, systematic reviews, or meta-analyses were downloaded and stored for further review. Articles not related to rectal cancer surgery were excluded during the classification phase of our study. The CONSORT statement published 2010 was studied by the reviewers, and the definitions of each item were noted.3,4 Secondary analyses and pooled analyses were excluded. All of the included RCTs were read by 2 independent reviewers who were not blinded to the article’s authors, institutions, and journals during the assessment and data collection. Scoring was performed using a CONSORT checklist, which consists of 27 elements; each item was scored using a 0-2 scale; 0 if the item was not reported, 1 if partially reported, or 2 if completely reported in the article. The score corresponding with each item was entered in an electronic database. All of the points were added to obtain the overall score, and a percentage score was calculated for each trial. The maximum score was 74 points (100%). The scores obtained by the reviewers were compared, and in case of incongruence, the article was re-examined by a third reviewer. We defined “review articles” as those that summarize evidence available in the literature without identification of the methodology used. “Systematic review” and “metaanalyses” were defined according to the Cochrane Handbook. A systematic review is a methodologic collection of evidence to answer a specific question through identification, selection, appraisal, collection, and analyses of data from relevant studies included in the review. A meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies.20
985
Diseases of the Colon & Rectum Volume 57: 8 (2014)
TABLE 1. Grading system
Grade A: excellent B: good C: deficient F: fail
TABLE 2. Types of publications
Checklist items, %
Numeric score for RCTs
>85 84-71 70-56 63 53-62 41-52 23 20-22 19-15 70% of the items evaluated. The remaining articles had severe deficiencies, especially in the Methods section, which is indispensable to allow duplication of the results. The most common deficiencies were sample size calculation (not reported: 39.8% of articles). Interim analyses and stopping guidelines were missing in 63.9%, and randomization technique including allocation mechanism and implementation of the randomization were missing in >50% of the articles. The CONSORT diagram was not present in 52.8% of the articles, and funding was not reported in 52.8%. The results of our study are similar to those reported previously in the literature8–12; Balasubramanian et al9 showed that only 13.3% of the trials clearly explained allocation, 43.5% had no description of sample size, and 58.0% did not state the method of randomization. These deficiencies increase the risk of bias and might be associated with exaggerated treatment effects.9,21 Sinha et al10
found that only 17% of the RCTs adequately reported adverse events, and only 36% of the articles included the flow diagram, which provides a graphic representation of the patients screened and randomized, as well as exclusions and losses at each step of the trial. This information is particularly important in complex trials. The number of screened people was rarely mentioned in the trials, and this affects the external validity, because it might be an indicator of the population included.4 Reporting of the randomization technique, allocation, and implementation is aimed to describe the methods used to prevent selection bias and to ensure an unpredictable assignment sequence, including who generated the sequence and enrolled the patients; it is expected that the person(s) involved in the generation of the sequence are not involved in the implementation to prevent introduction of bias.4 Our study showed that 73.1% of the articles did not mention whether blinding was used. Those that mentioned that blinding was not performed were given the maximum score (2 points), because it met the criteria for reporting. Similar results were identified by other
990
studies. Balasubramanian et al9 reported that, among surgical trials where blinding was possible, only 30% of the articles clearly explained the blinding techniques; Wenner et al8 showed that only 4 in 10 surgical articles reported blinding techniques; Sinha et al10 also described that blinding was rarely adequately reported. Although blinding in surgical clinical trials is often difficult, there are techniques that can be used to decrease the risk of bias: determination of adverse effects and outcomes is performed by a blinded assessor, and analysis of the data is performed by a blinded reviewer. Research shows that the use of a blinded independent outcome assessor is achievable in surgical clinical trials.8,21 This single item can improve surgical research by decreasing observer bias without increasing the risk to the patients. We identified some factors associated with the achieved score, year of publication, journal impact factor, multi-institutional studies, and declared funding. Some of these were also described by Balasubramanian et al,9 although Sinha et al10 did not identify such associations. The most important factor was year of publications over the last 10 years; the average score had a steady increase that was significant, and this can be partially attributed to the most recent update of the CONSORT statement in 2010 and the increase in CONSORT endorsement by surgical journals over the years. The impact factor of the journal relates to the peer-review process and relevance in the field. Although this correlates with the believe that the impact factor is a surrogate of the publication quality, there is some concern related to a positive results bias: trials with negative results are less likely to be accepted at high impact factor journals.22 Declared funding might relate to the type of institutions involved: large academic center or studies funded by national agencies require the funding grant number to be reported in their articles, and these studies would probably have larger funds and resources than smaller trials. One of the factors that affects the generalizability of the results is the surgeon; particularly, the type of training or type of techniques that were used to decrease variations between surgeons and institutions. In our study, 43% of the articles did not mention the institution, which is important because large volume hospitals tend to report better results. A similar result was described by Jacquier et al11;
Are Our Publications Failing the Inspection?: A Review of the Publications in Rectal Cancer Surgery Between 2002 and 2012 Maria C. Mora Pinzon, M.D. • Dana M. Hayden, M.D. • Darlene Ariel, B.S. Kimberly A. Bartosiak, B.S. • Michael V. Chiodo, B.S. • Konstantine Kosmidis, H.S. Ann Evans, M.D. • Theodore J. Saclarides, M.D. Division of Colon and Rectum Surgery, Department of Surgery, Loyola University Medical Center, Maywood, Illinois
BACKGROUND: Quality of publications is considered a
subjective measurement, and more weight is placed on prospective studies, especially randomized clinical trials and meta-analyses. OBJECTIVE: This study describes the type of publications and evaluates the quality of randomized clinical trials and review articles using an objective measurement. DATA SOURCES: Medline (PubMed) is the data source for
this work. STUDY SELECTION: We used the terms “rectal neoplasms/ surgery” and the filters “10 years,” “humans,” and “English.” MAIN OUTCOME MEASURES: We measured compliance with checklist items. Randomized clinical trials were reviewed using the Consolidates Standards of Reporting Trials statement; systematic reviews/metaanalyses were reviewed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. RESULTS: A total of 3603 articles were identified: 20.8% were case report/series, 20.5% were retrospective cohorts, 14.0% were reviews or meta-analyses, 16.4% were prospective cohorts, 14.0% were other types of articles (comments, letters, or editorials), 5.5% were clinical
trials (phase I/II), 4.2% were randomized clinical trials, and 4.4% were cross-sectional studies. We reviewed 108 randomized clinical trials; the maximum score possible was 74.0, the average score was 44.6 (range, 20.0-64.0), 4 (3.7%) were graded as “excellent,” 21 (19.4%) were “good,” 44 (40.7%) were “deficient,” and 39 (36.1%) were graded as “fail.” The predictors of higher scores for randomized clinical trials were year of publication after 2007 (p = 0.00), higher impact factor (p = 0.03), and declared funding (p = 0.01). Twenty-nine meta-analyses were reviewed; the average score was 19.64 (range, 12.025.0); 5 articles (17.2%) were graded as “excellent,” 12 (41.4%) were “good,” 10 (34.5%) were “deficient,” and 2 (6.9%) were “fail.” LIMITATIONS: Only 1 electronic database was used, so we lacked a validated score. In addition, the search terms did not include “colorectal.” CONCLUSIONS: A total of 20.8% of the articles published were case reports and 25.0% of the articles were prospective or clinical trials. Although randomized clinical trials and systematic reviews provide the highest level of evidence, publications with missing data limit replication of the study and affect the generalizability of results to other populations. To improve the quality of our publications, authors, reviewers, and journal editors should consider the endorsement of standardize checklists.
Financial Disclosure: None reported. Podium presentation at the meeting of The American Society of Colon and Rectal Surgeons, Phoenix, AZ, April 27 to May 1, 2013.
key words: CONSORT; PRISMA; Quality; Rectal cancer; Review.
Correspondence: Theodore J. Saclarides, M.D., 2160 S First Ave, Maywood, IL 60153. E-mail: [email protected]
R
Dis Colon Rectum 2014; 57: 983–992 DOI: 10.1097/DCR.0000000000000169 © The ASCRS 2014 Diseases of the Colon & Rectum Volume 57: 8 (2014)
ecently we have seen an increase in the number of publications that are regularly available in the surgical literature, thus increasing the amount of articles that a surgeon must read to keep up with technology and new discoveries. In general, we tend to prefer 983
984
randomized clinical trial (RCT) and review articles over retrospective or prospective cohorts, but there are few mechanisms available to determine which articles provide useful information or comply with the scientific method. To improve the reporting of RCTs in 1996, the first Consolidates Standards of Reporting Trials (CONSORT) guidelines were published1 and subsequently updated in 20012 and 2010.3,4 The aim of these guidelines was to increase the transparency of reporting and to improve the level of evidence provided by such articles. Since the introduction of CONSORT guidelines, an improvement of publications has been identified.5–7 Previous studies have reviewed the quality of reporting in surgical trials using the CONSORT statement or Jadad score, and deficiencies in reporting were common across specialties.7–12 Similar results have been described in the review of systematic reviews and meta-analysis,13,14 which led to the development of the Quality of Reporting of Meta-Analysis Statement in 1999, which was updated in 2009 as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement.15,16 Although many studies have evaluated the quality of reporting in meta-analyses,17–19 to our knowledge, this is the first that focuses on rectal cancer surgery. The purpose of our study was to describe the type of publications in rectal cancer surgery over the past 10 years and to evaluate the compliance with reporting guidelines (CONSORT and PRISMA statements) of highly evidenced publications (RCTs and systematic reviews/meta-analyses) in rectal cancer surgery. This systematic review will help to identify deficiencies inherent to rectal cancer reports, and its results can be used to improve the reports of other topics in colorectal surgery.
MATERIALS AND METHODS Our study is a systematic review of the articles published on rectal cancer surgery over the past 10 years. The study protocol was designed and performed by the authors with departmental funding, and no registration number was used. We conducted an electronic search of PubMed using the term “rectal neoplasms/surgery” (medical subject headings). Filters were “10 years,” “humans,” and “English.” The search was performed on May 26, 2012. The resulting list was downloaded to an electronic Microsoft Excel database containing the following variables: year of publication, journal name, article title, first author, affiliation, country of first author, publication type, number of participants, and country of publication. In addition, the following information was added to the database for all RCT and reviews: impact factor of the journal, number of references, number of citations, and number of institutions involved.
MORA PINZON ET AL: PUBLICATIONS IN RECTAL CANCER SURGERY
The authors decided to use PubMed with broad search categories because it is the largest and most commonly used database; therefore, a broad search would capture the most commonly requested articles in the specified topic. Any search strategy is at risk of missing articles. In our case, those can be considered randomly missed that would not affect the purpose of the study, which is to provide a general idea of the type and quality of articles published. The use of a librarian was not considered in the elaboration of the study, because the search terms were broad and did not refer to any specific pathology or treatment. All of the articles were classified according to publication type, which was obtained from the downloaded file; the titles and abstracts of those classified as “journal article” or “multi-institutional” were reviewed by 1 investigator to determine the type of study, and further classification into 1 of the following categories was performed: case reports/ series, retrospective cohorts, prospective cohorts, crosssectional, clinical trials, RCTs, population study, review or meta-analysis, and others. If there was insufficient information in the title and abstract to determine the category, the full article was reviewed. Those publications classified as RCTs, systematic reviews, or meta-analyses were downloaded and stored for further review. Articles not related to rectal cancer surgery were excluded during the classification phase of our study. The CONSORT statement published 2010 was studied by the reviewers, and the definitions of each item were noted.3,4 Secondary analyses and pooled analyses were excluded. All of the included RCTs were read by 2 independent reviewers who were not blinded to the article’s authors, institutions, and journals during the assessment and data collection. Scoring was performed using a CONSORT checklist, which consists of 27 elements; each item was scored using a 0-2 scale; 0 if the item was not reported, 1 if partially reported, or 2 if completely reported in the article. The score corresponding with each item was entered in an electronic database. All of the points were added to obtain the overall score, and a percentage score was calculated for each trial. The maximum score was 74 points (100%). The scores obtained by the reviewers were compared, and in case of incongruence, the article was re-examined by a third reviewer. We defined “review articles” as those that summarize evidence available in the literature without identification of the methodology used. “Systematic review” and “metaanalyses” were defined according to the Cochrane Handbook. A systematic review is a methodologic collection of evidence to answer a specific question through identification, selection, appraisal, collection, and analyses of data from relevant studies included in the review. A meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies.20
985
Diseases of the Colon & Rectum Volume 57: 8 (2014)
TABLE 1. Grading system
Grade A: excellent B: good C: deficient F: fail
TABLE 2. Types of publications
Checklist items, %
Numeric score for RCTs
>85 84-71 70-56 63 53-62 41-52 23 20-22 19-15 70% of the items evaluated. The remaining articles had severe deficiencies, especially in the Methods section, which is indispensable to allow duplication of the results. The most common deficiencies were sample size calculation (not reported: 39.8% of articles). Interim analyses and stopping guidelines were missing in 63.9%, and randomization technique including allocation mechanism and implementation of the randomization were missing in >50% of the articles. The CONSORT diagram was not present in 52.8% of the articles, and funding was not reported in 52.8%. The results of our study are similar to those reported previously in the literature8–12; Balasubramanian et al9 showed that only 13.3% of the trials clearly explained allocation, 43.5% had no description of sample size, and 58.0% did not state the method of randomization. These deficiencies increase the risk of bias and might be associated with exaggerated treatment effects.9,21 Sinha et al10
found that only 17% of the RCTs adequately reported adverse events, and only 36% of the articles included the flow diagram, which provides a graphic representation of the patients screened and randomized, as well as exclusions and losses at each step of the trial. This information is particularly important in complex trials. The number of screened people was rarely mentioned in the trials, and this affects the external validity, because it might be an indicator of the population included.4 Reporting of the randomization technique, allocation, and implementation is aimed to describe the methods used to prevent selection bias and to ensure an unpredictable assignment sequence, including who generated the sequence and enrolled the patients; it is expected that the person(s) involved in the generation of the sequence are not involved in the implementation to prevent introduction of bias.4 Our study showed that 73.1% of the articles did not mention whether blinding was used. Those that mentioned that blinding was not performed were given the maximum score (2 points), because it met the criteria for reporting. Similar results were identified by other
990
studies. Balasubramanian et al9 reported that, among surgical trials where blinding was possible, only 30% of the articles clearly explained the blinding techniques; Wenner et al8 showed that only 4 in 10 surgical articles reported blinding techniques; Sinha et al10 also described that blinding was rarely adequately reported. Although blinding in surgical clinical trials is often difficult, there are techniques that can be used to decrease the risk of bias: determination of adverse effects and outcomes is performed by a blinded assessor, and analysis of the data is performed by a blinded reviewer. Research shows that the use of a blinded independent outcome assessor is achievable in surgical clinical trials.8,21 This single item can improve surgical research by decreasing observer bias without increasing the risk to the patients. We identified some factors associated with the achieved score, year of publication, journal impact factor, multi-institutional studies, and declared funding. Some of these were also described by Balasubramanian et al,9 although Sinha et al10 did not identify such associations. The most important factor was year of publications over the last 10 years; the average score had a steady increase that was significant, and this can be partially attributed to the most recent update of the CONSORT statement in 2010 and the increase in CONSORT endorsement by surgical journals over the years. The impact factor of the journal relates to the peer-review process and relevance in the field. Although this correlates with the believe that the impact factor is a surrogate of the publication quality, there is some concern related to a positive results bias: trials with negative results are less likely to be accepted at high impact factor journals.22 Declared funding might relate to the type of institutions involved: large academic center or studies funded by national agencies require the funding grant number to be reported in their articles, and these studies would probably have larger funds and resources than smaller trials. One of the factors that affects the generalizability of the results is the surgeon; particularly, the type of training or type of techniques that were used to decrease variations between surgeons and institutions. In our study, 43% of the articles did not mention the institution, which is important because large volume hospitals tend to report better results. A similar result was described by Jacquier et al11;
