Neurourology and Urodynamics 35:412–416 (2016)

The Explicit Mentioning of Reporting Guidelines in Urogynecology Journals in 2013: A Bibliometric Study 1

Marianne Koch,1,2* Paul Riss,1,2 Wolfgang Umek,1,2 and Engelbert Hanzal1 Department of Obstetrics and Gynecology, Medical University Vienna, Vienna, Austria 2 Karl Landsteiner Institute of Special Gynecology and Obstetrics, Vienna, Austria

Aims: Poor reporting of research may limit critical appraisal and reproducibility, whereas adherence to reporting guidelines (RG) can guarantee completeness and transparency. We aimed to determine the explicit citing of RGs (CONSORT, PRISMA, STROBE) in urogynecology articles in 2013, the requirements of relevant journals and a potential difference between urogynecology and general gynecology journals. Methods: All urogynecologic articles published between January and December 2013 in the journals NAU, IUJ, FPMRS, GREEN, AJOG, and BJOG were included. Issues were searched for systematic reviews, RCTs, cohort studies, case-control studies and cross-sectional studies. Each electronic article was searched for the term PRISMA, CONSORT, or STROBE according to the study design. Instructions to Authors of the six journals were screened for requirement of using RGs. Results: We included 296 articles (243 observational studies, 40 RCTs, and 13 systematic reviews). The use of PRISMA guidelines was explicitly declared in 54% of systematic reviews, CONSORT guidelines were referenced in 25% of RCTs and STROBE in 1.2% of observational studies. The use of CONSORT is required by all journals except FPMRS. PRISMA and STROBE are only compulsory in the journals GREEN, AJOG, and BJOG. The overall rate of explicit mentioning of RGs comparing urogynecology and general gynecology journals was 6.7% versus 7.1%, respectively. Conclusions: The explicit mentioning of RGs was on a relatively low level. A slightly higher adherence was recognized among general gynecology journals compared to urogynecology journals. Stronger efforts should be taken to further promote the use of RGs in urogynecology. Neurourol. Urodynam. 35:412–416, 2016. # 2015 Wiley Periodicals, Inc. Key words: CONSORT; PRISMA; reporting guidelines; research design; STROBE; urogynecology

INTRODUCTION

Correct reporting of research is considered necessary for the assessment of its quality and of its potential reproducibility. The Helsinki Declaration, which demands ethical principles for medical research, states that authors shall adhere to commonly accepted guidelines for the reporting of research.1 Poor reporting of research might cover a poor trial design and conduct, whereas adherence to qualitative reporting guidelines (RG) can guarantee clarity, completeness and transparency of research publications.2 Transparent and complete reporting facilitates critical appraisal of a scientific work and can reveal deficiencies of trial designs.3–8 If a trial has been poorly conducted, it is likely to fail adequate reporting on the basis of a checklist, which will consequently facilitate critical appraisal and prevent the implementation of potentially biased research results into clinical practice.2 Researchers, by now, have access to RGs for different study designs and many journals require authors to edit their manuscripts according to these. In most cases, RGs comprise checklists, which consist of a varying numbers of items that are considered essential for the reporting of the respective scientific work. These checklists commonly provide the basic structure of an article by requiring the items ‘‘title, abstract, introduction, methods, results, discussion, and funding’’ in this order. According to the different research types, however, detailed instructions vary mainly concerning the description of methodology of the scientific work. The first RG to be introduced was the CONSORT (Consolidated Standards of Reporting Trials) statement for Randomized Controlled Trials (RCTs) in 1996, which was updated in 2001 and again revised and published in 2010. The CONSORT statement includes a checklist of 25 items as well as a flow #

2015 Wiley Periodicals, Inc.

diagram to report the progress of the different phases during a parallel-group randomized trial. These phases represent the participant flow during enrolment, intervention allocation, follow-up, and data analysis. The STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement, which was initiated in 2004 and published by 2007, covers three main study types: cohort studies, case-control studies, and cross-sectional studies. Eighteen of the 22 items presented in the STROBE checklist are applicable to all three study types, whereas four items are more specific. The most recent RG, PRISMA (Preferred Reporting Guidelines for Systematic Reviews and Meta-Analyses) was first initiated in 2005 and published in 2009 to facilitate the reporting of systematic reviews. PRISMA includes a 27-item checklist and a four-phase flow diagram to picture the information flow of the different phases of a systematic review (origin of records, number of records screened, number of full-text articles assessed, number of studies included in analysis/meta- analysis).9 Table I. All three RGs have been systematically developed by groups of experts, incorporate available evidence, and are therefore widely recognized. In recent years, attempts have been made by Christopher Chapple led the peer-review process as the Associate Editor responsible for the paper. Potential conflicts of interest: M. Koch: none. W. Umek: acceptance of paid travel expenses or honoraria (Astellas, Innovacell, Pfizer). P. Riss: Co-editor-in-chief of IUJ. E. Hanzal: acceptane of paid travel expenses or honoraria (Astellas, Innovacell). *Correspondence to: Marianne Koch, Department of Obstetrics and Gynecology, Medical University Vienna, Vienna, Austria. E-mail: [email protected] Received 25 September 2014; Accepted 25 November 2014 Published online 24 January 2015 in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/nau.22726

Reporting Guidelines in Urogynecology Articles

413

TABLE I. Listing of Investigated Reporting Guidelines Study type PRISMA

First publication

Reference

2009

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097 Schulz KF, Altman DG, Moher D, for the CONSORT Group (2010) CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials. PLoS Med 7(3): e1000251. doi:10.1371/journal.pmed.1000251 von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007 Oct 16;4(10):e296. PMID: 17941714

Review

CONSORT Randomized Controlled Trial

1996

STROBE

2004

Cohort study, case- control study, cross- sectional study (observational)

researchers to evaluate the completeness of articles and adherence to reporting guidelines among different medical specialties and biomedicine. Most come to the conclusion, that either the use of RGs is insufficiently required by publishing journals, or that the implementation of RGs in final articles is unsatisfactory.10–17 In this study, we aimed to determine to which extent relevant RGs are explicitly mentioned in published articles in the subspecialty urogynecology and whether these results would be in line with requirements for use of RGs in the respective journals’ instructions to authors. We furthermore aimed to determine whether the proportion of articles in which the use of RGs is explicitly mentioned differs between urogynecology journals and general gynecology and obstetrics journals. MATERIALS AND METHODS

All articles assigned to the sub-specialty ‘‘urogynecology,’’ which were published between January and December 2013 in the three urogynecology journals NAU (Neurourology and Urodynamics), IUJ (International Urogynecology Journal), FPMRS (Female Pelvic Medicine and Reconstructive Surgery), and the three general gynecology journals AJOG (American Journal of Obstetrics and Gynecology), GREEN (Obstetrics and Gynecology), and BJOG (An International Journal of Obstetrics and Gynaecology) were included in this research. NAU, IUJ, and FPMRS are to date the only journals, which are specialized on urogynecology, whereas GREEN, AJOG, and BJOG are general gynecology and obstetrics journals that frequently publish urogynecology articles, and which were found sufficiently representative for a comparison regarding the study question. Articles from NAU were taken into consideration if the study included females only, or males and females. The time frame January to December 2013 for publication date of included articles was chosen in order to depict the most recent status of explicit mentioning of reporting guidelines in urogynecology. Journals were accessed electronically and issues were searched for reviews, RCTs, cohort studies, case-control studies, and cross-sectional studies by two independent investigators (MK and PR). Other study designs were not included, as they are not reflected in the selected RGs. The study category was either indicated in the table of contents, in the title of the article, or assessed by the investigators. If the study type was not indicated in the article title, the abstract and methods section were screened for a description. If no study type was explicitly indicated within the article, the two investigators independently determined the study type according to their critical appraisal of the study methodology. Articles were only included if both investigators reached a consensus on the study type. In a second step, the electronic version (PDF) of all articles was searched for the term PRISMA, CONSORT, or STROBE according Neurourology and Urodynamics DOI 10.1002/nau

to the study design. This keyword search was conducted using the automated PDF search function within the PDF document ([cmd F] for Macintosh; [ctrl F] for Windows). All original articles and systematic reviews published in 2013 in the journals IUJ, NAU, FPMRS, and GREEN were screened (total n ¼ 573). In addition, we screened 37 urogynecology articles published in 2013 in the journals AJOG and BJOG. In these two journals, each issue is separated into different topic sections. Therefore, we only screened the sections, which were indicated as ‘‘urogynecology’’ (if available in the respective issue). Moreover, BJOG published one extra issue on urogynecology in 2013, of which all articles were screened. In total, 610 studies were assessed for eligibility. All articles, which could not be categorized as either ‘‘cohort study,’’ ‘‘case-control study,’’ ‘‘cross-sectional study,’’ ‘‘systematic review,’’ or ‘‘randomized controlled trial’’ were excluded (n ¼ 314). Finally, a total number of 296 studies could be included in the qualitative analysis (Fig. 1). The current ‘‘Instructions to Authors’’ of the six journals were screened for the requirement of using RGs. Institutional Review Board (IRB) approval was waived by the Ethics Committee of the Medical University Vienna (bibliometric study). RESULTS

The literature search for urogynecology articles, which were published in 2013 in the six selected journals yielded 296 articles, which conformed to the inclusion criteria (n ¼ 254 in urogynecology journals and n ¼ 42 in general gynecology and obstetrics journals). In total, 243 observational studies (including cohort studies, case-control studies and cross-sectional studies), 40 RCTs and 13 systematic reviews were identified and included in this study. One study which was published in NAU included males and females. All 13 urogynecologic systematic reviews were published by the journals NAU and IUJ (n ¼ 6 and n ¼ 7, respectively), whereas the remaining journals did not publish any in 2013. The use of PRISMA guidelines was explicitly declared in 54% of all systematic reviews. Urogynecologic RCTs were published by all journals except AJOG and the CONSORT guidelines were referenced in 10 out of 40 (25%). The most frequent study type of published urogynecology articles in 2013 was observational (n ¼ 243), of which only three articles referenced STROBE (1.2%). The overall rate of explicit mentioning of all RGs among the included articles was 6.7% for the three urogynecology journals and 7.1% for the three general gynecology and obstetrics journals. Table II. All selected journals except FPMRS require the use of CONSORT guidelines for reporting of RCTs in their instructions to authors. The use of PRISMA and STROBE checklists in the preparation of the manuscript are compulsory for submitting

414

Koch et al.

Fig. 1. Flow diagram of screening, exclusion, and inclusion of articles. *(IUJ) International Urogynecology journal, (NAU) Neurourologiy and Urodynamics, (FPMRS) Female Pelvic Medicine and Reconstructive Surgery, (GREEN) Obstetrics and Gynecology, (AJOG) American Journal of Obstetrics and Gynecology, (BJOG) An International Journal of Obstetrics and Gynaecology; all original article and systematic reviews were screened in IUJ, NAU, FPMRS, and GREEN. Only articles within the specific section ‘‘urogynecology’’ were screened in AJOG and BJOG.

authors in three out of six journals (GREEN, AJOG, BJOG). These three general gynecology journals had a higher 2013 Impact Factor (mean; 4.1 versus 2.3) than NAU and IUJ (FPMRS not yet listed). Table III. DISCUSSION

Among published articles attributed to the sub-specialty urogynecology, it was yet unclear to which extent relevant RGs are considered during the manuscript preparation. The explicit use of RGs in the investigated urogynecology articles in 2013 was generally low (overall 1–54%). We could, however, observe a tendency towards a more frequent referencing of PRISMA in systematic reviews compared to the remaining two article types. PRISMA was explicitly

mentioned in 54% of all investigated systematic reviews of 2013, whereas CONSORT was mentioned in 25% of RCTs and STROBE in only 1% of observational studies. As the aim of this study was to evaluate the current status of explicit mentioning of RGs in urogynecology articles, we consider the time frame of 1 year (2013) as sufficient for this purpose. It is commonly acknowledged, that the application of RGs in the course of writing up scientific work has an impact on its quality and facilitates its critical appraisal. Failure of adequate reporting of a trial can reveal deficiencies of a trial design.3–8 If a trial has been poorly conducted, it is likely to fail adequate reporting on the basis of a checklist. Consequently, implementation of potentially biased research results into clinical practice may be averted.2

TABLE II. Frequency of Explicit Declaration of Each Reporting Guideline (PRISMA, CONSORT, STROBE) Out of Total Number of Articles in the Respective Category (n/n) PRISMA (systematic review)

%

CONSORT (RCT)

%

STROBE (observational)

%

33 71 n/a

2/6 7/22 0/4

33 32 0

0/25 1/137 0/47

0 0.7 0

Totala

Urogynecology journals NAU IUJ FPMRS

2/6 5/7 0

4/37 13/166 0/51 6.7%

General gynecology journals GREEN AJOG BJOG

0 0 0

n/a n/a n/a

1/7 0 0/1b

14 n/a 0

0/8 0/12 2/14

0 0 14

Totalc

7/13

54

10/40

25

3/243

1.2

1/15 0/12 2/15 7.1%

(NAU) Neurourology and Urodynamics; (IUJ) International Urogynecology Journal; (FPMRS) Female Pelvic Medicine and Reconstructive Surgery (FPMRS); (GREEN) Obstetrics and Gynecology; (AJOG) American Journal of Obstetrics and Gynecology; (BJOG) An International Journal of Obstetrics and Gynaecology; n/a (not applicable); (RCT) Randomized Controlled Trial. a

Total frequency of declaration of all reporting guidelines per journal.

b

Follow-up study after RCT.

c

Total frequency of declaration of each reporting guideline in urogynecology articles.

Neurourology and Urodynamics DOI 10.1002/nau

Reporting Guidelines in Urogynecology Articles

415

TABLE III. Use of Reporting Guidelines Required in the ‘‘Instructions to Authors’’ of the Respective Journal

NAUb IUJ FPMRS GREEN AJOG BJOG

PRISMA

CONSORT

STROBE

Impact factor 2013

h5-Indexa

0 0 0 1 1 1

1 1 0 1 1 1

0 0 0 1 1 1

2,458 2,161 n/a 4,368 3,973 3,862

37 36 n/a 74 72 56

(NAU) Neurourology and Urodynamics; (IUJ) International Urogynecology Journal; (FPMRS) Female Pelvic Medicine and Reconstructive Surgery; (GREEN) Obstetrics & Gynecology; (AJOG) American Journal of Obstetrics and Gynecology; (BJOG) An International Journal of Obstetrics and Gyneacology. no (0), yes (1). a

Accessed on http://scholar.google.com/intl/en/scholar/metrics.html on August 2, 2014.

b

Authors are encouraged to consult reporting guidelines relevant to their specific research design.

Statements in the ‘‘instructions to authors’’ of the investigated journals do not seem to be related to the actual number of explicitly mentioned RGs in the respective articles. None of the investigated journals showed 100% explicit citing of reporting guidelines, even though CONSORT is required by all journals (except FPMRS), and PRISMA and STROBE are additionally required by GREEN, AJOG, and BJOG. CONSORT was explicitly mentioned in 0–33% of articles in journals where CONSORT was required (NAU, IUJ, GREEN, AJOG, BJOG), whereas PRISMA and STROBE were explicitly mentioned in 0–14% of articles in journals with requirement (GREEN, AJOG, BJOG). Even though the journals NAU and IUJ do not require the use of PRISMA and STROBE in their instructions to authors, there is still a high number of explicit mentioning of these RGs in the respective published article types (Table II). These findings put in question the efficacy of journal requirements, as according to our results, they might have not been adequately followed in 70–85% of articles. Some journals follow a policy, which demands authors to additionally submit an RG checklist according to their study type when submitting the manuscripts. Through this method, non-adherence to reporting guidelines during manuscript preparation would be uncovered at the time of submission, as the authors are forced to document page and line numbers where RG checklist items were included in the manuscript. Whereas on one hand, such rigid control mechanisms would raise standards of submitted manuscripts, it is also clear on the other hand, that not every journal may afford the necessary logistics to follow up on authors’ statements. When comparing the three urogynecology journals with three general gynecology and obstetrics journals, it showed a similar proportion of explicit mentioning of RGs (6.7% vs. 7.1%, respectively). These findings seem surprising, as we would have expected a larger difference between standard urogynecology journals and gynecologic top journals. It would therefore suggest, that the simple requirement for RGs in the instructions to authors is not sufficient and that it might be necessary for all investigated journals to set a stronger focus on the actual implementation of RGs when manuscripts are reviewed. By screening the articles for the keywords CONSORT, PRISMA, and STROBE, we can only conclude whether the RGs were explicitly cited within the article, but we are not able to determine whether the RGs were actually used in the preparation of the manuscripts. It is, therefore, not possible to draw a direct conclusion on the quality of an article. However, we assume that the explicit citing of a RG in an article is a good indication that the authors are aware of the respective RG and that they therefore may have implemented them in the preparation of the manuscript. We also found that most authors used flowcharts showing the flow of patients in Neurourology and Urodynamics DOI 10.1002/nau

manuscripts reporting RCTs even when they did not explicitly mention the reporting guidelines. It appears that the use of flowcharts has become common practice in manuscripts reporting RCTs and systematic reviews, even though they may not be the originally proposed templates provided by the CONSORT and PRISMA guidelines. Reporting guidelines can be important tools for designing a study protocol and for reporting the results of a study. They give structure to a manuscript, they remind authors not to forget key aspects of a study, and they facilitate comparability and reproducibility of a study.7–9 These two latter aspects are of particular importance. Study results are of very limited value if they cannot be and are not replicated, and the standardized reporting of results is a prerequisite for summarizing the findings from several studies in systematic reviews and meta-analyses. It is also for these reasons that efforts are under way to encourage researchers to use standardized outcome measures such as the CROWN initiative.18 Assessment of implementation and evaluation of the final impact of research findings is difficult. Such investigation would require very long follow-up and a potential ideal study design is in question. However, it should be possible to ‘‘retrospectively’’ draw conclusions on the quality of research when looking at the reporting of a trial. As already pointed out, if a trial cannot be adequately reported, it is likely that this trial had been inadequately designed and/or conducted. Thereafter, we assume that if a trial was inadequately conducted, any research findings evolving from this trial might be biased. If biased results are furthermore implemented in clinical practice, this might cause harm to future patients. Therefore, even though it is difficult to define quality of research, it should be possible to evaluate the quality of a study by assessing its implementation of reporting criteria. This raises the question, whether journals should continue to publish studies which fail to comply with reporting guidelines, or whether such studies should be categorically rejected for publication. As the study title indicates, we aimed to investigate the current explicit mentioning of RGs in the sub- specialty of urogynecology. A limitation of our study may be that we did not do a systematic literature search for any urogynecologic articles published in 2013, but restricted the inclusion of articles to the three specific urogynecology journals and three general gynecology and obstetrics journals as a representative comparison. However, based on the findings of the very variable use of RGs in different journals—even in those with a higher IF—we assume that our results would also apply to other journals. We also recognize that on one hand the explicit mentioning of reporting guidelines in a manuscript does not guarantee the actual use of these guidelines in the manuscript and that on the other hand

416

Koch et al.

manuscripts without reference to RGs can be of high quality. The strength of our study is that we looked at a cross section of general obstetrics and gynecology and urogynecology subspecialty journals and were able to assess the awareness of authors of the relevant RGs and the degree to which authors followed the requirements of journals in this regard. Even though we are not able to draw a conclusion as to whether the authors actually used the RG checklists during the preparation of their manuscripts and did not just cite the RG in the final paper, the results of this study still provide us with information on the extent to which RGs are explicitly mentioned in urogynecology articles, which may function as a reference point. These findings suggest that we should strive to increase awareness for reporting guidelines among authors, but also editors and reviewers in the field of urogynecology. Adherence to reporting guidelines when writing a manuscript should become standard among authors, and journal editors as well as reviewers should be further sensitized to rigorously follow up on RG inclusion. This study focuses on the awareness of the use of RGs among authors of urogynecology articles, whereas future studies may provide more detailed information on implementation of specific RG checklist items in the manuscripts. CONCLUSION

The correct reporting of research facilitates its critical appraisal and reproducibility, guarantees completeness and transparency and therefore eventually improves a study’s quality. We found a low number of explicit mentioning of reporting guidelines in urogynecology articles published in 2013 with proportions of 25% for CONSORT, 54% for PRISMA, and lowest of 1% for STROBE. These findings may suggest a lack of awareness for adherence to reporting guidelines, especially concerning observational studies (STROBE), among authors of urogynecology articles. Stronger efforts should be taken to further promote the use of RGs in this field. REFERENCES 1. WMA Declaration of Helsinki [cited 2014]. Available from: http://www.wma. net/en/30publications/10policies/b3/.

Neurourology and Urodynamics DOI 10.1002/nau

2. Schulz KF, Altman DG, Moher D. Group C. CONSORT statement: Updated guidelines for reporting parallel group randomised trials. PLoS Med 2010;7: e1000251. 3. Simera I, Moher D, Hirst A, et al. Transparent accurate reporting increases reliability utility impact of your research: Reporting guidelines the EQUATOR Network. BMC Med 2010; 8:24. 4. Simera I, Moher D, Hoey J, et al. The EQUATOR Network and reporting guidelines: Helping to achieve high standards in reporting health research studies. Maturitas 2009; 1:4–6. 5. Moher D, Simera I, Schulz KF, et al. Helping editors peer reviewers, authors improve the clarity, completeness, transparency of reporting health research. BMC Med 2008; 6:13. 6. Altman DG, Simera I. Responsible reporting of health research studies: Transparent complete accurate timely. J Antimicrob Chemother 2010; 65: 1–3. 7. Turner L, Shamseer L, Altman DG, et al. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic reviews 2012; 1:60. 8. Cobo E, Cortes J, Ribera JM, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: Masked randomised trial. Bmj. 2011; 343:d6783. 9. EQUATOR Network [cited 2014]. Available from: http://www.equatornetwork.org. 10. Serrano M, Gonzalvo MC, Sanchez-Pozo MC, et al. Adherence to reporting guidelines in observational studies concerning exposure to persistent organic pollutants and effects on semen parameters. Human reproduction 2014; 29:1122–33. 11. Vasilevsky NA, Brush MH, Paddock H, et al. On the reproducibility of science: Unique identification of research resources in the biomedical literature. PeerJ. 2013; 1:e148. 12. Florez-Vargas O, Bramhall M, Noyes H, et al. The quality of methods reporting in parasitology experiments. PloS ONE 2014; 9:e101131. 13. Cui Q, Tian J, Song X, Yang K. Does the CONSORT checklist for abstracts improve the quality of reports of randomized controlled trials on clinical pathways? J Eval Clin Pract 2014; (Epub ahead of print) PubMed PMID: 24916891. 14. Walther S, Schueler S, Tackmann R, et al. Compliance with STARD checklist among studies of coronary CT angiography: Systematic review. Radiology 2014; 271:74–86. 15. Peron J, Maillet D, Gan HK, et al. Adherence to CONSORT adverse event reporting guidelines in randomized clinical trials evaluating systemic cancer therapy: A systematic review. J Clin Oncol 2013; 31:3957–63. 16. Blakely ML, Kao LS, Tsao K, et al. Adherence of randomized trials within children’s surgical specialties published during 2000 to 2009 to standard reporting guidelines. J Am Coll Surg 2013; 217:394–9 e7. 17. Cavadas V, Branco F, Carvalho FL, et al. The quality of reporting of randomized controlled trials in pelvic organ prolapse. Int Urogynecol J 2011; 22:1117–25. 18. Dwyer PL, Riss P, Haylen BT. Does it matter what we measure? Core outcomes, the IUJ and the CROWN and COMET initiatives. Int Urogynecol J 2014; 25:1301–2.