This article was downloaded by: [University of Tennessee, Knoxville] On: 05 January 2015, At: 18:50 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Accountability in Research: Policies and Quality Assurance Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/gacr20
A Core-Item Reviewer Evaluation (CoRE) System for Manuscript Peer Review Adedayo A. Onitilo M.D. M.S.C.R. F.A.C.P. M.S.N. F.N.P.-B.C.
b c
a b c
, Jessica M. Engel c
, Sherry A. Salzman-Scott B.S. , Rachel V.
c
Stankowski Ph.D. & Suhail A. R. Doi F.R.C.P. Ph.D.
a
a
Clinical Epidemiology Unit , School of Population Health, University of Queensland , Brisbane , Australia b
Department of Hematology/Oncology , Marshfield Clinic Weston Center , Weston , Wisconsin , USA c
Click for updates
Marshfield Clinic Research Foundation , Marshfield , Wisconsin , USA Published online: 14 Nov 2013.
To cite this article: Adedayo A. Onitilo M.D. M.S.C.R. F.A.C.P. , Jessica M. Engel M.S.N. F.N.P.-B.C. , Sherry A. Salzman-Scott B.S. , Rachel V. Stankowski Ph.D. & Suhail A. R. Doi F.R.C.P. Ph.D. (2014) A Core-Item Reviewer Evaluation (CoRE) System for Manuscript Peer Review, Accountability in Research: Policies and Quality Assurance, 21:2, 109-121, DOI: 10.1080/08989621.2014.847664 To link to this article: http://dx.doi.org/10.1080/08989621.2014.847664
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Accountability in Research, 21:109–121, 2014 Copyright © Taylor & Francis Group, LLC ISSN: 0898-9621 print / 1545-5815 online DOI: 10.1080/08989621.2014.847664
A Core-Item Reviewer Evaluation (CoRE) System for Manuscript Peer Review Adedayo A. Onitilo, M.D., M.S.C.R., F.A.C.P.,1,2,3 Jessica M. Engel, M.S.N., F.N.P.-B.C.,2,3 Sherry A. Salzman-Scott, B.S.,3 Rachel V. Stankowski, Ph.D.,3 and Suhail A. R. Doi, F.R.C.P., Ph.D.1 1 Clinical Epidemiology Unit, School of Population Health, University of Queensland, Brisbane, Australia 2 Department of Hematology/Oncology, Marshfield Clinic Weston Center, Weston, Wisconsin, USA 3 Marshfield Clinic Research Foundation, Marshfield, Wisconsin, USA
Manuscript peer review is essential for ensuring accountability to all involved in the publication process, including authors, journals, and readers. Lack of consensus regarding what constitutes an accountable manuscript peer review process has resulted in varying practices from one journal to the next. Currently, reviewers are asked to make global judgments about various aspects of a paper for review irrespective of whether guided by a review checklist or not, and several studies have documented gross disagreement between reviewers of the same manuscript. We have previously proposed that the solution may be to direct reviewers to concrete items that do not require global judgments but rather provide a specific choice, along with referee justification for such choices. This study evaluated use of such a system via an international survey of health care professionals who had recently reviewed a health care–related manuscript. Results suggest that use of such a peer review system by reviewers does indeed improve interreviewer agreement, and thus, has the potential to support more consistent and effective peer review, if introduced into journal processes for peer review. Keywords: accountability, disagreement, peer review, rating
Address correspondence to Suhail A. R. Doi, Clinical Epidemiology Unit, School of Population Health, University of Queensland, Brisbane, Australia. E-mail: sardoi@ gmx.net
110
A. A. Onitilo et al.
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
INTRODUCTION Manuscript peer review is essential to ensuring accountability to all involved in the publication process, including authors, readers, and journals. Authors benefit by receipt of suggestions for improvement of their manuscript, readers benefit when journal standards are upheld, and journal editors benefit from guidance regarding various aspects of manuscript acceptability (Loonen et al., 2005). Many journals aim to achieve the latter by relying on a primarily subjective peer review process based on reviewer opinion and expertise. While general guidelines and instructions for reviewers do exist (Abbott, 2011; Curzon and Cleaton-Jones, 2011; Blomme, 2011), only a few refer to a specific tool for manuscript critique (Rajesh et al., 2013; Henly and Dougherty, 2009; Sosa et al., 2009; Ramulu et al., 2005). In the era of evidence-based medical practice, it is only ironic that many editorial decisions on whether or not to publish a manuscript are often based on subjective criteria, rather than evidence-based views rendered by reviewers. Standardization of the general review process alone has not been sufficient to improve reviewer agreement (Onitilo et al., 2013), probably because standardized manuscript review checklists (Duchesne and Jannin, 2008; Campion, 1993) provide exhaustive lists of items linked to a “good” paper, on which global judgments still need to be made. Heddle and Ness suggest that instead, questions should be designed to direct reviewers toward clearly describing deficiencies with suggestions for improvement (Heddle and Ness, 2009), thus shifting the focus to concrete concepts in lieu of global judgments. Indeed, Strayhorn et al. demonstrated that the reliability of ratings of scientific articles are improved if eliciting ratings of separate, concrete items rather than a global judgment (Strayhorn et al., 1993). We hypothesize that the latter problem accounted for the poor reviewer agreement we observed in a recent evaluation of a standardized review form used by the journal Clinical Medicine and Research. Each question in this form represented a global judgment rather than a concrete item, which may have resulted in continued reviewer disagreement despite standardization (Onitilo et al., 2013). In our previous paper (Onitilo et al., 2013), we suggested a prototype for a generic concrete, item-based, decision-driven checklist (Appendix 1) with the expectation that such a tool would encourage reviewers to rate these items quantitatively and require them to provide comments to justify low item scores. Quantifying responses and determining a total score would nullify the need for the reviewer to provide a publication recommendation while improving rank ordering of papers and linking of responses to an editorial decision. Here we report the results of a prospective evaluation of the reliability and consistency of this core-item reviewer evaluation (CoRE) checklist tool across a wide-spectrum of international reviewers.
Reviewer Ratings in Manuscript Peer Review
METHODS
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Reviewer Selection Participating reviewers were volunteers from the membership of two listservs. The EPIDEMIO-L listserv is sponsored by the University of Montreal and provides access to prominent members of the global epidemiological community. The Evidence Based Health (EBH) listserv is sponsored by JISC Netskills, Newcastle University, and provides access to teachers and practitioners of EBH interested in discussing issues and controversies surrounding implementation of EBH. A simple message was posted on each of these two listservs asking members who had recently reviewed a health-related journal article to complete an anonymous survey regarding that manuscript. A link to the survey hosted on surveygizmo.com was included in the message. The survey was open for 72 hrs from Sunday, April 14, 2013 through Tuesday, April 16, 2013.
Survey Tool Participants were asked to respond to 12 questions regarding the last manuscript they reviewed. Question 1 asked the participant about the type of manuscript reviewed. Questions 2–10 incorporated the eight items on the CoRE checklist shown in Appendix 1. Questions 2–5 focus on core items regarding research design and execution, while questions 6– 0 focus on other items related to manuscript structure. Question 11 asked participants to provide a recommendation regarding publication, and question 12 provided the option to add comments. Questions 2–10 were coded on a Likert scale from a minimum of -2 (weak) to a maximum of 2 (strong), depending on the question. In order to force a decision, a neutral category was not provided except for question 4 regarding study methods where the reviewer could select the option of “refer to statistician,” which was scored neutrally. Question 11 regarding publication recommendation provided the participant with three options including (1) reject/reject and resubmit, (2) re-evaluate after revision, or (3) potentially acceptable after revision/accept. Separate summary scores were created from questions in the core domain (questions 2–5) and questions in the structural domain (questions 6–10).
Statistical Analysis Agreement between summary scores and publication recommendation was assessed using a quadratic weighted kappa statistic. Cronbach’s alpha was used to determine reliability of the core and structural scales for predicting publication recommendation. Cronbach’s alpha is essentially the average measure of intraclass correlation coefficient [ICC(3,k)] and can also be used to assess
111
112
A. A. Onitilo et al.
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
agreement across questions within a scale where the questions are considered “raters” and k is the number of questions per scale. All analyses were stratified by publication recommendation and carried out using SPSS version 20 (IBM, Armonk, NY) and MedCalc version 11.6.1.0 (MedCalc Software, Ostend, Belgium).
RESULTS Seventy-two reviewers submitted a complete response to the survey sent via the listservs. Participating reviewers were from all continents, except Antarctica, and included 35 from North America, 17 from Europe, 5 from South America, 7 from Asia, and the remaining 8 from African, MiddleEastern, and Caribbean countries. Survey respondents reported 62 reviews of original research, 3 review articles, and 7 case reports or other miscellaneous manuscripts. Only those who reported review of original research were required to complete the core section of the survey (questions 2–5). Of the 14 manuscripts receiving a publication recommendation of reject, 13 were original research articles, and 1 was in the “other” category. Survey responses were assigned quantitative scores as shown in Appendix 1, and summary core and structural scores were generated by summing item scores. Item scores and summary scores for each of the two domains were distributed as shown in Figs. 1 and 2, categorized by reviewer recommendation for publication.
Scale Performance Tables 1 and 2 show the results of reliability analyses. Reliability of the questions was excellent (alpha ≥ .7) for the reject category using the core scale (Table 1) and for the re-evaluate after revision category using the structural scale (Table 2). Given that Cronbach’s alpha is equal to ICC(3,k) and considering within scale questions to be the “raters,” then reviewers whose opinions were reject tended to agree on their average core item scores, and reviewers whose opinions were revise tended to agree on their average structural item scores.
Agreement by Publication Recommendation Based on results of the reliability analysis, cutoff scores were set to delineate publication recommendation categories. A cutoff for the core score based on the 75th percentile within the reject category was used as the threshold for rejection (score > 0). Similarly, the 75th percentile within the revise category of the structural score was used as a threshold for acceptance (score ≥ 7). Manuscripts that did not meet the core score threshold (≤ 0) were considered weak; manuscripts that met the core threshold (≥ 1) but not the
Reviewer Ratings in Manuscript Peer Review A) Core items (original research only)
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Reject and/or reject and resubmit
Re-evaluate after revision (major or minor)
–2
–1
0
1
2
Potentially acceptable after revision or accept
–2
–1
0
1
2
relevance methods
design implications
Graphs by decision
*All P < 0.05 for comparison of each item across decision groups, Kruskal Wallis H. B) Structural items (all research types)
Reject and/or reject and resubmit
Re-evaluate after revision (major or minor)
–2
–1
0
1
2
Potentially acceptable after revision or accept
–2
–1
0
1
2
originality findings litreview
background discussion
Graphs by decision
*All P < 0.05 for comparison of each item across decision groups, Kruskal Wallis H. Figure 1: Distribution of items score by reviewer recommendation. Box plots depict the distribution of (A) core and (B) structural score items by publication recommendation. Lines indicate median value with box around the interquartile range (IQR). Whiskers represent non-outlier range, while absence indicates that scores do not change beyond the IQR. Absence of a line indicates that the median value is one end of the box. Dots represent outliers (color figure available online).
113
114
A. A. Onitilo et al. A) Core scores (original researchonly)
10
Core score
0
–5
–10 Reject and/or reject and resubmit
Potentially acceptable
Re-evaluate after revision
B) Structural scores (all research types)
10
5 Structural score
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
5
0
–5
–10 Reject and/or reject and resubmit
Potentially acceptable
Re-evaluate after revision Figure 2: Distribution of category scores by reviewer recommendation. Box plots demonstrate the distribution of (A) core and (B) structural scores by publication recommendation. Lines indicate median value with box around the interquartile range. Whiskers represent non-outlier range while absence indicates that scores beyond the IQR do not change. Dots represent outliers (color figure available online).
Reviewer Ratings in Manuscript Peer Review Table 1: Core Scale Reliability by Publication Recommendation (original research only) a) Reject/reject and resubmit Cronbach’s alpha 95% lower confidence limit
.8460 .6870
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Effect of dropping variables Variable dropped
Alpha
Change
Relevance Design Methods Implications
.9088 .7259 .8048 .7210
.0628 −.1201 −.0412 −.1250
b) Re-evaluate after revision Cronbach’s alpha 95% lower confidence limit
.5740 .3244
Effect of dropping variables Variable dropped
Alpha
Change
Relevance Design Methods Implications
.5854 .5407 .4961 .3595
.0114 −.0333 −.0779 −.2145
c) Potentially acceptable after revision/accept Cronbach’s alpha 95% lower confidence limit
.2714 −.3312
Effect of dropping variables Variable dropped
Alpha
Change
Relevance Design Methods Implications
.1745 .1468 .3915 .1435
−.09694 −.1246 .1201 −.1279
structural threshold (< 7) were considered intermediate, and manuscripts that met both thresholds (core ≥ 1, structural ≥ 7) were considered strong. Agreement between score-based and actual reviewer recommendation was measured using the quadratic weighted kappa and indicated good agreement (kappa = .67) even though all manuscripts and reviewers were different (Table 3).
DISCUSSION Guidelines created for the purpose of assisting reviewers and editors with the peer review process will only prove useful if they can be directly linked to the
115
116
A. A. Onitilo et al. Table 2: Structural Scale Reliability by Publication Recommendation (all research types) a) Reject/reject and resubmit Cronbach’s alpha 95% lower confidence limit
.5998 .2342
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Effect of dropping variables Variable dropped
Alpha
Change
Originality Background Findings Discussion Literature review
.5625 .4928 .5408 .4086 .6663
−.0373 −.1070 −.0590 −.1912 .0665
b) Re-evaluate after revision Cronbach’s alpha 95% lower confidence limit
.8081 .7091
Effect of dropping variables Variable dropped
Alpha
Change
Originality Background Findings Discussion Literature review
.7822 .7544 .7911 .7564 .7697
−.0259 −.0537 −.0170 −.0517 −.0384
c) Potentially acceptable after revision/accept Cronbach’s alpha 95% lower confidence limit
.6140 .3575
Effect of dropping variables Variable dropped
Alpha
Change
Originality Background Findings Discussion Literature review
.6549 .5453 .5180 .5232 .5220
.0409 −.0687 −.0960 −.0908 −.0920
editorial decision, and shift focus toward delineating specific deficiencies and providing suggestions for improvement. Such a process works to ensure that comments are constructive and avoids the common criticism that reviewers see themselves not as peers, but rather as superiors, in the hierarchy of science (Starbuck, 2003). However, neither general guidelines and instructions (Abbott, 2011; Curzon and Cleaton-Jones, 2011; Blomme, 2011) nor tools for manuscript critique (Henly and Dougherty, 2009; Sosa et al., 2009; Ramulu et al., 2005) have had much impact on reviewer reliability. Even a standardized tool implemented at the journal we help edit, Clinical Medicine and Research,
Reviewer Ratings in Manuscript Peer Review Table 3: Agreement between Score-Based Recommendation and Reviewer Recommendation (original research only) Reviewer Recommendation
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Score-Based Recommendation1
Weak Intermediate Strong Total Weighted Kappa2 Standard Error 95% CI
Reject/reject and resubmit
Re-evaluate after revision
Potentially acceptable after revision/accept
10 3 0 13 (21.0%)
10 14 7 31 (50.0%)
0 4 14 18 (29.0%)
Total
20 (32.3%) 21 (33.9%) 21 (33.9%) .666 .064 .541–.791
1 Score-based recommendation based on core and structural score thresholds: weak (core ≤ 0), intermediate (core ≥ 1, structural < 7), strong (core ≥ 1, structural ≥ 7). 2 Quadratic-weighted.
was unable to improve agreement among reviewers (Onitilo et al., 2013). Data presented here suggest that these previous attempts to refine the peer review process might have been unsuccessful because they relied on exhaustive lists of items linked to a “good” manuscript necessitating global judgments. Heddle and Ness proposed that questions need to be chosen such that they direct reviewers towards clearly stating deficiencies while providing suggestions for improvements (Heddle and Ness, 2009), and this approach has been shown to increase reviewer agreement, but only if eliciting ratings of separate, concrete items rather than a global judgment (Strayhorn et al., 1993). The peer review tool assessed in the current study was designed with these considerations in mind, and we demonstrate good reliability across a broad sample of reviewers and manuscripts. The items included in the questionnaire we have created and tested here are designed to lead reviewers and editors through the peer review process in a manner that is directly linked to two key issues. First, the peer review process is intended to provide constructive feedback for the improvement of scientific manuscripts. Requesting justification for poor scores takes a step toward ensuring that comments are constructive, fulfilling the first key issue of the peer review process. The second issue is to facilitate editorial decisions. Such a concrete item-based, decision-driven checklist, as we have described, has the potential to improve reviewer accountability by linking scores and comments to the publication recommendation. In such a system, reviewers would be expected to focus on rating the concrete items quantitatively and providing comments to explain low item scores, but would not be asked to make a publication recommendation. Quantifying checklist responses is a classic way of improving reliability since random errors tend to cancel out in the averaging
117
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
118
A. A. Onitilo et al.
process (Strayhorn et al., 1993), and the scores provide a measure that can be easily compared to some normative standard, as we have done here, and linked to an editorial decision. In practice, such normative standards would be set by the journal and would have a tendency to be revised with use and further evaluation of the checklist tool. The good agreement between checklist thresholds and reviewer recommendations demonstrated in this study suggests promise for the checklist tool proposed here, though further investigation is warranted. Editorial board members are often confronted with making publication decisions based on variable and very divergent reviews of the same manuscript. Even the most knowledgeable editors cannot be context or subject experts on every manuscript they make publication decisions on; thus they have to make their own decision informed by the quality of reviews they have received. Some reviews are detailed, thoughtful, helpful, and evidence-based. Others are brief, disjointed, and based on personal opinions. Oftentimes when editors are faced with disparate reviews, they make judgments based on factors extraneous to the manuscript, similar to those used in grant proposal evaluation. Such factors include reviewers’ and authors’ institution, qualification, country, publication history, and funding source for the research being evaluated for publication. A structured and grade-based review, as tested in our current study, with a concrete item-based, decision-driven checklist has the potential to improve the editorial decision-making process too. Study strengths include evaluation of the proposed peer review checklist tool in a heterogeneous, international population of reviewers regarding a wide variety of manuscripts, which makes these results highly generalizable. However, the overall sample was small and self-selected. There was also little qualitative feedback regarding usability of the tool via narrative comments, although the comments that were made did not suggest a specific problem. Another limitation of our study is that we do not have the final editorial decisions for the manuscripts evaluated. Since the thrust of our current study is on the reviewers’ ratings and interrater reliability, and not editors per se, we do not consider this a significant limitation. Future studies will examine the impact of the survey instrument tested in this study on the editorial decision.
CONCLUSION We demonstrate that there is considerable agreement across questions in a peer review checklist tool focused on concrete items. This is in contrast to the considerable disagreement we recently demonstrated with “usual” checklists that focus on exhaustive lists of items linked to “good” papers and ask reviewers to make global judgments (Onitilo et al., 2013). The proposed checklist asks reviewers to focus on specific, important, concrete items during the peer
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Reviewer Ratings in Manuscript Peer Review
review process and supports the provision of meaningful feedback for authors. In addition to benefiting the author, this type of review tool also provides meaningful feedback to editors, who could use quantitative results for assistance in rank ordering of articles and in the making of editorial decisions. Further testing and refinement of the CoRE checklist may result in an important tool that has the potential to improve the peer review process for authors, reviewers, and editors alike.
Conflict of Interest Statement AAO and SAS are associate and senior editor, respectively, of the journal Clinical Medicine and Research, and SAD is an editorial board member. JME and RVS are research division employees of Marshfield Clinic Research Foundation which publishes Clinical Medicine and Research.
REFERENCES Abbott, A. (2011). A how-to for peer review. Nature, 473: 17. Blomme, E. A. (2011). The ARRIVE guidelines: A resource for authors and reviewers to ensure that submissions to The Veterinary Journal meet minimal expectations of completeness, accuracy and transparency. Vet. J. 189: 237–238. Campion, M. A. (1993). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46: 705–718. Curzon, M. E. and Cleaton-Jones, P. E. (2011). Reviewing scientific manuscripts. Eur. Arch. Paediatr. Dent., 12: 184–187. Duchesne, S. and Jannin, P. (2008). Proposing a manuscript peer-review checklist. Neuroimage, 39: 1783–1787. Heddle, N. M. and Ness, P. M. (2009). Reviewing manuscripts: Tips and responsibilities. Transfusion, 49: 2265–2268. Henly, S. J. and Dougherty, M. C. (2009). Quality of manuscript reviews in nursing research. Nurs. Outlook, 57: 18–26. Loonen, M. P., Hage, J. J., and Kon, M. (2005). Who benefits from peer review? An analysis of the outcome of 100 requests for review by Plastic and Reconstructive Surgery. Plast. Reconstr. Surg., 116: 1461–1472; Discussion 1473–1475. Onitilo, A. A., Engel, J. M., Salzman-Scott, S. A., Stankowski, R. V., and Doi, S. A. (2013). Reliability of reviewer ratings in the manuscript peer review process: An opportunity for improvement. Account. Res., 20(4): 270–284. Rajesh, A., Cloud, G., and Harisinghani, M. G. (2013). Improving the quality of manuscript reviews: Impact of introducing a structured electronic template to submit reviews. Am. J. Roentgenol., 200: 20–23. Ramulu, V. G., Levine, R. B., Hebert, R. S., and Wright, S. M. (2005). Development of a case report review instrument. Int. J. Clin. Pract., 59: 457–461. Sosa, J. A., Mehta, P., Thomas, D. C., Berland, G., Gross, C., McNamara, R. L., Rosenthal, R., Udelsman, R., Bravata, D. M., and Roman, S. A. (2009). Evaluating
119
120
A. A. Onitilo et al. the surgery literature: Can standardizing peer-review today predict manuscript impact tomorrow? Ann. Surg. 250: 152–158. Starbuck, W. H. (2003). Turning lemons into lemonade: Where is the value in peer reviews? Journal of Management Inquiry. 12: 344–351.
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Strayhorn, J., Jr, McDermott, J. F., Jr., and Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. Am. J. Psychiatry, 150: 947–952.
APPENDIX 1: THE CORE-ITEM REVIEWER EVALUATION (CoRE) CHECKLIST
Scores
Question Core items (These were questions 2–5 of the online survey)
Selected score
Relevance, scientific or practical value 2 Significant impact on research/practice in this area 1 Adds new information to research/practice in this area −1 Confirms research/practice in this area or provides additional descriptive data −2 Of indirect interest to research/practice in this area Study design 2 Adequate for the purpose 1 Potentially adequate after necessary improvements −1 Probably inadequate (requires further clarification from author) 2 Clearly inadequate Experimental methods used within the study design 2 Adequate for the purpose 1 Potentially adequate after necessary improvements 0 Requires input from statistician/methodologist −1 Probably inadequate (requires further clarification from author) −2 Clearly inadequate Are the implications and recommendations warranted by the findings, and explained clearly, with appropriate caveats? 2 Adequate for the purpose 1 Potentially adequate after necessary improvements −1 Probably inadequate (requires further clarification from author) −2 Clearly inadequate Total (core)
/8
Structural items (These were questions 6–10 of the online survey)
2 1 −1
Originality New topic or new subtopic within established field New angle/update to old topic or subtopic or additional descriptive data on old topic Repetition of old work or untimely (Continued)
Reviewer Ratings in Manuscript Peer Review Appendix 1: (Continued)
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Scores
Structural items continued
Selected score
Does Introduction lay out a clear background? 2 Concise and to the point or minor edits required 1 Significant editing required −1 Too long and/or off point or too scanty and brief Presentation of results/findings 2 Adequate for the purpose 1 Potentially adequate after necessary improvements −1 Repetitive of tables/figures or unclear - significant editing required −2 Clearly inadequate – major restructuring required Discussion 2 Adequate 1 Appropriate themes but with a lot of repetition of results/findings or requires some revision for clarity or requires more focused content −1 Overly speculative beyond results/scope of manuscript −2 Wordy and poorly defined themes and/or not supported in data Does the study address the related literature adequately? 2 Adequate review of other studies that agree, disagree, or otherwise differ importantly from the concept under study 1 Addresses previous research to a large extent but requires revision for relevance and/or completeness and/or accuracy of citations −1 Inadequate review of previous research and/or major problems with citations Total (structural)
/10
Comments (Please include specific comments regarding negative scores) ∗ Item
scores are indicated by numbers next to the item. If, for example, the paper under review “adds significant impact to research/practice in this area,” then the score would be 2 for the relevance item.
121
Accountability in Research: Policies and Quality Assurance Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/gacr20
A Core-Item Reviewer Evaluation (CoRE) System for Manuscript Peer Review Adedayo A. Onitilo M.D. M.S.C.R. F.A.C.P. M.S.N. F.N.P.-B.C.
b c
a b c
, Jessica M. Engel c
, Sherry A. Salzman-Scott B.S. , Rachel V.
c
Stankowski Ph.D. & Suhail A. R. Doi F.R.C.P. Ph.D.
a
a
Clinical Epidemiology Unit , School of Population Health, University of Queensland , Brisbane , Australia b
Department of Hematology/Oncology , Marshfield Clinic Weston Center , Weston , Wisconsin , USA c
Click for updates
Marshfield Clinic Research Foundation , Marshfield , Wisconsin , USA Published online: 14 Nov 2013.
To cite this article: Adedayo A. Onitilo M.D. M.S.C.R. F.A.C.P. , Jessica M. Engel M.S.N. F.N.P.-B.C. , Sherry A. Salzman-Scott B.S. , Rachel V. Stankowski Ph.D. & Suhail A. R. Doi F.R.C.P. Ph.D. (2014) A Core-Item Reviewer Evaluation (CoRE) System for Manuscript Peer Review, Accountability in Research: Policies and Quality Assurance, 21:2, 109-121, DOI: 10.1080/08989621.2014.847664 To link to this article: http://dx.doi.org/10.1080/08989621.2014.847664
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Accountability in Research, 21:109–121, 2014 Copyright © Taylor & Francis Group, LLC ISSN: 0898-9621 print / 1545-5815 online DOI: 10.1080/08989621.2014.847664
A Core-Item Reviewer Evaluation (CoRE) System for Manuscript Peer Review Adedayo A. Onitilo, M.D., M.S.C.R., F.A.C.P.,1,2,3 Jessica M. Engel, M.S.N., F.N.P.-B.C.,2,3 Sherry A. Salzman-Scott, B.S.,3 Rachel V. Stankowski, Ph.D.,3 and Suhail A. R. Doi, F.R.C.P., Ph.D.1 1 Clinical Epidemiology Unit, School of Population Health, University of Queensland, Brisbane, Australia 2 Department of Hematology/Oncology, Marshfield Clinic Weston Center, Weston, Wisconsin, USA 3 Marshfield Clinic Research Foundation, Marshfield, Wisconsin, USA
Manuscript peer review is essential for ensuring accountability to all involved in the publication process, including authors, journals, and readers. Lack of consensus regarding what constitutes an accountable manuscript peer review process has resulted in varying practices from one journal to the next. Currently, reviewers are asked to make global judgments about various aspects of a paper for review irrespective of whether guided by a review checklist or not, and several studies have documented gross disagreement between reviewers of the same manuscript. We have previously proposed that the solution may be to direct reviewers to concrete items that do not require global judgments but rather provide a specific choice, along with referee justification for such choices. This study evaluated use of such a system via an international survey of health care professionals who had recently reviewed a health care–related manuscript. Results suggest that use of such a peer review system by reviewers does indeed improve interreviewer agreement, and thus, has the potential to support more consistent and effective peer review, if introduced into journal processes for peer review. Keywords: accountability, disagreement, peer review, rating
Address correspondence to Suhail A. R. Doi, Clinical Epidemiology Unit, School of Population Health, University of Queensland, Brisbane, Australia. E-mail: sardoi@ gmx.net
110
A. A. Onitilo et al.
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
INTRODUCTION Manuscript peer review is essential to ensuring accountability to all involved in the publication process, including authors, readers, and journals. Authors benefit by receipt of suggestions for improvement of their manuscript, readers benefit when journal standards are upheld, and journal editors benefit from guidance regarding various aspects of manuscript acceptability (Loonen et al., 2005). Many journals aim to achieve the latter by relying on a primarily subjective peer review process based on reviewer opinion and expertise. While general guidelines and instructions for reviewers do exist (Abbott, 2011; Curzon and Cleaton-Jones, 2011; Blomme, 2011), only a few refer to a specific tool for manuscript critique (Rajesh et al., 2013; Henly and Dougherty, 2009; Sosa et al., 2009; Ramulu et al., 2005). In the era of evidence-based medical practice, it is only ironic that many editorial decisions on whether or not to publish a manuscript are often based on subjective criteria, rather than evidence-based views rendered by reviewers. Standardization of the general review process alone has not been sufficient to improve reviewer agreement (Onitilo et al., 2013), probably because standardized manuscript review checklists (Duchesne and Jannin, 2008; Campion, 1993) provide exhaustive lists of items linked to a “good” paper, on which global judgments still need to be made. Heddle and Ness suggest that instead, questions should be designed to direct reviewers toward clearly describing deficiencies with suggestions for improvement (Heddle and Ness, 2009), thus shifting the focus to concrete concepts in lieu of global judgments. Indeed, Strayhorn et al. demonstrated that the reliability of ratings of scientific articles are improved if eliciting ratings of separate, concrete items rather than a global judgment (Strayhorn et al., 1993). We hypothesize that the latter problem accounted for the poor reviewer agreement we observed in a recent evaluation of a standardized review form used by the journal Clinical Medicine and Research. Each question in this form represented a global judgment rather than a concrete item, which may have resulted in continued reviewer disagreement despite standardization (Onitilo et al., 2013). In our previous paper (Onitilo et al., 2013), we suggested a prototype for a generic concrete, item-based, decision-driven checklist (Appendix 1) with the expectation that such a tool would encourage reviewers to rate these items quantitatively and require them to provide comments to justify low item scores. Quantifying responses and determining a total score would nullify the need for the reviewer to provide a publication recommendation while improving rank ordering of papers and linking of responses to an editorial decision. Here we report the results of a prospective evaluation of the reliability and consistency of this core-item reviewer evaluation (CoRE) checklist tool across a wide-spectrum of international reviewers.
Reviewer Ratings in Manuscript Peer Review
METHODS
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Reviewer Selection Participating reviewers were volunteers from the membership of two listservs. The EPIDEMIO-L listserv is sponsored by the University of Montreal and provides access to prominent members of the global epidemiological community. The Evidence Based Health (EBH) listserv is sponsored by JISC Netskills, Newcastle University, and provides access to teachers and practitioners of EBH interested in discussing issues and controversies surrounding implementation of EBH. A simple message was posted on each of these two listservs asking members who had recently reviewed a health-related journal article to complete an anonymous survey regarding that manuscript. A link to the survey hosted on surveygizmo.com was included in the message. The survey was open for 72 hrs from Sunday, April 14, 2013 through Tuesday, April 16, 2013.
Survey Tool Participants were asked to respond to 12 questions regarding the last manuscript they reviewed. Question 1 asked the participant about the type of manuscript reviewed. Questions 2–10 incorporated the eight items on the CoRE checklist shown in Appendix 1. Questions 2–5 focus on core items regarding research design and execution, while questions 6– 0 focus on other items related to manuscript structure. Question 11 asked participants to provide a recommendation regarding publication, and question 12 provided the option to add comments. Questions 2–10 were coded on a Likert scale from a minimum of -2 (weak) to a maximum of 2 (strong), depending on the question. In order to force a decision, a neutral category was not provided except for question 4 regarding study methods where the reviewer could select the option of “refer to statistician,” which was scored neutrally. Question 11 regarding publication recommendation provided the participant with three options including (1) reject/reject and resubmit, (2) re-evaluate after revision, or (3) potentially acceptable after revision/accept. Separate summary scores were created from questions in the core domain (questions 2–5) and questions in the structural domain (questions 6–10).
Statistical Analysis Agreement between summary scores and publication recommendation was assessed using a quadratic weighted kappa statistic. Cronbach’s alpha was used to determine reliability of the core and structural scales for predicting publication recommendation. Cronbach’s alpha is essentially the average measure of intraclass correlation coefficient [ICC(3,k)] and can also be used to assess
111
112
A. A. Onitilo et al.
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
agreement across questions within a scale where the questions are considered “raters” and k is the number of questions per scale. All analyses were stratified by publication recommendation and carried out using SPSS version 20 (IBM, Armonk, NY) and MedCalc version 11.6.1.0 (MedCalc Software, Ostend, Belgium).
RESULTS Seventy-two reviewers submitted a complete response to the survey sent via the listservs. Participating reviewers were from all continents, except Antarctica, and included 35 from North America, 17 from Europe, 5 from South America, 7 from Asia, and the remaining 8 from African, MiddleEastern, and Caribbean countries. Survey respondents reported 62 reviews of original research, 3 review articles, and 7 case reports or other miscellaneous manuscripts. Only those who reported review of original research were required to complete the core section of the survey (questions 2–5). Of the 14 manuscripts receiving a publication recommendation of reject, 13 were original research articles, and 1 was in the “other” category. Survey responses were assigned quantitative scores as shown in Appendix 1, and summary core and structural scores were generated by summing item scores. Item scores and summary scores for each of the two domains were distributed as shown in Figs. 1 and 2, categorized by reviewer recommendation for publication.
Scale Performance Tables 1 and 2 show the results of reliability analyses. Reliability of the questions was excellent (alpha ≥ .7) for the reject category using the core scale (Table 1) and for the re-evaluate after revision category using the structural scale (Table 2). Given that Cronbach’s alpha is equal to ICC(3,k) and considering within scale questions to be the “raters,” then reviewers whose opinions were reject tended to agree on their average core item scores, and reviewers whose opinions were revise tended to agree on their average structural item scores.
Agreement by Publication Recommendation Based on results of the reliability analysis, cutoff scores were set to delineate publication recommendation categories. A cutoff for the core score based on the 75th percentile within the reject category was used as the threshold for rejection (score > 0). Similarly, the 75th percentile within the revise category of the structural score was used as a threshold for acceptance (score ≥ 7). Manuscripts that did not meet the core score threshold (≤ 0) were considered weak; manuscripts that met the core threshold (≥ 1) but not the
Reviewer Ratings in Manuscript Peer Review A) Core items (original research only)
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Reject and/or reject and resubmit
Re-evaluate after revision (major or minor)
–2
–1
0
1
2
Potentially acceptable after revision or accept
–2
–1
0
1
2
relevance methods
design implications
Graphs by decision
*All P < 0.05 for comparison of each item across decision groups, Kruskal Wallis H. B) Structural items (all research types)
Reject and/or reject and resubmit
Re-evaluate after revision (major or minor)
–2
–1
0
1
2
Potentially acceptable after revision or accept
–2
–1
0
1
2
originality findings litreview
background discussion
Graphs by decision
*All P < 0.05 for comparison of each item across decision groups, Kruskal Wallis H. Figure 1: Distribution of items score by reviewer recommendation. Box plots depict the distribution of (A) core and (B) structural score items by publication recommendation. Lines indicate median value with box around the interquartile range (IQR). Whiskers represent non-outlier range, while absence indicates that scores do not change beyond the IQR. Absence of a line indicates that the median value is one end of the box. Dots represent outliers (color figure available online).
113
114
A. A. Onitilo et al. A) Core scores (original researchonly)
10
Core score
0
–5
–10 Reject and/or reject and resubmit
Potentially acceptable
Re-evaluate after revision
B) Structural scores (all research types)
10
5 Structural score
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
5
0
–5
–10 Reject and/or reject and resubmit
Potentially acceptable
Re-evaluate after revision Figure 2: Distribution of category scores by reviewer recommendation. Box plots demonstrate the distribution of (A) core and (B) structural scores by publication recommendation. Lines indicate median value with box around the interquartile range. Whiskers represent non-outlier range while absence indicates that scores beyond the IQR do not change. Dots represent outliers (color figure available online).
Reviewer Ratings in Manuscript Peer Review Table 1: Core Scale Reliability by Publication Recommendation (original research only) a) Reject/reject and resubmit Cronbach’s alpha 95% lower confidence limit
.8460 .6870
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Effect of dropping variables Variable dropped
Alpha
Change
Relevance Design Methods Implications
.9088 .7259 .8048 .7210
.0628 −.1201 −.0412 −.1250
b) Re-evaluate after revision Cronbach’s alpha 95% lower confidence limit
.5740 .3244
Effect of dropping variables Variable dropped
Alpha
Change
Relevance Design Methods Implications
.5854 .5407 .4961 .3595
.0114 −.0333 −.0779 −.2145
c) Potentially acceptable after revision/accept Cronbach’s alpha 95% lower confidence limit
.2714 −.3312
Effect of dropping variables Variable dropped
Alpha
Change
Relevance Design Methods Implications
.1745 .1468 .3915 .1435
−.09694 −.1246 .1201 −.1279
structural threshold (< 7) were considered intermediate, and manuscripts that met both thresholds (core ≥ 1, structural ≥ 7) were considered strong. Agreement between score-based and actual reviewer recommendation was measured using the quadratic weighted kappa and indicated good agreement (kappa = .67) even though all manuscripts and reviewers were different (Table 3).
DISCUSSION Guidelines created for the purpose of assisting reviewers and editors with the peer review process will only prove useful if they can be directly linked to the
115
116
A. A. Onitilo et al. Table 2: Structural Scale Reliability by Publication Recommendation (all research types) a) Reject/reject and resubmit Cronbach’s alpha 95% lower confidence limit
.5998 .2342
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Effect of dropping variables Variable dropped
Alpha
Change
Originality Background Findings Discussion Literature review
.5625 .4928 .5408 .4086 .6663
−.0373 −.1070 −.0590 −.1912 .0665
b) Re-evaluate after revision Cronbach’s alpha 95% lower confidence limit
.8081 .7091
Effect of dropping variables Variable dropped
Alpha
Change
Originality Background Findings Discussion Literature review
.7822 .7544 .7911 .7564 .7697
−.0259 −.0537 −.0170 −.0517 −.0384
c) Potentially acceptable after revision/accept Cronbach’s alpha 95% lower confidence limit
.6140 .3575
Effect of dropping variables Variable dropped
Alpha
Change
Originality Background Findings Discussion Literature review
.6549 .5453 .5180 .5232 .5220
.0409 −.0687 −.0960 −.0908 −.0920
editorial decision, and shift focus toward delineating specific deficiencies and providing suggestions for improvement. Such a process works to ensure that comments are constructive and avoids the common criticism that reviewers see themselves not as peers, but rather as superiors, in the hierarchy of science (Starbuck, 2003). However, neither general guidelines and instructions (Abbott, 2011; Curzon and Cleaton-Jones, 2011; Blomme, 2011) nor tools for manuscript critique (Henly and Dougherty, 2009; Sosa et al., 2009; Ramulu et al., 2005) have had much impact on reviewer reliability. Even a standardized tool implemented at the journal we help edit, Clinical Medicine and Research,
Reviewer Ratings in Manuscript Peer Review Table 3: Agreement between Score-Based Recommendation and Reviewer Recommendation (original research only) Reviewer Recommendation
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Score-Based Recommendation1
Weak Intermediate Strong Total Weighted Kappa2 Standard Error 95% CI
Reject/reject and resubmit
Re-evaluate after revision
Potentially acceptable after revision/accept
10 3 0 13 (21.0%)
10 14 7 31 (50.0%)
0 4 14 18 (29.0%)
Total
20 (32.3%) 21 (33.9%) 21 (33.9%) .666 .064 .541–.791
1 Score-based recommendation based on core and structural score thresholds: weak (core ≤ 0), intermediate (core ≥ 1, structural < 7), strong (core ≥ 1, structural ≥ 7). 2 Quadratic-weighted.
was unable to improve agreement among reviewers (Onitilo et al., 2013). Data presented here suggest that these previous attempts to refine the peer review process might have been unsuccessful because they relied on exhaustive lists of items linked to a “good” manuscript necessitating global judgments. Heddle and Ness proposed that questions need to be chosen such that they direct reviewers towards clearly stating deficiencies while providing suggestions for improvements (Heddle and Ness, 2009), and this approach has been shown to increase reviewer agreement, but only if eliciting ratings of separate, concrete items rather than a global judgment (Strayhorn et al., 1993). The peer review tool assessed in the current study was designed with these considerations in mind, and we demonstrate good reliability across a broad sample of reviewers and manuscripts. The items included in the questionnaire we have created and tested here are designed to lead reviewers and editors through the peer review process in a manner that is directly linked to two key issues. First, the peer review process is intended to provide constructive feedback for the improvement of scientific manuscripts. Requesting justification for poor scores takes a step toward ensuring that comments are constructive, fulfilling the first key issue of the peer review process. The second issue is to facilitate editorial decisions. Such a concrete item-based, decision-driven checklist, as we have described, has the potential to improve reviewer accountability by linking scores and comments to the publication recommendation. In such a system, reviewers would be expected to focus on rating the concrete items quantitatively and providing comments to explain low item scores, but would not be asked to make a publication recommendation. Quantifying checklist responses is a classic way of improving reliability since random errors tend to cancel out in the averaging
117
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
118
A. A. Onitilo et al.
process (Strayhorn et al., 1993), and the scores provide a measure that can be easily compared to some normative standard, as we have done here, and linked to an editorial decision. In practice, such normative standards would be set by the journal and would have a tendency to be revised with use and further evaluation of the checklist tool. The good agreement between checklist thresholds and reviewer recommendations demonstrated in this study suggests promise for the checklist tool proposed here, though further investigation is warranted. Editorial board members are often confronted with making publication decisions based on variable and very divergent reviews of the same manuscript. Even the most knowledgeable editors cannot be context or subject experts on every manuscript they make publication decisions on; thus they have to make their own decision informed by the quality of reviews they have received. Some reviews are detailed, thoughtful, helpful, and evidence-based. Others are brief, disjointed, and based on personal opinions. Oftentimes when editors are faced with disparate reviews, they make judgments based on factors extraneous to the manuscript, similar to those used in grant proposal evaluation. Such factors include reviewers’ and authors’ institution, qualification, country, publication history, and funding source for the research being evaluated for publication. A structured and grade-based review, as tested in our current study, with a concrete item-based, decision-driven checklist has the potential to improve the editorial decision-making process too. Study strengths include evaluation of the proposed peer review checklist tool in a heterogeneous, international population of reviewers regarding a wide variety of manuscripts, which makes these results highly generalizable. However, the overall sample was small and self-selected. There was also little qualitative feedback regarding usability of the tool via narrative comments, although the comments that were made did not suggest a specific problem. Another limitation of our study is that we do not have the final editorial decisions for the manuscripts evaluated. Since the thrust of our current study is on the reviewers’ ratings and interrater reliability, and not editors per se, we do not consider this a significant limitation. Future studies will examine the impact of the survey instrument tested in this study on the editorial decision.
CONCLUSION We demonstrate that there is considerable agreement across questions in a peer review checklist tool focused on concrete items. This is in contrast to the considerable disagreement we recently demonstrated with “usual” checklists that focus on exhaustive lists of items linked to “good” papers and ask reviewers to make global judgments (Onitilo et al., 2013). The proposed checklist asks reviewers to focus on specific, important, concrete items during the peer
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Reviewer Ratings in Manuscript Peer Review
review process and supports the provision of meaningful feedback for authors. In addition to benefiting the author, this type of review tool also provides meaningful feedback to editors, who could use quantitative results for assistance in rank ordering of articles and in the making of editorial decisions. Further testing and refinement of the CoRE checklist may result in an important tool that has the potential to improve the peer review process for authors, reviewers, and editors alike.
Conflict of Interest Statement AAO and SAS are associate and senior editor, respectively, of the journal Clinical Medicine and Research, and SAD is an editorial board member. JME and RVS are research division employees of Marshfield Clinic Research Foundation which publishes Clinical Medicine and Research.
REFERENCES Abbott, A. (2011). A how-to for peer review. Nature, 473: 17. Blomme, E. A. (2011). The ARRIVE guidelines: A resource for authors and reviewers to ensure that submissions to The Veterinary Journal meet minimal expectations of completeness, accuracy and transparency. Vet. J. 189: 237–238. Campion, M. A. (1993). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46: 705–718. Curzon, M. E. and Cleaton-Jones, P. E. (2011). Reviewing scientific manuscripts. Eur. Arch. Paediatr. Dent., 12: 184–187. Duchesne, S. and Jannin, P. (2008). Proposing a manuscript peer-review checklist. Neuroimage, 39: 1783–1787. Heddle, N. M. and Ness, P. M. (2009). Reviewing manuscripts: Tips and responsibilities. Transfusion, 49: 2265–2268. Henly, S. J. and Dougherty, M. C. (2009). Quality of manuscript reviews in nursing research. Nurs. Outlook, 57: 18–26. Loonen, M. P., Hage, J. J., and Kon, M. (2005). Who benefits from peer review? An analysis of the outcome of 100 requests for review by Plastic and Reconstructive Surgery. Plast. Reconstr. Surg., 116: 1461–1472; Discussion 1473–1475. Onitilo, A. A., Engel, J. M., Salzman-Scott, S. A., Stankowski, R. V., and Doi, S. A. (2013). Reliability of reviewer ratings in the manuscript peer review process: An opportunity for improvement. Account. Res., 20(4): 270–284. Rajesh, A., Cloud, G., and Harisinghani, M. G. (2013). Improving the quality of manuscript reviews: Impact of introducing a structured electronic template to submit reviews. Am. J. Roentgenol., 200: 20–23. Ramulu, V. G., Levine, R. B., Hebert, R. S., and Wright, S. M. (2005). Development of a case report review instrument. Int. J. Clin. Pract., 59: 457–461. Sosa, J. A., Mehta, P., Thomas, D. C., Berland, G., Gross, C., McNamara, R. L., Rosenthal, R., Udelsman, R., Bravata, D. M., and Roman, S. A. (2009). Evaluating
119
120
A. A. Onitilo et al. the surgery literature: Can standardizing peer-review today predict manuscript impact tomorrow? Ann. Surg. 250: 152–158. Starbuck, W. H. (2003). Turning lemons into lemonade: Where is the value in peer reviews? Journal of Management Inquiry. 12: 344–351.
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Strayhorn, J., Jr, McDermott, J. F., Jr., and Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. Am. J. Psychiatry, 150: 947–952.
APPENDIX 1: THE CORE-ITEM REVIEWER EVALUATION (CoRE) CHECKLIST
Scores
Question Core items (These were questions 2–5 of the online survey)
Selected score
Relevance, scientific or practical value 2 Significant impact on research/practice in this area 1 Adds new information to research/practice in this area −1 Confirms research/practice in this area or provides additional descriptive data −2 Of indirect interest to research/practice in this area Study design 2 Adequate for the purpose 1 Potentially adequate after necessary improvements −1 Probably inadequate (requires further clarification from author) 2 Clearly inadequate Experimental methods used within the study design 2 Adequate for the purpose 1 Potentially adequate after necessary improvements 0 Requires input from statistician/methodologist −1 Probably inadequate (requires further clarification from author) −2 Clearly inadequate Are the implications and recommendations warranted by the findings, and explained clearly, with appropriate caveats? 2 Adequate for the purpose 1 Potentially adequate after necessary improvements −1 Probably inadequate (requires further clarification from author) −2 Clearly inadequate Total (core)
/8
Structural items (These were questions 6–10 of the online survey)
2 1 −1
Originality New topic or new subtopic within established field New angle/update to old topic or subtopic or additional descriptive data on old topic Repetition of old work or untimely (Continued)
Reviewer Ratings in Manuscript Peer Review Appendix 1: (Continued)
Downloaded by [University of Tennessee, Knoxville] at 18:50 05 January 2015
Scores
Structural items continued
Selected score
Does Introduction lay out a clear background? 2 Concise and to the point or minor edits required 1 Significant editing required −1 Too long and/or off point or too scanty and brief Presentation of results/findings 2 Adequate for the purpose 1 Potentially adequate after necessary improvements −1 Repetitive of tables/figures or unclear - significant editing required −2 Clearly inadequate – major restructuring required Discussion 2 Adequate 1 Appropriate themes but with a lot of repetition of results/findings or requires some revision for clarity or requires more focused content −1 Overly speculative beyond results/scope of manuscript −2 Wordy and poorly defined themes and/or not supported in data Does the study address the related literature adequately? 2 Adequate review of other studies that agree, disagree, or otherwise differ importantly from the concept under study 1 Addresses previous research to a large extent but requires revision for relevance and/or completeness and/or accuracy of citations −1 Inadequate review of previous research and/or major problems with citations Total (structural)
/10
Comments (Please include specific comments regarding negative scores) ∗ Item
scores are indicated by numbers next to the item. If, for example, the paper under review “adds significant impact to research/practice in this area,” then the score would be 2 for the relevance item.
121
