Editorial Downloaded from www.ajronline.org by The University of Manchester on 01/23/15 from IP address 130.88.90.140. Copyright ARRS. For personal use only; all rights reserved
Reviewer Development: New Initiatives
W
e continue to examine new approaches to assist our more than 2000 American Journal of Roentgenology reviewers. Since 2011, we have dedicated eight editorials to reviewer assistance issues. The August through October 2014 editorials provided reviewer assistance focused on three separate topics [1–3]. Earlier editorials discussed the value of close communication between reviewers and editors, mentoring, and monitoring reviewer quality. All of these efforts are critical to ensure consistent quality in our peer review process [4]. Over the past few years, we have expanded our reviewer assistance programs to include a required interactive WebEx session for all new reviewers; improved our online reviewer assistance tools; and, beginning in 2015, we will present focused topics for reviewers at the Reviewers’ Luncheons at all ARRS Annual Meetings [5]. Providing feedback to reviewers and section editors is critical for our reviewer assistance and quality monitoring processes. The new reviewer orientation provides the opportunity to define the expectations for AJR reviewers and also enables the new or any attending reviewers to raise concerns and ask questions. In addition, the names of all new reviewers attending the orientation sessions are provided to the section editors so they can select new reviewers in their first year to accomplish at least four reviews. We will soon begin meeting with those who have provided reviews to discuss their progress and address any issues they have experienced in the process. Thus far, we have held six new reviewer orientations, with 58 attendees. The new reviewers have completed more than 144 reviews, with 11 more in progress. New sessions will begin soon for the 51 reviewers who have not been able to attend a previous session. In the past, section editors reviewed their reviewer pools at the end of each year to determine whether reviewers were active or inactive, compared reviewer scores, and assisted with selection of key reviewers for their sections. Inactive reviewers were contacted to determine whether they were still interested in reviewing, and all reviewers were given the opportunity to modify their focused areas of expertise. The latter can be
modified at any time by notifying the journal staff. Beginning in September 2014, we began a new reviewer monitoring system designed to provide the editor in chief the ability to evaluate all section reviewers on a monthly rotating basis. This system enables more organized and defined information for each section to assist with evaluation of the reviewer pools. Each section editor is provided with the following information: 1. Total number of reviewers 2. Reviewers with no invitations and percentage of total 3. Reviewers with scores less than 3 and percentage of total 4. Reviewers with scores 3.5 or above but less than 4 and percentage of total 5. Reviewers with scores of 4 (highest score possible) and percentage of total 6. Reviewers who have failed to accept invitations with associated data: for example, invitations 94, completed reviews 0, terminated after accepting due to noncompliance 4, uninvited 90 The names of the reviewers are also provided for categories 4, 5, and 6. In 2013, we developed a scoring system for reviewers ranging from 1 to 4, with 4 the highest, and associated criteria on the basis of work by Mark Kliewer et al. [6] and Aruguman Rajesh et al. [7]. This scale is still available for use by section editors. However, because we have several new section editors and because of the available reviewer data, we developed another set of guidelines for grading reviewers to improve scoring consistency. The new suggested guidelines are summarized as follows: 0 Points: Review does not follow reviewer guidelines No comments or summary for the editor No constructive comments or lack of author comments Unfavorable timeliness (> 14 days to complete review) Decision does not match manuscript grading questions 1 Point: Comment summary to editor and authors completed Review does not follow reviewer guidelines Lack of constructive or sophisticated
comments to authors Unfavorable timeliness (> 14 days to complete review) Decision does not match manuscript grading questions 2 Points: Comment summary somewhat helpful to the authors Lack of structure (no comments by section, page, or line numbers) Timely completion of the review (≤ 14 days) Decision matches manuscript grading questions 3 Points: Comment summary to editor completed and matches decision Comments to author helpful Timely completion of review (≤ 14 days) Decision matches manuscript grading questions 4 Points: Sophisticated detailed comment summary to author contains manuscript section detail with page and line documentation Comments enhance the merits and relevance of the work Concise organized comments to editor match decision Timely or early review completion of review (≤ 14 days) Decision matches manuscript grading questions Reviewers also need to keep in mind that to receive CME credit for their reviews they must be active members of the ARRS and achieve a reviewer grade of at least 2. Hopefully, these comments will assist reviewers and enable others involved in the publication process to understand the importance of quality reviews and timely decisions on submitted manuscripts. In addition to monitoring the quality of reviews, we make every attempt to reward excellent reviews or “above and beyond” efforts by our AJR reviewers. Section editors are encouraged to communicate with reviewers who provide excellent reviews and reviewers who need assistance to improve their reviewer scores. As described, we plan to meet with new reviewers after the first four reviews or their first year to provide them with feedback and address any issues or concerns they may be experiencing.
AJR:204, January 2015 1
Downloaded from www.ajronline.org by The University of Manchester on 01/23/15 from IP address 130.88.90.140. Copyright ARRS. For personal use only; all rights reserved
Editorial We also reward reviewers with scores greater than or equal to 3 and those who provide more than 12 reviews per year by placing their names on the masthead page of the journal [8]. Beginning with the January 2015 AJR EIC Newsletter, we are creating a Reviewer Honor Roll for reviewers who have performed above and beyond related to specific accomplishments. Examples may include prompt responses and reviews to special requests, consistent timeliness, specific subspecialty expertise, statistical analysis issues, and so on. We believe this new feature will provide additional positive feedback to our reviewers and will serve as an additional reminder of
the importance of consistent, thorough, and timely manuscript reviews. Thomas H. Berquist Editor in Chief [email protected] DOI:10.2214/AJR.14.14024
References 1. Berquist TH. Reviewer assistance: our support efforts must continue! AJR 2014; 203:233–234 2. Berquist TH. Reviewer assistance: how important is the invitation letter? AJR 2014; 203:467 3. Berquist TH. Reviewer assistance: focus on communication with authors and manuscript decisions. AJR 2014; 203:697–698
4. Berquist TH. Editors and reviewers: roles and responsibilities. AJR 2012; 198:245 5. Berquist TH. The new reviewer assistance program: a multipronged approach! AJR 2013; 201:1 6. Kliewer MA, Freed KS, De Long DM, Pickhardt PJ, Provenzale JM. Reviewing the reviewers: comparison of quality and reviewer characteristics at the American Journal of Roentgenology. AJR 2005; 184:1731–1735 7. Rajesh A, Cloud G, Harisinghani MG. Improving the quality of manuscript reviews: impact of submitting a structured electronic template to submit reviews. AJR 2013; 200:20–23 8. Berquist TH. AJR reviewers: we need to continue to improve our education and monitoring methods. AJR 2013; 200:1179–1180
F O R YO U R I N F O R M AT I O N
Mark your calendar for the following ARRS annual meetings: April 19–24, 2015—Toronto Convention Centre, Toronto, ON, Canada April 17–22, 2016—Los Angeles Convention Center, Los Angeles, CA April 30–May 5, 2017—Hyatt Regency New Orleans, New Orleans, LA April 22–27, 2018—Marriott Wardman Park Hotel, Washington DC
2
AJR:204, January 2015
Reviewer Development: New Initiatives
W
e continue to examine new approaches to assist our more than 2000 American Journal of Roentgenology reviewers. Since 2011, we have dedicated eight editorials to reviewer assistance issues. The August through October 2014 editorials provided reviewer assistance focused on three separate topics [1–3]. Earlier editorials discussed the value of close communication between reviewers and editors, mentoring, and monitoring reviewer quality. All of these efforts are critical to ensure consistent quality in our peer review process [4]. Over the past few years, we have expanded our reviewer assistance programs to include a required interactive WebEx session for all new reviewers; improved our online reviewer assistance tools; and, beginning in 2015, we will present focused topics for reviewers at the Reviewers’ Luncheons at all ARRS Annual Meetings [5]. Providing feedback to reviewers and section editors is critical for our reviewer assistance and quality monitoring processes. The new reviewer orientation provides the opportunity to define the expectations for AJR reviewers and also enables the new or any attending reviewers to raise concerns and ask questions. In addition, the names of all new reviewers attending the orientation sessions are provided to the section editors so they can select new reviewers in their first year to accomplish at least four reviews. We will soon begin meeting with those who have provided reviews to discuss their progress and address any issues they have experienced in the process. Thus far, we have held six new reviewer orientations, with 58 attendees. The new reviewers have completed more than 144 reviews, with 11 more in progress. New sessions will begin soon for the 51 reviewers who have not been able to attend a previous session. In the past, section editors reviewed their reviewer pools at the end of each year to determine whether reviewers were active or inactive, compared reviewer scores, and assisted with selection of key reviewers for their sections. Inactive reviewers were contacted to determine whether they were still interested in reviewing, and all reviewers were given the opportunity to modify their focused areas of expertise. The latter can be
modified at any time by notifying the journal staff. Beginning in September 2014, we began a new reviewer monitoring system designed to provide the editor in chief the ability to evaluate all section reviewers on a monthly rotating basis. This system enables more organized and defined information for each section to assist with evaluation of the reviewer pools. Each section editor is provided with the following information: 1. Total number of reviewers 2. Reviewers with no invitations and percentage of total 3. Reviewers with scores less than 3 and percentage of total 4. Reviewers with scores 3.5 or above but less than 4 and percentage of total 5. Reviewers with scores of 4 (highest score possible) and percentage of total 6. Reviewers who have failed to accept invitations with associated data: for example, invitations 94, completed reviews 0, terminated after accepting due to noncompliance 4, uninvited 90 The names of the reviewers are also provided for categories 4, 5, and 6. In 2013, we developed a scoring system for reviewers ranging from 1 to 4, with 4 the highest, and associated criteria on the basis of work by Mark Kliewer et al. [6] and Aruguman Rajesh et al. [7]. This scale is still available for use by section editors. However, because we have several new section editors and because of the available reviewer data, we developed another set of guidelines for grading reviewers to improve scoring consistency. The new suggested guidelines are summarized as follows: 0 Points: Review does not follow reviewer guidelines No comments or summary for the editor No constructive comments or lack of author comments Unfavorable timeliness (> 14 days to complete review) Decision does not match manuscript grading questions 1 Point: Comment summary to editor and authors completed Review does not follow reviewer guidelines Lack of constructive or sophisticated
comments to authors Unfavorable timeliness (> 14 days to complete review) Decision does not match manuscript grading questions 2 Points: Comment summary somewhat helpful to the authors Lack of structure (no comments by section, page, or line numbers) Timely completion of the review (≤ 14 days) Decision matches manuscript grading questions 3 Points: Comment summary to editor completed and matches decision Comments to author helpful Timely completion of review (≤ 14 days) Decision matches manuscript grading questions 4 Points: Sophisticated detailed comment summary to author contains manuscript section detail with page and line documentation Comments enhance the merits and relevance of the work Concise organized comments to editor match decision Timely or early review completion of review (≤ 14 days) Decision matches manuscript grading questions Reviewers also need to keep in mind that to receive CME credit for their reviews they must be active members of the ARRS and achieve a reviewer grade of at least 2. Hopefully, these comments will assist reviewers and enable others involved in the publication process to understand the importance of quality reviews and timely decisions on submitted manuscripts. In addition to monitoring the quality of reviews, we make every attempt to reward excellent reviews or “above and beyond” efforts by our AJR reviewers. Section editors are encouraged to communicate with reviewers who provide excellent reviews and reviewers who need assistance to improve their reviewer scores. As described, we plan to meet with new reviewers after the first four reviews or their first year to provide them with feedback and address any issues or concerns they may be experiencing.
AJR:204, January 2015 1
Downloaded from www.ajronline.org by The University of Manchester on 01/23/15 from IP address 130.88.90.140. Copyright ARRS. For personal use only; all rights reserved
Editorial We also reward reviewers with scores greater than or equal to 3 and those who provide more than 12 reviews per year by placing their names on the masthead page of the journal [8]. Beginning with the January 2015 AJR EIC Newsletter, we are creating a Reviewer Honor Roll for reviewers who have performed above and beyond related to specific accomplishments. Examples may include prompt responses and reviews to special requests, consistent timeliness, specific subspecialty expertise, statistical analysis issues, and so on. We believe this new feature will provide additional positive feedback to our reviewers and will serve as an additional reminder of
the importance of consistent, thorough, and timely manuscript reviews. Thomas H. Berquist Editor in Chief [email protected] DOI:10.2214/AJR.14.14024
References 1. Berquist TH. Reviewer assistance: our support efforts must continue! AJR 2014; 203:233–234 2. Berquist TH. Reviewer assistance: how important is the invitation letter? AJR 2014; 203:467 3. Berquist TH. Reviewer assistance: focus on communication with authors and manuscript decisions. AJR 2014; 203:697–698
4. Berquist TH. Editors and reviewers: roles and responsibilities. AJR 2012; 198:245 5. Berquist TH. The new reviewer assistance program: a multipronged approach! AJR 2013; 201:1 6. Kliewer MA, Freed KS, De Long DM, Pickhardt PJ, Provenzale JM. Reviewing the reviewers: comparison of quality and reviewer characteristics at the American Journal of Roentgenology. AJR 2005; 184:1731–1735 7. Rajesh A, Cloud G, Harisinghani MG. Improving the quality of manuscript reviews: impact of submitting a structured electronic template to submit reviews. AJR 2013; 200:20–23 8. Berquist TH. AJR reviewers: we need to continue to improve our education and monitoring methods. AJR 2013; 200:1179–1180
F O R YO U R I N F O R M AT I O N
Mark your calendar for the following ARRS annual meetings: April 19–24, 2015—Toronto Convention Centre, Toronto, ON, Canada April 17–22, 2016—Los Angeles Convention Center, Los Angeles, CA April 30–May 5, 2017—Hyatt Regency New Orleans, New Orleans, LA April 22–27, 2018—Marriott Wardman Park Hotel, Washington DC
2
AJR:204, January 2015
