thesis
Double-blind under review Can peer review be improved by withholding information from referees? There is some evidence to suggest it might be, but the jury is still out, reports Alastair Brown. Peer review is a hallmark of a scientific paper. The process, in which qualified individuals (‘peers’) check over a paper for technical and logical flaws, is widely accepted to provide an important quality filter. It encourages authors to be more rigorous and can identify problems when standards drop. So although it is, of course, not perfect 1, it is generally considered to be a good thing. But is there room for improvement? The most common approach to peer review is single-blind review, where the authors are known to the referees but the referees remain anonymous to the authors. This system protects the identity of the referees so that they feel they can comment frankly without fear of repercussions. In the current drive for change, there seems to be two distinct directions: a push for increased transparency (open peer review) and a push for increased objectivity (doubleblind peer review). In principle, full transparency is a good thing, provided that people are sufficiently rational. The argument in favour of preventing full transparency, and thus withholding the identities of the authors and referees (double-blind review), is that this isn’t always the case. In particular, there are concerns that the attributes of the authors, rather than just the science they present, could influence referees in their appraisal of a paper. Common concerns include bias related to gender, institutional membership, nationality and previous publication record. In practice, we all have preconceived ideas that can bias our judgments, despite our best efforts to avoid them2,3. This need to guard against ourselves is something that should be familiar to scientists. As Richard Feynman put it, “The first principle is that you must not fool yourself, and you are the easiest person to fool.” Indeed, it is precisely to avoid fooling ourselves that many of the systematic and rigorous approaches of science have been developed. Doubleblind, it could be argued attempts to apply such ideas to peer review and take a more scientific approach to the process. Although data on the extent to which bias operates in peer review is largely
lacking, surveys consistently show a high level of support for double-blind review across a range of fields4–6, which suggests that the perception of the potential for referee bias is widespread. Given the strong support for double-blind peer review in principle, two Nature research journals — Nature Climate Change (where I am an editor) and Nature Geoscience — have been trialling double-blind peer review as an option for authors since June 20137,8. Although it is too early to draw any firm conclusions, some interesting patterns have emerged. An ongoing survey of authors that have submitted manuscripts to the two journals indicates that over the first year of the trial around 80% of respondents were supportive of double-blind review in principle. However, uptake of the doubleblind option has been relatively low: taking the two journals together, the percentage of monthly submissions that have chosen this option has never risen above around 20%. There is clearly a disparity between support for the idea of double-blind and what authors are actually choosing for their own papers. There are a number of possible explanations for this. First, authors may be reluctant to have their work be part of a trial: like a new or unfamiliar technology, there are typically a small number of early adopters and most people only follow once the approach has had time to be proven reliable. Second, because the trial has not been running for very long, author awareness remains quite low. Consequently, authors frequently only become aware of the double-blind option once they have already finalized their manuscript. At this point, having to go back and remove author details and alter text (removing phrases like ‘as we showed previously’) represents additional work and can delay submission. Such concerns were frequently cited as a reason for not selecting double-blind in our author survey. It is difficult to know how large an effect this might be. However, if it is large we would expect a gradual increase in double-blind submissions as the trial continues and awareness grows — this has not been particularly evident so far.
NATURE NANOTECHNOLOGY | VOL 9 | NOVEMBER 2014 | www.nature.com/naturenanotechnology
© 2014 Macmillan Publishers Limited. All rights reserved
A third explanation is that because the trial is elective, authors believe that by choosing double-blind they are lumping themselves into a category that will be presumed to contain the work of less prominent individuals. A fairly typical survey comment in this regard is: “Because it was voluntary, that means that the reviewer will know that the authors are likely to have all or some of the characteristics that people bias against.” Again the size of this effect is difficult to estimate, but presumably it would diminish if a much higher proportion of authors chose double-blind review. Another common belief is that referees will be able to guess the identity of the authors, so a double-blind process will make no practical difference. From this perspective, single-blind is a more accurate and realistic model, so why change. The referees might correctly identify the authors in some cases, but as editors we know from past experience that attempts by authors to guess the identities of referees are very often wrong and it seems likely that referees will be no more successful. There is also some evidence that, even in small research fields, around 50% of referees do not suspect the identities of the authors during doubleblind review 5. However, the American Economic Review dropped double-blind review in 2011, citing the role of search engines in limiting their capacity to maintain author anonymity as a major reason for the decision. The accuracy with which referees can guess the identities of authors is likely to be quite field-specific and this is something we are looking to investigate as part of our ongoing trial. We also have some anecdotal feedback, from discussions with individuals, that hasn’t emerged from the survey. One of the recurring arguments can be summarized as ‘reputations should count’: that is, if you have worked in a field for a decade and produced thorough, reliable work then that is useful information for referees to know. For example, the efforts of referees needs to be targeted, so knowing that findings are from a laboratory that has a long history of accurate, well-calibrated measurement is 871
thesis useful and important information. The key question here is whether the loss of useful information associated with knowing the authors’ identities outweigh the potential benefits of a potentially fairer judgment of a paper’s content? This remains an open question at present. Surprisingly, given the important function that peer review is intended to perform in academic publishing, very little analysis has been brought to bear on the question of its efficacy in general or the relative merits of different peer-review
872
models in particular 9,10. There is reason to suspect that bias could be a problem in peer review and, even in the absence of firm supporting data, double-blind review is a sensible strategy to reduce this potential. Only time will tell whether double-blind proves popular in practice and whether it discernibly alters what gets published. ❐ Alastair Brown is a Senior Editor at Nature Climate Change. e-mail: [email protected]
References
1. Godlee, F., Gale, C. R. & Martyn, C. N. J. Am. Med. Assoc. 280, 237–240 (1998). 2. Goldin, C. & Rose, C. The Am. Economic Rev. 90, 715–741 (2001). 3. Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J. & Handelsman, J. Proc. Natl Acad. Sci. USA 109, 16474–16479 (2012). 4. Regehr, G. & Bordage, G. Med. Educ. 40, 832–839 (2006). 5. Jagsi, R. et al. Int. J. Radiation Oncol. Biol. Phys. 89, 940–946 (2014). 6. Nature Geosci. 5, 585 (2012). 7. Nature Geosci. 6, 413 (2013). 8. Nature Clim. Change 3, 525 (2013). 9. Darling, E. Conserv. Biol. http://dx.doi.org/10.1111/cobi.12333 (2014). 10. Jefferson, T. et al. J. Am. Med. Assoc. 287, 2784–2786 (2002).
Published online: 2 November 2014
NATURE NANOTECHNOLOGY | VOL 9 | NOVEMBER 2014 | www.nature.com/naturenanotechnology
© 2014 Macmillan Publishers Limited. All rights reserved
Double-blind under review Can peer review be improved by withholding information from referees? There is some evidence to suggest it might be, but the jury is still out, reports Alastair Brown. Peer review is a hallmark of a scientific paper. The process, in which qualified individuals (‘peers’) check over a paper for technical and logical flaws, is widely accepted to provide an important quality filter. It encourages authors to be more rigorous and can identify problems when standards drop. So although it is, of course, not perfect 1, it is generally considered to be a good thing. But is there room for improvement? The most common approach to peer review is single-blind review, where the authors are known to the referees but the referees remain anonymous to the authors. This system protects the identity of the referees so that they feel they can comment frankly without fear of repercussions. In the current drive for change, there seems to be two distinct directions: a push for increased transparency (open peer review) and a push for increased objectivity (doubleblind peer review). In principle, full transparency is a good thing, provided that people are sufficiently rational. The argument in favour of preventing full transparency, and thus withholding the identities of the authors and referees (double-blind review), is that this isn’t always the case. In particular, there are concerns that the attributes of the authors, rather than just the science they present, could influence referees in their appraisal of a paper. Common concerns include bias related to gender, institutional membership, nationality and previous publication record. In practice, we all have preconceived ideas that can bias our judgments, despite our best efforts to avoid them2,3. This need to guard against ourselves is something that should be familiar to scientists. As Richard Feynman put it, “The first principle is that you must not fool yourself, and you are the easiest person to fool.” Indeed, it is precisely to avoid fooling ourselves that many of the systematic and rigorous approaches of science have been developed. Doubleblind, it could be argued attempts to apply such ideas to peer review and take a more scientific approach to the process. Although data on the extent to which bias operates in peer review is largely
lacking, surveys consistently show a high level of support for double-blind review across a range of fields4–6, which suggests that the perception of the potential for referee bias is widespread. Given the strong support for double-blind peer review in principle, two Nature research journals — Nature Climate Change (where I am an editor) and Nature Geoscience — have been trialling double-blind peer review as an option for authors since June 20137,8. Although it is too early to draw any firm conclusions, some interesting patterns have emerged. An ongoing survey of authors that have submitted manuscripts to the two journals indicates that over the first year of the trial around 80% of respondents were supportive of double-blind review in principle. However, uptake of the doubleblind option has been relatively low: taking the two journals together, the percentage of monthly submissions that have chosen this option has never risen above around 20%. There is clearly a disparity between support for the idea of double-blind and what authors are actually choosing for their own papers. There are a number of possible explanations for this. First, authors may be reluctant to have their work be part of a trial: like a new or unfamiliar technology, there are typically a small number of early adopters and most people only follow once the approach has had time to be proven reliable. Second, because the trial has not been running for very long, author awareness remains quite low. Consequently, authors frequently only become aware of the double-blind option once they have already finalized their manuscript. At this point, having to go back and remove author details and alter text (removing phrases like ‘as we showed previously’) represents additional work and can delay submission. Such concerns were frequently cited as a reason for not selecting double-blind in our author survey. It is difficult to know how large an effect this might be. However, if it is large we would expect a gradual increase in double-blind submissions as the trial continues and awareness grows — this has not been particularly evident so far.
NATURE NANOTECHNOLOGY | VOL 9 | NOVEMBER 2014 | www.nature.com/naturenanotechnology
© 2014 Macmillan Publishers Limited. All rights reserved
A third explanation is that because the trial is elective, authors believe that by choosing double-blind they are lumping themselves into a category that will be presumed to contain the work of less prominent individuals. A fairly typical survey comment in this regard is: “Because it was voluntary, that means that the reviewer will know that the authors are likely to have all or some of the characteristics that people bias against.” Again the size of this effect is difficult to estimate, but presumably it would diminish if a much higher proportion of authors chose double-blind review. Another common belief is that referees will be able to guess the identity of the authors, so a double-blind process will make no practical difference. From this perspective, single-blind is a more accurate and realistic model, so why change. The referees might correctly identify the authors in some cases, but as editors we know from past experience that attempts by authors to guess the identities of referees are very often wrong and it seems likely that referees will be no more successful. There is also some evidence that, even in small research fields, around 50% of referees do not suspect the identities of the authors during doubleblind review 5. However, the American Economic Review dropped double-blind review in 2011, citing the role of search engines in limiting their capacity to maintain author anonymity as a major reason for the decision. The accuracy with which referees can guess the identities of authors is likely to be quite field-specific and this is something we are looking to investigate as part of our ongoing trial. We also have some anecdotal feedback, from discussions with individuals, that hasn’t emerged from the survey. One of the recurring arguments can be summarized as ‘reputations should count’: that is, if you have worked in a field for a decade and produced thorough, reliable work then that is useful information for referees to know. For example, the efforts of referees needs to be targeted, so knowing that findings are from a laboratory that has a long history of accurate, well-calibrated measurement is 871
thesis useful and important information. The key question here is whether the loss of useful information associated with knowing the authors’ identities outweigh the potential benefits of a potentially fairer judgment of a paper’s content? This remains an open question at present. Surprisingly, given the important function that peer review is intended to perform in academic publishing, very little analysis has been brought to bear on the question of its efficacy in general or the relative merits of different peer-review
872
models in particular 9,10. There is reason to suspect that bias could be a problem in peer review and, even in the absence of firm supporting data, double-blind review is a sensible strategy to reduce this potential. Only time will tell whether double-blind proves popular in practice and whether it discernibly alters what gets published. ❐ Alastair Brown is a Senior Editor at Nature Climate Change. e-mail: [email protected]
References
1. Godlee, F., Gale, C. R. & Martyn, C. N. J. Am. Med. Assoc. 280, 237–240 (1998). 2. Goldin, C. & Rose, C. The Am. Economic Rev. 90, 715–741 (2001). 3. Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J. & Handelsman, J. Proc. Natl Acad. Sci. USA 109, 16474–16479 (2012). 4. Regehr, G. & Bordage, G. Med. Educ. 40, 832–839 (2006). 5. Jagsi, R. et al. Int. J. Radiation Oncol. Biol. Phys. 89, 940–946 (2014). 6. Nature Geosci. 5, 585 (2012). 7. Nature Geosci. 6, 413 (2013). 8. Nature Clim. Change 3, 525 (2013). 9. Darling, E. Conserv. Biol. http://dx.doi.org/10.1111/cobi.12333 (2014). 10. Jefferson, T. et al. J. Am. Med. Assoc. 287, 2784–2786 (2002).
Published online: 2 November 2014
NATURE NANOTECHNOLOGY | VOL 9 | NOVEMBER 2014 | www.nature.com/naturenanotechnology
© 2014 Macmillan Publishers Limited. All rights reserved
