Letters to the Editor Effect of Prevalence Expectations on Radiologists’ Behavior From: Vikas Chaudhary, MD, Shahina Bano, MD From the Department of Radiodiagnosis, Employees’ State Insurance Corporation (ESIC) Model Hospital, Sector-9A, Gurgaon 122001, Haryana, India; Department of Radiodiagnosis, Post Graduate Institute of Medical Education & Research (PGIMER), Dr. Ram Manohar Lohia Hospital, New Delhi 110001, India. We read with great interest the article, ‘‘Can prevalence expectations drive radiologists’ behavior?’’ by Reed et al. (1) in the April 2014 issue of the Academic Radiology. We would like to laud Reed et al. for their excellent work, which can be an eye opener for many of us. It apparently seems that the study has the potential to be used in improving the radiologists’ vision for the target. The authors have very nicely put the subtle difference between the effect of ‘‘increased prevalence expectation’’ and ‘‘increased prevalence’’ on radiologists’ interpretative behavior. As stated, previous studies exploring the correlation between ‘‘disease prevalence’’ and radiologists’ performance have shown conflicting results. On the other hand, the present study, searching for the effect of ‘‘increased prevalence expectation’’ on radiologists’ interpretation of the images, has shown that increased prevalence expectation prolongs image scrutiny time and decreases the radiologists’ confidence in reporting a normal image. Furthermore, the study also shows that increased prevalence expectation increases the number of false positives but by a nonsignificant amount. It should be noted that the present study and a similar previous study have been done with the expert radiologists (1,2). It would be very interesting to see what would be the effect of the prevalence phenomenon on learning radiologists’ (ie, radiology residents/postgraduate students) image interpretation behavior (3). As per our understanding on the subject, we predict that if similar studies are done with the learning radiologists, the effect of prevalence expectation may increase the false positives and false negatives by a significant amount. This aspect becomes much relevant in context of the developing countries such as India where much of the radiology work, especially in government departments, is performed by the residents/postgraduate students. An important scenario can be ‘‘reporting chest skiagrams in India.’’ It is
Acad Radiol 2014; 21:1220–1221 ªAUR, 2014 http://dx.doi.org/10.1016/j.acra.2014.04.018
1220
worth mentioning here that most residents/students in India have a big background impression of high prevalence expectation for tuberculosis. Therefore, we would urge Reed et al. and other researchers to generate more knowledge on this subject so that the appropriate measures can be taken to solve this issue.
REFERENCES 1. Reed WM, Chow SL, Chew LE, et al. Can prevalence expectations drive radiologists’ behavior? Acad Radiol 2014; 21:450–456. 2. Reed WM, Ryan JT, McEntee MF, et al. The effect of abnormalityprevalence expectation on expert observer performance and visual search. Radiology 2011; 258:938–943. 3. Monnier-Cholley L, Carrat F, Cholley BP, et al. Detection of lung cancer on radiographs: receiver operating characteristic analyses of radiologists’, pulmonologists’, and anesthesiologists’ performance. Radiology 2004; 233:799–805.
Assessing the Impact of Prevalence Expectations on Radiologists’ Behavior From: Warren M. Reed, PhD, Suet Ling Candice Chow, BSc (Hons), Lay Ee Chew, BSc (Hons), Patrick C. Brennan, PhD From the Medical Image Optimisation and Perception Group, Discipline of Medical Radiation Sciences, Faculty of Health Sciences, The University of Sydney, Lidcombe, New South Wales 1825, Australia. Dear Sir, We would like to thank Dr Vikas Chaudhary and Dr Shahina Bano for their interest and encouraging comments regarding our recently published article (1). We also appreciate the opportunity to discuss and clarify some points of possible interest, concerning this article and a previous referenced article (2), for the readers. As mentioned, we have distinguished between altering prevalence expectations by interpreting an identical set of images rather than changing the actual target prevalence with different sets of images as has been the norm in previous studies (3). Our results have found no evidence that the accuracy of experienced radiologists is altered because of changing explicit abnormality expectation rates. However, confidence ratings and visual search appear to be affected, and this impact appears more apparent for normal images. Also, although a previous study by our group of naive readers did not provide evidence that the overall performance of the student radiographers was affected by the changing prevalence expectations in terms of overall receiver operating
Academic Radiology, Vol 21, No 9, September 2014
characteristic (ROC) (Az) measurement, a trend was observed where their sensitivity increased and their specificity decreased at higher abnormality prevalence expectations (4). Interestingly, it has been shown elsewhere that false negatives tend to decrease at higher prevalence because the primary effect of prevalence results in a criterion shift where an observer in a higher prevalence situation is more likely to call an ambiguous finding as positive and often less likely to terminate visual search (5–7). We believe that a critical question for future research is to determine the extent that prevalence expectation affects the detectability of abnormalities when presenting the same case sets to radiologists but rather than stating actual prevalence values as we have used here, present the cases as belonging to specific patient prevalence conditions, such as tuberculosis, as suggested by Dr Vikas Chaudhary and Dr Shahina Bano. It is clear that further work is required to understand the impact of prevalence expectation on different types of images and readers (eg, ‘‘learning radiologists’’) to fully understand the
LETTERS TO THE EDITOR
context of image interpretation in perceived high and low prevalence conditions.
REFERENCES 1. Reed WM, Chow SL, Chew LE, et al. Can prevalence expectations drive radiologists’ behavior? Acad Radiol 2014; 21:450–456. 2. Reed WM, Ryan JT, McEntee MF, et al. The effect of abnormalityprevalence expectation on expert observer performance and visual search. Radiology 2011; 258:938–943. 3. Wolfe JM, Horowitz TS, Kenner NM. Cognitive psychology: rare items often missed in visual searches. Nature 2005; 435(7041):439–440. 4. Nocum DJ, Brennan PC, Huang RT, et al. The effect of abnormalityprevalence expectation on na€ıve observer performance and visual search. Radiography 2013; 19(3):196–199. 5. Wolfe JM, Horowitz TS, Van Wert MJ, et al. Low target prevalence is a stubborn source of errors in visual search tasks. J Exp Psychol Gen 2007; 136(4):623–638. 6. Wolfe JM, Van Wert MJ. Varying target prevalence reveals two dissociable decision criteria in visual search. Curr Biol 2010; 20(2):121–124. 7. Evans KK, Birdwell RL, Wolfe JM. If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLoS One 2013; 8(5):e64366.
1221
Acad Radiol 2014; 21:1220–1221 ªAUR, 2014 http://dx.doi.org/10.1016/j.acra.2014.04.018
1220
worth mentioning here that most residents/students in India have a big background impression of high prevalence expectation for tuberculosis. Therefore, we would urge Reed et al. and other researchers to generate more knowledge on this subject so that the appropriate measures can be taken to solve this issue.
REFERENCES 1. Reed WM, Chow SL, Chew LE, et al. Can prevalence expectations drive radiologists’ behavior? Acad Radiol 2014; 21:450–456. 2. Reed WM, Ryan JT, McEntee MF, et al. The effect of abnormalityprevalence expectation on expert observer performance and visual search. Radiology 2011; 258:938–943. 3. Monnier-Cholley L, Carrat F, Cholley BP, et al. Detection of lung cancer on radiographs: receiver operating characteristic analyses of radiologists’, pulmonologists’, and anesthesiologists’ performance. Radiology 2004; 233:799–805.
Assessing the Impact of Prevalence Expectations on Radiologists’ Behavior From: Warren M. Reed, PhD, Suet Ling Candice Chow, BSc (Hons), Lay Ee Chew, BSc (Hons), Patrick C. Brennan, PhD From the Medical Image Optimisation and Perception Group, Discipline of Medical Radiation Sciences, Faculty of Health Sciences, The University of Sydney, Lidcombe, New South Wales 1825, Australia. Dear Sir, We would like to thank Dr Vikas Chaudhary and Dr Shahina Bano for their interest and encouraging comments regarding our recently published article (1). We also appreciate the opportunity to discuss and clarify some points of possible interest, concerning this article and a previous referenced article (2), for the readers. As mentioned, we have distinguished between altering prevalence expectations by interpreting an identical set of images rather than changing the actual target prevalence with different sets of images as has been the norm in previous studies (3). Our results have found no evidence that the accuracy of experienced radiologists is altered because of changing explicit abnormality expectation rates. However, confidence ratings and visual search appear to be affected, and this impact appears more apparent for normal images. Also, although a previous study by our group of naive readers did not provide evidence that the overall performance of the student radiographers was affected by the changing prevalence expectations in terms of overall receiver operating
Academic Radiology, Vol 21, No 9, September 2014
characteristic (ROC) (Az) measurement, a trend was observed where their sensitivity increased and their specificity decreased at higher abnormality prevalence expectations (4). Interestingly, it has been shown elsewhere that false negatives tend to decrease at higher prevalence because the primary effect of prevalence results in a criterion shift where an observer in a higher prevalence situation is more likely to call an ambiguous finding as positive and often less likely to terminate visual search (5–7). We believe that a critical question for future research is to determine the extent that prevalence expectation affects the detectability of abnormalities when presenting the same case sets to radiologists but rather than stating actual prevalence values as we have used here, present the cases as belonging to specific patient prevalence conditions, such as tuberculosis, as suggested by Dr Vikas Chaudhary and Dr Shahina Bano. It is clear that further work is required to understand the impact of prevalence expectation on different types of images and readers (eg, ‘‘learning radiologists’’) to fully understand the
LETTERS TO THE EDITOR
context of image interpretation in perceived high and low prevalence conditions.
REFERENCES 1. Reed WM, Chow SL, Chew LE, et al. Can prevalence expectations drive radiologists’ behavior? Acad Radiol 2014; 21:450–456. 2. Reed WM, Ryan JT, McEntee MF, et al. The effect of abnormalityprevalence expectation on expert observer performance and visual search. Radiology 2011; 258:938–943. 3. Wolfe JM, Horowitz TS, Kenner NM. Cognitive psychology: rare items often missed in visual searches. Nature 2005; 435(7041):439–440. 4. Nocum DJ, Brennan PC, Huang RT, et al. The effect of abnormalityprevalence expectation on na€ıve observer performance and visual search. Radiography 2013; 19(3):196–199. 5. Wolfe JM, Horowitz TS, Van Wert MJ, et al. Low target prevalence is a stubborn source of errors in visual search tasks. J Exp Psychol Gen 2007; 136(4):623–638. 6. Wolfe JM, Van Wert MJ. Varying target prevalence reveals two dissociable decision criteria in visual search. Curr Biol 2010; 20(2):121–124. 7. Evans KK, Birdwell RL, Wolfe JM. If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLoS One 2013; 8(5):e64366.
1221
