Journal of the ICRU Vol 8 No 1 (2008) Report 79 Oxford University Press
doi:10.1093/jicru/ndn006
Executive Summary are interpreted by multiple observers using a standard imaging modality and the new modality being evaluated. The preferred statistical methodology for the analysis of the response data from an evaluative study is called the multireader, multicase (MRMC) method. It uses a balanced data collection design and standard ROC curve-fitting techniques. It then uses a jackknife technique to develop a family of ROC pseudovalues that are used in an analysis of variance to calculate the contribution to any observed difference between modalities of the case sample and the readers. It is currently the most efficient method for dealing with appropriately collected decision data, and it is presented in detail. From a practical standpoint, the ROC analysis is still incomplete. It cannot effectively deal with multiple abnormalities per case or with accounting for the location of abnormalities. Progress on these fertile fields for investigation is described in the present report. The final section of the report deals with some of the practical aspects of experimental design that can minimize the biases that are inherent to the methodology. The ROC analysis is just a statistical methodology that is applied to appropriately structured tables of data. The conclusions that can be reached from the analysis can be greatly influenced by the design of the data collection experiment. Three appendices have been included as aids to potential users of the methodology. There is a list of the current web sites from which the ROC analysis software can be downloaded, a description of a well-designed and executed ROC experiment, and a worked-out example of a statistical power calculation based on the MRMC method. It provides some guidelines for estimating the number of cases and the number of observers needed in an evaluation in order to reach statistically valid conclusions.
# International Commission on Radiation Units and Measurements 2008
Downloaded from http://jicru.oxfordjournals.org/ at University of Arizona on June 1, 2015
Receiver operating characteristic (ROC) analysis is used extensively for analyzing human decision making in imaging, although it is equally applicable to other domains including the decisions made by computer programs. It is based on a formal model derived from the statistical decision theory. The observer’s decision data are used to compute the model parameters that are used to construct an ROC curve, which is a plot of the truepositive fraction (sensitivity) against the falsepositive fraction (1 – specificity) as the observer’s decision criterion is varied from strict to lenient. The model parameters are also used to compute the area under the ROC curve, which is a singlevalued, unbiased measure of diagnostic accuracy and an index of detectability, which is a measure of the observer’s signal-to-noise ratio. The present report describes the practical and theoretical aspects of the ROC methodology as it is used currently in image system evaluation and some new developments that are still works in progress. Diagnostic accuracy is an important attribute of an imaging system but it is only one aspect of the total spectrum of information that determines efficacy. Measures of accuracy must be preceded by measures of the physical properties of the system and must be considered when making higher-level decisions about the value added to patient care. The ROC analysis provides the best available assessment of diagnostic accuracy. The present report contains a nontechnical description of the production of an ROC curve from the response data of a single observer. It was included to provide a more intuitive understanding of the meaning of the curve. It also provides an introduction to a more detailed somewhat technical description of the statistical decision theory model. The overall methodology for evaluating imaging systems usually includes the development of a set of test cases that
doi:10.1093/jicru/ndn006
Executive Summary are interpreted by multiple observers using a standard imaging modality and the new modality being evaluated. The preferred statistical methodology for the analysis of the response data from an evaluative study is called the multireader, multicase (MRMC) method. It uses a balanced data collection design and standard ROC curve-fitting techniques. It then uses a jackknife technique to develop a family of ROC pseudovalues that are used in an analysis of variance to calculate the contribution to any observed difference between modalities of the case sample and the readers. It is currently the most efficient method for dealing with appropriately collected decision data, and it is presented in detail. From a practical standpoint, the ROC analysis is still incomplete. It cannot effectively deal with multiple abnormalities per case or with accounting for the location of abnormalities. Progress on these fertile fields for investigation is described in the present report. The final section of the report deals with some of the practical aspects of experimental design that can minimize the biases that are inherent to the methodology. The ROC analysis is just a statistical methodology that is applied to appropriately structured tables of data. The conclusions that can be reached from the analysis can be greatly influenced by the design of the data collection experiment. Three appendices have been included as aids to potential users of the methodology. There is a list of the current web sites from which the ROC analysis software can be downloaded, a description of a well-designed and executed ROC experiment, and a worked-out example of a statistical power calculation based on the MRMC method. It provides some guidelines for estimating the number of cases and the number of observers needed in an evaluation in order to reach statistically valid conclusions.
# International Commission on Radiation Units and Measurements 2008
Downloaded from http://jicru.oxfordjournals.org/ at University of Arizona on June 1, 2015
Receiver operating characteristic (ROC) analysis is used extensively for analyzing human decision making in imaging, although it is equally applicable to other domains including the decisions made by computer programs. It is based on a formal model derived from the statistical decision theory. The observer’s decision data are used to compute the model parameters that are used to construct an ROC curve, which is a plot of the truepositive fraction (sensitivity) against the falsepositive fraction (1 – specificity) as the observer’s decision criterion is varied from strict to lenient. The model parameters are also used to compute the area under the ROC curve, which is a singlevalued, unbiased measure of diagnostic accuracy and an index of detectability, which is a measure of the observer’s signal-to-noise ratio. The present report describes the practical and theoretical aspects of the ROC methodology as it is used currently in image system evaluation and some new developments that are still works in progress. Diagnostic accuracy is an important attribute of an imaging system but it is only one aspect of the total spectrum of information that determines efficacy. Measures of accuracy must be preceded by measures of the physical properties of the system and must be considered when making higher-level decisions about the value added to patient care. The ROC analysis provides the best available assessment of diagnostic accuracy. The present report contains a nontechnical description of the production of an ROC curve from the response data of a single observer. It was included to provide a more intuitive understanding of the meaning of the curve. It also provides an introduction to a more detailed somewhat technical description of the statistical decision theory model. The overall methodology for evaluating imaging systems usually includes the development of a set of test cases that
