BMJ 2014;348:g4438 doi: 10.1136/bmj.g4438 (Published 7 July 2014)
Page 1 of 3
Endgames
ENDGAMES STATISTICAL QUESTION
Measuring the performance of screening tests Philip Sedgwick reader in medical statistics and medical education Institute of Medical and Biomedical Education, St George’s, University of London, London, UK
Researchers assessed the performance of the SCOFF questionnaire as a screening tool in primary care for eating disorders. The questionnaire consists of five questions related to the core features of anorexia nervosa and bulimia nervosa. The study participants were 341 consecutive female attendees aged 18-50 years at two general practices. All women completed the SCOFF questionnaire and underwent a clinical diagnostic interview. If a woman responded “yes” to two or more of the five questions on the SCOFF questionnaire she was identified as “positive” and at “high risk” of eating disorders; otherwise she was identified as “negative” and at “low risk” of eating disorders. A diagnosis of eating disorders was confirmed through a clinical diagnostic interview for eating disorders based on criteria from the Diagnostic and Statistical Manual of Mental Disorders (fourth edition).1
diagnostic interview. The SCOFF questionnaire indicates whether a woman is at high (positive result) or low risk (negative result) of an eating disorder. Any woman identified at high risk of an eating disorder could then be invited for a clinical diagnostic interview to confirm the diagnosis. Those women identified as at low risk of an eating disorder would not be invited for a further diagnostic interview. The SCOFF questionnaire is quick and easy to complete. By inviting only those women at high risk of an eating disorder for a diagnostic clinical interview, time and money will be saved.
Which of the following statements, if any, are true? a) The sensitivity is the proportion of women with a diagnosed eating disorder identified as positive on screening
As a screening test the SCOFF questionnaire will not have correctly predicted the diagnostic status of all women. The results of the SCOFF questionnaire in relation to the clinical diagnostic interview for the above study are shown in the table⇓. Those women for whom their SCOFF questionnaire result correctly predicted their clinical diagnosis are referred to as “true positives” and “true negatives” in the table. A true positive was a woman diagnosed with an eating disorder on clinical interview who was correctly identified by the SCOFF questionnaire as positive. A true negative was a woman not diagnosed with an eating disorder on clinical interview who was correctly identified by the SCOFF questionnaire as negative.
The SCOFF questionnaire had a sensitivity of 84.6%, specificity of 89.6%, positive predictive value of 24.4%, and negative predictive value of 99.3%.
b) The specificity is the proportion of women without a diagnosed eating disorder identified as negative on screening c) The positive predictive value is the proportion of women with a positive result on screening who had a diagnosed eating disorder d) The negative predictive value is the proportion of women with a negative result on screening who were found not to have a diagnosed eating disorder
Answers
Statements a, b, c, and d are all true.
It would be impractical for all women in primary care to have a clinical diagnostic interview for eating disorders. Not only would this be time consuming and expensive, but it would probably be inconvenient and unnecessary for most women. The purpose of the SCOFF questionnaire is to predict the result of the clinical diagnostic interview. The SCOFF questionnaire is not a diagnostic test and is not as accurate as the clinical
In the above study, all women completed the SCOFF questionnaire and underwent a clinical diagnostic interview. This was so that the performance of the SCOFF questionnaire as a screening test in predicting the clinical diagnostic status of eating disorders could be evaluated as described below. However, in clinical practice only those women identified at high risk of an eating disorder (positive result) would be invited for a clinical diagnostic interview to confirm diagnosis.
Those women whose SCOFF questionnaire result did not correctly predict their clinical diagnosis are referred to as “false negatives” and “false positives” in the table. A false negative was a woman diagnosed with an eating disorder on clinical interview who was incorrectly identified as negative by the SCOFF questionnaire. A false positive was a woman who was not diagnosed as having an eating disorder who was incorrectly identified as positive by the SCOFF questionnaire. The performance of the SCOFF questionnaire as a screening test for predicting diagnosed eating disorders was quantified by
[email protected] For personal use only: See rights and reprints http://www.bmj.com/permissions
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g4438 doi: 10.1136/bmj.g4438 (Published 7 July 2014)
Page 2 of 3
ENDGAMES
the indices of sensitivity, specificity, positive predictive value, and negative predictive value. The sensitivity of the SCOFF questionnaire is the proportion of women, expressed as a percentage, with diagnosed eating disorders correctly identified with a positive result (true positive) on the SCOFF questionnaire (a is true). In the study above, 13 women had a diagnosed eating disorder on clinical interview, of whom 11 (84.6%) were correctly identified with a positive result. The remaining two women were incorrectly identified by the SCOFF questionnaire with a negative result (false negative).
The specificity of the SCOFF questionnaire is the proportion of women, expressed as a percentage, without a diagnosed eating disorder correctly identified with a negative result (true negative) on the SCOFF questionnaire (b is true). In the study above, 328 women were found not to have a diagnosed eating disorder, of whom 294 (89.6%) were correctly identified with a negative result on the SCOFF questionnaire. The remaining 34 women were incorrectly identified as positive by the SCOFF questionnaire (false positive result). The positive predictive value is the accuracy of a positive result on the SCOFF questionnaire. It is the proportion of women, expressed as a percentage, identified as positive on the SCOFF questionnaire who had a diagnosed eating disorder (c is true). Of the 45 women identified as positive on the SCOFF questionnaire, 11 (24.4%) were correctly identified because they were subsequently diagnosed as having an eating disorder (true positive result). The remaining 34 women were incorrectly identified (false positive result) by the SCOFF questionnaire—they were subsequently not diagnosed as having an eating disorder on the clinical diagnostic interview.
The negative predictive value is the accuracy of a negative result on the SCOFF questionnaire. It is the proportion of women, expressed as a percentage, identified by the SCOFF questionnaire as negative who did not have a diagnosed eating disorder (d is true). Of the 296 women identified as negative on the SCOFF questionnaire, 294 (99.3%) were correctly identified (true negative result) because they were subsequently diagnosed as not having an eating disorder. The remaining two women were incorrectly identified (false negative result) by the SCOFF questionnaire—they were subsequently diagnosed as having an eating disorder on the clinical diagnostic interview. The indices of performance for a screening test described above are often confused, particularly sensitivity and positive predictive value (statements a and c). The sensitivity and specificity of the SCOFF questionnaire (statements a and b) describe the proportion of women with and without diagnosed eating disorders who were correctly identified by the screening test, respectively. The positive predictive value and negative
For personal use only: See rights and reprints http://www.bmj.com/permissions
predictive value (statements c and d) describe the accuracy of a positive and negative result on the SCOFF questionnaire—that is, the proportion of women with a positive and negative result, respectively, on screening whose diagnostic status was correctly predicted. The above indices are traditionally calculated for a screening test. The positive likelihood ratio and negative likelihood ratio, described in a previous question,2 are often also used to describe the performance of a screening test. The positive likelihood ratio and negative likelihood ratio will be described further in a future question.
The SCOFF questionnaire consists of five questions. Women who responded “yes” to two or more of the five questions were identified as positive and at high risk of eating disorders; otherwise they were identified as negative and at low risk. The researchers investigated the optimal number of questions a woman needed to respond positively to in order to be identified as positive. A positive response on one to five questions was taken successively as the cut-off point between a negative and positive screening test result; all scores greater than or equal to the cut-off score were considered positive and all others as negative. The optimal cut-off score was investigated using a receiver operating characteristic curve; such an approach has been described in a previous question.3 Sensitivity and specificity should, ideally, be as close to 100% as possible. However, in the above study, the choice of the cut-off number of questions meant that a compromise between sensitivity and specificity was needed, because as one index increased the other decreased. Therefore, ultimately the choice of cut-off score influenced the number of women incorrectly identified (false negatives and false positives) on screening. The use of the words positive and negative to describe the results of a screening test is best avoided. Although this is common practice in laboratory reports and textbooks, evidence suggests that it causes confusion for the patient. Generally the lay interpretation of the term positive reflects a good outcome, which is different from the outcome of a positive screening test result. Therefore, the phrases high risk and low risk are recommended instead. Competing interests: None declared. 1 2 3
Luck AJ, Morgan JF, Reid F, O’Brien A, Brunton J, Price C, et al. The SCOFF questionnaire and clinical interview for eating disorders in general practice: comparative study. BMJ 2002;325:755-6. Sedgwick P. Screening tests: likelihood ratios. BMJ 2011;342:d3986. Sedgwick P. Receiver operating characteristic curves. BMJ 2013;346:f2493.
Cite this as: BMJ 2014;349:g4438 © BMJ Publishing Group Ltd 2014
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g4438 doi: 10.1136/bmj.g4438 (Published 7 July 2014)
Page 3 of 3
ENDGAMES
Table Table 1| Cross tabulation of the results of the SCOFF questionnaire, categorised as “negative”” or “positive,” against the clinical diagnosis
of eating disorder
Diagnosed eating disorder Yes ≥2 positive responses: positive or “high risk”
No
Total
11 (true positives) 34 (false positives)
45
0 or 1 positive response: negative or “low risk” 2 (false negatives) 294 (true negatives)
296
Total
13
For personal use only: See rights and reprints http://www.bmj.com/permissions
328
Subscribe: http://www.bmj.com/subscribe
Page 1 of 3
Endgames
ENDGAMES STATISTICAL QUESTION
Measuring the performance of screening tests Philip Sedgwick reader in medical statistics and medical education Institute of Medical and Biomedical Education, St George’s, University of London, London, UK
Researchers assessed the performance of the SCOFF questionnaire as a screening tool in primary care for eating disorders. The questionnaire consists of five questions related to the core features of anorexia nervosa and bulimia nervosa. The study participants were 341 consecutive female attendees aged 18-50 years at two general practices. All women completed the SCOFF questionnaire and underwent a clinical diagnostic interview. If a woman responded “yes” to two or more of the five questions on the SCOFF questionnaire she was identified as “positive” and at “high risk” of eating disorders; otherwise she was identified as “negative” and at “low risk” of eating disorders. A diagnosis of eating disorders was confirmed through a clinical diagnostic interview for eating disorders based on criteria from the Diagnostic and Statistical Manual of Mental Disorders (fourth edition).1
diagnostic interview. The SCOFF questionnaire indicates whether a woman is at high (positive result) or low risk (negative result) of an eating disorder. Any woman identified at high risk of an eating disorder could then be invited for a clinical diagnostic interview to confirm the diagnosis. Those women identified as at low risk of an eating disorder would not be invited for a further diagnostic interview. The SCOFF questionnaire is quick and easy to complete. By inviting only those women at high risk of an eating disorder for a diagnostic clinical interview, time and money will be saved.
Which of the following statements, if any, are true? a) The sensitivity is the proportion of women with a diagnosed eating disorder identified as positive on screening
As a screening test the SCOFF questionnaire will not have correctly predicted the diagnostic status of all women. The results of the SCOFF questionnaire in relation to the clinical diagnostic interview for the above study are shown in the table⇓. Those women for whom their SCOFF questionnaire result correctly predicted their clinical diagnosis are referred to as “true positives” and “true negatives” in the table. A true positive was a woman diagnosed with an eating disorder on clinical interview who was correctly identified by the SCOFF questionnaire as positive. A true negative was a woman not diagnosed with an eating disorder on clinical interview who was correctly identified by the SCOFF questionnaire as negative.
The SCOFF questionnaire had a sensitivity of 84.6%, specificity of 89.6%, positive predictive value of 24.4%, and negative predictive value of 99.3%.
b) The specificity is the proportion of women without a diagnosed eating disorder identified as negative on screening c) The positive predictive value is the proportion of women with a positive result on screening who had a diagnosed eating disorder d) The negative predictive value is the proportion of women with a negative result on screening who were found not to have a diagnosed eating disorder
Answers
Statements a, b, c, and d are all true.
It would be impractical for all women in primary care to have a clinical diagnostic interview for eating disorders. Not only would this be time consuming and expensive, but it would probably be inconvenient and unnecessary for most women. The purpose of the SCOFF questionnaire is to predict the result of the clinical diagnostic interview. The SCOFF questionnaire is not a diagnostic test and is not as accurate as the clinical
In the above study, all women completed the SCOFF questionnaire and underwent a clinical diagnostic interview. This was so that the performance of the SCOFF questionnaire as a screening test in predicting the clinical diagnostic status of eating disorders could be evaluated as described below. However, in clinical practice only those women identified at high risk of an eating disorder (positive result) would be invited for a clinical diagnostic interview to confirm diagnosis.
Those women whose SCOFF questionnaire result did not correctly predict their clinical diagnosis are referred to as “false negatives” and “false positives” in the table. A false negative was a woman diagnosed with an eating disorder on clinical interview who was incorrectly identified as negative by the SCOFF questionnaire. A false positive was a woman who was not diagnosed as having an eating disorder who was incorrectly identified as positive by the SCOFF questionnaire. The performance of the SCOFF questionnaire as a screening test for predicting diagnosed eating disorders was quantified by
[email protected] For personal use only: See rights and reprints http://www.bmj.com/permissions
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g4438 doi: 10.1136/bmj.g4438 (Published 7 July 2014)
Page 2 of 3
ENDGAMES
the indices of sensitivity, specificity, positive predictive value, and negative predictive value. The sensitivity of the SCOFF questionnaire is the proportion of women, expressed as a percentage, with diagnosed eating disorders correctly identified with a positive result (true positive) on the SCOFF questionnaire (a is true). In the study above, 13 women had a diagnosed eating disorder on clinical interview, of whom 11 (84.6%) were correctly identified with a positive result. The remaining two women were incorrectly identified by the SCOFF questionnaire with a negative result (false negative).
The specificity of the SCOFF questionnaire is the proportion of women, expressed as a percentage, without a diagnosed eating disorder correctly identified with a negative result (true negative) on the SCOFF questionnaire (b is true). In the study above, 328 women were found not to have a diagnosed eating disorder, of whom 294 (89.6%) were correctly identified with a negative result on the SCOFF questionnaire. The remaining 34 women were incorrectly identified as positive by the SCOFF questionnaire (false positive result). The positive predictive value is the accuracy of a positive result on the SCOFF questionnaire. It is the proportion of women, expressed as a percentage, identified as positive on the SCOFF questionnaire who had a diagnosed eating disorder (c is true). Of the 45 women identified as positive on the SCOFF questionnaire, 11 (24.4%) were correctly identified because they were subsequently diagnosed as having an eating disorder (true positive result). The remaining 34 women were incorrectly identified (false positive result) by the SCOFF questionnaire—they were subsequently not diagnosed as having an eating disorder on the clinical diagnostic interview.
The negative predictive value is the accuracy of a negative result on the SCOFF questionnaire. It is the proportion of women, expressed as a percentage, identified by the SCOFF questionnaire as negative who did not have a diagnosed eating disorder (d is true). Of the 296 women identified as negative on the SCOFF questionnaire, 294 (99.3%) were correctly identified (true negative result) because they were subsequently diagnosed as not having an eating disorder. The remaining two women were incorrectly identified (false negative result) by the SCOFF questionnaire—they were subsequently diagnosed as having an eating disorder on the clinical diagnostic interview. The indices of performance for a screening test described above are often confused, particularly sensitivity and positive predictive value (statements a and c). The sensitivity and specificity of the SCOFF questionnaire (statements a and b) describe the proportion of women with and without diagnosed eating disorders who were correctly identified by the screening test, respectively. The positive predictive value and negative
For personal use only: See rights and reprints http://www.bmj.com/permissions
predictive value (statements c and d) describe the accuracy of a positive and negative result on the SCOFF questionnaire—that is, the proportion of women with a positive and negative result, respectively, on screening whose diagnostic status was correctly predicted. The above indices are traditionally calculated for a screening test. The positive likelihood ratio and negative likelihood ratio, described in a previous question,2 are often also used to describe the performance of a screening test. The positive likelihood ratio and negative likelihood ratio will be described further in a future question.
The SCOFF questionnaire consists of five questions. Women who responded “yes” to two or more of the five questions were identified as positive and at high risk of eating disorders; otherwise they were identified as negative and at low risk. The researchers investigated the optimal number of questions a woman needed to respond positively to in order to be identified as positive. A positive response on one to five questions was taken successively as the cut-off point between a negative and positive screening test result; all scores greater than or equal to the cut-off score were considered positive and all others as negative. The optimal cut-off score was investigated using a receiver operating characteristic curve; such an approach has been described in a previous question.3 Sensitivity and specificity should, ideally, be as close to 100% as possible. However, in the above study, the choice of the cut-off number of questions meant that a compromise between sensitivity and specificity was needed, because as one index increased the other decreased. Therefore, ultimately the choice of cut-off score influenced the number of women incorrectly identified (false negatives and false positives) on screening. The use of the words positive and negative to describe the results of a screening test is best avoided. Although this is common practice in laboratory reports and textbooks, evidence suggests that it causes confusion for the patient. Generally the lay interpretation of the term positive reflects a good outcome, which is different from the outcome of a positive screening test result. Therefore, the phrases high risk and low risk are recommended instead. Competing interests: None declared. 1 2 3
Luck AJ, Morgan JF, Reid F, O’Brien A, Brunton J, Price C, et al. The SCOFF questionnaire and clinical interview for eating disorders in general practice: comparative study. BMJ 2002;325:755-6. Sedgwick P. Screening tests: likelihood ratios. BMJ 2011;342:d3986. Sedgwick P. Receiver operating characteristic curves. BMJ 2013;346:f2493.
Cite this as: BMJ 2014;349:g4438 © BMJ Publishing Group Ltd 2014
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g4438 doi: 10.1136/bmj.g4438 (Published 7 July 2014)
Page 3 of 3
ENDGAMES
Table Table 1| Cross tabulation of the results of the SCOFF questionnaire, categorised as “negative”” or “positive,” against the clinical diagnosis
of eating disorder
Diagnosed eating disorder Yes ≥2 positive responses: positive or “high risk”
No
Total
11 (true positives) 34 (false positives)
45
0 or 1 positive response: negative or “low risk” 2 (false negatives) 294 (true negatives)
296
Total
13
For personal use only: See rights and reprints http://www.bmj.com/permissions
328
Subscribe: http://www.bmj.com/subscribe
![[Comparison of evaluation scales and performance tests for measuring the severity of cyclothymic depressions].](/assets/img/hold.png)
