QJM Advance Access published May 5, 2014
Clinical Prediction Rules in Community-Acquired Pneumonia: Lies, Damn Lies, and Statistics Michael S. Abers1, Daniel M. Musher1,2 1
Baylor College of Medicine, Houston, Texas
2
Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
Abstract
A variety of prediction scores have been developed to identify at the time of presentation patients with community-acquired pneumonia (CAP) are at risk for ICU admission or death within 30 days. The effectiveness of each scoring score is typically assessed by calculation of the area under the receiver-operator characteristic curve (AUROC). While this statistical parameter is helpful in determining the discriminatory value of a score, it assumes equal importance of false negatives and false positives in the tradeoff between sensitivity and specificity. Because patient safety takes precedence over cost, the balance between limiting false negatives (unnecessarily strict ICU admission policy) and false positives (unnecessarily liberal ICU admission policy) should favor the reduction of false negatives. Instead of using AUROC as the primary measure to evaluate prediction rules, we propose the use of sensitivity as a more appropriate alternative.
Clinical prediction rules purport to determine which patients, at the time of presentation, for community-acquired pneumonia (CAP) will require ICU admission or will die within 30 days. These scoring systems (PSI, CURB-65, and SMART-COP) have been validated in prospective trials, retrospective databases, and meta-analyses. Their © The Author 2014. Published by Oxford University Press on behalf of the Association of Physicians. All rights reserved. For Permissions, please email: [email protected]
1
importance is demonstrated by a prominent inclusion in CAP guidelines.1-3 Rigorous statistical analyses and recommendations by prestigious organizations notwithstanding, we believe that these scores have certain limitations that may compromise patient safety when emphasized over clinical judgment. The present article examines CAP scoring systems, emphasizing clinical utility rather than statistical validity. Our critique of CAP scoring systems rests on two principles of good medical
is statistically valid, this parameter relies on false assumptions about how clinical decisions ought to be made. AUROC is predicated on the assumption that sensitivity and specificity are of equal clinical importance. In the case of CAP, this would imply that clinicians should apply equal weight to the risk to patient safety of failing to admit to an ICU a patient who needs ICU care (false negative) on the one hand, and to the risk of excessive use of ICU facilities when a patient who does not require ICU care is admitted to an ICU (false positive) on the other. Although limited resources necessitate a careful consideration of potential false positives, good clinical practice demands that greater emphasis be placed on patient safety by limiting false negatives. A second, and related problem is that each of these scoring systems yields an unacceptable false negative rate, at the recommended cutoff value. In a recent metaanalysis, a PSI score > 4 had a sensitivity of 74% as a predictor of ICU admission.4 Thus, PSI
as a tool to guide ICU admission decisions leads to a potentially inadequate level of care in 26% of patients who ultimately require ICU admission. If one were to decrease the cutoff value to reach an appropriate false negative rate, the false positive rate would be
2
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
practice. First, although the area under the receiver-operator characteristic curve (AUROC)
exceptionally low (10-15%). CURB-65 is an even less desirable score, with a sensitivity of approximately 50%.4 This analysis leads us to three conclusions regarding the appropriate application of prediction rules to CAP patients. First, because of the substantial variability, with many false positives and negatives, no scoring system can be universally recommended to evaluate patients with CAP. Clinicians should only rely upon a prediction score when there
rules should probably be used only when the clinician’s judgment does not specifically favor the need for either higher or lower acuity of care. Second, CURB-65, although popular due to its simplicity and despite its relatively high specificity, might be abandoned altogether as a screening tool; if it is retained, it should only be used if positive because a positive result is highly likely to indicate the need for ICU care. Third, if the clinician chooses to use a prediction rule, it should probably be the PSI. These recommendations are based on the concept that the false negative rate is the parameter of greatest consequence in triaging CAP patients. In summary, when considering the predictive value of a scoring system, a statistician may apply equal weight to the risk of false negatives and false positives, but a clinician should not. False negatives are associated with a decreased level of care and potentially increased adverse outcomes. In the case of CAP, this translates into human lives. False positives are associated with a potentially increased burden on facilities and even wasteful use of limited resources, which society certainly wishes to avoid. Evidencebased medicine deals analytically with statistics. Use of the AUROC curve gives statistically valid data, but, by giving equal weight to false positives and false negatives, AUROC may
3
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
is an intermediate pretest probability of a negative outcome. In practical terms, prediction
potentially endanger patients. Clinicians should only calculate a prediction score when there is an intermediate pretest probability of a negative outcome. In practical terms, prediction rules should probably be used only when the clinician’s judgment does not specifically favor the need for either higher or lower acuity of care. While the above discussion pertains to clinical scoring systems for CAP, our argument can be applied to any prediction rule in which an imminent outcome may be death. Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
Funding: none. Acknowledgements: none. Conflicts of interest: none. Contact information: Michael Abers. One Baylor Plaza, Houston, TX 77030. E-mail: [email protected]
References 1
Lim WS, Baudouin SV, George RC, Hill AT, Jamieson C, Le Jeune I, et al. British Thoracic
Society guidelines for the management of community acquired pneumonia in adults: update 2009. Thorax 2009; 64(Suppl III):iii1–iii55. 2
Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell GD, Dean NC, et al. Infectious
Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis 2007; 44 Suppl 2:S27-72.
4
3
Spindler C, Strålin K, Eriksson L, Hjerdt-Goscinski G, Holmberg H, Lidman C, et al. Swedish
guidelines on the management of community-acquired pneumonia in immunocompetent adults-Swedish Society of Infectious Diseases 2012. Scand J Infect Dis 2012; 44:885-902. 4
Chalmers JD, Mandal P, Singanayagam A, Akram AR, Choudhury G, Short PM, et al. Severity
assessment tools to guide ICU admission in community-acquired pneumonia: systematic review and meta-analysis. Intensive Care Med 2011; 37:1409-20. Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
5
Acronyms CAP: community-acquired pneumonia AUROC: area under the receiver-operator characteristic curve ICU: intensive care unit PSI: pneumonia severity index
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
Clinical Prediction Rules in Community-Acquired Pneumonia: Lies, Damn Lies, and Statistics Michael S. Abers1, Daniel M. Musher1,2 1
Baylor College of Medicine, Houston, Texas
2
Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
Abstract
A variety of prediction scores have been developed to identify at the time of presentation patients with community-acquired pneumonia (CAP) are at risk for ICU admission or death within 30 days. The effectiveness of each scoring score is typically assessed by calculation of the area under the receiver-operator characteristic curve (AUROC). While this statistical parameter is helpful in determining the discriminatory value of a score, it assumes equal importance of false negatives and false positives in the tradeoff between sensitivity and specificity. Because patient safety takes precedence over cost, the balance between limiting false negatives (unnecessarily strict ICU admission policy) and false positives (unnecessarily liberal ICU admission policy) should favor the reduction of false negatives. Instead of using AUROC as the primary measure to evaluate prediction rules, we propose the use of sensitivity as a more appropriate alternative.
Clinical prediction rules purport to determine which patients, at the time of presentation, for community-acquired pneumonia (CAP) will require ICU admission or will die within 30 days. These scoring systems (PSI, CURB-65, and SMART-COP) have been validated in prospective trials, retrospective databases, and meta-analyses. Their © The Author 2014. Published by Oxford University Press on behalf of the Association of Physicians. All rights reserved. For Permissions, please email: [email protected]
1
importance is demonstrated by a prominent inclusion in CAP guidelines.1-3 Rigorous statistical analyses and recommendations by prestigious organizations notwithstanding, we believe that these scores have certain limitations that may compromise patient safety when emphasized over clinical judgment. The present article examines CAP scoring systems, emphasizing clinical utility rather than statistical validity. Our critique of CAP scoring systems rests on two principles of good medical
is statistically valid, this parameter relies on false assumptions about how clinical decisions ought to be made. AUROC is predicated on the assumption that sensitivity and specificity are of equal clinical importance. In the case of CAP, this would imply that clinicians should apply equal weight to the risk to patient safety of failing to admit to an ICU a patient who needs ICU care (false negative) on the one hand, and to the risk of excessive use of ICU facilities when a patient who does not require ICU care is admitted to an ICU (false positive) on the other. Although limited resources necessitate a careful consideration of potential false positives, good clinical practice demands that greater emphasis be placed on patient safety by limiting false negatives. A second, and related problem is that each of these scoring systems yields an unacceptable false negative rate, at the recommended cutoff value. In a recent metaanalysis, a PSI score > 4 had a sensitivity of 74% as a predictor of ICU admission.4 Thus, PSI
as a tool to guide ICU admission decisions leads to a potentially inadequate level of care in 26% of patients who ultimately require ICU admission. If one were to decrease the cutoff value to reach an appropriate false negative rate, the false positive rate would be
2
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
practice. First, although the area under the receiver-operator characteristic curve (AUROC)
exceptionally low (10-15%). CURB-65 is an even less desirable score, with a sensitivity of approximately 50%.4 This analysis leads us to three conclusions regarding the appropriate application of prediction rules to CAP patients. First, because of the substantial variability, with many false positives and negatives, no scoring system can be universally recommended to evaluate patients with CAP. Clinicians should only rely upon a prediction score when there
rules should probably be used only when the clinician’s judgment does not specifically favor the need for either higher or lower acuity of care. Second, CURB-65, although popular due to its simplicity and despite its relatively high specificity, might be abandoned altogether as a screening tool; if it is retained, it should only be used if positive because a positive result is highly likely to indicate the need for ICU care. Third, if the clinician chooses to use a prediction rule, it should probably be the PSI. These recommendations are based on the concept that the false negative rate is the parameter of greatest consequence in triaging CAP patients. In summary, when considering the predictive value of a scoring system, a statistician may apply equal weight to the risk of false negatives and false positives, but a clinician should not. False negatives are associated with a decreased level of care and potentially increased adverse outcomes. In the case of CAP, this translates into human lives. False positives are associated with a potentially increased burden on facilities and even wasteful use of limited resources, which society certainly wishes to avoid. Evidencebased medicine deals analytically with statistics. Use of the AUROC curve gives statistically valid data, but, by giving equal weight to false positives and false negatives, AUROC may
3
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
is an intermediate pretest probability of a negative outcome. In practical terms, prediction
potentially endanger patients. Clinicians should only calculate a prediction score when there is an intermediate pretest probability of a negative outcome. In practical terms, prediction rules should probably be used only when the clinician’s judgment does not specifically favor the need for either higher or lower acuity of care. While the above discussion pertains to clinical scoring systems for CAP, our argument can be applied to any prediction rule in which an imminent outcome may be death. Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
Funding: none. Acknowledgements: none. Conflicts of interest: none. Contact information: Michael Abers. One Baylor Plaza, Houston, TX 77030. E-mail: [email protected]
References 1
Lim WS, Baudouin SV, George RC, Hill AT, Jamieson C, Le Jeune I, et al. British Thoracic
Society guidelines for the management of community acquired pneumonia in adults: update 2009. Thorax 2009; 64(Suppl III):iii1–iii55. 2
Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell GD, Dean NC, et al. Infectious
Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis 2007; 44 Suppl 2:S27-72.
4
3
Spindler C, Strålin K, Eriksson L, Hjerdt-Goscinski G, Holmberg H, Lidman C, et al. Swedish
guidelines on the management of community-acquired pneumonia in immunocompetent adults-Swedish Society of Infectious Diseases 2012. Scand J Infect Dis 2012; 44:885-902. 4
Chalmers JD, Mandal P, Singanayagam A, Akram AR, Choudhury G, Short PM, et al. Severity
assessment tools to guide ICU admission in community-acquired pneumonia: systematic review and meta-analysis. Intensive Care Med 2011; 37:1409-20. Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
5
Acronyms CAP: community-acquired pneumonia AUROC: area under the receiver-operator characteristic curve ICU: intensive care unit PSI: pneumonia severity index
Downloaded from http://qjmed.oxfordjournals.org/ at University of Windsor on July 17, 2014
