Assessment of Prediction of Mortality by Using the APACHE II Scoring System in Intensive-Care Units
H. MICHAEL MARSH, M.B.,B.S.,* Critical Care Service; IQBAL KRISHAN, M.D., Section of Health Services Evaluation and Division of Preventive Medicine and Internal Medicine; JAMES M. NAESSENS, M.P.H., Section ofBiostatistics; ROBERT A. STRICKLAND, M.D., DOUGLAS R. GRACEY, M.D., Critical Care Service; MARY E. CAMPION, R.N.C., M.S., Section ofHealth Services Evaluation; FRED T. NOBREGA, M.D., Division ofPreventive Medicine and Internal Medicine; PETER A. SOUTHORN, M.B.,B.S., Critical Care Service; JOHN C. McMICHAN, M.B.,B.S., Ph.D.,t Department ofAnesthesiology; MARY P. KELLY, R.R.A., Section of Health Services Evaluation
Some investigators have suggested that information on quality of care in intensivecare units (ICUs) may be inferred from mortality rates. Specifically, the ratio ofactual to predicted hospital mortality (AlP) has been proposed as a valid measure for comparing ICU outcomes when predicted mortality has been derived from data collected during the first 24 hours of ICU therapy with use of a severity scoring tool, APACHE II (acute physiology and chronic health evaluation). We present a comparison ofmortality ratios (AlP) in four ICUs under common management, in two hospitals within a single institution. Significant differences in AlP were detected for nonoperative patients (0.99 versus 0.67;P =0.014) between the two hospitals. This variation was traced to uneven representation of a subset of patients who had chronic health problems related to diseases that necessitated admission to the hematology-oncology or hepatology service. No differences in AlP were seen between the two hospitals for operative patients or for nonoperative patients on services other than hematologyoncology or hepatology. Thus, differences in AlP detected by using the APACHE II system not only may reside in operational factors within the ICU organization but also may be related to weaknesses in the APACHE II model to measure factors intrinsic to the disease process in some patients. We suggest that case-mix must be examined in detail before concluding that differences in AlP are caused by differences in quality of care.
The assessment of quality of care for hospitalized patients is attracting increasing attention from physicians, hospital administrators, third*Current address: Henry Ford Hospital, Detroit, Michigan. tMayo Clinic Scottsdale, Scottsdale, Arizona. Address reprint requests to Dr. R. A. Strickland, Critical Care Service, Mayo Clinic, Rochester, MN 55905. Mayo Clin Proc 65:1549-1557,1990
party payers, public activists, and governmental agencies. Some authors have proposed that quality of care can be assessed from mortality data and is inversely proportional to the ratio of actual mortality to predicted mortality (AlP). 1-3 The rational use of mortality data as a key indicator of quality necessitates that the biases caused by selection of patients and prognostic variability be addressed through some method
1549
1550 ASSESSMENT OF MORTALTIY PREDICTION
of severity scoring to allow risk adjustment for comparison of different patient samples. For patients in intensive-care units (ICUs), the most concise, readily available tool with a large published data base is the APACHE II (acute physiology and chronic health evaluation) scoring system.l-" Other tools such as diagnosis-related groups," Medisgroups," Medicus," Computerized Severity Index," patient management categories," and staging of disease as developed by Gonnella and associates" would not be anticipated to be useful for prediction of mortality or risk adjustment for ICU patients because they have not been used extensively in intensive-care settings. In the APACHE II system, scores derived from age, physiologic status, chronic health problems, and reasons for admission to the ICU (a weighted diagnostic component) are combined to predict the risk of death for each patient. 1 In a recent study, differences in AlP were measured in 13 tertiary-care hospitals in the United States. 2 The authors concluded that, within specific diagnostic categories, variations in adjusted mortality rates may reflect differences in the way ICUs are organized. They hypothesized that these organizational factors might include the intensity of staff interaction, coordination, and use of protocols for care and thus may be related to outcome. In this report, we examine the differences in ICU outcome as measured by AlP in a setting where organization differences are not expected. The care for critically ill patients in two separate Mayo-affiliated hospitals (Rochester Methodist Hospital and Saint Marys Hospital) is provided by the same medical staff (rotating consultant staff physicians and residents) and by similarly trained nursing and paramedical staffs. Protocols of care are comparable in the four general medical-surgical adult ICUs in these two hospitals. Against this background of similarity, however, are differences in case-mix that distinguish the two Rochester Methodist Hospital ICUs from the two Saint Marys Hospital ICUs. All oncology and preoperative liver transplantation patients are admitted to Rochester Methodist Hospital. This pattern of triage (and the
Mayo
cue Proc, December 1990, Vol 65
constancy of protocolized management and organizational functions between the two hospitals) enabled us to examine the influence of diagnostic components and case chronicity on variations in APACHE II scores.
METHODS
Selection and Management of Patients.Patients were admitted to these four general medical-surgical ICUs on the basis of the presence of either actual or impending organ system failure or after a major operation (predominantly vascular or thoracic procedures). Critically ill patients excluded from this study were adults admitted either to coronary-care units or to cardiac surgical or neurosurgical ICUs and pediatric patients younger than 12 years of age. All hematology-oncology service patients and hepatology service patients were treated at one hospital, the Rochester Methodist Hospital. In 1987, at the time of this study, the four ICUs had the following varied patterns ofadmission of operative and nonoperative patients: Saint Marys Hospital unit 1 (12 beds) had 926 admitted patients, with 21% postoperative, 51% older than age 65 years, and an overall mortality of6%; Saint Marys Hospital unit 2 (24 beds) had 1,863 admitted patients, with 97% postoperative, 54% older than age 65 years, and an overall mortalityof3.5%; Rochester Methodist Hospital unit 3 (14 beds) had 808 admitted patients, with 57% postoperative, 50% older than age 65 years, and an overall mortality of 9%; and Rochester Methodist Hospital unit 4 (12 beds) had 584 admitted patients with 97% postoperative, 54% older than age 65 years, and an overall mortality of 2%. Mechanical ventilation for more than 48 hours was used for 9.4%, 10.9%, 16.0%, and 5.2% of admissions to each of the four units listed consecutively, and the mean durations of stay were 4.5, 4.6, 4.6, and 3.9 days, respectively. The physical facilities in each of these four units were similar, and patient care was provided in each unit by a dedicated critical-care team that consisted of a consultant from the critical-care service (anesthesiologists and pulmonary internists with special expertise in critical care), a senior resident or critical-care fellow,
Mayo Clin Proc, December 1990, Vol 65
and three to four junior residents who worked closely with the bedside registered nurses (8- to 12-hour shifts) and the respiratory therapists dedicated to that unit. The critical-care service was primarily responsible for all supportive care, such as ventilatory support and hemodynamic monitoring, whereas the primary service of record was responsible for visiting the patient at least once or twice daily and for consulting with the critical-care service on the overall strategy for treating the underlying illness. The consultants and residents had 3-week rotating assignments in these units, whereas the nursing and technical support staff remained largely dedicated to a single unit. Group education and practice meetings were held frequently and involved all members of all units. Protocols for care and procedures were as uniform as possible among the four units. Collection of Data.-Data were collected from consecutive admissions to the four ICUs at the two Mayo-affiliated hospitals in Rochester in 1985 and 1987. APACHE II scores (maximum, 71 points) were derived by using age (maximum, 6 points), acute physiology score based on 12 scaled physiologic variables (maximum, 60 points), and chronic health evaluation (maximum, 5 points) for each patient. The acute physiology score was obtained at the time of admission of the patient to the unit and for the worst data during the first 24 hours of the ICU stay. Patients whose stay in the ICU was less than 17 hours were excluded from further analysis. When specific physiologic data were unavailable, that variable was assumed to be within normal values, and a score of zero was assigned to that variable in determining the acute physiology score. Hospital mortality or survival was determined at the time of hospital dismissal. After a pilot study of 141 patients in 1985, a trained team of three qualified data collectors (registered nurses) amassed data from 1,285 patients admitted to these four ICUs in 1987. Overview by the investigators was maintained to ensure the accuracy of data analysis and diagnostic group assignment (reason for ICU admission).
ASSESSMENT OF MORTALITY PREDICTION 1551
Statistical Analysis.-The primary hypothesis of comparison of the AlP ratios between hospitals was made separately for surgical and nonsurgical patients. These comparisons were based on the following logistic model:
in which A is an indicator for death, R is the predicted risk for hospital death of the individual patient based on the APACHE II score, and H is an indicator for hospital. Ifthe 132 coefficient was significantly different from zero, the hospitals were thought to be different. The risk of hospital death, R, was calculated by the formula suggested by Knaus and colleagues,' as follows: In [R/(1- R)]
= -3.517 + 0.146AII + 0.603
(if emergency operation) + diagnostic category weight
in which R is the probability of death, AIl is the total APACHE II score, and the weights for various diagnostic categories are as previously reported in the appendix to the article by Knaus and co-workers. 1 Other secondary comparisons between groups were based on X2 tests for nominal data, Wilcoxon rank sum tests for ordinal and skewed continuous variables, and two-sample t tests for other continuous variables. All tests were two-tailed. Statistical analysis was performed with the SAS package. 10
RESULTS
Comparison of Actual and Predicted Mortality Rates.-The AlP for nonoperative cases at Rochester Methodist Hospital (0.99) was significantly higher (P = 0.014) than that for nonoperative cases at Saint Marys Hospital (0.67) (Table 1). Nonoperative cases at Rochester Methodist Hospital also had significantly higher mean APACHE II scores (P = 0.006) and a higher risk of hospital death (P = 0.002). In contrast, no statistically significant difference was found in AlP for operative cases between Rochester Methodist Hospital and Saint Marys Hospital, although Saint Marys Hospital
1552 ASSESSMENT OF MORTALTIY PREDICTION
Mayo
cue Proc, December 1990, Vol 65
Table I.-Ratio of Actual to Predicted Mortality in Nonoperative and Operative Patients in General Intensive-Care Units at Two Mayo Clinic-Affiliated Hospitals in 1987*
Cases Nonoperative RMH SMH Operative RMH SMH Combined
NP
APACHE II score (mean± SD)
(%)
54 41
0.99t 0.67
17.3 ± s.ir 15.1 ± 7.5
28.0t 22.6
18 22 135
0.49 0.61 0.72
10.9 ± 4.9t 12.3 ± 5.5 13.1 ± 6.6
8.2t 10.3 14.9
Patients
Deaths (no.)
197 272 459 357 1,285
(no.)
R
*NP = ratio of actual to predicted hospital deaths for all
patients with intensivecare unit stays of more than 17 hours (all missing physiologic data were calculated as 0 acute physiology score points); APACHE = acute physiology and chronic health evaluation; R = probability of death as percent of predicted mortality; RMH = Rochester Methodist Hospital; SMH = Saint Marys Hospital. tP
H. MICHAEL MARSH, M.B.,B.S.,* Critical Care Service; IQBAL KRISHAN, M.D., Section of Health Services Evaluation and Division of Preventive Medicine and Internal Medicine; JAMES M. NAESSENS, M.P.H., Section ofBiostatistics; ROBERT A. STRICKLAND, M.D., DOUGLAS R. GRACEY, M.D., Critical Care Service; MARY E. CAMPION, R.N.C., M.S., Section ofHealth Services Evaluation; FRED T. NOBREGA, M.D., Division ofPreventive Medicine and Internal Medicine; PETER A. SOUTHORN, M.B.,B.S., Critical Care Service; JOHN C. McMICHAN, M.B.,B.S., Ph.D.,t Department ofAnesthesiology; MARY P. KELLY, R.R.A., Section of Health Services Evaluation
Some investigators have suggested that information on quality of care in intensivecare units (ICUs) may be inferred from mortality rates. Specifically, the ratio ofactual to predicted hospital mortality (AlP) has been proposed as a valid measure for comparing ICU outcomes when predicted mortality has been derived from data collected during the first 24 hours of ICU therapy with use of a severity scoring tool, APACHE II (acute physiology and chronic health evaluation). We present a comparison ofmortality ratios (AlP) in four ICUs under common management, in two hospitals within a single institution. Significant differences in AlP were detected for nonoperative patients (0.99 versus 0.67;P =0.014) between the two hospitals. This variation was traced to uneven representation of a subset of patients who had chronic health problems related to diseases that necessitated admission to the hematology-oncology or hepatology service. No differences in AlP were seen between the two hospitals for operative patients or for nonoperative patients on services other than hematologyoncology or hepatology. Thus, differences in AlP detected by using the APACHE II system not only may reside in operational factors within the ICU organization but also may be related to weaknesses in the APACHE II model to measure factors intrinsic to the disease process in some patients. We suggest that case-mix must be examined in detail before concluding that differences in AlP are caused by differences in quality of care.
The assessment of quality of care for hospitalized patients is attracting increasing attention from physicians, hospital administrators, third*Current address: Henry Ford Hospital, Detroit, Michigan. tMayo Clinic Scottsdale, Scottsdale, Arizona. Address reprint requests to Dr. R. A. Strickland, Critical Care Service, Mayo Clinic, Rochester, MN 55905. Mayo Clin Proc 65:1549-1557,1990
party payers, public activists, and governmental agencies. Some authors have proposed that quality of care can be assessed from mortality data and is inversely proportional to the ratio of actual mortality to predicted mortality (AlP). 1-3 The rational use of mortality data as a key indicator of quality necessitates that the biases caused by selection of patients and prognostic variability be addressed through some method
1549
1550 ASSESSMENT OF MORTALTIY PREDICTION
of severity scoring to allow risk adjustment for comparison of different patient samples. For patients in intensive-care units (ICUs), the most concise, readily available tool with a large published data base is the APACHE II (acute physiology and chronic health evaluation) scoring system.l-" Other tools such as diagnosis-related groups," Medisgroups," Medicus," Computerized Severity Index," patient management categories," and staging of disease as developed by Gonnella and associates" would not be anticipated to be useful for prediction of mortality or risk adjustment for ICU patients because they have not been used extensively in intensive-care settings. In the APACHE II system, scores derived from age, physiologic status, chronic health problems, and reasons for admission to the ICU (a weighted diagnostic component) are combined to predict the risk of death for each patient. 1 In a recent study, differences in AlP were measured in 13 tertiary-care hospitals in the United States. 2 The authors concluded that, within specific diagnostic categories, variations in adjusted mortality rates may reflect differences in the way ICUs are organized. They hypothesized that these organizational factors might include the intensity of staff interaction, coordination, and use of protocols for care and thus may be related to outcome. In this report, we examine the differences in ICU outcome as measured by AlP in a setting where organization differences are not expected. The care for critically ill patients in two separate Mayo-affiliated hospitals (Rochester Methodist Hospital and Saint Marys Hospital) is provided by the same medical staff (rotating consultant staff physicians and residents) and by similarly trained nursing and paramedical staffs. Protocols of care are comparable in the four general medical-surgical adult ICUs in these two hospitals. Against this background of similarity, however, are differences in case-mix that distinguish the two Rochester Methodist Hospital ICUs from the two Saint Marys Hospital ICUs. All oncology and preoperative liver transplantation patients are admitted to Rochester Methodist Hospital. This pattern of triage (and the
Mayo
cue Proc, December 1990, Vol 65
constancy of protocolized management and organizational functions between the two hospitals) enabled us to examine the influence of diagnostic components and case chronicity on variations in APACHE II scores.
METHODS
Selection and Management of Patients.Patients were admitted to these four general medical-surgical ICUs on the basis of the presence of either actual or impending organ system failure or after a major operation (predominantly vascular or thoracic procedures). Critically ill patients excluded from this study were adults admitted either to coronary-care units or to cardiac surgical or neurosurgical ICUs and pediatric patients younger than 12 years of age. All hematology-oncology service patients and hepatology service patients were treated at one hospital, the Rochester Methodist Hospital. In 1987, at the time of this study, the four ICUs had the following varied patterns ofadmission of operative and nonoperative patients: Saint Marys Hospital unit 1 (12 beds) had 926 admitted patients, with 21% postoperative, 51% older than age 65 years, and an overall mortality of6%; Saint Marys Hospital unit 2 (24 beds) had 1,863 admitted patients, with 97% postoperative, 54% older than age 65 years, and an overall mortalityof3.5%; Rochester Methodist Hospital unit 3 (14 beds) had 808 admitted patients, with 57% postoperative, 50% older than age 65 years, and an overall mortality of 9%; and Rochester Methodist Hospital unit 4 (12 beds) had 584 admitted patients with 97% postoperative, 54% older than age 65 years, and an overall mortality of 2%. Mechanical ventilation for more than 48 hours was used for 9.4%, 10.9%, 16.0%, and 5.2% of admissions to each of the four units listed consecutively, and the mean durations of stay were 4.5, 4.6, 4.6, and 3.9 days, respectively. The physical facilities in each of these four units were similar, and patient care was provided in each unit by a dedicated critical-care team that consisted of a consultant from the critical-care service (anesthesiologists and pulmonary internists with special expertise in critical care), a senior resident or critical-care fellow,
Mayo Clin Proc, December 1990, Vol 65
and three to four junior residents who worked closely with the bedside registered nurses (8- to 12-hour shifts) and the respiratory therapists dedicated to that unit. The critical-care service was primarily responsible for all supportive care, such as ventilatory support and hemodynamic monitoring, whereas the primary service of record was responsible for visiting the patient at least once or twice daily and for consulting with the critical-care service on the overall strategy for treating the underlying illness. The consultants and residents had 3-week rotating assignments in these units, whereas the nursing and technical support staff remained largely dedicated to a single unit. Group education and practice meetings were held frequently and involved all members of all units. Protocols for care and procedures were as uniform as possible among the four units. Collection of Data.-Data were collected from consecutive admissions to the four ICUs at the two Mayo-affiliated hospitals in Rochester in 1985 and 1987. APACHE II scores (maximum, 71 points) were derived by using age (maximum, 6 points), acute physiology score based on 12 scaled physiologic variables (maximum, 60 points), and chronic health evaluation (maximum, 5 points) for each patient. The acute physiology score was obtained at the time of admission of the patient to the unit and for the worst data during the first 24 hours of the ICU stay. Patients whose stay in the ICU was less than 17 hours were excluded from further analysis. When specific physiologic data were unavailable, that variable was assumed to be within normal values, and a score of zero was assigned to that variable in determining the acute physiology score. Hospital mortality or survival was determined at the time of hospital dismissal. After a pilot study of 141 patients in 1985, a trained team of three qualified data collectors (registered nurses) amassed data from 1,285 patients admitted to these four ICUs in 1987. Overview by the investigators was maintained to ensure the accuracy of data analysis and diagnostic group assignment (reason for ICU admission).
ASSESSMENT OF MORTALITY PREDICTION 1551
Statistical Analysis.-The primary hypothesis of comparison of the AlP ratios between hospitals was made separately for surgical and nonsurgical patients. These comparisons were based on the following logistic model:
in which A is an indicator for death, R is the predicted risk for hospital death of the individual patient based on the APACHE II score, and H is an indicator for hospital. Ifthe 132 coefficient was significantly different from zero, the hospitals were thought to be different. The risk of hospital death, R, was calculated by the formula suggested by Knaus and colleagues,' as follows: In [R/(1- R)]
= -3.517 + 0.146AII + 0.603
(if emergency operation) + diagnostic category weight
in which R is the probability of death, AIl is the total APACHE II score, and the weights for various diagnostic categories are as previously reported in the appendix to the article by Knaus and co-workers. 1 Other secondary comparisons between groups were based on X2 tests for nominal data, Wilcoxon rank sum tests for ordinal and skewed continuous variables, and two-sample t tests for other continuous variables. All tests were two-tailed. Statistical analysis was performed with the SAS package. 10
RESULTS
Comparison of Actual and Predicted Mortality Rates.-The AlP for nonoperative cases at Rochester Methodist Hospital (0.99) was significantly higher (P = 0.014) than that for nonoperative cases at Saint Marys Hospital (0.67) (Table 1). Nonoperative cases at Rochester Methodist Hospital also had significantly higher mean APACHE II scores (P = 0.006) and a higher risk of hospital death (P = 0.002). In contrast, no statistically significant difference was found in AlP for operative cases between Rochester Methodist Hospital and Saint Marys Hospital, although Saint Marys Hospital
1552 ASSESSMENT OF MORTALTIY PREDICTION
Mayo
cue Proc, December 1990, Vol 65
Table I.-Ratio of Actual to Predicted Mortality in Nonoperative and Operative Patients in General Intensive-Care Units at Two Mayo Clinic-Affiliated Hospitals in 1987*
Cases Nonoperative RMH SMH Operative RMH SMH Combined
NP
APACHE II score (mean± SD)
(%)
54 41
0.99t 0.67
17.3 ± s.ir 15.1 ± 7.5
28.0t 22.6
18 22 135
0.49 0.61 0.72
10.9 ± 4.9t 12.3 ± 5.5 13.1 ± 6.6
8.2t 10.3 14.9
Patients
Deaths (no.)
197 272 459 357 1,285
(no.)
R
*NP = ratio of actual to predicted hospital deaths for all
patients with intensivecare unit stays of more than 17 hours (all missing physiologic data were calculated as 0 acute physiology score points); APACHE = acute physiology and chronic health evaluation; R = probability of death as percent of predicted mortality; RMH = Rochester Methodist Hospital; SMH = Saint Marys Hospital. tP
