Annals of Epidemiology xxx (2014) 1e6
Contents lists available at ScienceDirect
Annals of Epidemiology journal homepage: www.annalsofepidemiology.org
Original article
Comparison of comorbidity classification methods for predicting outcomes in a population-based cohort of adults with human immunodeficiency virus infection Tony Antoniou PhD a, b, c, d, *, Ryan Ng MSc d, Richard H. Glazier MD, MPH a, b, c, d, e, f, g, Alexander Kopp BA d, Peter C. Austin PhD d, f a
Department of Family and Community Medicine, St. Michael’s Hospital, Toronto, Ontario, Canada The Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada d Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada e Centre for Research on Inner City Health, St. Michael’s Hospital, Toronto, Ontario, Canada f Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada g Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada b c
a r t i c l e i n f o
a b s t r a c t
Article history: Received 15 August 2013 Accepted 7 April 2014 Available online XXX
Purpose: We compared the John’s Hopkins’ Aggregated Diagnosis Groups (ADGs), which are derived using inpatient and outpatient records, with the hospital record-derived Charlson and Elixhauser comorbidity indices for predicting outcomes in human immunodeficiency virus (HIV)-infected patients. Methods: We used a validated algorithm to identify HIV-infected adults (n ¼ 14,313) in Ontario, Canada, and randomly divided the sample into derivation and validation samples 100 times. The primary outcome was all-cause mortality within 1 year, and secondary outcomes included hospital admission and all-cause mortality within 1e2 years. Results: The ADG, Elixhauser, and Charlson methods had comparable discriminative performance for predicting 1-year mortality, with median c-statistics of 0.785, 0.767, and 0.788, respectively, across the 100 validation samples. All methods had lower predictive accuracy for all-cause mortality within 1e2 years. For hospital admission, the ADG method had greater discriminative performance than either the Elixhauser or Charlson methods, with median c-statistics of 0.727, 0.678, and 0.668, respectively. All models displayed poor calibration for each outcome. Conclusions: In patients with HIV, the ADG, Charlson, and Elixhauser methods are comparable for predicting 1-year mortality. However, poor calibration limits the use of these methods for provider profiling and clinical application. Ó 2014 Elsevier Inc. All rights reserved.
Keywords: Comorbidity Databases Factual Diagnosis-related groups Risk adjustment HIV Predictive value of tests ROC curve Logistic models
Introduction Advances in the management of human immunodeficiency virus (HIV) infection have transformed the illness from one associated with high rates of mortality and inpatient utilization into a chronic illness characterized by an aging cohort of ambulatory patients living Competing interests: All authors declare (1) no support from any company for the submitted work; (2) no relationships with any companies that might have an interest in the submitted work in the previous 3 years; (3) their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and (4) no nonfinancial interests that may be relevant to the submitted work. * Corresponding author. Department of Family and Community Medicine, St. Michael’s Hospital, 410 Sherbourne Street, 4th Floor, Toronto, ON M4X 1K2 Canada. Tel.: þ1 416 867 7460x8344; fax: þ1 416 867 3726. E-mail address: [email protected] (T. Antoniou).
with multiple comorbid diseases [1e6]. In this context, there is an increased interest in the use of administrative health care databases to conduct population-based research examining patterns of health services utilization and health outcomes among persons living with HIV [7e9]. Because these studies are typically observational in nature, risk adjustment for morbidity burden is required. Among the various methods available for summarizing the comorbidity of a population, the Charlson and Elixhauser comorbidity indices are commonly used for risk adjustment in observational studies conducted with administrative databases. The Charlson index was originally developed to predict mortality in hospitalized patients using data abstracted from the charts of general medicine inpatients and was subsequently adapted for use with administrative databases using International Classification of Diseases, ninth revision, (ICD-9CM) diagnosis and procedure codes [10,11]. The weighted index is
1047-2797/$ e see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.annepidem.2014.04.002
Please cite this article in press as: Antoniou T, et al., Comparison of comorbidity classification methods for predicting outcomes in a populationbased cohort of adults with human immunodeficiency virus infection, Annals of Epidemiology (2014), http://dx.doi.org/10.1016/ j.annepidem.2014.04.002
2
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
based on diagnosis codes for 17 conditions, each of which are assigned a value of 1, 2, 3, or 6, from which a summary score is computed for each patient. Like the Charlson index, the Elixhauser comorbidity index was developed to quantify the burden of comorbidity among hospitalized patients [12]. However, in lieu of a summary score, the Elixhauser index is computed using binary indicators (i.e. present or absent) for 30 different medical conditions. Although both the Charlson and Elixhauser methods have been validated in various settings and have been updated for use with International Classification of Diseases, 10th revision (ICD-10) codes [13e17], both measures were designed for use solely with inpatient administrative data, thereby limiting their utility in patients who are managed predominantly in outpatient settings. In contrast, the Johns Hopkins Adjusted Clinical Group (ACG) Case-Mix System uses administrative data that are derived from inpatient and outpatient encounters, thereby permitting risk adjustment in ambulatory and hospitalized patients [18e20]. The ACG system assigns each ICD-9 or ICD-10 diagnosis to one of 32 diagnosis clusters known as Aggregated Diagnosis Groups (ADGs). ADGs are similar in terms of severity, persistence over time, and expected need for health care utilization. Each individual may have between zero and 32 ADGs [21]. Although several risk-adjustment indices have been developed and validated for use in HIV patients, these measures have been developed using clinical data that are not routinely available in administrative databases [22e27]. There is therefore a need to compare the predictive validity of comorbidity measures derived from administrative health care data in a population of persons with HIV. Accordingly, we compared the ability of the ADG, Charlson, and Elixhauser methods to predict mortality and hospitalization in persons with HIV. We speculated that by virtue of being derived from both inpatient and outpatient administrative patient records, the ADG method would have superior predictive performance relative to either the Charlson or Elixhauser comorbidity indices.
discharge. These databases were deterministically linked in an anonymous fashion using encrypted health card numbers and have been previously used for the validation of comorbidity indices in populations of Ontario residents with chronic diseases [28e31].
Methods
Outcomes
Data sources
The primary outcome was death from any cause within 365 days of the index date (April 1, 2009). Secondary outcomes included hospitalization from any cause within 365 days of the index date and death from any cause between days 366 and 730 after the index date among individuals who survived for at least 1 year.
We obtained data from Ontario’s administrative health care databases, which are available at the Institute for Clinical Evaluative Sciences through a data sharing agreement with the Ontario Ministry of Health and Long-Term Care. Specifically, we used the Ontario Health Insurance Plan (OHIP) database to identify claims submitted by physicians to the provincial universal health insurance program. To receive payment for services rendered, physicians must submit the name, date of birth, and OHIP card number of the individual patient seen, the service provided (i.e., a service code), and a single diagnosis code on each claim. For OHIP claims, the diagnosis code is a truncated three-digit version of the corresponding ICD-9 code. We obtained hospitalization data from the Canadian Institute for Health Information Discharge Abstract Database (CIHI DAD), which contains information from all acute care hospital separations in Ontario. Each hospitalization record includes the patient OHIP number, dates of admission and discharge from the hospital and up to 25 diagnoses contributing to a given admission, coded by trained health information professionals using standard ICD-10 diagnosis and procedure codes. We obtained basic demographic information, including age, sex, and date of death, from the Registered Persons Database, a registry of all Ontario residents eligible for health insurance. Finally, we used the Ontario Mental Health Reporting System (OMHRS) database to identify admissions to adult-designated inpatient mental health beds in psychiatric and nonpsychiatric facilities in Ontario. Each record in the OMHRS contains data regarding psychiatric and nonpsychiatric diagnoses, as well as reasons for admission and
Study population We identified adults in Ontario aged 18 years and older who were living with HIV as of April 1, 2009 from the Ontario HIV database, an administrative data registry of Ontario residents with diagnosed HIV infection, which was generated using a previously validated case-finding algorithm [32]. The definition of three physician claims with an ICD-9 code for HIV infection (042, 043, and 044) within a 3-year period has a sensitivity and specificity of 96.2% (95 confidence intervals 95.2%e97.9%) and 99.6% (95% confidence intervals 99.1%e99.8%), respectively, for identifying persons with HIV who are regular users of primary care. Due to the use of administrative health care databases, complete follow-up was available for all individuals. For each patient, we determined the Charlson comorbidity score and presence of Elixhauser comorbidities using hospitalization data obtained from the CIHI DAD for all admissions occurring in the 2 years preceding the index date, April 1, 2009. We obtained mental health and addiction diagnoses for the Elixhauser comorbidities from the OMHRS database in addition to the CIHI DAD and OHIP physician claims database. Following common practice, individuals who had not been hospitalized in the previous 2 years had their Charlson score set to zero and values for each of the 30 Elixhauser comorbidities designated as absent [33]. We used the Johns Hopkins ACGs case-mix assignment software (Sun Microsystems Inc., Santa Clara, CA) to determine the presence or absence of each of the 32 ADGs for a given patient using all diagnostic codes listed in the CIHI DAD and OHIP databases in the 2 years preceding the index date.
Statistical analysis To evaluate the predictive performance of each comorbidity classification scheme in a sample that was independent of the sample in which regression models were derived, we used a random number generator to divide the overall sample into approximately equal-sized derivation and validation samples. In the derivation sample, we used a series of “base” and “comorbidity-adjusted” multivariable logistic regression models to assess the predictive performance of each comorbidity measure. Specifically, the base model for each outcome included age (in years), sex, neighborhood income quintile, and number of years in the Ontario HIV database, whereas comorbidityadjusted models included all the variables in the base model in addition to either the Charlson comorbidity index (as a continuous variable), indicator variables denoting the presence or absence of the 30 Elixhauser comorbidities or indicator variables denoting the presence or absence of the 32 ADGs. The discriminatory performance of each model developed in the derivation sample was assessed in the derivation and validations samples using the c-statistic, which is equivalent to the area under the receiver operating characteristic curve for dichotomous outcomes [34]. We repeated the process of randomly splitting the original sample into derivation and validation samples 100 times, applying the coefficients from each derivation
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
sample to the corresponding validation sample, and obtaining a distribution of values for the c-statistic from which we determined the median and interquartile range (IQR). The c-statistic ranges in value from 0.5 (chance prediction) to 1 (perfect prediction), with values of 0.8 representing inadequate, acceptable, and excellent discriminative performance, respectively [35]. We compared the differences in c-statistics between the base and comorbidity-adjusted models using the Wilcoxon signed-ranks test. We assessed model calibration in the validation samples graphically. Specifically, we stratified the validation sample into 10 approximately equally sized groups defined by the deciles of risk for each outcome, and determined the mean predicted (from logistic regression models) and observed probabilities of each outcome within each stratum. We then constructed calibration plots of observed versus predicted probabilities for each outcome using a loess smoothing algorithm [36]. For a well-calibrated model, the observed and expected probabilities will align approximately along a 45 line. Finally, we evaluated predictive accuracy using the Brier score, which provides a measure of overall model accuracy [37]. As a relative measure for comparison, a lower Brier score indicates a model with less prediction error. All statistical analyses were conducted using SAS version 9.3 (SAS institute, Cary, NC). Ethics approval We obtained ethics approval for this study from the Research Ethics Board of Sunnybrook Health Sciences Centre. Results We identified 14,313 individuals diagnosed and living with HIV infection as of April 1, 2009 (Table 1). The median age of the cohort was 45 years (IQR, 39 to 51 years), and approximately 20% were women. During the 1-year follow-up period, 205 individuals (1.4%) died and 1002 (7.0%) were hospitalized at least once. In the validation samples, the addition of each comorbidity measure to the base model improved predictive performance for 1year mortality (P < .001 for each comparison; Table 2).
3
Discriminative performance for 1-year mortality was comparable among the comorbidity-adjusted models, with median c-statistics across the 100 validation samples of 0.785 (IQR, 0.765e0.799), 0.788 (IQR, 0.744e0.797) and 0.767 (IQR, 0.746e0.797) for the ADG, Charlson, and Elixhauser methods, respectively. The median Brier score for all models was 0.014, indicating a very low prediction error. Model discrimination for each of the secondary outcomes in the validation sample was inferior to that observed for 1-year mortality (Table 2). Among the 13,946 individuals who survived at least 1 year, 162 (1.1%) died between days 366 and 730 after the index date. The addition of either the Charlson or the ADG model resulted in statistically significant increases (P < .001) in the c-statistic for this outcome relative to the base model. The predictive performance of the Charlson (median c-statistic 0.733, IQR, 0.713e0.750) and ADG (median c-statistic 0.721, IQR, 0.703e0.740) models were similar. For hospital admission within 1 year, the median c-statistic for the base model was 0.569 (IQR 0.563e0.576). Although the inclusion of each comorbidity measure improved the predictive performance of the base model for this outcome, only the ADG model had a median cstatistic value (0.727, IQR 0.723e0.735) that exceeded the threshold for acceptable discrimination of 0.7 (Table 2). In a formal comparison of c-statistics for this outcome, the ADG model had significantly greater discrimination than both the Charlson (P < .001) and Elixhauser (P < .001) models. Overall calibration was poor for all models (Figs. 1e3). For the primary outcome, the Elixhauser model demonstrates the best concordance between the mean predicted probability of death and the proportion of individuals that died within 1 year of the index date among those subjects with a predicted probability of this outcome of less than 0.3 (Fig. 1). For mortality within 1e2 years, the Elixhauser, base and ADG models have approximately equal calibration for the lower risk strata, although all models have poor calibration at higher predicted probabilities (Fig. 2). The ADG model demonstrates better calibration than the comparator models for hospital admission among those subjects with a predicted probability of this outcome of less than 0.4 (Fig. 3), although, as with the other outcomes, calibration deteriorates at higher predicted probabilities of hospitalization. Discussion
Table 1 Baseline characteristics Variable Age (median, IQR) Age group, n (%) 18e30 31e45 46e64 65þ Female, n (%) Years with HIV, (median, IQR) Residence, n (%) Urban Rural Missing Income quintile, n (%) 1 (lowest) 2 3 4 5 Missing Death (within 1 year), n (%) Hospitalization (within 1 year), n (%) Death (between 1 and 2 years), n (%)* IQR ¼ interquartile range. * Number in denominator: 14,108.
Full cohort (n ¼ 14,313) 45 (39e51) 1136 6474 6036 667 2791 9.5
(7.9) (45.2) (42.2) (4.7) (19.5) (4.4e14.3)
13,453 (94.0) 574 (4.0) 286 (2.0) 4451 2881 2285 2090 2196 410 205 1002 162
(31.1) (20.1) (16.0) (14.6) (15.3) (2.9) (1.4) (7.0) (1.1)
In our study, we found that risk-adjustment with either the Charlson, Elixhauser, or Johns Hopkins’ ADGs improved the discriminative performance of logistic regression models based on age, sex, neighborhood income quintile and years in the Ontario HIV database when predicting mortality in persons with HIV, and that all three models had similar discriminative performance for this outcome. In addition, we found that the ADG method performed best for predicting the probability of hospital admission within 1 year and was comparable with the Charlson index for predicting the probability of death between 1 and 2 years. Because discriminative ability is the most relevant model performance measure for research purposes, our study supports the utility of the ADG measure as an alternative to the Charlson and Elixhauser measures as a method of risk and comorbidity adjustment in a population-based cohort of persons with HIV. However, the poor calibration of the models indicates that they would not be useful for prognostic application in individual patient cases or comparing outcomes between health providers. Because persons with HIV are managed predominantly in the ambulatory setting, the improved discriminative performance of the ADG method in this population for predicting hospitalization may reflect the incorporation of data generated by both inpatient and outpatient encounters. In contrast, the Charlson and Elixhauser comorbidity indices rely solely on diagnosis codes from hospital
4
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
Table 2 Predictive accuracy of different models for death and hospitalization in a cohort of persons with HIV Model
Death within 1 year Base model* Base model þ Charlson Base model þ Elixhauser Base model þ ADG Hospitalization (1 year) Base model* Base model þ Charlson Base model þ Elixhauser Base model þ ADG Death (between 1 and 2 years) Base model* Base model þ Charlson Base model þ Elixhauser Base model þ ADG *
Derivation sample
Validation sample
Median c-statistic (IQR)
Median change in c-statistic (IQR)
Median Brier score
Median c-statistic (IQR)
Median change in c-statistic (IQR)
Median Brier score
0.683 0.809 0.842 0.846
(0.668e0.692) (0.800e0.825) (0.832e0.852) (0.838e0.858)
d 0.131 (0.115e0.141) 0.162 (0.145e0.172) 0.166 (0.151e0.182)
0.013 0.014 0.013 0.013
0.651 0.788 0.767 0.785
(0.635e0.662) (0.774e0.797) (0.746e0.783) (0.765e0.799)
d 0.138 (0.125e0.153) 0.117 (0.101e0.134) 0.137 (0.115e0.151)
0.014 0.014 0.014 0.014
0.584 0.681 0.700 0.750
(0.576e0.590) (0.674e0.687) (0.693e0.707) (0.743e0.755)
d 0.100 (0.092e0.103) 0.118 (0.110e0.124) 0.265 (0.255e0.278)
0.066 0.062 0.061 0.061
0.569 0.668 0.678 0.727
(0.563e0.576) (0.663e0.677) (0.666e0.685) (0.723e0.735)
d 0.100 (0.094e0.105) 0.108 (0.098e0.112) 0.158 (0.151e0.165)
0.066 0.063 0.063 0.062
0.703 0.758 0.790 0.823
(0.685e0.717) (0.743e0.777) (0.779e0.802) (0.812e0.835)
d 0.060 (0.044e0.073) 0.087 (0.075e0.108) 0.120 (0.105e0.138)
0.012 0.011 0.011 0.011
0.666 0.733 0.658 0.721
(0.649e0.688) (0.713e0.750) (0.630e0.684) (0.703e0.740)
d 0.066 (0.052e0.075) 0.007 (0.038 to 0.015) 0.056 (0.034e0.076)
0.012 0.011 0.012 0.012
Logistic regression model with age, sex, neighborhood income quintile, and years in Ontario HIV database.
discharge abstracts, and may therefore miss important comorbid conditions recorded on outpatient claims. Although some investigators have addressed this problem by using both inpatient and outpatient data to compute the Charlson summary score, this modification has not universally improved the predictive performance of this index [38e40]. Furthermore, both the Charlson and Elixhauser indices were initially derived and validated in hospitalized patients and were developed for use with administrative inpatient data [10,12]. We therefore elected to examine the predictive validity of these indices in a manner that reflects their original derivation and validation. In addition, differences in the performance of the various measures could reflect the use of a grouping algorithm by the Johns Hopkins ACG case-mix software, whereby similar conditions are clustered into ADGs based on physician opinion about the expected duration of the illness, potential for disability and reduced longevity, certainty of diagnosis, etiology, and expected need for specialist care and hospitalization. However, the choice of comorbidity measure depends not only on factors such as validity and choice of outcome but also on practical considerations such as the availability of data or the need for specialized software to create the measure. The latter is a possible limitation to the use of Johns Hopkins ADGs in some settings, whereas the Charlson and Elixhauser methods are nonproprietary and therefore do not require a fee for use. Furthermore, with 32 indicator variables, use of ADGs may result in model overfitting in small data sets with few outcomes. However, the same limitation applies to Elixhauser comorbidities, and it is possible to model ADGs more parsimoniously by collapsing the 32 categories into 12 clinically cogent groupings known as collapsed ADGS. Furthermore, a prior study demonstrated how the 32 ADGs can be collapsed into a single weighted score that reflects the risk of 1-year death in the general adult population [41]. Previous studies examining the predictive validity of different comorbidity measures in persons with HIV have focused on the development and application of measures derived from clinical data [22e27]. However, because many of these indices were developed before the introduction of potent antiretroviral therapies and typically emphasized viral load, CD4þ count and the presence or absence of AIDS-defining illnesses when predicting outcomes in persons with HIV, their applicability to the contemporary era of HIV management may be limited. To address this concern, investigators from the Veterans Aging Cohort Study (VACS) have developed an index that supplements traditional markers of risk in persons with HIV with a variety of routinely monitored laboratory tests and
hepatitis C coinfection status [27]. The discriminative performance of the VACS index has been determined for various outcomes in persons with HIV, with a c-statistic of 0.78e0.81 for all-cause mortality at 5 years among patients followed in the North American AIDS Cohort Collaboration [42]. Moreover, the VACS index demonstrated good concordance between predicted probability of death and observed death, perhaps reflecting the importance of HIV and organ system biomarkers when developing well-calibrated predictive models for persons with HIV. Furthermore, each 5point increment in the VACS index score was associated with a 10% increased risk of hospitalization and intensive care unit admission over a 2-year follow-up period among HIV-infected veterans [43]. Because the VACS index is based on laboratory tests routinely used in the management of persons with HIV, it has the advantage of being readily adaptable for use to the clinical setting. However, alternative methods of risk-adjustment are required when using administrative databases to examine outcomes in persons with HIV, since clinical data are typically not available to investigators who work with these data sets. To the best of our knowledge, ours is the first study comparing the predictive validity of comorbidity measures that can be derived from administrative data in a large population of persons with HIV.
Fig. 1. Calibration plot: comparison of observed and predicted probability of death within 1 year.
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
5
validated algorithm with excellent test characteristics for discriminating between HIV-infected and noninfected individuals to assemble our cohort [32]. Conclusions We found that the ADG, Charlson, and Elixhauser methods were all comparable for predicting 1-year mortality in a population of persons with HIV, and that the ADG method demonstrated superior discriminative performance for hospitalization relative to the other measures in these subjects. However, all methods displayed generally poor calibration for each outcome. Thus, when focusing on hospitalizations, the ADG method may be the preferred method for risk-adjustment in observational studies of inpatient utilization among persons living with HIV, but like other measures, may be unreliable for medical decision making or provider profiling. Further research evaluating the predictive validity of various comorbidity classification methods in persons with HIV is needed. Fig. 2. Calibration plot: comparison of observed and predicted probability of death between 1 and 2 years.
The main strength of our study is the use of comprehensive population-based data, thereby allowing us to evaluate the predictive performance of various comorbidity measures in all persons diagnosed with HIV. However, several limitations of our work merit emphasis. First, we had no access to clinical data and prescription drug records for our population of persons with HIV, rendering it impossible to account for important predictors of death and hospitalization, such as smoking, use of antiretroviral therapy, and the components of the VACS index. In addition, the lack of data regarding prescription drug use precluded an examination of the predictive performance of comorbidity measures, which are derived using these data, including the Chronic Disease Score and the number of unique drugs prescribed [40,44]. We used years in the Ontario HIV database as a proxy for the number of years diagnosed with HIV, which underestimates this variable for people diagnosed before 1991. To prevent model overfitting, we did not test for interactions between candidate predictors in our analyses. Finally, the potential for misclassification is always a consideration when using administrative data. To address this concern, we used a
Acknowledgment Funding: This project was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care. The sponsors had no role in the design or conduct of the study; in the collection, analysis or interpretation of the data; or in the preparation, review or approval of the manuscript. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding source. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. T.A. is supported by a new investigator award from the Canadian Institutes of Health Research and the Ontario HIV Treatment Network. R.H.G. is a Clinician Scientist in the Department of Family and Community Medicine at the University of Toronto and St. Michael’s Hospital. P.C.A. is supported by a Career Investigator award from the Heart and Stroke Foundation. Contributors: All authors contributed to the concept and design of the study. T.A., R.N., and A.K. acquired the data, and all authors were involved in the analysis and interpretation of the data. T.A. drafted the manuscript, and all authors were involved in critical revision of the manuscript. All authors approved the manuscript submitted for publication. T.A. is the guarantor for the article. References
Fig. 3. Calibration plot: comparison of observed and predicted probability of hospitalization within 1 year.
[1] Antiretroviral Therapy Cohort Collaboration, Zwahlen M, Harris R, May M, Hogg R, Costagliola D, et al. Mortality of HIV-infected patients starting potent antiretroviral therapy: comparison with the general population in nine industrialized countries. Int J Epidemiol 2009;38:1624e33. [2] Hasse B, Ledergerber B, Furrer H, Battegay M, Hirschel B, Cavassini M, et al. Morbidity and aging in HIV-infected persons: the Swiss HIV Cohort Study. Clin Infect Dis 2011;53:1130e9. [3] Van Sighem AI, Gras LA, Reiss P, Brinkman K, de Wolf F. ATHENA National Observational Cohort Study. Life expectancy of recently diagnosed asymptomatic HIV-infected patients approaches that of uninfected individuals. AIDS 2010;24:1527e35. [4] Berry SA, Fleishman JA, Moore RD, Gebo KA. Trends in reasons for hospitalizations in a multisite United States Cohort of Persons Living with HIV, 20012008. J Acquir Immune Defic Syndr 2012;59:368e75. [5] Goulet JL, Fultz SL, Rimland D, Butt A, Gibert C, Rodriguez-Barradas M, et al. Aging and infectious diseases: do patterns of comorbidity vary by HIV status, age, and HIV severity? Clin Infect Dis 2007;45:1593e601. [6] Antoniou T, Zagorski B, Loutfy MR, Strike C, Glazier RH. Socio-economic and sex-related disparities in rates of hospital admission among patients with HIV infection in Ontario: a population-based study. Open Med 2012;6:e146e54. [7] Fultz SL, Skanderson M, Mole LA, Gandhi N, Bryant K, Crystal S, et al. Development and verification of a “virtual” cohort using the national VA health information system. Med Care 2006;44:S25e30. [8] Walkup JT, Wei W, Sambamoorthi U, Crystal S. Sensitivity of an AIDS casefinding algorithm: who are we missing? Med Care 2004;42:756e63.
6
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
[9] Justice AC, Landefeld CS, Asch SM, Gifford AL, Whalen CC, Covinsky KE. Justification for a new cohort study of people aging with and without HIV infection. J Clin Epidemiol 2001;54(Suppl 1):S3e8. [10] Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373e83. [11] Deyo RA, Cherken DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613e9. [12] Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8e27. [13] Stukenborg GJ, Wagner DP, Connors AF. Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations. Med Care 2001;39:727e39. [14] Quan H, Sundararajan V, Halfon P, Fong A, Burnard B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005;43:1130e9. [15] Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol 2004;57:1288e94. [16] Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/ Deyo methods of comorbidity measurement in administrative data. Med Care 2004;42:355e60. [17] Quail JM, Lix LM, Osman BA, Teare GF. Comparing comorbidity measures for predicting mortality and hospitalization in three population-based cohorts. BMC Health Servi Res 2011;11:146. [18] Johns Hopkins University. Johns Hopkins ACG Case-Mix Adjustment System. Available at: https://acg.jhsph.org. [19] Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management. Health Serv Res 1991;26:53e74. [20] Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory case case-mix. Med Care 1991;29:452e72. [21] Sibley LM, Moineddin R, Agha MM, Glazier RH. Risk adjustment using administrative data-based and survey-derived methods for explaining physician utilization. Med Care 2010;48:175e82. [22] Turner BJ, Kelly JV, Ball JK. A severity classification system for AIDS hospitalizations. Med Care 1989;27:423e37. [23] Justice AC, Feinstein AR, Wells CK. A new prognostic staging system for the acquired immunodeficiency syndrome. N Engl J Med 1989;320:1388e93. [24] Egger M, May M, Chêne G, Phillips AN, Ledergerber B, Dabis F, et al. Prognosis of HIV-1 infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet 2002;360:119e29. [25] Mocroft A, Kirk O, Barton SE, Dietrich M, Proenca R, Colebunders R, et al. Anaemia is an independent predictive marker for clinical prognosis in HIVinfected patients from across Europe. EuroSIDA study group. AIDS 1999;13:943e50. [26] Lundgren JD, Mocroft A, Gatell JM, Ledergerber B, D’Arminio Monforte A, Hermans P, et al. A clinically prognostic scoring system for patients receiving highly active antiretroviral therapy: results from the EuroSIDA study. J Infect Dis 2002;185:178e87. [27] Justice AC, McGinnis KA, Skanderson M, Chang CC, Gibert CL, Goetz MB, et al. Towards a combined prognostic index for survival in HIV infection: the role of ‘non-HIV’ biomarkers. HIV Med 2009;11:143e51.
[28] Austin PC, Shah BR, Newman A, Anderson GM. Using the Johns Hopkins’ Aggregated Diagnosis Groups (ADGs) to predict 1-year mortality in population-based cohorts of patients with diabetes in Ontario, Canada. Diabet Med 2012;29:1134e41. [29] Austin PC, van Walraven C, Wodchis WP, Newman A, Anderson GM. Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a general adult population cohort in Ontario, Canada. Med Care 2011;49:932e9. [30] Austin PC, Newman A, Kurdyak PM. Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a population-based cohort of subjects with schizophrenia in Ontario, Canada. Psychiatry Res 2012;196:32e7. [31] Austin PC, Stanbrook MB, Anderson GM, Newman A, Gershon AS. Comparative ability of comorbidity classification methods for administrative data to predict outcomes in patients with chronic obstructive pulmonary disease. Ann Epidemiol 2012;22:881e7. [32] Antoniou T, Zagorski B, Loutfy MR, Strike C, Glazier RH. Validation of casefinding algorithms derived from administrative data for identifying adults living with human immunodeficiency virus infection. PLoS One 2011;6:e21748. [33] Perkins AJ, Kroenke K, Unützer J, Katon W, Williams Jr JW, Hope C, et al. Common comorbidity scales were similar in their ability to predict health care costs and mortality. J Clin Epidemiol 2004;57:1040e8. [34] Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128e38. [35] Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York, NY: John Wiley & Sons, Inc; 2000. [36] Austin PC, Steryerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014;33:517e35. [37] Steyerberg EW. Clinical prediction models. A practical approach to development, validation, and updating. New York, NY: Springer; 2009. [38] Klabunde CN, Potosky AL, Legler JM, Warren JL. Development of a comorbidity index using physician claims data. J Clin Epidemiol 2000;53:1258e67. [39] Fan VS, Maciejewski ML, Liu CF, McDonell MB, Fihn SD. Comparison of risk adjustment measures based on self-report, administrative data, and pharmacy records to predict clinical outcomes. Health Serv Outcomes Res Method 2006;6:21e36. [40] Schneeweiss S, Seeger JD, Maclure M, Wang PS, Avorn J, Glynn RJ. Performance of comorbidity scores to control for confounding in epidemiologic studies using claims data. Am J Epidemiol 2001;154(9):854e64. [41] Austin PC, Walraven CV. The mortality risk score and the ADG score: two points-based scoring systems for the Johns Hopkins aggregated diagnosis groups to predict mortality in a general adult population cohort in Ontario. Can Med Care 2011;49:940e7. [42] Justice AC, Modur SP, Tate JP, Althoff KN, Jacobson LP, Gebo KA, et al. Predictive accuracy of the Veterans Aging Cohort Study Index for mortality with HIV infection: a North American cross cohort analysis. J Acquir Immune Defic Syndr 2013;62:149e63. [43] Akgün KM, Gordon K, Pisani M, Fired T, McGinnis KA, Tate JP, et al. Risk factors for hospitalization and medical intensive care unit (MICU) admission among HIV-infected veterans. J Acquir Immune Defic Syndr 2013;62:52e9. [44] Von Korff M, Wagner EH, Sunders K. A chronic disease score from automated pharmacy data. J Clin Epidemiol 1992;45:197e203.
Contents lists available at ScienceDirect
Annals of Epidemiology journal homepage: www.annalsofepidemiology.org
Original article
Comparison of comorbidity classification methods for predicting outcomes in a population-based cohort of adults with human immunodeficiency virus infection Tony Antoniou PhD a, b, c, d, *, Ryan Ng MSc d, Richard H. Glazier MD, MPH a, b, c, d, e, f, g, Alexander Kopp BA d, Peter C. Austin PhD d, f a
Department of Family and Community Medicine, St. Michael’s Hospital, Toronto, Ontario, Canada The Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada d Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada e Centre for Research on Inner City Health, St. Michael’s Hospital, Toronto, Ontario, Canada f Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada g Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada b c
a r t i c l e i n f o
a b s t r a c t
Article history: Received 15 August 2013 Accepted 7 April 2014 Available online XXX
Purpose: We compared the John’s Hopkins’ Aggregated Diagnosis Groups (ADGs), which are derived using inpatient and outpatient records, with the hospital record-derived Charlson and Elixhauser comorbidity indices for predicting outcomes in human immunodeficiency virus (HIV)-infected patients. Methods: We used a validated algorithm to identify HIV-infected adults (n ¼ 14,313) in Ontario, Canada, and randomly divided the sample into derivation and validation samples 100 times. The primary outcome was all-cause mortality within 1 year, and secondary outcomes included hospital admission and all-cause mortality within 1e2 years. Results: The ADG, Elixhauser, and Charlson methods had comparable discriminative performance for predicting 1-year mortality, with median c-statistics of 0.785, 0.767, and 0.788, respectively, across the 100 validation samples. All methods had lower predictive accuracy for all-cause mortality within 1e2 years. For hospital admission, the ADG method had greater discriminative performance than either the Elixhauser or Charlson methods, with median c-statistics of 0.727, 0.678, and 0.668, respectively. All models displayed poor calibration for each outcome. Conclusions: In patients with HIV, the ADG, Charlson, and Elixhauser methods are comparable for predicting 1-year mortality. However, poor calibration limits the use of these methods for provider profiling and clinical application. Ó 2014 Elsevier Inc. All rights reserved.
Keywords: Comorbidity Databases Factual Diagnosis-related groups Risk adjustment HIV Predictive value of tests ROC curve Logistic models
Introduction Advances in the management of human immunodeficiency virus (HIV) infection have transformed the illness from one associated with high rates of mortality and inpatient utilization into a chronic illness characterized by an aging cohort of ambulatory patients living Competing interests: All authors declare (1) no support from any company for the submitted work; (2) no relationships with any companies that might have an interest in the submitted work in the previous 3 years; (3) their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and (4) no nonfinancial interests that may be relevant to the submitted work. * Corresponding author. Department of Family and Community Medicine, St. Michael’s Hospital, 410 Sherbourne Street, 4th Floor, Toronto, ON M4X 1K2 Canada. Tel.: þ1 416 867 7460x8344; fax: þ1 416 867 3726. E-mail address: [email protected] (T. Antoniou).
with multiple comorbid diseases [1e6]. In this context, there is an increased interest in the use of administrative health care databases to conduct population-based research examining patterns of health services utilization and health outcomes among persons living with HIV [7e9]. Because these studies are typically observational in nature, risk adjustment for morbidity burden is required. Among the various methods available for summarizing the comorbidity of a population, the Charlson and Elixhauser comorbidity indices are commonly used for risk adjustment in observational studies conducted with administrative databases. The Charlson index was originally developed to predict mortality in hospitalized patients using data abstracted from the charts of general medicine inpatients and was subsequently adapted for use with administrative databases using International Classification of Diseases, ninth revision, (ICD-9CM) diagnosis and procedure codes [10,11]. The weighted index is
1047-2797/$ e see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.annepidem.2014.04.002
Please cite this article in press as: Antoniou T, et al., Comparison of comorbidity classification methods for predicting outcomes in a populationbased cohort of adults with human immunodeficiency virus infection, Annals of Epidemiology (2014), http://dx.doi.org/10.1016/ j.annepidem.2014.04.002
2
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
based on diagnosis codes for 17 conditions, each of which are assigned a value of 1, 2, 3, or 6, from which a summary score is computed for each patient. Like the Charlson index, the Elixhauser comorbidity index was developed to quantify the burden of comorbidity among hospitalized patients [12]. However, in lieu of a summary score, the Elixhauser index is computed using binary indicators (i.e. present or absent) for 30 different medical conditions. Although both the Charlson and Elixhauser methods have been validated in various settings and have been updated for use with International Classification of Diseases, 10th revision (ICD-10) codes [13e17], both measures were designed for use solely with inpatient administrative data, thereby limiting their utility in patients who are managed predominantly in outpatient settings. In contrast, the Johns Hopkins Adjusted Clinical Group (ACG) Case-Mix System uses administrative data that are derived from inpatient and outpatient encounters, thereby permitting risk adjustment in ambulatory and hospitalized patients [18e20]. The ACG system assigns each ICD-9 or ICD-10 diagnosis to one of 32 diagnosis clusters known as Aggregated Diagnosis Groups (ADGs). ADGs are similar in terms of severity, persistence over time, and expected need for health care utilization. Each individual may have between zero and 32 ADGs [21]. Although several risk-adjustment indices have been developed and validated for use in HIV patients, these measures have been developed using clinical data that are not routinely available in administrative databases [22e27]. There is therefore a need to compare the predictive validity of comorbidity measures derived from administrative health care data in a population of persons with HIV. Accordingly, we compared the ability of the ADG, Charlson, and Elixhauser methods to predict mortality and hospitalization in persons with HIV. We speculated that by virtue of being derived from both inpatient and outpatient administrative patient records, the ADG method would have superior predictive performance relative to either the Charlson or Elixhauser comorbidity indices.
discharge. These databases were deterministically linked in an anonymous fashion using encrypted health card numbers and have been previously used for the validation of comorbidity indices in populations of Ontario residents with chronic diseases [28e31].
Methods
Outcomes
Data sources
The primary outcome was death from any cause within 365 days of the index date (April 1, 2009). Secondary outcomes included hospitalization from any cause within 365 days of the index date and death from any cause between days 366 and 730 after the index date among individuals who survived for at least 1 year.
We obtained data from Ontario’s administrative health care databases, which are available at the Institute for Clinical Evaluative Sciences through a data sharing agreement with the Ontario Ministry of Health and Long-Term Care. Specifically, we used the Ontario Health Insurance Plan (OHIP) database to identify claims submitted by physicians to the provincial universal health insurance program. To receive payment for services rendered, physicians must submit the name, date of birth, and OHIP card number of the individual patient seen, the service provided (i.e., a service code), and a single diagnosis code on each claim. For OHIP claims, the diagnosis code is a truncated three-digit version of the corresponding ICD-9 code. We obtained hospitalization data from the Canadian Institute for Health Information Discharge Abstract Database (CIHI DAD), which contains information from all acute care hospital separations in Ontario. Each hospitalization record includes the patient OHIP number, dates of admission and discharge from the hospital and up to 25 diagnoses contributing to a given admission, coded by trained health information professionals using standard ICD-10 diagnosis and procedure codes. We obtained basic demographic information, including age, sex, and date of death, from the Registered Persons Database, a registry of all Ontario residents eligible for health insurance. Finally, we used the Ontario Mental Health Reporting System (OMHRS) database to identify admissions to adult-designated inpatient mental health beds in psychiatric and nonpsychiatric facilities in Ontario. Each record in the OMHRS contains data regarding psychiatric and nonpsychiatric diagnoses, as well as reasons for admission and
Study population We identified adults in Ontario aged 18 years and older who were living with HIV as of April 1, 2009 from the Ontario HIV database, an administrative data registry of Ontario residents with diagnosed HIV infection, which was generated using a previously validated case-finding algorithm [32]. The definition of three physician claims with an ICD-9 code for HIV infection (042, 043, and 044) within a 3-year period has a sensitivity and specificity of 96.2% (95 confidence intervals 95.2%e97.9%) and 99.6% (95% confidence intervals 99.1%e99.8%), respectively, for identifying persons with HIV who are regular users of primary care. Due to the use of administrative health care databases, complete follow-up was available for all individuals. For each patient, we determined the Charlson comorbidity score and presence of Elixhauser comorbidities using hospitalization data obtained from the CIHI DAD for all admissions occurring in the 2 years preceding the index date, April 1, 2009. We obtained mental health and addiction diagnoses for the Elixhauser comorbidities from the OMHRS database in addition to the CIHI DAD and OHIP physician claims database. Following common practice, individuals who had not been hospitalized in the previous 2 years had their Charlson score set to zero and values for each of the 30 Elixhauser comorbidities designated as absent [33]. We used the Johns Hopkins ACGs case-mix assignment software (Sun Microsystems Inc., Santa Clara, CA) to determine the presence or absence of each of the 32 ADGs for a given patient using all diagnostic codes listed in the CIHI DAD and OHIP databases in the 2 years preceding the index date.
Statistical analysis To evaluate the predictive performance of each comorbidity classification scheme in a sample that was independent of the sample in which regression models were derived, we used a random number generator to divide the overall sample into approximately equal-sized derivation and validation samples. In the derivation sample, we used a series of “base” and “comorbidity-adjusted” multivariable logistic regression models to assess the predictive performance of each comorbidity measure. Specifically, the base model for each outcome included age (in years), sex, neighborhood income quintile, and number of years in the Ontario HIV database, whereas comorbidityadjusted models included all the variables in the base model in addition to either the Charlson comorbidity index (as a continuous variable), indicator variables denoting the presence or absence of the 30 Elixhauser comorbidities or indicator variables denoting the presence or absence of the 32 ADGs. The discriminatory performance of each model developed in the derivation sample was assessed in the derivation and validations samples using the c-statistic, which is equivalent to the area under the receiver operating characteristic curve for dichotomous outcomes [34]. We repeated the process of randomly splitting the original sample into derivation and validation samples 100 times, applying the coefficients from each derivation
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
sample to the corresponding validation sample, and obtaining a distribution of values for the c-statistic from which we determined the median and interquartile range (IQR). The c-statistic ranges in value from 0.5 (chance prediction) to 1 (perfect prediction), with values of 0.8 representing inadequate, acceptable, and excellent discriminative performance, respectively [35]. We compared the differences in c-statistics between the base and comorbidity-adjusted models using the Wilcoxon signed-ranks test. We assessed model calibration in the validation samples graphically. Specifically, we stratified the validation sample into 10 approximately equally sized groups defined by the deciles of risk for each outcome, and determined the mean predicted (from logistic regression models) and observed probabilities of each outcome within each stratum. We then constructed calibration plots of observed versus predicted probabilities for each outcome using a loess smoothing algorithm [36]. For a well-calibrated model, the observed and expected probabilities will align approximately along a 45 line. Finally, we evaluated predictive accuracy using the Brier score, which provides a measure of overall model accuracy [37]. As a relative measure for comparison, a lower Brier score indicates a model with less prediction error. All statistical analyses were conducted using SAS version 9.3 (SAS institute, Cary, NC). Ethics approval We obtained ethics approval for this study from the Research Ethics Board of Sunnybrook Health Sciences Centre. Results We identified 14,313 individuals diagnosed and living with HIV infection as of April 1, 2009 (Table 1). The median age of the cohort was 45 years (IQR, 39 to 51 years), and approximately 20% were women. During the 1-year follow-up period, 205 individuals (1.4%) died and 1002 (7.0%) were hospitalized at least once. In the validation samples, the addition of each comorbidity measure to the base model improved predictive performance for 1year mortality (P < .001 for each comparison; Table 2).
3
Discriminative performance for 1-year mortality was comparable among the comorbidity-adjusted models, with median c-statistics across the 100 validation samples of 0.785 (IQR, 0.765e0.799), 0.788 (IQR, 0.744e0.797) and 0.767 (IQR, 0.746e0.797) for the ADG, Charlson, and Elixhauser methods, respectively. The median Brier score for all models was 0.014, indicating a very low prediction error. Model discrimination for each of the secondary outcomes in the validation sample was inferior to that observed for 1-year mortality (Table 2). Among the 13,946 individuals who survived at least 1 year, 162 (1.1%) died between days 366 and 730 after the index date. The addition of either the Charlson or the ADG model resulted in statistically significant increases (P < .001) in the c-statistic for this outcome relative to the base model. The predictive performance of the Charlson (median c-statistic 0.733, IQR, 0.713e0.750) and ADG (median c-statistic 0.721, IQR, 0.703e0.740) models were similar. For hospital admission within 1 year, the median c-statistic for the base model was 0.569 (IQR 0.563e0.576). Although the inclusion of each comorbidity measure improved the predictive performance of the base model for this outcome, only the ADG model had a median cstatistic value (0.727, IQR 0.723e0.735) that exceeded the threshold for acceptable discrimination of 0.7 (Table 2). In a formal comparison of c-statistics for this outcome, the ADG model had significantly greater discrimination than both the Charlson (P < .001) and Elixhauser (P < .001) models. Overall calibration was poor for all models (Figs. 1e3). For the primary outcome, the Elixhauser model demonstrates the best concordance between the mean predicted probability of death and the proportion of individuals that died within 1 year of the index date among those subjects with a predicted probability of this outcome of less than 0.3 (Fig. 1). For mortality within 1e2 years, the Elixhauser, base and ADG models have approximately equal calibration for the lower risk strata, although all models have poor calibration at higher predicted probabilities (Fig. 2). The ADG model demonstrates better calibration than the comparator models for hospital admission among those subjects with a predicted probability of this outcome of less than 0.4 (Fig. 3), although, as with the other outcomes, calibration deteriorates at higher predicted probabilities of hospitalization. Discussion
Table 1 Baseline characteristics Variable Age (median, IQR) Age group, n (%) 18e30 31e45 46e64 65þ Female, n (%) Years with HIV, (median, IQR) Residence, n (%) Urban Rural Missing Income quintile, n (%) 1 (lowest) 2 3 4 5 Missing Death (within 1 year), n (%) Hospitalization (within 1 year), n (%) Death (between 1 and 2 years), n (%)* IQR ¼ interquartile range. * Number in denominator: 14,108.
Full cohort (n ¼ 14,313) 45 (39e51) 1136 6474 6036 667 2791 9.5
(7.9) (45.2) (42.2) (4.7) (19.5) (4.4e14.3)
13,453 (94.0) 574 (4.0) 286 (2.0) 4451 2881 2285 2090 2196 410 205 1002 162
(31.1) (20.1) (16.0) (14.6) (15.3) (2.9) (1.4) (7.0) (1.1)
In our study, we found that risk-adjustment with either the Charlson, Elixhauser, or Johns Hopkins’ ADGs improved the discriminative performance of logistic regression models based on age, sex, neighborhood income quintile and years in the Ontario HIV database when predicting mortality in persons with HIV, and that all three models had similar discriminative performance for this outcome. In addition, we found that the ADG method performed best for predicting the probability of hospital admission within 1 year and was comparable with the Charlson index for predicting the probability of death between 1 and 2 years. Because discriminative ability is the most relevant model performance measure for research purposes, our study supports the utility of the ADG measure as an alternative to the Charlson and Elixhauser measures as a method of risk and comorbidity adjustment in a population-based cohort of persons with HIV. However, the poor calibration of the models indicates that they would not be useful for prognostic application in individual patient cases or comparing outcomes between health providers. Because persons with HIV are managed predominantly in the ambulatory setting, the improved discriminative performance of the ADG method in this population for predicting hospitalization may reflect the incorporation of data generated by both inpatient and outpatient encounters. In contrast, the Charlson and Elixhauser comorbidity indices rely solely on diagnosis codes from hospital
4
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
Table 2 Predictive accuracy of different models for death and hospitalization in a cohort of persons with HIV Model
Death within 1 year Base model* Base model þ Charlson Base model þ Elixhauser Base model þ ADG Hospitalization (1 year) Base model* Base model þ Charlson Base model þ Elixhauser Base model þ ADG Death (between 1 and 2 years) Base model* Base model þ Charlson Base model þ Elixhauser Base model þ ADG *
Derivation sample
Validation sample
Median c-statistic (IQR)
Median change in c-statistic (IQR)
Median Brier score
Median c-statistic (IQR)
Median change in c-statistic (IQR)
Median Brier score
0.683 0.809 0.842 0.846
(0.668e0.692) (0.800e0.825) (0.832e0.852) (0.838e0.858)
d 0.131 (0.115e0.141) 0.162 (0.145e0.172) 0.166 (0.151e0.182)
0.013 0.014 0.013 0.013
0.651 0.788 0.767 0.785
(0.635e0.662) (0.774e0.797) (0.746e0.783) (0.765e0.799)
d 0.138 (0.125e0.153) 0.117 (0.101e0.134) 0.137 (0.115e0.151)
0.014 0.014 0.014 0.014
0.584 0.681 0.700 0.750
(0.576e0.590) (0.674e0.687) (0.693e0.707) (0.743e0.755)
d 0.100 (0.092e0.103) 0.118 (0.110e0.124) 0.265 (0.255e0.278)
0.066 0.062 0.061 0.061
0.569 0.668 0.678 0.727
(0.563e0.576) (0.663e0.677) (0.666e0.685) (0.723e0.735)
d 0.100 (0.094e0.105) 0.108 (0.098e0.112) 0.158 (0.151e0.165)
0.066 0.063 0.063 0.062
0.703 0.758 0.790 0.823
(0.685e0.717) (0.743e0.777) (0.779e0.802) (0.812e0.835)
d 0.060 (0.044e0.073) 0.087 (0.075e0.108) 0.120 (0.105e0.138)
0.012 0.011 0.011 0.011
0.666 0.733 0.658 0.721
(0.649e0.688) (0.713e0.750) (0.630e0.684) (0.703e0.740)
d 0.066 (0.052e0.075) 0.007 (0.038 to 0.015) 0.056 (0.034e0.076)
0.012 0.011 0.012 0.012
Logistic regression model with age, sex, neighborhood income quintile, and years in Ontario HIV database.
discharge abstracts, and may therefore miss important comorbid conditions recorded on outpatient claims. Although some investigators have addressed this problem by using both inpatient and outpatient data to compute the Charlson summary score, this modification has not universally improved the predictive performance of this index [38e40]. Furthermore, both the Charlson and Elixhauser indices were initially derived and validated in hospitalized patients and were developed for use with administrative inpatient data [10,12]. We therefore elected to examine the predictive validity of these indices in a manner that reflects their original derivation and validation. In addition, differences in the performance of the various measures could reflect the use of a grouping algorithm by the Johns Hopkins ACG case-mix software, whereby similar conditions are clustered into ADGs based on physician opinion about the expected duration of the illness, potential for disability and reduced longevity, certainty of diagnosis, etiology, and expected need for specialist care and hospitalization. However, the choice of comorbidity measure depends not only on factors such as validity and choice of outcome but also on practical considerations such as the availability of data or the need for specialized software to create the measure. The latter is a possible limitation to the use of Johns Hopkins ADGs in some settings, whereas the Charlson and Elixhauser methods are nonproprietary and therefore do not require a fee for use. Furthermore, with 32 indicator variables, use of ADGs may result in model overfitting in small data sets with few outcomes. However, the same limitation applies to Elixhauser comorbidities, and it is possible to model ADGs more parsimoniously by collapsing the 32 categories into 12 clinically cogent groupings known as collapsed ADGS. Furthermore, a prior study demonstrated how the 32 ADGs can be collapsed into a single weighted score that reflects the risk of 1-year death in the general adult population [41]. Previous studies examining the predictive validity of different comorbidity measures in persons with HIV have focused on the development and application of measures derived from clinical data [22e27]. However, because many of these indices were developed before the introduction of potent antiretroviral therapies and typically emphasized viral load, CD4þ count and the presence or absence of AIDS-defining illnesses when predicting outcomes in persons with HIV, their applicability to the contemporary era of HIV management may be limited. To address this concern, investigators from the Veterans Aging Cohort Study (VACS) have developed an index that supplements traditional markers of risk in persons with HIV with a variety of routinely monitored laboratory tests and
hepatitis C coinfection status [27]. The discriminative performance of the VACS index has been determined for various outcomes in persons with HIV, with a c-statistic of 0.78e0.81 for all-cause mortality at 5 years among patients followed in the North American AIDS Cohort Collaboration [42]. Moreover, the VACS index demonstrated good concordance between predicted probability of death and observed death, perhaps reflecting the importance of HIV and organ system biomarkers when developing well-calibrated predictive models for persons with HIV. Furthermore, each 5point increment in the VACS index score was associated with a 10% increased risk of hospitalization and intensive care unit admission over a 2-year follow-up period among HIV-infected veterans [43]. Because the VACS index is based on laboratory tests routinely used in the management of persons with HIV, it has the advantage of being readily adaptable for use to the clinical setting. However, alternative methods of risk-adjustment are required when using administrative databases to examine outcomes in persons with HIV, since clinical data are typically not available to investigators who work with these data sets. To the best of our knowledge, ours is the first study comparing the predictive validity of comorbidity measures that can be derived from administrative data in a large population of persons with HIV.
Fig. 1. Calibration plot: comparison of observed and predicted probability of death within 1 year.
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
5
validated algorithm with excellent test characteristics for discriminating between HIV-infected and noninfected individuals to assemble our cohort [32]. Conclusions We found that the ADG, Charlson, and Elixhauser methods were all comparable for predicting 1-year mortality in a population of persons with HIV, and that the ADG method demonstrated superior discriminative performance for hospitalization relative to the other measures in these subjects. However, all methods displayed generally poor calibration for each outcome. Thus, when focusing on hospitalizations, the ADG method may be the preferred method for risk-adjustment in observational studies of inpatient utilization among persons living with HIV, but like other measures, may be unreliable for medical decision making or provider profiling. Further research evaluating the predictive validity of various comorbidity classification methods in persons with HIV is needed. Fig. 2. Calibration plot: comparison of observed and predicted probability of death between 1 and 2 years.
The main strength of our study is the use of comprehensive population-based data, thereby allowing us to evaluate the predictive performance of various comorbidity measures in all persons diagnosed with HIV. However, several limitations of our work merit emphasis. First, we had no access to clinical data and prescription drug records for our population of persons with HIV, rendering it impossible to account for important predictors of death and hospitalization, such as smoking, use of antiretroviral therapy, and the components of the VACS index. In addition, the lack of data regarding prescription drug use precluded an examination of the predictive performance of comorbidity measures, which are derived using these data, including the Chronic Disease Score and the number of unique drugs prescribed [40,44]. We used years in the Ontario HIV database as a proxy for the number of years diagnosed with HIV, which underestimates this variable for people diagnosed before 1991. To prevent model overfitting, we did not test for interactions between candidate predictors in our analyses. Finally, the potential for misclassification is always a consideration when using administrative data. To address this concern, we used a
Acknowledgment Funding: This project was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care. The sponsors had no role in the design or conduct of the study; in the collection, analysis or interpretation of the data; or in the preparation, review or approval of the manuscript. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding source. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. T.A. is supported by a new investigator award from the Canadian Institutes of Health Research and the Ontario HIV Treatment Network. R.H.G. is a Clinician Scientist in the Department of Family and Community Medicine at the University of Toronto and St. Michael’s Hospital. P.C.A. is supported by a Career Investigator award from the Heart and Stroke Foundation. Contributors: All authors contributed to the concept and design of the study. T.A., R.N., and A.K. acquired the data, and all authors were involved in the analysis and interpretation of the data. T.A. drafted the manuscript, and all authors were involved in critical revision of the manuscript. All authors approved the manuscript submitted for publication. T.A. is the guarantor for the article. References
Fig. 3. Calibration plot: comparison of observed and predicted probability of hospitalization within 1 year.
[1] Antiretroviral Therapy Cohort Collaboration, Zwahlen M, Harris R, May M, Hogg R, Costagliola D, et al. Mortality of HIV-infected patients starting potent antiretroviral therapy: comparison with the general population in nine industrialized countries. Int J Epidemiol 2009;38:1624e33. [2] Hasse B, Ledergerber B, Furrer H, Battegay M, Hirschel B, Cavassini M, et al. Morbidity and aging in HIV-infected persons: the Swiss HIV Cohort Study. Clin Infect Dis 2011;53:1130e9. [3] Van Sighem AI, Gras LA, Reiss P, Brinkman K, de Wolf F. ATHENA National Observational Cohort Study. Life expectancy of recently diagnosed asymptomatic HIV-infected patients approaches that of uninfected individuals. AIDS 2010;24:1527e35. [4] Berry SA, Fleishman JA, Moore RD, Gebo KA. Trends in reasons for hospitalizations in a multisite United States Cohort of Persons Living with HIV, 20012008. J Acquir Immune Defic Syndr 2012;59:368e75. [5] Goulet JL, Fultz SL, Rimland D, Butt A, Gibert C, Rodriguez-Barradas M, et al. Aging and infectious diseases: do patterns of comorbidity vary by HIV status, age, and HIV severity? Clin Infect Dis 2007;45:1593e601. [6] Antoniou T, Zagorski B, Loutfy MR, Strike C, Glazier RH. Socio-economic and sex-related disparities in rates of hospital admission among patients with HIV infection in Ontario: a population-based study. Open Med 2012;6:e146e54. [7] Fultz SL, Skanderson M, Mole LA, Gandhi N, Bryant K, Crystal S, et al. Development and verification of a “virtual” cohort using the national VA health information system. Med Care 2006;44:S25e30. [8] Walkup JT, Wei W, Sambamoorthi U, Crystal S. Sensitivity of an AIDS casefinding algorithm: who are we missing? Med Care 2004;42:756e63.
6
T. Antoniou et al. / Annals of Epidemiology xxx (2014) 1e6
[9] Justice AC, Landefeld CS, Asch SM, Gifford AL, Whalen CC, Covinsky KE. Justification for a new cohort study of people aging with and without HIV infection. J Clin Epidemiol 2001;54(Suppl 1):S3e8. [10] Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373e83. [11] Deyo RA, Cherken DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613e9. [12] Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8e27. [13] Stukenborg GJ, Wagner DP, Connors AF. Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations. Med Care 2001;39:727e39. [14] Quan H, Sundararajan V, Halfon P, Fong A, Burnard B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005;43:1130e9. [15] Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol 2004;57:1288e94. [16] Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/ Deyo methods of comorbidity measurement in administrative data. Med Care 2004;42:355e60. [17] Quail JM, Lix LM, Osman BA, Teare GF. Comparing comorbidity measures for predicting mortality and hospitalization in three population-based cohorts. BMC Health Servi Res 2011;11:146. [18] Johns Hopkins University. Johns Hopkins ACG Case-Mix Adjustment System. Available at: https://acg.jhsph.org. [19] Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management. Health Serv Res 1991;26:53e74. [20] Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory case case-mix. Med Care 1991;29:452e72. [21] Sibley LM, Moineddin R, Agha MM, Glazier RH. Risk adjustment using administrative data-based and survey-derived methods for explaining physician utilization. Med Care 2010;48:175e82. [22] Turner BJ, Kelly JV, Ball JK. A severity classification system for AIDS hospitalizations. Med Care 1989;27:423e37. [23] Justice AC, Feinstein AR, Wells CK. A new prognostic staging system for the acquired immunodeficiency syndrome. N Engl J Med 1989;320:1388e93. [24] Egger M, May M, Chêne G, Phillips AN, Ledergerber B, Dabis F, et al. Prognosis of HIV-1 infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet 2002;360:119e29. [25] Mocroft A, Kirk O, Barton SE, Dietrich M, Proenca R, Colebunders R, et al. Anaemia is an independent predictive marker for clinical prognosis in HIVinfected patients from across Europe. EuroSIDA study group. AIDS 1999;13:943e50. [26] Lundgren JD, Mocroft A, Gatell JM, Ledergerber B, D’Arminio Monforte A, Hermans P, et al. A clinically prognostic scoring system for patients receiving highly active antiretroviral therapy: results from the EuroSIDA study. J Infect Dis 2002;185:178e87. [27] Justice AC, McGinnis KA, Skanderson M, Chang CC, Gibert CL, Goetz MB, et al. Towards a combined prognostic index for survival in HIV infection: the role of ‘non-HIV’ biomarkers. HIV Med 2009;11:143e51.
[28] Austin PC, Shah BR, Newman A, Anderson GM. Using the Johns Hopkins’ Aggregated Diagnosis Groups (ADGs) to predict 1-year mortality in population-based cohorts of patients with diabetes in Ontario, Canada. Diabet Med 2012;29:1134e41. [29] Austin PC, van Walraven C, Wodchis WP, Newman A, Anderson GM. Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a general adult population cohort in Ontario, Canada. Med Care 2011;49:932e9. [30] Austin PC, Newman A, Kurdyak PM. Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a population-based cohort of subjects with schizophrenia in Ontario, Canada. Psychiatry Res 2012;196:32e7. [31] Austin PC, Stanbrook MB, Anderson GM, Newman A, Gershon AS. Comparative ability of comorbidity classification methods for administrative data to predict outcomes in patients with chronic obstructive pulmonary disease. Ann Epidemiol 2012;22:881e7. [32] Antoniou T, Zagorski B, Loutfy MR, Strike C, Glazier RH. Validation of casefinding algorithms derived from administrative data for identifying adults living with human immunodeficiency virus infection. PLoS One 2011;6:e21748. [33] Perkins AJ, Kroenke K, Unützer J, Katon W, Williams Jr JW, Hope C, et al. Common comorbidity scales were similar in their ability to predict health care costs and mortality. J Clin Epidemiol 2004;57:1040e8. [34] Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128e38. [35] Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York, NY: John Wiley & Sons, Inc; 2000. [36] Austin PC, Steryerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014;33:517e35. [37] Steyerberg EW. Clinical prediction models. A practical approach to development, validation, and updating. New York, NY: Springer; 2009. [38] Klabunde CN, Potosky AL, Legler JM, Warren JL. Development of a comorbidity index using physician claims data. J Clin Epidemiol 2000;53:1258e67. [39] Fan VS, Maciejewski ML, Liu CF, McDonell MB, Fihn SD. Comparison of risk adjustment measures based on self-report, administrative data, and pharmacy records to predict clinical outcomes. Health Serv Outcomes Res Method 2006;6:21e36. [40] Schneeweiss S, Seeger JD, Maclure M, Wang PS, Avorn J, Glynn RJ. Performance of comorbidity scores to control for confounding in epidemiologic studies using claims data. Am J Epidemiol 2001;154(9):854e64. [41] Austin PC, Walraven CV. The mortality risk score and the ADG score: two points-based scoring systems for the Johns Hopkins aggregated diagnosis groups to predict mortality in a general adult population cohort in Ontario. Can Med Care 2011;49:940e7. [42] Justice AC, Modur SP, Tate JP, Althoff KN, Jacobson LP, Gebo KA, et al. Predictive accuracy of the Veterans Aging Cohort Study Index for mortality with HIV infection: a North American cross cohort analysis. J Acquir Immune Defic Syndr 2013;62:149e63. [43] Akgün KM, Gordon K, Pisani M, Fired T, McGinnis KA, Tate JP, et al. Risk factors for hospitalization and medical intensive care unit (MICU) admission among HIV-infected veterans. J Acquir Immune Defic Syndr 2013;62:52e9. [44] Von Korff M, Wagner EH, Sunders K. A chronic disease score from automated pharmacy data. J Clin Epidemiol 1992;45:197e203.
