pharmacoepidemiology and drug safety 2015; 24: 943–950 Published online 24 June 2015 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3808
ORIGINAL REPORT
Diagnostic accuracy of algorithms to identify hepatitis C status, AIDS status, alcohol consumption and illicit drug use among patients living with HIV in an administrative healthcare database† Madeleine Durand1*, Yishu Wang2, François Venne3, Jacques Lelorier3, Cécile L. Tremblay4 and Michal Abrahamowicz2 1
Department of Department of 3 Department of 4 Department of 2
Internal Medicine, Centre Hospitalier de l’Unvisersité de Montréal, Montréal, Canada Epidemiology and Biostatistics, McGill University, Montréal, Canada Medicine, Université de Montréal, Montréal, Canada Microbiology, CHUM Research Center, Montréal, Canada
ABSTRACT Purpose This study aims to develop and evaluate diagnostic algorithms for AIDS, hepatitis C status, alcohol abuse and illicit drug use in the administrative healthcare database of the Province of Quebec, Canada (Régie de l’assurance-maladie du Québec (RAMQ)). Methods We selected HIV-positive patients contributing to both the RAMQ database and a local clinical database, which was used as gold standard. We developed algorithms to identify the diagnoses of interest in RAMQ using data from hospital discharge summaries and medical and pharmaceutical claims databases. We estimated and compared sensitivity, specificity, positive predictive and negative predictive values and area under receiver operating curve for each algorithm. Results Four hundred twenty patients contributed to both databases. Prevalence of conditions of interest in the clinical database was as follows: AIDS 233 (55%), hepatitis C infection 105 (25%), alcohol abuse 106 (25%), illicit drug use 144 (34%) and intravenous drug use 107 (25%). Sensitivity to detect AIDS, hepatitis C, alcohol abuse, illicit drug use and intravenous drug use was 46% [95%CI: 39–53], 26% [18–35], 50% [37–57], 64% [55–72] and 70% [61–79], respectively. Specificity to detect these conditions was 91% [86–95], 97% [94–98], 92% [88–95], 95% [92–97] and 90% [87–93], respectively. Positive predictive values were 87% [80–92], 71% [54–85], 68% [56–78], 87% [79–93] and 72% [62–80], respectively. Area under receiver operating curve varied from 0.62 [0.57–0.65] for hepatitis C to 0.80 [0.76–0.85] for intravenous drug use. Conclusions Sensitivity was low to detect AIDS, alcohol abuse, illicit drug use and especially hepatitis C in RAMQ. Researchers must be aware of the potential for residual confounding and must consider additional methods to control for confounding. Copyright © 2015 John Wiley & Sons, Ltd. key words—HIV AIDS; epidemiology; RAMQ database; administrative healthcare database; diagnostic accuracy; sensitivity; specificity; pharmacoepidemiology Received 5 March 2015; Revised 1 May 2015; Accepted 2 May 2015
INTRODUCTION The clinical picture of human immunodeficiency virus (HIV) infection has changed greatly since the *Correspondence to: M. Durand, CHUM Research Center, 850, rue Saint-Denis, Montreal, Québec, H2X 0A9, Canada. E-mail: madeleine.durand.chum@ssss. gouv.qc.ca † Prior presentations: This research was conducted at the centre de recherche de l’UHRESS du CHUM and at the Pharmacoeconomy and pharmacoepidemiology research unit of the CHUM, Montréal, Canada. Results have been presented at the Canadian association for AIDS research annual meeting in St-John, Newfoundland, Canada in April 2013 (poster presentation) and at the International Conference of Pharmacoepidemiology in Taipei, Taiwan in October 2014 (oral poster presentation).
Copyright © 2015 John Wiley & Sons, Ltd.
introduction of highly active antiretroviral therapy (HAART), and prolonged survival due to HAART has led to an increase in incidence of chronic, noninfectious complications such as cardiovascular and metabolic diseases.1–5 This shift in disease burden requires a paradigm shift in research methodology, with more emphasis on long-term studies of chronic, complex and rarer diseases. The use of administrative healthcare data is appealing, in order to study large population of patients living with HIV. However, some important variables for the study of HIV and its comorbidities have not yet been validated in administrative healthcare databases, so the sensitivity,
944
m. durand et al.
specificity and positive and negative predictive values of coding algorithms are unknown. In a prior work, we showed that AIDS status, hepatitis C status, alcohol and illicit drug use were important confounders of the relationship between HIV infection and important health outcomes.6,7 Yet, without a precise estimate of diagnostic accuracy of algorithms used to identify these variables in administrative healthcare databases, it is difficult to assess whether, and to what extent, adjustment for possibly inaccurate measurements of these variables leads to residual confounding.8 The Régie de l’assurance maladie du Québec (RAMQ) database is the administrative healthcare database of the province of Québec, Canada. It contains routinely collected data on medical and pharmacological care for the population of Québec, Canada, in the setting of a universal healthcare insurance coverage. We aimed to develop and validate algorithms in the RAMQ to diagnose AIDS, hepatitis C, alcohol and illicit drug use in a cohort of HIV-infected individuals. We then aimed to compare the test characteristics (sensitivity, specificity, positive predictive value and negative predictive value), as well as the area under the receiver operating curves (AUCs), of the alternative algorithms, to select the best algorithm to be used in future studies. METHODS Study design and data sources Study design. This is a diagnostic accuracy study, performed using secondary analyses of linked data from two prospective cohort studies. It was conducted and is reported in accordance with the Statement for Reporting Studies of Diagnostic Accuracy (STARD) consensus.9
CHUM HIV database. All HIV-positive patients followed at the Centre Hospitalier de l’Université de Montréal (CHUM) from 1985 until present contribute to a clinical database. The CHUM is a tertiary, urban reference center with a dedicated HIV clinic. Comprehensive data are available on sociodemographic factors, HIV risk factors, habits, CD4 counts, viral load, treatments, vaccination status, opportunistic infections and comorbidities. A standardized chart is used in the follow-up of HIV-positive patients at the CHUM, which ensures completeness and coherence of collected data. This database is updated every time a patient has a medical encounter and is linkable to the hospital medical file.10 In our analyses, the CHUM-HIV database was considered as the gold standard for establishing the diagnoses of interest, which were coded Copyright © 2015 John Wiley & Sons, Ltd.
as present or absent. Hence, the gold standard diagnoses were based on medical charts, completed by HIV specialists in a dedicated clinic. No additional independent arbitration was performed.
RAMQ database. The RAMQ administrative database provides information generated by medical fee-forservice billing for all residents of the province of Québec, with ICD-9 codes for each service. For publicly insured Quebec residents, it also contains pharmaceutical information, including all dispensed prescriptions. This represents 43% of Québec’s population, including everyone over 65 years of age, people on social assistance and subscribers to the public drug insurance plan. RAMQ database also contains the discharge summary for all acute care hospitalizations for all Quebec residents, with ICD-9 and ICD-10 codes for all discharge diagnoses. Information is available from 1987 to present. Several validation studies have been conducted using RAMQ database,11–19 and these data have been used in numerous peer-reviewed publications, including studies of HIV.6,7
Study population Identification of HIV in RAMQ. We identified the cohort of all persons living with HIV within the RAMQ database using ICD-9 codes 042–044 and Québecspecific ICD-9 code 7958. The HIV+ cohort included all subjects who had at least one relevant code between 1 January 1988 and 31 December 2007. The date of the first HIV diagnosis was defined as entry into the cohort, and patients were followed until death, end of drug insurance coverage, or 31 December 2007. Patients had to have at least one year of public drug insurance coverage prior to their cohort entry. This cohort has been described elsewhere.6
Definition of the validation population. Patients contributing data to both CHUM HIV and RAMQ databases were identified using a unique anonymized identifier. Only overlapping periods of follow-up from both databases were included. Patients were excluded if (i) there was no overlap in follow-up periods, (ii) they were followed in another out-patient clinic for routine follow-up and only came to the CHUM for acute hospitalizations, (c) they had less than two visits recorded in the CHUM-HIV database during RAMQ follow-up or (d) they were deceased before year Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
945
validation of administrative healthcare database for study of hiv
2000. Data were collected in both databases prior to enrollment in this study.
Table 1. database
Algorithms used to define conditions of interest in RAMQ
Condition of interest
Definition of diagnoses in CHUM HIV database. AIDS-defining diagnoses, as well as hepatitis C status, alcohol and illicit drug use, are systematically recorded in the database and were retrieved for participating patients. In case of missing data, queries were sent to treating physicians, and hospital charts were retrieved and reviewed. AIDS-defining illnesses were defined as per the list of the Center for Disease Control and Prevention.20 CD4 thresholds were not used to diagnose AIDS, as this information cannot be retrieved in RAMQ. Alcohol abuse was defined as consumption of more than 9 alcohol units per week for women and more than 14 alcohol units per week for men, as per Health Canada’s recommendations.21 Illicit drug use was defined as consumption of any illicit drug (excluding inhaled or oral marijuana derivatives), and intravenous drug administration was recorded when present. Both alcohol abuse and illicit drug use were time-fixed binary variables, coded as present if they happened at any point during follow-up.
Diagnoses in RAMQ. For each diagnosis, four different algorithms were developed and their test characteristics were assessed using CHUM HIV cohort data as a “gold standard”. This was carried out in order to identify the best possible algorithm to diagnose the conditions of interest in RAMQ. Algorithms used outpatients diagnostic claims data (ICD-9 codes), inpatient diagnostic codes data (ICD-9 and 10 codes) and/or pharmaceutical data (using common drug denomination codes). For each diagnosis of interest, the best algorithm was selected by comparing the AUCs. Table 1 describes the different algorithms used to diagnose each condition, and Supplementary Appendix 1 lists the ICD-9, ICD10 and medication codes used in algorithms.
Variation of test characteristics with follow-up time. Reporting of conditions in RAMQ is passive, in the sense that it only happens at times of medical encounters. Therefore, it is probable that test characteristics vary with length of follow-up in RAMQ. To evaluate this, we stratified the validation population (defined above) into mutually exclusive periods of follow-up, using terciles of the follow-up time distribution as the cut-off points (less than 4 years, 4–8 years and Copyright © 2015 John Wiley & Sons, Ltd.
AIDS
Algorithm
Description
1
Presence of at least 1 inpatient code for AIDS defining condition Presence of at least 1 outpatient code for AIDS defining condition Presence of a least 2 outpatient code for AIDS defining condition Presence of either 1 inpatient OR 1 outpatient code for AIDS defining condition Presence of at least 1 inpatient code for HCV Presence of at least 1 outpatient code for HCV Dispensation of medication specific to treat HCV Presence of either 1 inpatient, 1 outpatient or 1 dispensation of medication specific for HCV Presence of at least 1 inpatient code for alcohol-related health problem Presence of at least 1 outpatient code for alcohol-related health problem Presence of at least 2 outpatient code for alcohol-related health problem Presence of either 1 inpatient OR 1 outpatient code for alcohol related health problem Presence of at least 1 inpatient code for illicit drug use-related health problem Presence of at least 1 outpatient code for illicit drug use-related health problem Dispensation of methadone Presence of at least 1 inpatient or outpatient code for illicit drug use-related health problem, or dispensation of methadone
2 3 4 Hepatitis C
1 2 3 4
Alcohol Abuse
1 2 3 4
Illicit drug use and intravenous drug use
1 2 3 4
Most discriminative algorithms (defined as highest area under curve) are in boldface.
more than 8 years) and re-tested the diagnostic algorithms separately in the three resulting strata.
Statistical analysis Appropriate descriptive statistics were used to describe the population’s sociodemographic characteristics. For each diagnosis algorithm tested, sensitivity was estimated as the proportion of true positives, that is, patients correctly identified by the algorithm with the condition of interest in the RAMQ database, among all those identified as having the condition in the CHUM HIV cohort (gold standard). (This corresponds to the ratio of true positives over the sum of true positives + false negatives.) Specificity was estimated as the proportion of true negatives, that is, patients correctly identified without the condition by the algorithms in the RAMQ, among all those who did not have the condition of interest in the Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
946
m. durand et al.
CHUM HIV cohort (gold standard). This corresponds to the ratio of true negatives over the sum of true negatives + false positives. AUC for a binary diagnostic test was estimated as (sensitivity + specificity)/2.22 Confidence intervals for AUC were obtained using the DeLong nonparametric method.23 Positive predictive value (PPV) was measured as the ratio of true positive (positive in both RAMQ and CHUM HIV) over total number of positives identified by the algorithm in RAMQ (true positives + false positives). Negative predictive value (NPV) was measured as the ratio of true negatives (negative both in RAMQ and CHUM HIV) over total number of negatives in RAMQ (true negatives + false negatives). Exact 95% confidence intervals for proportions were computed for all aforementioned test characteristics. A one degree of freedom chi-square test for trend was used to assess, in separate analyses, whether sensitivity and specificity varied across the three subpopulations corresponding to increasing duration of follow-up in RAMQ. Statistical analyses were performed with SAS version 9.224 and Stata version 11.25 RESULTS Description of validation population There were 1476 persons living with HIV in the CHUM HIV database, and 8057 HIV-positive patients in the RAMQ database, among whom 493 (6.2%) patients were matched across the RAMQ and CHUM HIV databases using the unique identifier. Seventythree (15%) of the matched patients were excluded because (i) their follow-up periods in the two databases did not overlap (n = 25), (ii) they had most of their clinical follow-up in another center (n = 27), (iii) they had less than two visits recorded in the CHUM HIV database during their RAMQ follow-up (n = 16) or (iv) they died before year 2000 (n = 5). The remaining 420 (85%) patients were included in the validation sample. Mean age of participants was 52.8 (SD 9.2) years. Eighty-two patients (19.5%) died during follow-up. Of the patients, 318 (75.7%) were male. Ethnic origin was Caucasian for 281 (66.9%) participants, African, African-American or Haitian in 94 (22.4%), and others in 45 (10.7%). During the study period from May 1988 to December 2007, the median overlap in follow-up time in both databases was 6.2 years (interquartile range 3.0–9.3 years.) The dates of HIV diagnoses ranged from 1983 to 2007 but were unknown for 90 (21%) patients. Stages of HIV disease, at the beginning of follow-up, according to the classification of the Center for Disease Control were A (mildly or asymptomatic) in 112 (27%), B (moderately symptomatic) in 117 (28%) and C (severely symptomatic) in 191 (45%). Copyright © 2015 John Wiley & Sons, Ltd.
Prevalence, at any time during follow-up (with exact 95% confidence intervals) of the conditions of interest in the CHUM HIV dataset (“gold standard”), was as follows: 233 (55% [51–60%]) for AIDS, 105 (25% [21–29%]) for hepatitis C infection, 106 (25% [21– 30%]) for alcohol abuse, 144 (34% [30–39%]) for illicit drug use and 107 (25% [21–30%]) for intravenous drug use. Test characteristics of predefined algorithms. Table 2 details the test characteristics, with 95% confidence intervals, for each of the pre-specified algorithms, along with the AUC for each algorithm. (AUC can be interpreted as the overall discrimination capacity of the algorithm: it combines sensitivity and specificity in one figure.) For all algorithms except for AIDS status, the highest AUC is obtained with the most inclusive algorithm (last column of Table 2), which uses combinations of outpatient, inpatient and, where applicable, pharmacological information (Table 1). To determine AIDS status, however, the inclusion in the algorithm of outpatient codes lowered the specificity with an insufficient increase in sensitivity, so algorithm 1, which relies on inpatient codes only, offers the best sensitivity/specificity trade-off (Table 2). Overall, specificity was high: above 90% for all algorithms. Sensitivity to detect the conditions of interest in the RAMQ administrative database, on the other hand, varied widely across algorithms and was generally low. It was very low even for the most discriminative algorithms for hepatitis C (26%) and low for both AIDS (46%) and alcohol abuse (50%), but substantially higher for intravenous drug use (70%). AUC were low for most diagnoses: 0.69 95%CI [0.65–0.72] for AIDS, 0.62 [0.57–0.65] for hepatitis C, and 0.71 [0.66–0.76] for alcohol abuse. Only the algorithms to identify illicit drug use and intravenous drug use reached values of 0.80 for AUC (0.80 [0.76–0.85] for intravenous drug use). Positive and Negative Predictive Values: PPVs were very high for AIDS and for illicit drug use (87% for both diagnoses) and moderately high for hepatitis C (71%), for alcohol abuse (68%) and for intravenous drug use (72%). NPVs were also high for hepatitis C (80%), alcohol abuse (85%), illicit drug use (83%) and intravenous drug use (90%) but lower for AIDS (57%). This much lower NPV for AIDS is explained by the higher prevalence of this condition in our population (55%). Table 2 gives PPV and NPV for all tested algorithms, along with 95% confidence intervals. Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
947
validation of administrative healthcare database for study of hiv
Table 2. Prevalence of conditions in CHUM’s clinical database and according to various algorithms in RAMQ, with tests characteristics and area under ROC (for 420 patients)
AIDS CHUM Algo 1 Algo 2 Algo 3 Algo 4 Hepatitis C CHUM Algo 1 Algo 2 Algo 3 Algo 4 Alcohol abuse CHUM Algo 1 Algo 2 Algo 3 Algo 4 Illicit drug use CHUM Algo 1 Algo 2 Algo 3 Algo 4 IV drug use CHUM Algo 1 Algo 2 Algo 3 Algo 4
Prevalence n (%)
Sensitivity [95%CI]
Specificity [95%CI]
Positive predictive value [95%CI]
Negative predictive Value [95%CI]
Area under ROC [95%CI]
233 (55) 123 (30) 64 (15) 31 (7) 148 (35)
Ref 46% [39–53] 21% [16–27] 12% [8–16] 50% [44–57]
Ref 91% [86–95] 92% [87–95] 98% [95–99] 83% [77–88]
Ref 87% [80–92] 77% [64–86] 87% [70–96] 79% [72–85]
Ref 57% [52–63] 48% [43–54] 47% [42–52] 57% [51–63]
0.69 [0.65–0.72] 0.57 [0.53–0.60] 0.55 [0.52–0.57] 0.67 [0.63–0.71]
105 (25) 30 (7) 17 (4) 6 (1) 38 (9)
Ref 22% [14–31] 11% [6–19] 5% [2–11] 26% [18–35]
Ref 98% [95–99] 98% [96–99] 100% [99–100]* 97% [94–98]
Ref 77% [58–90] 71% [44–90] 83% [34–100] 71% [54–85]
Ref 79% [75–83] 77% [72–81] 76% [71–80] 80% [75–84]
0.60 [0.56–0.64] 0.55 [0.52–0.58] 0.53 [0.50–0.54] 0.62 [0.57–0.65]
106 (25) 58 (13) 59 (14) 32 (8) 78 (19)
Ref 43% [34–53] 38% [29–48] 26% [18–36] 50% [37–57]
Ref 96% [93–98] 94% [91–96] 99% [96–100] 92% [88–95]
Ref 79% [67–89] 68% [54–79] 88% [71–96] 68% [56–78]
Ref 83% [79–87] 82% [77–86] 80% [76–84] 85% [80–88]
0.70 [0.65–0.75] 0.66 [0.61–0.71] 0.63 [0.58–0.67] 0.71 [0.66–0.76]
144 (34) 80 (19) 78 (19) 1 (0.2) 106 (25)
Ref 52% [44–60] 47% [39–56] 1% [0–4] 64% [55–72]
Ref 98% [96–99] 96% [93–98] 100% [99–100]* 95% [92–97]
Ref 94% [86–98] 87% [78–94] 100% [5–100]* 87% [79–93]
Ref 80% [75–84] 78% [73–82] 66% [61–70] 83% [79–87]
0.75 [0.71–0.79] 0.72 [0.68–0.76] 0.51 [0.50–0.51] 0.80 [0.75–0.83]
108 (26) 80 (19) 78 (19) 1 (0.2) 106 (25)
Ref 57% [48–67] 53% [43–62] 1% [0–5] 70% [61–79]
Ref 94% [91–97] 93% [90–96] 100% [99–100]* 90% [87–93]
Ref 78% [67–86] 73% [62–82] 100% [5–100]* 72% [62–80]
Ref 86% [82–90] 85% [81–79] 74% [70–79] 90% [86–93]
0.76 [0.71–0.81] 0.73 [0.68–0.78] 0.51 [0.50–0.51] 0.80 [0.76–0.85]
Most discriminative algorithms (defined as highest area under curve) are in boldface. *One-sided 95%CI.
Table 3. Evolution of sensitivity/specificity with duration of follow-up, for best diagnostic algorithm
AIDS 7 Sensitivity Specificity AUC Hepatitis C Sensitivity Specificity AUC Alcohol abuse Sensitivity Specificity AUC Illicit drug use Sensitivity Specificity AUC IV drug use Sensitivity Specificity AUC
Less than 4 years of follow-up (n = 138)
4 to 8 years of follow-up (n = 148)
More than 8 years of follow-up (n = 134)
1 df p-value for trend
38 [27–50] 92 [83–97] 0.65 [0.59–0.72]
42 [31–55] 95 [87–98] 0.69 [0.62–0.75]
55 [44–66] 85 [72–94] 0.70 [0.63–0.77]
0.08 0.18
13 [4–27] 99 [94–100] 0.56 [0.50–0.61]
31 [16–50] 97 [93–99] 0.64 [0.56–0.73]
36 [20–55] 93 [86–97] 0.65 [0.56–0.73]
0.01 0.02
31 [17–49] 93 [86–97] 0.62 [0.54–0.70]
61 [42–78] 93 [87–97] 0.77 [0.68–0.86]
57 [41–73] 89 [81–95] 0.73 [0.65–0.82]
0.03 0.32
54 [39–68] 99 [94–100] 0.77 [0.69–0.83]
62 [46–76] 95 [89–98] 0.79 [0.71–0.86]
76 [62–87] 90 [82–96] 0.83 [0.76–0.90]
0.02 0.01
60 [43–75] 95 [88–98] 0.78 [0.69–0.85]
71 [53–85] 94 [88–97] 0.83 [0.74–0.90]
82 [65–93] 82 [73–89] 0.82 [0.75–0.90]
0.04 0.002
AUC, area under receiver operating curve. Numbers in square brackets are 95% confidence intervals.
Copyright © 2015 John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
948
m. durand et al.
Variation of test characteristics with increasing duration of follow-up. Table 3 shows the variation of test characteristics for the optimal algorithm across the three subgroups with gradually increasing follow-up duration. As expected, for all diagnoses, there are statistically significant improvements in sensitivity with increased follow-up time (p < 0.05 for all tests for trend). For some conditions (hepatitis C, illicit drug use and intravenous drug use), this is associated with a significant decrease in specificity. AUC generally increased with increasing follow-up time, suggesting that overall performance of the proposed diagnostic algorithms in the RAMQ database improves when longer follow-up periods are available. However, even with longer follow-up, AUC remained below 0.80 for AIDS, hepatitis C infection and alcohol abuse. DISCUSSION In this validation study of an administrative healthcare database, we estimated the test characteristics for prespecified diagnostic algorithms for conditions of particular interest to the study of HIV/AIDS, by using a clinical database as a “gold standard”. We found overall low sensitivity of diagnostic algorithms to identify AIDS, alcohol abuse and illicit drug use, and especially hepatitis C infection (46%, 50%, 64% and 26%, respectively). Specificity to identify the same conditions was relatively high (83%, 92%, 90% and 97%, respectively). We also showed that, as expected, test characteristics vary with follow-up time in the administrative database, with marked increases in sensitivity and relatively insignificant losses in specificity with longer follow-up times. Our results are consistent with previous validation studies in administrative healthcare databases. For example, Landry et al. found a sensitivity of 45–52% and specificity of 98% to diagnose broncho-pulmonary dysplasia (a chronic respiratory disease) in pre-term infants.15 In a systematic review, Chung et al. present nine studies reporting test characteristics for algorithms to identify rheumatoid arthritis and report sensitivities from 51% to 65% in general administrative healthcare databases comparable with RAMQ.26 Therefore, we believe the sensitivities found to detect AIDS, alcohol abuse and illicit drug use are comparable with those found in this type of data for other chronic conditions. The sensitivity to detect hepatitis C, however, is markedly lower. In the general population, it is estimated that a significant proportion of Hepatitis C infections are undiagnosed, although a precise estimate is not available.27,28 In our “gold standard” population, patients undergo routine screening for hepatitis C Copyright © 2015 John Wiley & Sons, Ltd.
infection, so their treating physicians are aware of their diagnosis. The low detection of this comorbidity in RAMQ, in spite of the patient’s physicians being aware of it, is not surprising. First, most patients chronically infected with hepatitis C are asymptomatic. Second, RAMQ coding for outpatient visits only allows one diagnostic code to be used per medical visit. Therefore, in patients co-infected with HIV, it is likely that the treating physicians will report HIV infection rather than hepatitis C infection. In the future, as less toxic and more accessible treatments for hepatitis C infection become available and treatment rates for hepatitis C increase, it is likely that its capture will be improved in administrative healthcare databases with available pharmacological data, such as RAMQ. Our work has important implications for the use of administrative databases in the study of HIV. Indeed, as persons living with HIV live longer, this condition has become a chronic disease in developed countries.1 Large databases have become an important tool for studies of long-term effects and toxicities of medications and emergence of rare conditions.29–33 Clinical trials or smaller cohorts are often ill-suited to study such outcomes. Our study assessed the accuracy of two types of variables relevant for such investigations: (i) AIDS status is important to establish in order to situate patients within the spectrum of HIV infection, while (ii) hepatitis C, alcohol abuse and illicit drug will often be considered as potential confounders of the associations between HIV and health outcomes. HIV infection has a very broad clinical spectrum, ranging from asymptomatic to profound immunosuppression, which must be taken into account. In the absence of data on viral load and CD4 lymphocytes values in administrative healthcare databases, establishing AIDS status is important and challenging. We have shown in prior work7 that an association between HIV infection and intracranial hemorrhage exists in the entire population of persons living with HIV. However, when stratified according to AIDS status, the association disappears in non-AIDS population. Failure to take the AIDS status into account leads to biases in pharmacoepidemiological studies. For example, if a new drug is given preferentially to AIDS patients, because it is believed to be more potent, then a kind of indication bias (bias by severity, or channeling bias) will occur.27,28 Second, according to published studies, alcohol abuse34,35 is more prevalent in persons living with HIV than in the general population. Illicit drug use is a recognized risk factor for HIV infection,36 and hepatitis C infection is a frequent co-infection in persons living with HIV.37 Each of these three health Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
validation of administrative healthcare database for study of hiv
conditions is associated with numerous health outcomes. As such, they are classical confounders of the associations between HIV and various health outcomes and must be adjusted for. This work shows that studies about HIV performed in the RAMQ database are vulnerable to residual confounding because of misclassification of those variables. Further studies are necessary to assess if, and to what extent, our findings apply to other administrative databases. This study has several limitations. First, the prevalence of the conditions of interest in our population (people living with HIV) differs from that of the general population, so results might not be extrapolated when studying HIV-negative controls. Not only is prevalence of conditions different in HIV-negative population but physicians’ interest in those conditions differs, arguably affecting diagnosis and reporting. Our results can also be affected by spectrum bias: sensitivity for a condition can be higher if the condition is more severe.9,38,39 This can be especially true for the diagnoses of AIDS, as the natural course of HIV infection has greatly changed in the period covered by our cohort. AIDS manifestations were more likely to be present in the earlier years of the epidemic and were more likely to be sustained and severe. Sensitivity might be lower to detect AIDS status in the highly active antiretroviral therapy era. Finally, the gold standard used in our study is a clinical database, which can be imprecise because of less than 100% sensitivity and specificity in clinical characteristics detection by clinicians. Strengths of this study include its large validation sample and the review of both the clinical database and patient’s charts. It emphasizes that conditions such as alcohol abuse and illicit drug use can be captured (although imperfectly) by administrative healthcare data, with test characteristics comparable with those of chronic medical conditions. We also found very high PPV for some conditions of interest (87% for AIDS and intravenous drug use) and high PPV for others (71% for hepatitis C and 68% for alcohol abuse). Patients identified by the algorithms therefore have a high probability of truly having the condition. Analysis of the effect of follow-up time on test characteristics also adds an interesting perspective to analysis of routinely collected healthcare data. Restricting study cohorts to patients with longer follow-up would introduce bias by truncating the sicker patients, who are more likely to die before completing longer follow-up. On the other hand, researchers must recognize that diagnostic algorithms perform better for patients with longer follow-up. Copyright © 2015 John Wiley & Sons, Ltd.
949
Hence, there will be greater misclassification of covariates in patients with shorter follow-up, which in turn will increase the residual confounding for these patients. Furthermore, if the exposure of interest is associated with follow-up length, then the degree of misclassification will be differential according to the exposure status and might bias the effect estimate in any direction. Relying on time-dependent, as opposed to time-fixed, diagnostic algorithms for covariates would address this concern. Overall, this study shows that the RAMQ database is limited to capture AIDS status, hepatitis C infection, alcohol abuse and illicit drug use. Sensitivity tends to be low, especially for hepatitis C, but specificity, as expected, is high. Prior work showed that the knowledge of test characteristics, derived from a sub-sample of a larger cohort, coupled with simulation, can reduce bias in large administrative healthcare databases.40 Conclusions. This work establishes test characteristics for key variables for the study of HIV/AIDS using Québec’s administrative healthcare database. Our findings, which test characteristics of algorithms to detect important variables are insufficient to adequately control for their confounding effects, should lead researchers to develop and explore novel methods to adjust for residual confounding when using administrative data for health research purposes.
CONFLICTS OF INTEREST Y. Wang, F. Venne and M. Abrahamowicz have no conflict of interest to disclose. M. Abrahamowicz is a James McGill Professor of Biostatistics at McGill University. J. Lelorier and C. L. Tremblay receive research funds and are consultants for various sponsors (detailed in disclosure forms). KEY POINTS Administrative databases are a potentially important data source for study of health issues related to HIV infection. • Hepatitis C status, AIDS status, alcohol and drug abuse are important potential confounders for the study of HIV. • Specificity to identify those variables is high, but the sensitivity is low; additional method to address residual confounding should be considered when using administrative data for study of HIV.
•
Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
950
m. durand et al.
ETHICS STATEMENT This study was approved by the CHUM ethics committee. Permission to use the data from RAMQ and to perform the matching with the CHUM-HIV database was obtained from the Commission d’Accès à l’information (CAI). All study participants gave informed consent to contribute data to the CHUM-HIV database. ACKNOWLEDGEMENTS Funds to conduct this study came from the Canadian Association for AIDS Research and in part from the Canadian Institutes for Health Research (CIHR) grant MOP 81275. Funds to acquire the RAMQ data used in this study were provided by an unrestricted grant from Boeringher-Ingelheim. They had no part in deciding study design, data analysis, manuscript preparation and decision to submit for publication. M. Durand is supported by a post-doctoral fellowship from the CIHR HIV Clinical Trial Network. REFERENCES 1. Antiretroviral Therapy Cohort Collaboration. Life expectancy of individuals on combination antiretroviral therapy in high-income countries: a collaborative analysis of 14 cohort studies. Lancet 2008; 372(9635): 293–299. 2. Lohse N, Hansen AB, Pedersen G, et al. Survival of persons with and without HIV infection in Denmark, 1995–2005. Ann Intern Med 2007; 146(2): 87–95. 3. Martinez E, Milinkovic A, Buira E, et al. Incidence and causes of death in HIVinfected persons receiving highly active antiretroviral therapy compared with estimates for the general population of similar age and from the same geographical area. HIV Med 2007; 8(4): 251–258. 4. van Sighem AI, Gras LA, Reiss P, Brinkman K, de Wolf F. Life expectancy of recently diagnosed asymptomatic HIV-infected patients approaches that of uninfected individuals. AIDS 2010; 24(10): 1527–1535. 5. Zwahlen M, Harris R, May M, et al. Mortality of HIV-infected patients starting potent antiretroviral therapy: comparison with the general population in nine industrialized countries. Int J Epidemiol 2009; 38(6): 1624–1633. 6. Durand M, Sheehy O, Baril JG, Lelorier J, Tremblay CL. Association between HIV infection, antiretroviral therapy, and risk of acute myocardial infarction: a cohort and nested case–control study using Quebec’s public health insurance database. J Acquir Immune Defic Syndr 2011; 57(3): 245–253. 7. Durand M, Sheehy O, Baril JG, Lelorier J, Tremblay CL. Risk of spontaneous intracranial hemorrhage in HIV-infected individuals: a population-based cohort study. J Stroke Cerebrovasc Dis 2013; 22(7): e34–41. 8. Brenner H. A potential pitfall in control of covariates in epidemiologic studies. Epidemiology (Cambridge, Mass) 1998; 9(1): 68–71. 9. Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 2003; 138(1): W1–W12. 10. Durand M, Toma E, Phaneuf D, et al. Description of participant’s characteristics and mortality in the UHRESS database: data from 1648 HIV infected individuals followed at the CHUM from 1982 to 2011. Abstract Number CDB056 http://pag. ias2011.org/abstracts.aspx?aid=4428. poster presentation at the 2011 International AIDS Society meeting Rome; [2011]. 11. Berard A, Lacasse A. Validity of perinatal pharmacoepidemiologic studies using data from the RAMQ administrative database. Can J Clin Pharmacol 2009; 16(2): e360–e369. 12. Firoozi F, Lemiere C, Beauchesne MF, Forget A, Blais L. Development and validation of database indexes of asthma severity and control. Thorax 2007; 62(7): 581–587. 13. Jean S, Candas B, Belzile E, et al. Algorithms can be used to identify fragility fracture cases in physician-claims databases. Osteoporos Int 2012; 23(2): 483–501. 14. Lacasse Y, Montori VM, Lanthier C, Maltis F. The validity of diagnosing chronic obstructive pulmonary disease from a large administrative database. Can Respir J 2005; 12(5): 251–256. 15. Landry JS, Croitoru D, Menzies D. Validation of ICD-9 diagnostic codes for bronchopulmonary dysplasia in Quebec’s provincial health care databases. Chronic Dis Inj Can 2012; 33(1): 47–52.
Copyright © 2015 John Wiley & Sons, Ltd.
16. Monfared AA, Lelorier J. Accuracy and validity of using medical claims data to identify episodes of hospitalizations in patients with COPD. Pharmacoepidemiol Drug Saf 2006; 15(1): 19–29. 17. Tagalakis V, Kahn SR. Determining the test characteristics of claims-based diagnostic codes for the diagnosis of venous thromboembolism in a medical service claims database. Pharmacoepidemiol Drug Saf 2011; 20(3): 304–307. 18. Tairou F, De Wals P, Bastide A. Validity of death and stillbirth certificates and hospital discharge summaries for the identification of neural tube defects in Quebec City. Chronic Dis Can 2006; 27(3): 120–124. 19. Wyse JM, Joseph L, Barkun AN, Sewitch MJ. Accuracy of administrative claims data for polypectomy. CMAJ 2011; 183(11): E743–E747. 20. Schneider E, Whitmore S, Glynn KM, Dominguez K, Mitsch A, McKenna MT. Revised surveillance case definitions for HIV infection among adults, adolescents, and children aged
ORIGINAL REPORT
Diagnostic accuracy of algorithms to identify hepatitis C status, AIDS status, alcohol consumption and illicit drug use among patients living with HIV in an administrative healthcare database† Madeleine Durand1*, Yishu Wang2, François Venne3, Jacques Lelorier3, Cécile L. Tremblay4 and Michal Abrahamowicz2 1
Department of Department of 3 Department of 4 Department of 2
Internal Medicine, Centre Hospitalier de l’Unvisersité de Montréal, Montréal, Canada Epidemiology and Biostatistics, McGill University, Montréal, Canada Medicine, Université de Montréal, Montréal, Canada Microbiology, CHUM Research Center, Montréal, Canada
ABSTRACT Purpose This study aims to develop and evaluate diagnostic algorithms for AIDS, hepatitis C status, alcohol abuse and illicit drug use in the administrative healthcare database of the Province of Quebec, Canada (Régie de l’assurance-maladie du Québec (RAMQ)). Methods We selected HIV-positive patients contributing to both the RAMQ database and a local clinical database, which was used as gold standard. We developed algorithms to identify the diagnoses of interest in RAMQ using data from hospital discharge summaries and medical and pharmaceutical claims databases. We estimated and compared sensitivity, specificity, positive predictive and negative predictive values and area under receiver operating curve for each algorithm. Results Four hundred twenty patients contributed to both databases. Prevalence of conditions of interest in the clinical database was as follows: AIDS 233 (55%), hepatitis C infection 105 (25%), alcohol abuse 106 (25%), illicit drug use 144 (34%) and intravenous drug use 107 (25%). Sensitivity to detect AIDS, hepatitis C, alcohol abuse, illicit drug use and intravenous drug use was 46% [95%CI: 39–53], 26% [18–35], 50% [37–57], 64% [55–72] and 70% [61–79], respectively. Specificity to detect these conditions was 91% [86–95], 97% [94–98], 92% [88–95], 95% [92–97] and 90% [87–93], respectively. Positive predictive values were 87% [80–92], 71% [54–85], 68% [56–78], 87% [79–93] and 72% [62–80], respectively. Area under receiver operating curve varied from 0.62 [0.57–0.65] for hepatitis C to 0.80 [0.76–0.85] for intravenous drug use. Conclusions Sensitivity was low to detect AIDS, alcohol abuse, illicit drug use and especially hepatitis C in RAMQ. Researchers must be aware of the potential for residual confounding and must consider additional methods to control for confounding. Copyright © 2015 John Wiley & Sons, Ltd. key words—HIV AIDS; epidemiology; RAMQ database; administrative healthcare database; diagnostic accuracy; sensitivity; specificity; pharmacoepidemiology Received 5 March 2015; Revised 1 May 2015; Accepted 2 May 2015
INTRODUCTION The clinical picture of human immunodeficiency virus (HIV) infection has changed greatly since the *Correspondence to: M. Durand, CHUM Research Center, 850, rue Saint-Denis, Montreal, Québec, H2X 0A9, Canada. E-mail: madeleine.durand.chum@ssss. gouv.qc.ca † Prior presentations: This research was conducted at the centre de recherche de l’UHRESS du CHUM and at the Pharmacoeconomy and pharmacoepidemiology research unit of the CHUM, Montréal, Canada. Results have been presented at the Canadian association for AIDS research annual meeting in St-John, Newfoundland, Canada in April 2013 (poster presentation) and at the International Conference of Pharmacoepidemiology in Taipei, Taiwan in October 2014 (oral poster presentation).
Copyright © 2015 John Wiley & Sons, Ltd.
introduction of highly active antiretroviral therapy (HAART), and prolonged survival due to HAART has led to an increase in incidence of chronic, noninfectious complications such as cardiovascular and metabolic diseases.1–5 This shift in disease burden requires a paradigm shift in research methodology, with more emphasis on long-term studies of chronic, complex and rarer diseases. The use of administrative healthcare data is appealing, in order to study large population of patients living with HIV. However, some important variables for the study of HIV and its comorbidities have not yet been validated in administrative healthcare databases, so the sensitivity,
944
m. durand et al.
specificity and positive and negative predictive values of coding algorithms are unknown. In a prior work, we showed that AIDS status, hepatitis C status, alcohol and illicit drug use were important confounders of the relationship between HIV infection and important health outcomes.6,7 Yet, without a precise estimate of diagnostic accuracy of algorithms used to identify these variables in administrative healthcare databases, it is difficult to assess whether, and to what extent, adjustment for possibly inaccurate measurements of these variables leads to residual confounding.8 The Régie de l’assurance maladie du Québec (RAMQ) database is the administrative healthcare database of the province of Québec, Canada. It contains routinely collected data on medical and pharmacological care for the population of Québec, Canada, in the setting of a universal healthcare insurance coverage. We aimed to develop and validate algorithms in the RAMQ to diagnose AIDS, hepatitis C, alcohol and illicit drug use in a cohort of HIV-infected individuals. We then aimed to compare the test characteristics (sensitivity, specificity, positive predictive value and negative predictive value), as well as the area under the receiver operating curves (AUCs), of the alternative algorithms, to select the best algorithm to be used in future studies. METHODS Study design and data sources Study design. This is a diagnostic accuracy study, performed using secondary analyses of linked data from two prospective cohort studies. It was conducted and is reported in accordance with the Statement for Reporting Studies of Diagnostic Accuracy (STARD) consensus.9
CHUM HIV database. All HIV-positive patients followed at the Centre Hospitalier de l’Université de Montréal (CHUM) from 1985 until present contribute to a clinical database. The CHUM is a tertiary, urban reference center with a dedicated HIV clinic. Comprehensive data are available on sociodemographic factors, HIV risk factors, habits, CD4 counts, viral load, treatments, vaccination status, opportunistic infections and comorbidities. A standardized chart is used in the follow-up of HIV-positive patients at the CHUM, which ensures completeness and coherence of collected data. This database is updated every time a patient has a medical encounter and is linkable to the hospital medical file.10 In our analyses, the CHUM-HIV database was considered as the gold standard for establishing the diagnoses of interest, which were coded Copyright © 2015 John Wiley & Sons, Ltd.
as present or absent. Hence, the gold standard diagnoses were based on medical charts, completed by HIV specialists in a dedicated clinic. No additional independent arbitration was performed.
RAMQ database. The RAMQ administrative database provides information generated by medical fee-forservice billing for all residents of the province of Québec, with ICD-9 codes for each service. For publicly insured Quebec residents, it also contains pharmaceutical information, including all dispensed prescriptions. This represents 43% of Québec’s population, including everyone over 65 years of age, people on social assistance and subscribers to the public drug insurance plan. RAMQ database also contains the discharge summary for all acute care hospitalizations for all Quebec residents, with ICD-9 and ICD-10 codes for all discharge diagnoses. Information is available from 1987 to present. Several validation studies have been conducted using RAMQ database,11–19 and these data have been used in numerous peer-reviewed publications, including studies of HIV.6,7
Study population Identification of HIV in RAMQ. We identified the cohort of all persons living with HIV within the RAMQ database using ICD-9 codes 042–044 and Québecspecific ICD-9 code 7958. The HIV+ cohort included all subjects who had at least one relevant code between 1 January 1988 and 31 December 2007. The date of the first HIV diagnosis was defined as entry into the cohort, and patients were followed until death, end of drug insurance coverage, or 31 December 2007. Patients had to have at least one year of public drug insurance coverage prior to their cohort entry. This cohort has been described elsewhere.6
Definition of the validation population. Patients contributing data to both CHUM HIV and RAMQ databases were identified using a unique anonymized identifier. Only overlapping periods of follow-up from both databases were included. Patients were excluded if (i) there was no overlap in follow-up periods, (ii) they were followed in another out-patient clinic for routine follow-up and only came to the CHUM for acute hospitalizations, (c) they had less than two visits recorded in the CHUM-HIV database during RAMQ follow-up or (d) they were deceased before year Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
945
validation of administrative healthcare database for study of hiv
2000. Data were collected in both databases prior to enrollment in this study.
Table 1. database
Algorithms used to define conditions of interest in RAMQ
Condition of interest
Definition of diagnoses in CHUM HIV database. AIDS-defining diagnoses, as well as hepatitis C status, alcohol and illicit drug use, are systematically recorded in the database and were retrieved for participating patients. In case of missing data, queries were sent to treating physicians, and hospital charts were retrieved and reviewed. AIDS-defining illnesses were defined as per the list of the Center for Disease Control and Prevention.20 CD4 thresholds were not used to diagnose AIDS, as this information cannot be retrieved in RAMQ. Alcohol abuse was defined as consumption of more than 9 alcohol units per week for women and more than 14 alcohol units per week for men, as per Health Canada’s recommendations.21 Illicit drug use was defined as consumption of any illicit drug (excluding inhaled or oral marijuana derivatives), and intravenous drug administration was recorded when present. Both alcohol abuse and illicit drug use were time-fixed binary variables, coded as present if they happened at any point during follow-up.
Diagnoses in RAMQ. For each diagnosis, four different algorithms were developed and their test characteristics were assessed using CHUM HIV cohort data as a “gold standard”. This was carried out in order to identify the best possible algorithm to diagnose the conditions of interest in RAMQ. Algorithms used outpatients diagnostic claims data (ICD-9 codes), inpatient diagnostic codes data (ICD-9 and 10 codes) and/or pharmaceutical data (using common drug denomination codes). For each diagnosis of interest, the best algorithm was selected by comparing the AUCs. Table 1 describes the different algorithms used to diagnose each condition, and Supplementary Appendix 1 lists the ICD-9, ICD10 and medication codes used in algorithms.
Variation of test characteristics with follow-up time. Reporting of conditions in RAMQ is passive, in the sense that it only happens at times of medical encounters. Therefore, it is probable that test characteristics vary with length of follow-up in RAMQ. To evaluate this, we stratified the validation population (defined above) into mutually exclusive periods of follow-up, using terciles of the follow-up time distribution as the cut-off points (less than 4 years, 4–8 years and Copyright © 2015 John Wiley & Sons, Ltd.
AIDS
Algorithm
Description
1
Presence of at least 1 inpatient code for AIDS defining condition Presence of at least 1 outpatient code for AIDS defining condition Presence of a least 2 outpatient code for AIDS defining condition Presence of either 1 inpatient OR 1 outpatient code for AIDS defining condition Presence of at least 1 inpatient code for HCV Presence of at least 1 outpatient code for HCV Dispensation of medication specific to treat HCV Presence of either 1 inpatient, 1 outpatient or 1 dispensation of medication specific for HCV Presence of at least 1 inpatient code for alcohol-related health problem Presence of at least 1 outpatient code for alcohol-related health problem Presence of at least 2 outpatient code for alcohol-related health problem Presence of either 1 inpatient OR 1 outpatient code for alcohol related health problem Presence of at least 1 inpatient code for illicit drug use-related health problem Presence of at least 1 outpatient code for illicit drug use-related health problem Dispensation of methadone Presence of at least 1 inpatient or outpatient code for illicit drug use-related health problem, or dispensation of methadone
2 3 4 Hepatitis C
1 2 3 4
Alcohol Abuse
1 2 3 4
Illicit drug use and intravenous drug use
1 2 3 4
Most discriminative algorithms (defined as highest area under curve) are in boldface.
more than 8 years) and re-tested the diagnostic algorithms separately in the three resulting strata.
Statistical analysis Appropriate descriptive statistics were used to describe the population’s sociodemographic characteristics. For each diagnosis algorithm tested, sensitivity was estimated as the proportion of true positives, that is, patients correctly identified by the algorithm with the condition of interest in the RAMQ database, among all those identified as having the condition in the CHUM HIV cohort (gold standard). (This corresponds to the ratio of true positives over the sum of true positives + false negatives.) Specificity was estimated as the proportion of true negatives, that is, patients correctly identified without the condition by the algorithms in the RAMQ, among all those who did not have the condition of interest in the Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
946
m. durand et al.
CHUM HIV cohort (gold standard). This corresponds to the ratio of true negatives over the sum of true negatives + false positives. AUC for a binary diagnostic test was estimated as (sensitivity + specificity)/2.22 Confidence intervals for AUC were obtained using the DeLong nonparametric method.23 Positive predictive value (PPV) was measured as the ratio of true positive (positive in both RAMQ and CHUM HIV) over total number of positives identified by the algorithm in RAMQ (true positives + false positives). Negative predictive value (NPV) was measured as the ratio of true negatives (negative both in RAMQ and CHUM HIV) over total number of negatives in RAMQ (true negatives + false negatives). Exact 95% confidence intervals for proportions were computed for all aforementioned test characteristics. A one degree of freedom chi-square test for trend was used to assess, in separate analyses, whether sensitivity and specificity varied across the three subpopulations corresponding to increasing duration of follow-up in RAMQ. Statistical analyses were performed with SAS version 9.224 and Stata version 11.25 RESULTS Description of validation population There were 1476 persons living with HIV in the CHUM HIV database, and 8057 HIV-positive patients in the RAMQ database, among whom 493 (6.2%) patients were matched across the RAMQ and CHUM HIV databases using the unique identifier. Seventythree (15%) of the matched patients were excluded because (i) their follow-up periods in the two databases did not overlap (n = 25), (ii) they had most of their clinical follow-up in another center (n = 27), (iii) they had less than two visits recorded in the CHUM HIV database during their RAMQ follow-up (n = 16) or (iv) they died before year 2000 (n = 5). The remaining 420 (85%) patients were included in the validation sample. Mean age of participants was 52.8 (SD 9.2) years. Eighty-two patients (19.5%) died during follow-up. Of the patients, 318 (75.7%) were male. Ethnic origin was Caucasian for 281 (66.9%) participants, African, African-American or Haitian in 94 (22.4%), and others in 45 (10.7%). During the study period from May 1988 to December 2007, the median overlap in follow-up time in both databases was 6.2 years (interquartile range 3.0–9.3 years.) The dates of HIV diagnoses ranged from 1983 to 2007 but were unknown for 90 (21%) patients. Stages of HIV disease, at the beginning of follow-up, according to the classification of the Center for Disease Control were A (mildly or asymptomatic) in 112 (27%), B (moderately symptomatic) in 117 (28%) and C (severely symptomatic) in 191 (45%). Copyright © 2015 John Wiley & Sons, Ltd.
Prevalence, at any time during follow-up (with exact 95% confidence intervals) of the conditions of interest in the CHUM HIV dataset (“gold standard”), was as follows: 233 (55% [51–60%]) for AIDS, 105 (25% [21–29%]) for hepatitis C infection, 106 (25% [21– 30%]) for alcohol abuse, 144 (34% [30–39%]) for illicit drug use and 107 (25% [21–30%]) for intravenous drug use. Test characteristics of predefined algorithms. Table 2 details the test characteristics, with 95% confidence intervals, for each of the pre-specified algorithms, along with the AUC for each algorithm. (AUC can be interpreted as the overall discrimination capacity of the algorithm: it combines sensitivity and specificity in one figure.) For all algorithms except for AIDS status, the highest AUC is obtained with the most inclusive algorithm (last column of Table 2), which uses combinations of outpatient, inpatient and, where applicable, pharmacological information (Table 1). To determine AIDS status, however, the inclusion in the algorithm of outpatient codes lowered the specificity with an insufficient increase in sensitivity, so algorithm 1, which relies on inpatient codes only, offers the best sensitivity/specificity trade-off (Table 2). Overall, specificity was high: above 90% for all algorithms. Sensitivity to detect the conditions of interest in the RAMQ administrative database, on the other hand, varied widely across algorithms and was generally low. It was very low even for the most discriminative algorithms for hepatitis C (26%) and low for both AIDS (46%) and alcohol abuse (50%), but substantially higher for intravenous drug use (70%). AUC were low for most diagnoses: 0.69 95%CI [0.65–0.72] for AIDS, 0.62 [0.57–0.65] for hepatitis C, and 0.71 [0.66–0.76] for alcohol abuse. Only the algorithms to identify illicit drug use and intravenous drug use reached values of 0.80 for AUC (0.80 [0.76–0.85] for intravenous drug use). Positive and Negative Predictive Values: PPVs were very high for AIDS and for illicit drug use (87% for both diagnoses) and moderately high for hepatitis C (71%), for alcohol abuse (68%) and for intravenous drug use (72%). NPVs were also high for hepatitis C (80%), alcohol abuse (85%), illicit drug use (83%) and intravenous drug use (90%) but lower for AIDS (57%). This much lower NPV for AIDS is explained by the higher prevalence of this condition in our population (55%). Table 2 gives PPV and NPV for all tested algorithms, along with 95% confidence intervals. Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
947
validation of administrative healthcare database for study of hiv
Table 2. Prevalence of conditions in CHUM’s clinical database and according to various algorithms in RAMQ, with tests characteristics and area under ROC (for 420 patients)
AIDS CHUM Algo 1 Algo 2 Algo 3 Algo 4 Hepatitis C CHUM Algo 1 Algo 2 Algo 3 Algo 4 Alcohol abuse CHUM Algo 1 Algo 2 Algo 3 Algo 4 Illicit drug use CHUM Algo 1 Algo 2 Algo 3 Algo 4 IV drug use CHUM Algo 1 Algo 2 Algo 3 Algo 4
Prevalence n (%)
Sensitivity [95%CI]
Specificity [95%CI]
Positive predictive value [95%CI]
Negative predictive Value [95%CI]
Area under ROC [95%CI]
233 (55) 123 (30) 64 (15) 31 (7) 148 (35)
Ref 46% [39–53] 21% [16–27] 12% [8–16] 50% [44–57]
Ref 91% [86–95] 92% [87–95] 98% [95–99] 83% [77–88]
Ref 87% [80–92] 77% [64–86] 87% [70–96] 79% [72–85]
Ref 57% [52–63] 48% [43–54] 47% [42–52] 57% [51–63]
0.69 [0.65–0.72] 0.57 [0.53–0.60] 0.55 [0.52–0.57] 0.67 [0.63–0.71]
105 (25) 30 (7) 17 (4) 6 (1) 38 (9)
Ref 22% [14–31] 11% [6–19] 5% [2–11] 26% [18–35]
Ref 98% [95–99] 98% [96–99] 100% [99–100]* 97% [94–98]
Ref 77% [58–90] 71% [44–90] 83% [34–100] 71% [54–85]
Ref 79% [75–83] 77% [72–81] 76% [71–80] 80% [75–84]
0.60 [0.56–0.64] 0.55 [0.52–0.58] 0.53 [0.50–0.54] 0.62 [0.57–0.65]
106 (25) 58 (13) 59 (14) 32 (8) 78 (19)
Ref 43% [34–53] 38% [29–48] 26% [18–36] 50% [37–57]
Ref 96% [93–98] 94% [91–96] 99% [96–100] 92% [88–95]
Ref 79% [67–89] 68% [54–79] 88% [71–96] 68% [56–78]
Ref 83% [79–87] 82% [77–86] 80% [76–84] 85% [80–88]
0.70 [0.65–0.75] 0.66 [0.61–0.71] 0.63 [0.58–0.67] 0.71 [0.66–0.76]
144 (34) 80 (19) 78 (19) 1 (0.2) 106 (25)
Ref 52% [44–60] 47% [39–56] 1% [0–4] 64% [55–72]
Ref 98% [96–99] 96% [93–98] 100% [99–100]* 95% [92–97]
Ref 94% [86–98] 87% [78–94] 100% [5–100]* 87% [79–93]
Ref 80% [75–84] 78% [73–82] 66% [61–70] 83% [79–87]
0.75 [0.71–0.79] 0.72 [0.68–0.76] 0.51 [0.50–0.51] 0.80 [0.75–0.83]
108 (26) 80 (19) 78 (19) 1 (0.2) 106 (25)
Ref 57% [48–67] 53% [43–62] 1% [0–5] 70% [61–79]
Ref 94% [91–97] 93% [90–96] 100% [99–100]* 90% [87–93]
Ref 78% [67–86] 73% [62–82] 100% [5–100]* 72% [62–80]
Ref 86% [82–90] 85% [81–79] 74% [70–79] 90% [86–93]
0.76 [0.71–0.81] 0.73 [0.68–0.78] 0.51 [0.50–0.51] 0.80 [0.76–0.85]
Most discriminative algorithms (defined as highest area under curve) are in boldface. *One-sided 95%CI.
Table 3. Evolution of sensitivity/specificity with duration of follow-up, for best diagnostic algorithm
AIDS 7 Sensitivity Specificity AUC Hepatitis C Sensitivity Specificity AUC Alcohol abuse Sensitivity Specificity AUC Illicit drug use Sensitivity Specificity AUC IV drug use Sensitivity Specificity AUC
Less than 4 years of follow-up (n = 138)
4 to 8 years of follow-up (n = 148)
More than 8 years of follow-up (n = 134)
1 df p-value for trend
38 [27–50] 92 [83–97] 0.65 [0.59–0.72]
42 [31–55] 95 [87–98] 0.69 [0.62–0.75]
55 [44–66] 85 [72–94] 0.70 [0.63–0.77]
0.08 0.18
13 [4–27] 99 [94–100] 0.56 [0.50–0.61]
31 [16–50] 97 [93–99] 0.64 [0.56–0.73]
36 [20–55] 93 [86–97] 0.65 [0.56–0.73]
0.01 0.02
31 [17–49] 93 [86–97] 0.62 [0.54–0.70]
61 [42–78] 93 [87–97] 0.77 [0.68–0.86]
57 [41–73] 89 [81–95] 0.73 [0.65–0.82]
0.03 0.32
54 [39–68] 99 [94–100] 0.77 [0.69–0.83]
62 [46–76] 95 [89–98] 0.79 [0.71–0.86]
76 [62–87] 90 [82–96] 0.83 [0.76–0.90]
0.02 0.01
60 [43–75] 95 [88–98] 0.78 [0.69–0.85]
71 [53–85] 94 [88–97] 0.83 [0.74–0.90]
82 [65–93] 82 [73–89] 0.82 [0.75–0.90]
0.04 0.002
AUC, area under receiver operating curve. Numbers in square brackets are 95% confidence intervals.
Copyright © 2015 John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
948
m. durand et al.
Variation of test characteristics with increasing duration of follow-up. Table 3 shows the variation of test characteristics for the optimal algorithm across the three subgroups with gradually increasing follow-up duration. As expected, for all diagnoses, there are statistically significant improvements in sensitivity with increased follow-up time (p < 0.05 for all tests for trend). For some conditions (hepatitis C, illicit drug use and intravenous drug use), this is associated with a significant decrease in specificity. AUC generally increased with increasing follow-up time, suggesting that overall performance of the proposed diagnostic algorithms in the RAMQ database improves when longer follow-up periods are available. However, even with longer follow-up, AUC remained below 0.80 for AIDS, hepatitis C infection and alcohol abuse. DISCUSSION In this validation study of an administrative healthcare database, we estimated the test characteristics for prespecified diagnostic algorithms for conditions of particular interest to the study of HIV/AIDS, by using a clinical database as a “gold standard”. We found overall low sensitivity of diagnostic algorithms to identify AIDS, alcohol abuse and illicit drug use, and especially hepatitis C infection (46%, 50%, 64% and 26%, respectively). Specificity to identify the same conditions was relatively high (83%, 92%, 90% and 97%, respectively). We also showed that, as expected, test characteristics vary with follow-up time in the administrative database, with marked increases in sensitivity and relatively insignificant losses in specificity with longer follow-up times. Our results are consistent with previous validation studies in administrative healthcare databases. For example, Landry et al. found a sensitivity of 45–52% and specificity of 98% to diagnose broncho-pulmonary dysplasia (a chronic respiratory disease) in pre-term infants.15 In a systematic review, Chung et al. present nine studies reporting test characteristics for algorithms to identify rheumatoid arthritis and report sensitivities from 51% to 65% in general administrative healthcare databases comparable with RAMQ.26 Therefore, we believe the sensitivities found to detect AIDS, alcohol abuse and illicit drug use are comparable with those found in this type of data for other chronic conditions. The sensitivity to detect hepatitis C, however, is markedly lower. In the general population, it is estimated that a significant proportion of Hepatitis C infections are undiagnosed, although a precise estimate is not available.27,28 In our “gold standard” population, patients undergo routine screening for hepatitis C Copyright © 2015 John Wiley & Sons, Ltd.
infection, so their treating physicians are aware of their diagnosis. The low detection of this comorbidity in RAMQ, in spite of the patient’s physicians being aware of it, is not surprising. First, most patients chronically infected with hepatitis C are asymptomatic. Second, RAMQ coding for outpatient visits only allows one diagnostic code to be used per medical visit. Therefore, in patients co-infected with HIV, it is likely that the treating physicians will report HIV infection rather than hepatitis C infection. In the future, as less toxic and more accessible treatments for hepatitis C infection become available and treatment rates for hepatitis C increase, it is likely that its capture will be improved in administrative healthcare databases with available pharmacological data, such as RAMQ. Our work has important implications for the use of administrative databases in the study of HIV. Indeed, as persons living with HIV live longer, this condition has become a chronic disease in developed countries.1 Large databases have become an important tool for studies of long-term effects and toxicities of medications and emergence of rare conditions.29–33 Clinical trials or smaller cohorts are often ill-suited to study such outcomes. Our study assessed the accuracy of two types of variables relevant for such investigations: (i) AIDS status is important to establish in order to situate patients within the spectrum of HIV infection, while (ii) hepatitis C, alcohol abuse and illicit drug will often be considered as potential confounders of the associations between HIV and health outcomes. HIV infection has a very broad clinical spectrum, ranging from asymptomatic to profound immunosuppression, which must be taken into account. In the absence of data on viral load and CD4 lymphocytes values in administrative healthcare databases, establishing AIDS status is important and challenging. We have shown in prior work7 that an association between HIV infection and intracranial hemorrhage exists in the entire population of persons living with HIV. However, when stratified according to AIDS status, the association disappears in non-AIDS population. Failure to take the AIDS status into account leads to biases in pharmacoepidemiological studies. For example, if a new drug is given preferentially to AIDS patients, because it is believed to be more potent, then a kind of indication bias (bias by severity, or channeling bias) will occur.27,28 Second, according to published studies, alcohol abuse34,35 is more prevalent in persons living with HIV than in the general population. Illicit drug use is a recognized risk factor for HIV infection,36 and hepatitis C infection is a frequent co-infection in persons living with HIV.37 Each of these three health Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
validation of administrative healthcare database for study of hiv
conditions is associated with numerous health outcomes. As such, they are classical confounders of the associations between HIV and various health outcomes and must be adjusted for. This work shows that studies about HIV performed in the RAMQ database are vulnerable to residual confounding because of misclassification of those variables. Further studies are necessary to assess if, and to what extent, our findings apply to other administrative databases. This study has several limitations. First, the prevalence of the conditions of interest in our population (people living with HIV) differs from that of the general population, so results might not be extrapolated when studying HIV-negative controls. Not only is prevalence of conditions different in HIV-negative population but physicians’ interest in those conditions differs, arguably affecting diagnosis and reporting. Our results can also be affected by spectrum bias: sensitivity for a condition can be higher if the condition is more severe.9,38,39 This can be especially true for the diagnoses of AIDS, as the natural course of HIV infection has greatly changed in the period covered by our cohort. AIDS manifestations were more likely to be present in the earlier years of the epidemic and were more likely to be sustained and severe. Sensitivity might be lower to detect AIDS status in the highly active antiretroviral therapy era. Finally, the gold standard used in our study is a clinical database, which can be imprecise because of less than 100% sensitivity and specificity in clinical characteristics detection by clinicians. Strengths of this study include its large validation sample and the review of both the clinical database and patient’s charts. It emphasizes that conditions such as alcohol abuse and illicit drug use can be captured (although imperfectly) by administrative healthcare data, with test characteristics comparable with those of chronic medical conditions. We also found very high PPV for some conditions of interest (87% for AIDS and intravenous drug use) and high PPV for others (71% for hepatitis C and 68% for alcohol abuse). Patients identified by the algorithms therefore have a high probability of truly having the condition. Analysis of the effect of follow-up time on test characteristics also adds an interesting perspective to analysis of routinely collected healthcare data. Restricting study cohorts to patients with longer follow-up would introduce bias by truncating the sicker patients, who are more likely to die before completing longer follow-up. On the other hand, researchers must recognize that diagnostic algorithms perform better for patients with longer follow-up. Copyright © 2015 John Wiley & Sons, Ltd.
949
Hence, there will be greater misclassification of covariates in patients with shorter follow-up, which in turn will increase the residual confounding for these patients. Furthermore, if the exposure of interest is associated with follow-up length, then the degree of misclassification will be differential according to the exposure status and might bias the effect estimate in any direction. Relying on time-dependent, as opposed to time-fixed, diagnostic algorithms for covariates would address this concern. Overall, this study shows that the RAMQ database is limited to capture AIDS status, hepatitis C infection, alcohol abuse and illicit drug use. Sensitivity tends to be low, especially for hepatitis C, but specificity, as expected, is high. Prior work showed that the knowledge of test characteristics, derived from a sub-sample of a larger cohort, coupled with simulation, can reduce bias in large administrative healthcare databases.40 Conclusions. This work establishes test characteristics for key variables for the study of HIV/AIDS using Québec’s administrative healthcare database. Our findings, which test characteristics of algorithms to detect important variables are insufficient to adequately control for their confounding effects, should lead researchers to develop and explore novel methods to adjust for residual confounding when using administrative data for health research purposes.
CONFLICTS OF INTEREST Y. Wang, F. Venne and M. Abrahamowicz have no conflict of interest to disclose. M. Abrahamowicz is a James McGill Professor of Biostatistics at McGill University. J. Lelorier and C. L. Tremblay receive research funds and are consultants for various sponsors (detailed in disclosure forms). KEY POINTS Administrative databases are a potentially important data source for study of health issues related to HIV infection. • Hepatitis C status, AIDS status, alcohol and drug abuse are important potential confounders for the study of HIV. • Specificity to identify those variables is high, but the sensitivity is low; additional method to address residual confounding should be considered when using administrative data for study of HIV.
•
Pharmacoepidemiology and Drug Safety, 2015; 24: 943–950 DOI: 10.1002/pds
950
m. durand et al.
ETHICS STATEMENT This study was approved by the CHUM ethics committee. Permission to use the data from RAMQ and to perform the matching with the CHUM-HIV database was obtained from the Commission d’Accès à l’information (CAI). All study participants gave informed consent to contribute data to the CHUM-HIV database. ACKNOWLEDGEMENTS Funds to conduct this study came from the Canadian Association for AIDS Research and in part from the Canadian Institutes for Health Research (CIHR) grant MOP 81275. Funds to acquire the RAMQ data used in this study were provided by an unrestricted grant from Boeringher-Ingelheim. They had no part in deciding study design, data analysis, manuscript preparation and decision to submit for publication. M. Durand is supported by a post-doctoral fellowship from the CIHR HIV Clinical Trial Network. REFERENCES 1. Antiretroviral Therapy Cohort Collaboration. Life expectancy of individuals on combination antiretroviral therapy in high-income countries: a collaborative analysis of 14 cohort studies. Lancet 2008; 372(9635): 293–299. 2. Lohse N, Hansen AB, Pedersen G, et al. Survival of persons with and without HIV infection in Denmark, 1995–2005. Ann Intern Med 2007; 146(2): 87–95. 3. Martinez E, Milinkovic A, Buira E, et al. Incidence and causes of death in HIVinfected persons receiving highly active antiretroviral therapy compared with estimates for the general population of similar age and from the same geographical area. HIV Med 2007; 8(4): 251–258. 4. van Sighem AI, Gras LA, Reiss P, Brinkman K, de Wolf F. Life expectancy of recently diagnosed asymptomatic HIV-infected patients approaches that of uninfected individuals. AIDS 2010; 24(10): 1527–1535. 5. Zwahlen M, Harris R, May M, et al. Mortality of HIV-infected patients starting potent antiretroviral therapy: comparison with the general population in nine industrialized countries. Int J Epidemiol 2009; 38(6): 1624–1633. 6. Durand M, Sheehy O, Baril JG, Lelorier J, Tremblay CL. Association between HIV infection, antiretroviral therapy, and risk of acute myocardial infarction: a cohort and nested case–control study using Quebec’s public health insurance database. J Acquir Immune Defic Syndr 2011; 57(3): 245–253. 7. Durand M, Sheehy O, Baril JG, Lelorier J, Tremblay CL. Risk of spontaneous intracranial hemorrhage in HIV-infected individuals: a population-based cohort study. J Stroke Cerebrovasc Dis 2013; 22(7): e34–41. 8. Brenner H. A potential pitfall in control of covariates in epidemiologic studies. Epidemiology (Cambridge, Mass) 1998; 9(1): 68–71. 9. Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 2003; 138(1): W1–W12. 10. Durand M, Toma E, Phaneuf D, et al. Description of participant’s characteristics and mortality in the UHRESS database: data from 1648 HIV infected individuals followed at the CHUM from 1982 to 2011. Abstract Number CDB056 http://pag. ias2011.org/abstracts.aspx?aid=4428. poster presentation at the 2011 International AIDS Society meeting Rome; [2011]. 11. Berard A, Lacasse A. Validity of perinatal pharmacoepidemiologic studies using data from the RAMQ administrative database. Can J Clin Pharmacol 2009; 16(2): e360–e369. 12. Firoozi F, Lemiere C, Beauchesne MF, Forget A, Blais L. Development and validation of database indexes of asthma severity and control. Thorax 2007; 62(7): 581–587. 13. Jean S, Candas B, Belzile E, et al. Algorithms can be used to identify fragility fracture cases in physician-claims databases. Osteoporos Int 2012; 23(2): 483–501. 14. Lacasse Y, Montori VM, Lanthier C, Maltis F. The validity of diagnosing chronic obstructive pulmonary disease from a large administrative database. Can Respir J 2005; 12(5): 251–256. 15. Landry JS, Croitoru D, Menzies D. Validation of ICD-9 diagnostic codes for bronchopulmonary dysplasia in Quebec’s provincial health care databases. Chronic Dis Inj Can 2012; 33(1): 47–52.
Copyright © 2015 John Wiley & Sons, Ltd.
16. Monfared AA, Lelorier J. Accuracy and validity of using medical claims data to identify episodes of hospitalizations in patients with COPD. Pharmacoepidemiol Drug Saf 2006; 15(1): 19–29. 17. Tagalakis V, Kahn SR. Determining the test characteristics of claims-based diagnostic codes for the diagnosis of venous thromboembolism in a medical service claims database. Pharmacoepidemiol Drug Saf 2011; 20(3): 304–307. 18. Tairou F, De Wals P, Bastide A. Validity of death and stillbirth certificates and hospital discharge summaries for the identification of neural tube defects in Quebec City. Chronic Dis Can 2006; 27(3): 120–124. 19. Wyse JM, Joseph L, Barkun AN, Sewitch MJ. Accuracy of administrative claims data for polypectomy. CMAJ 2011; 183(11): E743–E747. 20. Schneider E, Whitmore S, Glynn KM, Dominguez K, Mitsch A, McKenna MT. Revised surveillance case definitions for HIV infection among adults, adolescents, and children aged
