Receiver operating characteristic curve estimation for time to event with semicompeting risks and interval censoring.

Statistical Methods in Medical Research http://smm.sagepub.com/

Receiver operating characteristic curve estimation for time to event with semicompeting risks and interval censoring Hélène Jacqmin-Gadda, Paul Blanche, Emilie Chary, Célia Touraine and Jean-François Dartigues Stat Methods Med Res published online 6 May 2014 DOI: 10.1177/0962280214531691 The online version of this article can be found at: http://smm.sagepub.com/content/early/2014/05/06/0962280214531691

Published by: http://www.sagepublications.com

Additional services and information for Statistical Methods in Medical Research can be found at: Email Alerts: http://smm.sagepub.com/cgi/alerts Subscriptions: http://smm.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav

>> OnlineFirst Version of Record - May 6, 2014 What is This?

Downloaded from smm.sagepub.com at LOUISIANA STATE UNIV on September 3, 2014

XML Template (2014) [25.4.2014–3:35pm] [1–17] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140042/APPFile/SG-SMMJ140042.3d (SMM) [PREPRINTER stage]

Article

Receiver operating characteristic curve estimation for time to event with semicompeting risks and interval censoring

Statistical Methods in Medical Research 0(0) 1–17 ! The Author(s) 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/0962280214531691 smm.sagepub.com

He´le`ne Jacqmin-Gadda,1,2 Paul Blanche,1,2 Emilie Chary,1,2 Ce´lia Touraine1,2 and Jean-François Dartigues1,2

Abstract Semicompeting risks and interval censoring are frequent in medical studies, for instance when a disease may be diagnosed only at times of visit and disease onset is in competition with death. To evaluate the ability of markers to predict disease onset in this context, estimators of discrimination measures must account for these two issues. In recent years, methods for estimating the time-dependent receiver operating characteristic curve and the associated area under the ROC curve have been extended to account for right censored data and competing risks. In this paper, we show how an approximation allows to use the inverse probability of censoring weighting estimator for semicompeting events with interval censored data. Then, using an illnessdeath model, we propose two model-based estimators allowing to rigorously handle these issues. The first estimator is fully model based whereas the second one only uses the model to impute missing observations due to censoring. A simulation study shows that the bias for inverse probability of censoring weighting remains modest and may be less than the one of the two parametric estimators when the model is misspecified. We finally recommend the nonparametric inverse probability of censoring weighting estimator as main analysis and the imputation estimator based on the illness-death model as sensitivity analysis. Keywords area under the curve, interval censoring, imputation, illness-death model, inverse probability of censoring weighting, semicompeting risks

1 Introduction In epidemiology and clinical studies, semicompeting risks arise when subjects are at risk of both an intermediate event and a terminal event. The most classical example is disease and death: the disease may occur only if subjects are alive but subjects may die before or after disease onset. Analysis of 1 2

Universite´ Bordeaux Segalen, ISPED, Centre INSERM U897, Bordeaux, France INSERM, Centre INSERM U-897, F-33000 Bordeaux, France

Corresponding author: He´le`ne Jacqmin-Gadda, Universite´ Bordeaux Segalen, case 11, ISPED, Centre INSERM U-897, 146 rue Leó Saignat, 33076 Bordeaux, France. Email: [email protected]



2

Statistical Methods in Medical Research 0(0)

such data is complicated when the intermediate event is interval censored, for instance when the diagnosis of the disease is assessed only at the times of visit and it is not possible to identify retrospectively the date of disease onset. When a subject disease free at the last visit dies, the exact date of death can generally be collected but it is not known whether or not the subject had developed the disease between the last visit and death. This situation is frequent for chronic diseases with insidious onset such as dementia. In the French population-based prospective cohort, called Paquid, about cognitive aging, dementia diagnosis is assessed at each visit every 2 or 3 years.1 Thus, time of dementia is interval censored. When developing prediction models for dementia, the competing risk of death cannot be neglected because death is a very frequent event in the elderly and the risk of death is much higher among demented compared to nondemented subjects.2 Thus, the proportion of subjects died with dementia but without dementia diagnosis is not negligible and increases when the interval between visits is large due to the protocol of the study or to intermittent missing data as it is frequent in Paquid. In this context, estimation of the risk of dementia without bias requires joint modeling of the risk of death and the risk of dementia using an illness-death model. Model parameters must be estimated by maximizing the likelihood accounting for interval censoring.2 The motivation for the present work was to quantify the ability of cognitive tests measured at baseline to identify subjects who will become demented in the next 10 years. Indeed, as treatments given after the diagnosis were found poorly efficient, identifying subjects at high risk of dementia is essential to implement preventive intervention trials.3 However, to our knowledge, no measure of discrimination has been proposed to evaluate prognostic marker in the context of semicompeting risks and interval censoring. Sensitivity and specificity are standard criteria to evaluate the ability of markers to discriminate between diseased and nondiseased subjects. For diagnostic studies, these two quantities are respectively estimated by the proportion of true positive among diseased subjects and the proportion of true negative among nondiseased subjects. When the marker is quantitative, the receiver operating characteristic (ROC) curve displays sensitivity versus 1– specificity for all the possible threshold values of the marker and the area under the ROC curve (AUC) is a very convenient discrimination measure as it may be interpreted as the probability that a randomly chosen diseased subject has a higher marker value than a randomly chosen nondiseased subject.4 Time-dependent definitions of sensitivity and specificity have been proposed for prognostic studies that aim at discriminating between future diseased and nondiseased subjects. Using the cumulative/ dynamic definition,5 for a given threshold c, the sensitivity at time t is the probability that the marker value be above c given that the subject was ill before time t and the specificity at time t is the probability that the marker value be below c given that the subject is free of the disease at time t. However, estimation of these time-dependent sensitivity and specificity is complicated by right censoring since the disease status of subjects censored before time t is unknown. In the past 10 years, several estimators of the time-dependent ROC curve, denoted ROC(t), have been proposed accounting for right censoring.5–9 Then, the definition of sensitivity has been adapted to the competing risks setting and it was shown that two different definitions of specificity may be used depending on the way to consider subjects who met the competing event.10,11 Estimators accounting for right censoring have been proposed for these three measures of discrimination.10–12 However, none of these estimators handle semicompeting risks with interval censored data. For interval censored data without competing event, Li and Ma13 estimate ROC(t) by deleting all the subjects whose censoring interval includes t. When the times of visits are common for all the subjects, this issue may be avoided by computing the curves only for the times of visits. Otherwise, as



Jacqmin-Gadda et al.

3

an approximation, the time of disease may be imputed as the middle of the interval. In the semicompeting risks setting, interval censoring raises a more serious issue as often it is not possible to know which of the two events occurs first. Using an inverse probability of censoring weighting (IPCW) approach, we recently proposed nonparametric estimators of the ROC(t) curve for right censored time-to-event data with competing risks as well as a test for comparison of AUC(t) for different markers.12 This method does not handle interval censoring. Indeed, for each subject with an observed event, the method requires to know which of the two competing events arises first, whereas, with interval censored data, when a subject free of dementia at the last visit dies, we do not know if he has developed dementia between the last visit and date of death. For applying the IPCW estimator to the Paquid data, we previously used the following decision rule12: if the subject was seen and diagnosed as free of dementia in the last 2 years before death, the subject was considered as free of dementia at death time; if the subject was not visited in the last 2 years, the subject was considered as right censored at the last visit (and the information on death was not used). In this paper, we propose two estimators of sensitivity, specificity and ROC(t) curve based on the illness-death model that deal carefully with interval censoring and semicompeting risks. The first approach, denoted ‘‘fully model based,’’ is an extension for the illness-death model of the semiparametric estimator previously proposed for the Cox proportional hazard model.7,11,14 As it was previously shown that parametric estimators of the ROC curve were sensitive to model misspecification,15 we propose also an imputation estimator that was expected to be more robust since only the unknown disease status at time t is imputed using the illness-death model. The three estimators are compared in well specified and misspecified situations through a simulation study. The organization of the paper is as follows. Section 2 recalls the two definitions of the ROC(t) curve in the competing risks setting and the corresponding IPCW estimators for right censored data. Section 3 describes the two estimators based on the illness-death model for semicompeting risks and interval censoring. The simulation study is presented in Section 4 and the three methods are applied to the Paquid cohort in Section 5 before concluding.

2 ROC curve estimation with competing risks and right censored data 2.1 Definitions of ROC(t) curves Let us denote with M a quantitative marker measured at baseline. Without loss of generality, we assume that larger values of the marker M are associated with higher risks of events. To handle competing risks, we denote with T the time of the first event and ¼ 1, 2 indicates the type of event (1 for the main event and 2 for the competing event). In this setting, cumulative sensitivity for a window of prediction t and a cutpoint c for the marker is the probability to be positive for the marker (M 4 c) given that the main event occurs before t10–12 Seðc, tÞ ¼ PðM 4 cjT t, ¼ 1Þ Two definitions were proposed for dynamic specificity with competing risks. The first one is the probability to be negative for the marker (M c) given that the subject is free of any event at time t10–12 Sp ðc, tÞ ¼ PðM cjT 4 tÞ


ð1Þ


4


The second one is the probability to be negative for the marker ðM cÞ given that the subject is free of any event at time t or met the competing event before time t11,12 Spðc, tÞ ¼ PðM cjfT 4 tg [ fT t, ¼ 2gÞ

ð2Þ

The ROC(t) curve displays Seðc, tÞ versus 1 Sp ðc, tÞ or 1 Spðc, tÞ for all the possible values of c and AUC(t) is the area under this curve. Thereafter, we will denote AUC ðtÞ and AUC(t) the AUC computed, respectively, with definition (1) and definition (2) of the specificity. Thus AUC ðtÞ ¼ P Mi 4 Mj jTi t, i ¼ 1, Tj 4 t

ð3Þ

AUCðtÞ ¼ P Mi 4 Mj jTi t, i ¼ 1, Tj 4 t [ Tj t, j ¼ 2

ð4Þ

and

where i and j represent the indices of two independent subjects. With the dementia and death example, Sp evaluates the discrimination between future demented subjects and subjects alive and free of dementia at the end of the window of prediction t while Sp evaluates the discrimination between future demented subjects and subjects alive and free of dementia at t or deceased without dementia before t. Given that cognitive markers are associated with the risks of both dementia and death, we anticipate that the associated ROC(t) curves will be different. When high values of the marker increase the risk of the two events, AUC(t) is expected to be smaller than AUC ðtÞ:

2.2

IPCW estimator

When data are only right censored, we denote C the censoring time, ¼ 1ðTCÞ the failure indicator, and T~ ¼ minðT, CÞ the observed time; ~ ¼ : is equal to 0 for censoring times, 1 for the main event, and 2 for the competing event. We denote by D1i ðtÞ ¼ 1ðT~ i t,~i ¼1Þ the indicator that the main event was observed before t, D2i ðtÞ ¼ 1ðT~ i t,~ i ¼2Þ the indicator that the competing event was observed before t, and Di ðtÞ ¼ 1ðT~ i tÞ the indicator that one event was observed before t. Assuming that the censoring time C is independent of ðT, , MÞ, Blanche et al.12 proposed nonparametric estimators for Seðc, tÞ, Sp ðc, tÞ, and Spðc, tÞ using the IPCW approach. The idea of the IPCW approach is to use the standard estimator computed on subjects with known status at time t (for sensitivity this is the number of positive cases divided by the total number of cases) but instead of having a contribution of 1, the contribution of each subject is the inverse of the ^ 4 tÞ denote the inverse of probability to be observed. More precisely, let wðtÞ ¼ 1=PðC ^ 4 tÞ computed by Kaplan–Meier estimator, the probability to be uncensored at time t, with PðC the IPCW estimators are P ~ i:Mi 4c D1i ðtÞwðTi Þ b tÞ ¼ P Seðc, ð5Þ n ~ i¼1 D1i ðtÞwðTi Þ




5 P i:Mi c ð1 Di ðtÞÞwðtÞ c ðc, tÞ ¼ P Sp n ð1 Di ðtÞÞwðtÞ P i¼1 i:Mi c ð1 Di ðtÞÞ ¼ P n i¼1 ð1 Di ðtÞÞ

and c tÞ ¼ Spðc,

P

ð6Þ

~

i:M c ð1 Di ðtÞÞwðtÞ þ D2i ðtÞwðTi Þ Pni ~ i¼1 ð1 Di ðtÞÞwðtÞ þ D2i ðtÞwðTi Þ

ð7Þ

The two AUCs may be estimated by numerical integration under the corresponding ROC curves or equivalently by the direct formulas (8) or (9) in Blanche et al.12 These IPCW estimators have several advantages. They are fully nonparametric and have wellestablished large sample properties allowing development of confidence intervals and tests to compare the AUCs of two markers.12 They lead to standard estimators when there is no censoring and were found the most robust to marker-dependent censoring among three other estimators that do not account explicitly for marker-dependent censoring.16 However, when applied to semicompeting events with interval censored data, an approximation is required. A window of time has to be chosen to classify subjects who died before disease diagnosis: if the subject was not seen in this window preceding the death, the subject is considered as censored at the last visit and if the subject was seen in the window, the subject is considered as free of the disease at the time of death.

3 ROC curve estimators with semicompeting risks and interval censoring 3.1 Notations for interval censored data In practice, in cohort studies time of death and time of illness often have different censoring schemes needing additional notations. The time of illness denoted by T1 may be interval censored by the times of visit while the time of death, denoted by T2, is only right censored. We denote with L and R the left and right limits of the interval of censoring of the illness: L is the time of the last visit without illness and R is the time of the first visit with illness or þ1 if the subject is right censored. On the other hand, C2 denotes the censoring time for death with T~ 2 ¼ minðT2 , C2 Þ and the death indicator 2 ¼ 1ðT2 C2 Þ . With these notations, the time of the first event T ¼ minðT1 , T2 Þ, ¼ 1 if the first event is illness (T1 5 T2 ) and ¼ 2 if the first event is death (T2 5 T1 ). Often, time of death is only censored by the end of the study since vital status is easy to obtain without visiting the subject (for instance by phone call to the subject, a relative, or the practitioner, etc.). In any case, Li 5 T2i and Li C2i as subjects must be alive and uncensored to be visited. In the following, we assume noninformative censoring.

3.2

The illness-death model

To estimate the risk of semicompeting events without bias when data are interval censored, we need to estimate simultaneously the three transition intensities of the illness-death model displayed in Figure 1. The model includes three states: 0¼health, 1 ¼ illness, and 2 ¼ death. Using a nonhomogeneous Markov model, the transition intensity between state j and state k is modeled as a function of the marker Mi using a proportional intensity model jk ðtjMi Þ ¼ 0jk ðtÞ expðjk Mi Þ



6


Figure 1. The illness-death model.

where 0jk ðtÞ is the baseline intensity transition function and jk a regression parameter. The cumulative transition intensity between states j and k between times s and t is Zt Ajk ðs, tjMi Þ ¼ jk ðujMi Þdu for j ¼ 0, 1, k ¼ 1, 2 and j 6¼ k s

From this model, transition probabilities Pjk ðs, tÞ from state j at time s to state k at time t may be obtained by P00 ðs, tÞ ¼ eA01 ðs, tjMi ÞA02 ðs, tjMi Þ P11 ðs, tÞ ¼ eA12 ðs, tjMi Þ and Z P01 ðs, tÞ ¼

t

P00 ðs, uÞ01 ðujMi ÞP11 ðu, tÞdu s

P00 ðs, tÞ is the probability to remain healthy from time s to time t (that is the probability of not doing transition 0–1 or 0–2), P11 ðs, tÞ is the probability to remain ill from time s to time t (probability of not dying for a diseased subject), and P01 ðs, tÞ is the probability to be healthy at time s and alive and ill at time t. The likelihood of this model for interval censored data was detailed previously.2,17 Typically, the contribution to the likelihood for a subject i who remained healthy until visit k (time Vik) and died at time T2i 4 Vik (Li ¼ Vik and Ri ¼ þ1) is the sum of two terms P00 ð0, T2i Þ02 ðT2i jMi Þ þ P00 ð0, Li ÞP01 ðLi , T2i Þ12 ðT2i jMi Þ The first term is the probability to remain healthy until T2i and to die at T2i , the second term is the probability to remain healthy until Li, become ill between Li and T2i and die with the illness at T2i . Using parametric distributions for the baseline transition intensities 0jk ðtÞ, the model may be estimated by maximum likelihood. Semiparametric estimators may be obtained by maximizing the penalized likelihood where the roughness penalty function is the sum of the squared norms of




7

the second derivatives of the intensities, and the intensities are approximated on a basis of cubic splines.2

3.3

The fully model-based estimators

Following Chambless and Diao’s7 and Zheng et al.’s11 idea, a model-based estimator of sensitivity may be computed using maximum likelihood estimator (MLE) or maximum penalized likelihood estimator (MPLE) of the parameters from the illness-death model. By the Bayes theorem, we know that R þ1 PðTi t, i ¼ 1jmÞ fM ðmÞdm PðMi 4 c, Ti t, i ¼ 1Þ ¼ Rcþ1 Seðc, tÞ ¼ PðTi t, i ¼ 1Þ 1 PðTi t, i ¼ 1jmÞ fM ðmÞdm where fM ð:Þ is the density of M. Both the numerator and the denominator may be estimated as the ^ i t, i ¼ 1jMi Þ. Thus, the fully model-based sample mean of the predicted probabilities PðT estimator of the sensitivity is defined as P ^ i:Mi 4c PðTi t, i ¼ 1jMi Þ b tÞ ¼ P Seðc, ^ i PðTi t, i ¼ 1jMi Þ where ^ i t, i ¼ 1jMi Þ ¼ PðT

Z

t

P^ 00 ð0, sÞ^ 01 ðsjMi Þds

ð8Þ

0

By the same way, the specificity estimator according to definition (1) is P ^ i:Mi c PðTi 4 tjMi Þ c ðc, tÞ ¼ P ^ i 4 tjMi Þ ¼ P^ 00 ð0, tÞ where PðT Sp ^ i PðTi 4 tjMi Þ and, according to definition (2) of specificity, we have h i P ^ i t, i ¼ 1jMi Þ 1 PðT i:Mi c c tÞ ¼ i Spðc, P h ^ i t, i ¼ 1jMi Þ 1 PðT i ^ i t, i ¼ 1jMi Þ ¼ PðfT ^ i 4 tg [ fTi t, i ¼ 2gjMi Þ which is estimated using since 1 PðT equation (8). These estimators handle interval censoring and, being based on MLE or MPLE of jk ðtjMi Þ, they are robust to marker-dependent censoring. Moreover, they do not require distributional assumptions for the marker. However, they suffer from drawbacks of all the ROC curve estimators based on parametric or semiparametric models:15 they are sensitive to model misspecification. This is a crucial issue as goodness-of-fit evaluation is very difficult for interval-censored data.

3.4

The imputation estimators

As an alternative, we propose an imputation approach. When the status of the subject at time t is known, the observation is used. When the status is unknown, we impute the conditional probability to be a case or a control given all the available information for the subject until time t and using estimates from the illness-death model.



8


As illustrated in Figure 2, when estimating the predictive accuracy of the marker M to predict disease onset by time t, the subject is known to be a case if Ri t and the subject is known to be a control if Li t (for both definitions of specificity). The disease status is unknown when Li 5 t 5 Ri . This corresponds to three kinds of subjects: (A) subjects healthy at the last visit before t and alive at t (Li 5 t, T~ 2i 4 t) (denoted A for alive), (D) subjects healthy at the last visit who died before t (Li 5 t, T~ 2i t, 2i ¼ 1) (denoted D for dead), (C) subjects healthy at the last visit and censored for death before t (Li 5 t, T~ 2i t, 2i ¼ 0) (denoted C for censored). As previously explained, in many cohort studies, time of death may be only censored by the end of the study and thus subjects of type C are not possible. Thus, in the following, we consider that the vital status is known for every subject at time t and develop formulas for imputation only for subjects of type A and D. Formulas for subjects of type C are given in Appendix 1. When computing sensitivity, the unobserved disease status is replaced by the imputed probability that the subject becomes diseased before time t given the observations, denoted by Pi ðtÞ P ^ i:Mi 4c ½1ðRi tÞ þ 1ðLi 5t5Ri Þ Pi ðtÞ b Seðc, tÞ ¼ P n ^ i¼1 ½1ðRi tÞ þ 1ðLi 5t5Ri Þ Pi ðtÞ where ^D P^ i ðtÞ ¼ 1ðT~ 2i 4 tÞ P^ A i ðtÞ þ 1ðT~ 2i t, 2i ¼1Þ Pi ðtÞ

ð9Þ

For a subject of type A, the conditional probability to develop the disease between Li (the last visit before t) and t given the subject was healthy at Li and alive at t is estimated by ^ P^ A i ðtÞ ¼ PðT1i tjT1i 4 Li , T2i 4 tÞ P^ 01 ðLi , tÞ ¼ P^ 00 ðLi , tÞ þ P^ 01 ðLi , tÞ

ð10Þ

Indeed, PA i ðtÞ is the probability to become ill between Li and t divided by the sum of the probabilities of the two possible trajectories between Li and t for a subject healthy at Li and alive at t. For a subject of type D, the conditional probability to develop the disease between Li and T~ 2i given the subject was healthy at Li and died at T~ 2i is estimated by ^ ~ ~ P^ D i ðtÞ ¼ PðT1i T2i jT1i 4 Li , T2i ¼ T2i Þ P^ 01 ðLi , T~ 2i Þ^ 12 ðT~ 2i jMi Þ ¼ P^ 00 ðLi , T~ 2i Þ^ 02 ðT~ 2i jMi Þ þ P^ 01 ðLi , T~ 2i Þ^ 12 ðT~ 2i jMi Þ

ð11Þ

For estimating the specificity according to definition (1), subjects who died before t are not used. Thus, we have only to impute the probability to be healthy at t for subjects censored for the disease before t and alive at t (subjects A). The estimator is given by P Â i:Mi 5c ½1ðLi tÞ þ 1ðLi 5t,T~ 2i 4 tÞ ð1 Pi ðtÞÞ c ðc, tÞ ¼ P Sp n Â i¼1 ½1ðLi tÞ þ 1ðLi 5t,T~ 2i 4 tÞ ð1 Pi ðtÞÞ




9

Figure 2. Five subjects with different patterns according to the information available at time t for interval censored data (the shadowed area is the period of time before t where the subject may have developed the disease without being observed).

where P^ A i ðtÞ is given by equation (10). For the specificity as defined in equation (2), for subjects A, we impute the probability 1 P^ A i ðtÞ to be healthy at time t and, for subjects D, we impute the probability 1 P^ D ðtÞ to be free of the i disease at the time of death. The estimator is given by P ^ i:Mi 5c ½1ðLi tÞ þ 1ðLi 5t5Ri Þ ð1 Pi ðtÞÞ c tÞ ¼ P Spðc, n ^ i¼1 ½1ðLi tÞ þ 1ðLi 5t5Ri Þ ð1 Pi ðtÞÞ where P^ i ðtÞ is computed by equations (9)–(11). This imputation approach makes the best use of all the available information at time t and the model is used only to predict the status of the subject after the censoring time given the observations. For comparison, in the previous approach, the ROC curve estimation is fully model based while in the nonparametric IPCW approach the censored subjects are used only for estimating the censoring probability (even when they are censored late before the date of interest t) and interval censoring cannot be handled rigorously. For both, the fully model based and the imputation estimators, AUC ðtÞ and AUCðtÞ are estimated b Sp c , Sp, c AUC d , by numerical integration under the corresponding ROC curve and the variances of Se, d may be estimated by bootstrap. However, the bootstrap approach is computationally and AUC expensive since the illness-death model must be estimated on each bootstrap sample.

4 Simulation study The aim of the simulation study was to compare the behavior of the three AUC(t) estimators with semicompeting risks and interval censoring. When the model is correct, the fully model-based and imputation estimators are expected to be less biased since they rigorously account for interval censoring



10


but we are interested in evaluating the bias of the IPCW estimator according to the interval between visits. When the model is misspecified, we aim at comparing the bias of the three estimators.

4.1

Design

To mimic the digit symbol substitution test (DSST) analyzed in Section 5, the marker was generated according to a Gaussian distribution rounded to the closest integer. The mean and variance were those of the DSST in the Paquid sample (mean ¼ 27.45 and standard deviation ¼ 11.29). The maximum follow-up time was 12 years with visits either every year, 2 years, or 4 years. For each set of parameters, we simulated 1000 samples of n ¼ 200 subjects. Times of dementia and times to death were generated according to the illness-death model (Figure 1). In the well-specified case, proportional intensity models with Weibull baseline intensities were simulated for the three transitions jk ðtjMi Þ ¼ 0jk ðtÞ expðjk Mi Þ with j ¼ 0, 1, k ¼ 1, 2 and j 6¼ k

ð12Þ

Three sets of regression parameters were used in the well-specified case: 1a. 01 ¼ 0:08, 02 ¼ 0:03, and 12 ¼ 0:02: As observed in Paquid, the risk of dementia and the risk of death among healthy subjects decreased for high values of the marker at baseline while the risk of death among demented subjects increased for high marker values at baseline. 1b. 01 ¼ 0:08, 02 ¼ 0:03, and 12 ¼ 0:02: Unlike Paquid, the risk of death among demented decreases for increasing values of the marker. 1c. 01 ¼ 0:08, 02 ¼ 0:06, and 12 ¼ 0:02: This is similar to 1a but with a stronger association between the marker and the risk of death among healthy subjects. Two misspecified cases were investigated (1) Same parameters as 1a except that a 50–50 mixture model was used for the first transition with 01 ¼ 0 in the first half of the population and 01 ¼ 0:08 in the second half of the population. (2) Same as 1a but another covariate Zi Nð0, SD ¼ 11:29Þ rounded at the closest integer and correlated with the marker (corrðMi , Zi Þ ¼ 0:5) was added in the transition intensity models: jk ðtjMi , Zi Þ ¼ 0jk ðtÞ expðjk Mi þ jk Zi Þ with jk ¼ jk =2. In any of the considered cases, time to the first event was generated according to the method described in Beyersmann et al.18 (p. 44) and then the time to death was generated for all the demented subjects. The right censoring time for dementia was uniformly distributed between 0 and 30 and the time of dementia was then interval censored by the times of visits. Death time was only censored by the end of the study (C2i ¼ 12). The AUC for the ROC(t ¼ 12) curve was estimated using the three methods and the two definitions of specificity (denoted AUC ð12Þ and AUC(12)). The imputation and fully model-based estimators were computed using the MLE of the illness-death model with proportional transition intensities defined by equation (12) and Weibull baseline intensities including only the marker under study as explanatory variable whatever the model used for data generation. Estimations were also performed with MPLE of the illness-death model but the results were very similar to those obtained with MLE and are not presented. For the IPCW estimator calculation, a subject who died without dementia diagnosis less than 2 years after the last visit was considered as death without dementia and if the time




11

between the last visit and the death was longer than this window (denoted w thereafter), this subject was considered as censored at the last visit. When the interval between visits was 4 years, we compared IPCW estimates obtained using a window w ¼ 2 years and a window w ¼ 4 years for classifying subjects deceased without dementia diagnosis. As the true AUC was unknown, the bias was computed with respect to the ideal estimator computed on the uncensored data.

4.2

Results

d ð12Þ Tables 1, 2, and 3 display the simulation results (relative bias and standard deviation for AUC d and AUCð12Þ) for, respectively, the well-specified case (1a, 1b, 1c), the misspecified case using a mixture model (2), and the misspecified case due to a missing covariate (3). With the chosen values for the Weibull baseline intensity functions and the set of parameters (1a), about 56% of the simulated subjects developed the disease during the 12 years of follow-up, about 35% died with the disease and 24% died without the disease, and 20% were alive and nondemented at t ¼ 12. Due to right and interval censoring, the proportion of demented subjects who died before diagnosis were, respectively, about 7, 14, and 27% when the visits were every year, 2 years, or 4 years. When the model is well specified (Table 1), the imputation and fully model-based estimators performed well and gave slightly less biased estimates than the IPCW estimator. For AUC , the IPCW estimator has negligible bias but its standard deviation is slightly larger than those of the parametric estimators. As expected, the fully model-based one has the smallest variance. For AUC, the bias of the IPCW estimator remains small (less than 4% for w ¼ 2 years). The parametric estimators are unbiased but their variance may increase for the largest interval between visits (4 years). These simulations highlight that IPCW underestimates AUC when the window w is equal or larger than the interval between visits, with an increasing bias when the interval between visits enlarges. Indeed, when this interval increases, as the diagnosis may be assessed only at the visits, demented subjects are more likely to die before diagnosis. With the first definition of specificity, Sp , subjects who died without diagnosis contribute only to the computation of the weights (the probabilities to be uncensored). Thus, the between-visits time and w have little impact on IPCW estimates of AUC . On the other hand, for the second definition of specificity, Sp, subjects who died without diagnosis are either considered as control (dead without dementia if seen in the last w years) or as censored at the last visit (if not seen in the last w years). When the window w increases, these subjects are more frequently classified as controls. When w is larger than the between-visits interval, the time between the last visit and the date of death is always lower than w (except for some of the right censored subjects) and almost all these subjects are classified as controls although some of them are demented, leading to a systematic bias. When w is shorter than the between-visits interval, the misclassification may be in different directions (cases considered as controls or as censored and controls considered as censored) and thus the bias is expected to be lower but it is not possible to anticipate if the AUC will be under or over-estimated. When the fully model-based estimator is computed with a standard proportional intensity model while data are generated according to a mixture model (Table 2), it is clearly more biased than the others whatever the frequency of the visits (for w ¼ 2). As expected, the imputation estimator is less biased than the fully model-based one. The bias for the imputation estimator is similar or slightly larger than IPCW with w ¼ 2; its standard deviation is always larger for AUC and lower for AUC . In situation (3), the misspecification due to an omitted quantitative covariate correlated to Mi is less severe and the bias for the two parametric estimators is very low (Table 3). The behavior of the three estimators in this case is similar to the well-specified case (Table 1).



12


Table 1. Simulation results for the well-specified model. Data are generated with proportional intensity illness-death models. The imputation and fully model-based estimators are estimated using the same model (1000 replications). AUC ð12Þ Visits every

d AUC

Estimator

Case (1a): ð01 , 02 , 12 Þ ¼ ð0:08, 1 year Ideal IPCW (w ¼ 2*) Imputation Model based 2 years Ideal IPCW (w ¼ 2*) Imputation Model based 4 years Ideal IPCW (w ¼ 2*) IPCW (w ¼ 4*) Imputation Model based Case (1b): ð01 , 02 , 12 Þ ¼ ð0:08, 1 year Ideal IPCW (w ¼ 2*) Imputation Model based 2 years Ideal IPCW (w ¼ 2*) Imputation Model based 4 years Ideal IPCW (w ¼ 2*) IPCW (w ¼ 4*) Imputation Model based Case (1c): ð01 , 02 , 12 Þ ¼ ð0:08, 1 year Ideal IPCW (w ¼ 2*) Imputation Model based 2 years Ideal IPCW (w ¼ 2*) Imputation Model based 4 years Ideal IPCW (w ¼ 2*) IPCW (w ¼ 4*) Imputation Model based

Bias (%)

0:03, 0:02Þ 0.832 0.835 0.4 0.832 0 0.835 0.4 0.831 0.837 0.7 0.830 –0.1 0.834 0.3 0.831 0.835 0.5 0.842 1.4 0.830 –0.2 0.832 0.2 0:03, 0:02Þ 0.837 0.836 –0.1 0.837 0 0.837 0.1 0.837 0.835 –0.3 0.837 0 0.837 0 0.837 0.827 –1.2 0.832 –0.6 0.836 0 0.837 0 0:06, 0:02Þ 0.800 0.804 0.4 0.800 0 0.814 1.8 0.800 0.807 0.9 0.801 0.1 0.813 1.7 0.800 0.806 0.8 0.811 1.4 0.800 0.1 0.811 1.4

AUC(12) SD

d AUC

0.035 0.042 0.034 0.026 0.035 0.044 0.035 0.027 0.035 0.046 0.044 0.037 0.031

0.741 0.724 0.742 0.744 0.741 0.713 0.740 0.742 0.741 0.747 0.693 0.739 0.740

0.035 0.035 0.034 0.024 0.034 0.035 0.034 0.024 0.034 0.037 0.036 0.035 0.026

0.746 0.733 0.747 0.746 0.746 0.721 0.747 0.746 0.746 0.751 0.699 0.747 0.746

0.035 0.035 0.035 0.025 0.035 0.035 0.036 0.026 0.035 0.037 0.036 0.039 0.030

0.731 0.714 0.732 0.741 0.731 0.703 0.732 0.741 0.731 0.743 0.684 0.729 0.735

*Window between last visit and date of death used to classify subjects who died without diagnosis.


Bias (%)

–2.3 0.1 0.4 –3.8 –0.2 0.1 0.8 –6.5 –0.2 –0.1

–1.7 0.1 0.1 –3.3 0.1 0 0.7 –6.4 0.1 0

–2.3 0 1.3 –3.9 0.1 1.3 1.7 –6.5 –0.3 0.6

SD 0.035 0.039 0.039 0.036 0.035 0.041 0.045 0.042 0.035 0.045 0.041 0.056 0.054 0.035 0.035 0.036 0.032 0.035 0.036 0.038 0.033 0.035 0.039 0.036 0.043 0.039 0.037 0.037 0.043 0.040 0.037 0.037 0.048 0.046 0.037 0.039 0.037 0.060 0.059



13

Table 2. Simulation results for the misspecified model (2). Data are generated with a mixture of illness-death models while the imputation and fully model-based estimators used the proportional intensity illness-death model (1000 replications). AUC ð12Þ Visits every 1 year

2 years

4 years

Estimator

d AUC

Ideal IPCW (w ¼ 2*) Imputation Model based Ideal IPCW (w ¼ 2*) Imputation Model based Ideal IPCW (w ¼ 2*) IPCW (w ¼ 4*) Imputation Model based

0.769 0.771 0.758 0.741 0.769 0.775 0.754 0.735 0.768 0.784 0.785 0.752 0.735

Bias (%) 0.3 –1.4 –3.5 0.9 –1.8 –4.3 2.1 2.2 –2.1 –4.2

AUC(12) SD

d AUC

0.049 0.062 0.051 0.043 0.049 0.063 0.051 0.043 0.050 0.064 0.063 0.052 0.045

0.657 0.635 0.638 0.616 0.657 0.628 0.630 0.610 0.657 0.672 0.617 0.628 0.617

Bias (%) –3.3 –2.9 –6.2 –4.5 –4.1 –7.1 2.3 –6.1 –4.5 –6.1

SD 0.044 0.047 0.056 0.058 0.044 0.046 0.064 0.066 0.044 0.056 0.044 0.088 0.088


Table 3. Simulation results for the misspecified model (3). Data are generated with proportional intensity illnessdeath models with an additional predictor which is missed for estimating the imputation and fully model-based estimators (1000 replications). AUC ð12Þ Visits every 1 year

2 years

4 years

Estimator

d AUC

Ideal IPCW (w ¼ 2*) Imputation Model based Ideal IPCW (w ¼ 2*) Imputation Model based Ideal IPCW (w ¼ 2*) IPCW (w ¼ 4*) Imputation Model based

0.841 0.846 0.844 0.849 0.841 0.849 0.844 0.848 0.841 0.848 0.855 0.842 0.845

Bias (%) 0.6 0.3 0.9 1.0 0.3 0.8 0.8 1.6 0.1 0.4

AUC(12) SD

d AUC

0.033 0.039 0.033 0.025 0.033 0.040 0.033 0.026 0.032 0.042 0.041 0.035 0.029

0.766 0.750 0.768 0.772 0.766 0.742 0.768 0.771 0.767 0.774 0.722 0.767 0.768

Bias (%) –2.1 0.2 0.7 –3.2 0.2 0.6 1.0 –5.8 0.1 0.3

SD 0.035 0.039 0.038 0.033 0.035 0.040 0.040 0.036 0.035 0.044 0.042 0.050 0.046


5 Application to dementia prediction The Paquid project is a population-based prospective cohort study launched in 1988 with the aim to study risk factors and natural history of cognitive aging and dementia. The sample included 3777 subjects aged 65 years and older living at home at baseline in southwest of France. Subjects were



14


visited at home at inclusion and thereafter every 2 or 3 years over 20 years by a trained psychologist and completed a questionnaire and a battery of cognitive tests as well as scales of autonomy. At each visit, dementia was assessed using a two-phase procedure with (i) a screening by the psychologist using standardized criteria for dementia and (ii) a clinical examination by a neurologist for all the subjects screened as positive at the first stage. For all subjects who died during the follow-up, the exact date of death was collected. The aim of the present analysis is to evaluate the ability of a cognitive test (the DSST) and a scale of autonomy (instrumental activity of daily living, IADL) measured at baseline to identify subjects who will develop dementia in the next 10 years. The DSST measures attention and speed of information processing.19 Because of the time limitation for performing this test (90 s), the score in the Paquid sample ranges from 0 to 76 points, whereas the theoretical maximum is 90 points. IADL were assessed by the French version of Lawton’s scale.20 Four of the eight instrumental activities assessed by this scale shown to be associated with cognitive performance and free of confounding by sex were considered (telephone use, transportation, medication, and domestic finances).21 A score was calculated by summing the grades of dependency (1–4) for these four activities. The score ranges from 4 to 16 with 4 meaning no dependency and 78.7% of the sample has a score of 4 at inclusion. We excluded from the sample subjects demented, blind, deaf, or confined to bed at the initial visit, and those who did not complete the cognitive evaluation at baseline leading to a sample of 2795 subjects. For this analysis, we used only information on dementia diagnosis and death collected until the 10 year follow-up (at visits T1, T3, T5, T8, and T10). During the first 10 years of follow-up, 265 participants were diagnosed as demented, 1015 subjects died before dementia diagnosis, and 1515 subjects were alive at T10 and not demented at their last visit. Figure 3 displays the ROC(10) curves estimated for the two markers by IPCW (with a window w ¼ 2 years), imputation, and fully model-based estimators (based on the illness-death model with proportional intensities and Weibull baseline intensities) for the two definitions of specificity. For the DSST score and for the two definitions of specificity, the three estimators lead to very close ROC curves (Figure 2). For the IADL score, the ROC curves estimated by IPCW and imputation are similar but we observe slight discrepancies with the fully model-based estimators. Indeed, due to the log-linearity assumption, the fully model-based estimator assumes a strictly concave shape for the ROC curve whereas the log-linear assumption for the effect of IADL on both dementia and death is probably not valid. This illustrates the better robustness of the imputation estimator. Table 4 presents the estimated AUC(10) and standard error (computed by bootstrap for the two d is always higher than AUC. d Indeed, for the two markers, parametric estimators). As expected, AUC the associations marker-dementia and marker-death are of the same sign: high values of DSST are associated with a lower risk of both dementia (^01 ¼ 0.08, se ¼ 0.007) and death without dementia (^02 ¼ 0:03, se ¼ 0.004) while high IADL scores are associated with an increased risk of dementia (^01 ¼ 0:25, se ¼ 0.03) and death without dementia (^02 ¼ 0:27, se ¼ 0.02). Thus, with the two markers, the discrimination between future demented subjects and subjects who will die without dementia is more difficult than the discrimination between future demented subjects and subjects who will remain alive and free of dementia for 10 years. As a conclusion, DSST clearly exhibits a better ability than IADL score alone to identify subjects at high risk of dementia in the next 10 years. In addition, although the number of deaths without diagnosis is large and the time elapsed between the last visit and death is quite long (mean time ¼ 2.7 years, inter-quartile range: 1.0–3.4), ROC curve estimates with IPCW and imputation are very close.




15

6 Discussion Although interval censoring and semicompeting risks are frequent in cohort studies, at our knowledge this is the first time the impact of these two issues on ROC curve estimation is investigated. As for estimating incidence of dementia or its association with risk factors, rigorous handling of these two issues for estimating ROC curve requires estimation of an illness-death model by maximization of the likelihood accounting for interval censoring.2 We developed formulae for computing illness-death model based estimators of the ROC curves. However, for standard right censored data, simulation studies have shown that fully model-based estimation of ROC curves may be biased for misspecified models.15 As an alternative, we proposed to use the model only for imputing the missing information conditionally on the observed one. We compared these two estimators with the nonparametric IPCW estimator, previously proposed to deal with right censoring and competing risks, using a simple rule to impute the status of subjects who died before the diagnosis of the disease.

(a)

(b)

Figure 3. Estimated ROC curves for DSST and IADL at t ¼ 10 years for definition (1) of specificity (discrimination from nondemented survivors, panel (a)) and for definition (2) of specificity (discrimination in the whole population, panel (b)). For all the curves, the AUC is larger in panel (a). Paquid, n ¼ 2795.

Table 4. Estimated AUC(10) (with standard error) for DSST and IADL scores with the two definitions of the specificity and the three estimators (Paquid sample, n ¼ 2795). Marker

Estimator

d AUC

SE

d AUC

SE

DSST

IPCW Imputation Model based

0.758 0.741 0.753

0.016 0.016 0.015

0.713 0.699 0.706

0.017 0.018 0.017

IADL

IPCW Imputation Model based

0.608 0.599 0.576

0.015 0.014 0.011

0.575 0.565 0.545

0.015 0.015 0.011



16


Our first result is that the bias for IPCW is modest and it may be less than the two parametric estimators when the model is misspecified. However, the IPCW behavior depends on the imputation rule for the subjects who died before diagnosis. They are considered as deceased without the disease if they have been seen in a specified window of time before the death. Otherwise, they are considered as right censored at their last visit. We showed this window must be smaller than the interval between visits to avoid systematic error of classification of subjects dying quickly after disease onset who would be classified as dead without the disease. As expected, the imputation estimator is more robust to model misspecification than the fully model-based one. As a conclusion, we recommend to use first the IPCW estimator with the simple imputation rule discussed earlier because IPCW is a nonparametric estimator with well demonstrated asymptotic properties. Indeed, variance and confidence interval estimators as well as a test for comparison of AUCs were previously proposed with this approach.12 These methods are available in the R package timeROC (http://cran.r-project.org/web/packages/timeROC/index.html). Then a sensitivity analysis of the estimates may be performed thanks to the imputation estimator that may be computed from the R packages SmoothHazard (http://cran.r-project.org/web/packages/SmoothHazard/index.html) and timeROCIntCens (to be finalized). The two main weaknesses of the imputation estimator are that the bootstrap variance estimator is very time consuming and, even if it appears robust, it relies on parametric assumptions from the illness-death model which are not easy to check. Nevertheless, for a sensitivity analysis, variance estimation is not required. On the other hand, some parametric assumptions of the illness-death model may be relaxed using the MPLE for more flexible baseline transition intensities or using a nonlinear function of Mi to relax the log-linear assumption. However, the proportional intensity assumption remains. In the future, development of nonparametric estimators of the ROC curve for competing risks and interval-censored data would be useful. Although substantial research advances would be required, some extensions of recent work by Frydman and Szarek22 and Frydman et al.23 could be an interesting direction for such future developments. Funding This work was funded by a grant from France Alzheimer awarded to He´le`ne Jacqmin-Gadda in 2009. The PAQUID study is funded by IPSEN and Novartis laboratories.

References 1. Dartigues JF, Gagnon M, Barberger-Gateau P, et al. The Paquid epidemiological program on brain ageing. Neuroepidemiology 1992; 11: 14–18. 2. Joly P, Commenges D, Helmer C, et al. A penalized likelihood approach for an illness-death model with interval-censored data: Application to agespecific incidence of dementia. Biostatistics 2002; 3: 433–443. 3. Aisen P, Andrieu S, Sampaio C, et al. Report of the task force on designing clinical trials in early (predementia) Alzheimer disease. Neurology 2011; 76: 280–286. 4. Pepe M. The statistical evaluation of medical tests for evaluation and prediction. Oxford: Oxford University Press, 2003. 5. Heagerty P and Zheng Y. Survival model predictive accuracy and ROC curves. Biometrics 2005; 61: 92–105. 6. Heagerty P, Lumley T and Pepe M. Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics 2000; 56: 337–344.

7. Chambless LE and Diao G. Estimation of time-dependent area under the roc curve for long-term risk prediction. Stat Med 2006; 25: 3474–3486. 8. Hung H and Chiang C. Optimal composite markers for time-dependent receiver operating characteristic curves with censored survival data. Scand J Stat 2010; 37: 664–679. 9. Uno H, Cai T, Tian L, et al. Evaluating prediction rules for t-year survivors with censored regression models. J Am Stat Assoc 2007; 102: 527–537. 10. Saha P and Heagerty P. Time-dependent predictive accuracy in the presence of competing risks. Biometrics 2010; 66: 999–1011. 11. Zheng Y, Cai T, Jin Y, et al. Evaluating prognostic accuracy of biomarkers under competing risks. Biometrics 2012; 68: 388–396. 12. Blanche P, Dartigues J and Jacqmin-Gadda H. Estimating and comparing time-dependent areas under roc curves for censored event times with competing risks. Stat Med 2013; 32: 5381–5397.




17

13. Li J and Ma S. Time-dependent roc analysis under diverse censoring patterns. Stat Med 2011; 30: 1266–1277. 14. Song X and Zhou XH. A semiparametric approach for the covariate specific roc curve with survival outcome. Stat Sin 2008; 18: 947–965. 15. Viallon V and Latouche A. Discrimination measures for survival outcomes: Connection between the AUC and the predictiveness curve. Biometrical J 2011; 53: 217–236. 16. Blanche P, Dartigues JF and Jacqmin-Gadda H. Review and comparison of ROC curve estimators for a timedependent outcome with marker-dependent censoring. Biometrical J 2013; 55: 687–704. 17. Touraine C, Helmer C and Joly P. Predictions in an illnessdeath model. Stat Methods Med Res 2013. Epub ahead of print 22 May 2013. DOI: 10.1177/0962280213489234. 18. Beyersmann J, Allignol A and Schumacher M. Competing risks and multistate models with R. New York: Springer. 19. Wechsler D. Wechsler adult intelligence scale. rev. New York: Psychological Corporation, 1981.

20. Lawton MP and Brody BE. Assessment of older people: Self-maintaining and instrumental activities of daily living. Gerontologist 1969; 9: 179–186. 21. Barberger-Gateau P, Commenges D, Gagnon M, et al. Instrumental activities of daily living as a screening tool for cognitive impairment and dementia in elderly community dwellers. J Am Geriatr Soc 1992; 40: 1129–1134. 22. Frydman H and Szarek M. Nonparametric estimation in a Markov illness-death process from interval censored observations with missing intermediate transition status. Biometrics 2009; 65: 143–151. 23. Frydman H, Gerds T, Gron R, et al. Nonparametric estimation in an illness-death model when all transition times are interval censored. Biometrical J 2013; 55: 823–843.

Appendix 1: Imputation probabilities when Li 5 t 5 Ri , T~ 2i 5 t and 2i ¼ 0 When the time of death may be censored before the end of the study, subjects may be undiagnosed and right censored for death before the time of interest t: Li 5 t 5 Ri , T~ 2i t and 2i ¼ 0 (subjects of type C). In this context, three kinds of subjects (A, D, and C) have an unknown status at t and thus formula (9) for the imputed probability to be a case must be replaced by ^D ^C P^ i ðtÞ ¼ 1ðT~ 2i 4 tÞ P^ A i ðtÞ þ 1ðT~ 2i t, 2i ¼1Þ Pi ðtÞ þ 1ðT~ 2i t, 2i ¼0Þ Pi ðtÞ

ð13Þ

P^ C i ðtÞ is the estimated probability to develop the disease before t given that the subject was undiagnosed at Li and censored for death at T~ 2i and it is given by ^ ~ P^ C i ðtÞ ¼ PðT1i tjT1i 4 Li , T2i 4 T2i Þ Rt P^ 00 ð0, Li Þ½P^ 01 ðLi , tÞ þ T~ 2i P^ 01 ðLi , sÞ^ 12 ðsjMi Þds ¼ P^ 00 ð0, Li Þ½P^ 00 ðLi , T~ 2i Þ þ P^ 01 ðLi , T~ 2i Þ Rt P^ 01 ðLi , tÞ þ T~ 2i P^ 01 ðLi , sÞ^ 12 ðsjMi Þds ¼ P^ 00 ðLi , T~ 2i Þ þ P^ 01 ðLi , T~ 2i Þ

ð14Þ

For computing Sp , the imputation probability is the estimated probability to be free of any event given the data ^ 1i 4 t, T2i 4 tjT1i 4 Li , T2i 4 T~ 2i Þ PðT P^ 00 ðLi , tÞ ¼ P^ 00 ðLi , T~ 2i Þ þ P^ 01 ðLi , T~ 2i Þ

ð15Þ

For computing Sp, the imputation probability is the estimated probability to be free of any event or to die without the disease before t which is given by 1 P^ C i ðtÞ.


How to read a receiver operating characteristic curve.

Fully non-parametric receiver operating characteristic curve estimation for random-effects meta-analysis.

Receiver operating characteristic curve generalization for non-monotone relationships.

Editorial: Radiology and the receiver operating characteristic (ROC) curve.

Weighted Area Under the Receiver Operating Characteristic Curve and Its Application to Gene Selection.

The precision--recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases.

RE: "RECEIVER OPERATING CHARACTERISTIC CURVE INFERENCE FROM A SAMPLE WITH A LIMIT OF DETECTION".

Binary classification using multivariate receiver operating characteristic curve for continuous data.

A Linear Regression Framework for the Receiver Operating Characteristic (ROC) Curve Analysis.

CT.

Receiver operating characteristic curves: a basic understanding.

The average receiver operating characteristic curve in multireader multicase imaging studies.

Evaluation of the predictive performance of nutritional indicators by receiver-operating characteristic curve analysis.

Jackknife variance of the partial area under the empirical receiver operating characteristic curve.

Effects of 'matched filter' smoothing as measured by receiver operating characteristic curve.

Use of the receiver operating characteristic curve to evaluate sensitivity, specificity, and accuracy of methods for detection of peaks in hormone time series.

Oxidative status in different settings and with different methodological approaches compared by Receiver Operating Characteristic curve analysis.

Receiver Operating Characteristic curve analysis determines association of individual potato foliage volatiles with onion thrips preference, cultivar and plant age.

Generalization of the receiver-operating characteristic curve to determine the normal hemoglobin range cutoff points in pregnant women.

Comparison of semiparametric receiver operating characteristic models on observer data.

Understanding receiver-operating-characteristic curves: a graphic approach.

Study on diagnosis criteria of fire-heat syndrome based on receiver operating characteristic curve and principal component analysis.

Joint modeling approach for semicompeting risks data with missing nonterminal event status.

Evidence Based Emergency Medicine; Part 5 Receiver Operating Curve and Area under the Curve.