Research Improved ischemic stroke outcome prediction using model estimation of outcome probability: The THRIVE-c calculation Alexander C. Flint1*, Vivek A. Rao1, Sheila L. Chan1, Sean P. Cullen1, Bonnie S. Faigeles1, Wade S. Smith2, Philip M. Bath3, Nils Wahlgren4, Niaz Ahmed4, Geoff A. Donnan5, and S. Claiborne Johnston6 on behalf of the SITS International and VISTA-plus investigators Background and purpose The Totaled Health Risks in Vascular Events (THRIVE) score is a previously validated ischemic stroke outcome prediction tool. Although simplified scoring systems like the THRIVE score facilitate ease-of-use, when computers or devices are available at the point of care, a more accurate and patient-specific estimation of outcome probability should be possible by computing the logistic equation with patientspecific continuous variables. Methods We used data from 12 207 subjects from the Virtual International Stroke Trials Archive and the Safe Implementation of Thrombolysis in Stroke – Monitoring Study to develop and validate the performance of a model-derived estimation of outcome probability, the THRIVE-c calculation. Models were built with logistic regression using the underlying predictors from the THRIVE score: age, National Institutes of Health Stroke Scale score, and the Chronic Disease Scale (presence of hypertension, diabetes mellitus, or atrial fibrillation). Receiver operator characteristics analysis was used to assess model performance and compare the THRIVE-c model to the traditional THRIVE score, using a two-tailed Chi-squared test. Results The THRIVE-c model performed similarly in the randomly chosen development cohort (n = 6194, area under the curve = 0·786, 95% confidence interval 0·774–0·798) and validation cohort (n = 6013, area under the curve = 0·784, 95% confidence interval 0·772–0·796) (P = 0·79). Similar performance was also seen in two separate external validation Correspondence: Alexander C. Flint*, Department of Neuroscience, Kaiser Permanente, Redwood City, 1150 Veterans Blvd, Redwood City, CA 94025, USA. E-mail: [email protected] Twitter: @neuroicudoc 1 Department of Neuroscience, Kaiser Permanente, Redwood City, CA, USA 2 Department of Neurology, University of California San Francisco, San Francisco, CA, USA 3 Division of Stroke, University of Nottingham, Nottingham, UK 4 Department of Clinical Neurosciences, Karolinska Institutet, Karolinska University Hospital, Stockholm, Sweden 5 Florey Institute of Neuroscience and Mental Health, University of Melbourne, Melbourne, Australia 6 Dell Medical School, University of Texas, Austin, TX, USA Received: 27 April 2015; Accepted: 25 February 2015; Published online 04 Jun 2015 Conflict of interest: None declared. Author contributions statement: ACF conceived the study. ACF and SCJ provided statistical guidance, and ACF performed all statistical work for the study. NW and NA are members of the SITS International group, one of the data sources for the present study, and represent the SITS International investigators on this project. PMB is a member of the VISTA-Plus group, the other data source for the present study, and represents the VISTA-Plus investigators on this project. ACF drafted the manuscript and all authors contributed substantially to its revision. ACF takes responsibility for the paper as a whole. DOI: 10.1111/ijs.12529 © 2015 World Stroke Organization
cohorts. The THRIVE-c model (area under the curve = 0·785, 95% confidence interval 0·777–0·793) had superior performance when compared with the traditional THRIVE score (area under the curve = 0·746, 95% confidence interval 0·737–0·755) (P < 0·001). Conclusion By computing the logistic equation with patientspecific continuous variables in the THRIVE-c calculation, outcomes at the individual patient level are more accurately estimated. Given the widespread availability of computers and devices at the point of care, such calculations can be easily performed with a simple user interface. Key words: cerebral infarction, ischemic stroke, methodology, stroke, therapy, vascular events
Introduction The Totaled Health Risks in Vascular Events (THRIVE) score is a simple-to-use ischemic stroke outcome prediction tool based on clinical variables available at the time of stroke presentation [National Institutes of Health Stroke Scale (NIHSS), age, and presence of hypertension (HTN), diabetes mellitus (DM), or atrial fibrillation (AF)]. The THRIVE score was originally developed with data from the MERCI and Multi-MERCI trials (1) and has been validated in the Merci Registry (2), the National Institute of Neurological Disorders and Stroke (NINDS) tissue plasminogen activator (tPA) trial (3), the Virtual International Stroke Trials Archive (VISTA) (4), the Safe Implementation of Thrombolysis in Stroke-Monitoring Study (SITS-MOST), the TREVO-2 trial (5), and the SWIFT and STAR trials (6). The THRIVE score trichotomizes age and NIHSS scores, assigning 0, 1, or 2 points for trichotomized age (≤59, 60–79, or ≥80) and 0, 2, or 4 points for trichotomized NIHSS score (≤10, 11–20, or ≥21), and then combines these elements with a Chronic Disease Scale (CDS, 1 point each for presence of HTN, DM, or AF) (1). The composite THRIVE score (0–9 points) predicts increased chances of poor outcome [modified Rankin Scale (mRS) 3–6] or death by 90 days (1–6). The original THRIVE score is thus similar to many other clinical outcome prediction scores, such as the ABCD2 score (7) or the CHADS2 score (8), in that continuous variables such as age are dichotomized or trichotomized to facilitate score calculation. Such simplifications are reasonable if the typical clinician using a predictive score would not be expected to have ready access to a computer-based calculator, but these simplifications come with a potential loss of accuracy. For example, if one patient is 81 years old and another is 99 years old, they could have the same THRIVE score, because age is cut at 80 in the traditional THRIVE score. Therefore, in this example, the trichotomization of age in the THRIVE score would spuriously Vol 10, August 2015, 815–821
815
Research yield a similar estimation of outcome for these two patients with substantially different ages. The widespread availability of computers and Internetconnected devices at the point of care means that clinical prediction tools no longer have to be simplified by cutting continuous predictor variables like age. Instead, binary outcomes may be modeled using multivariable logistic regression using large datasets, and the obtained regression coefficients may be entered into the logistic equation along with individual subject data to yield an estimated probability of outcome (9). The user may enter the same data they would already enter to determine the traditional predictive score, and the computer can generate both the traditional score as well as the more precise estimation of outcome probability using continuous variables and the logistic equation. Here, we use a large cohort of subjects from the VISTA (10) and the SITS-MOST (11), to build and validate a tool for continuous prediction of outcome probability, the THRIVE-c calculation, and compare the performance of this tool to the traditional THRIVE score.
A. C. Flint et al. Statistical analysis Multivariable logistic regression was performed using standard techniques. For the THRIVE-c calculation, model-predicted probabilities were estimated using the logistic equation. Receiver operator characteristics (ROC) analysis for the discrete THRIVE score was performed as previously reported (2,4,6). ROC analysis of THRIVE-c logistic regression models was performed using standard post-estimation techniques. Statistical comparisons of ROC curve area under the curve (AUC, C-statistic) was performed using a two-tailed Chi-squared test as previously described (2,4,6). To perform separate analysis in a development and validation cohort, the overall dataset was randomized into two group of similar size (development cohort n = 6194, validation cohort n = 6013). Randomization at the individual subject level was performed using integer data derived from a true random source (http:// www.random.org). Patients with missing data on THRIVE components or outcomes were excluded from analysis. For determination of the THRIVE score and THRIVE-c across all permutations of age (40–90), NIHSS (0–42), and CDS (0–3), custom code was written in Python 2·7·8 (Python Software Foundation; http://python.org). All statistical analyses were performed using Stata MP version 12·1 (Stata Corp., College Station, TX).
Methods Data source and subjects We obtained demographic data, clinical data, three-month functional outcome on the mRS, and three-month mortality from VISTA (n = 5724) (10) and from the SITS-MOST study (n = 6483) (11). VISTA contains pooled data from multiple clinical trials (10), and SITS-MOST was a large, multicenter, openlabel post-market study of tPA administration for ischemic stroke (11). Performance of the original THRIVE score in VISTA and SITS-MOST has been previously described in detail (4,12). Institutional review board and regulatory approvals for both data sources were obtained, where appropriate, from the originating centers in the original data collection. For the present analysis, deidentified VISTA and SITS-MOST data were transmitted to the investigators via VISTA-plus (10). For confirmatory external validation of THRIVE-c calculation performance, we employed two datasets in which the THRIVE score has previously been validated: the NINDS tPA trial and the combined MERCI Trial, Multi MERCI Trial, and Merci Registry datasets, as previously described in detail (1–3). Measurements The traditional THRIVE score is calculated from age, initial stroke severity on the NIHSS score, and CDS (the presence or absence of HTN, DM, or AF). The THRIVE score assigns 1 point for age 60–79 years, 2 points for age ≥80 years, 2 points for NIHSS score 11–20, 4 points for NIHSS score ≥21, and 1 point for each CDS component (HTN, DM, and AF) (1). The THRIVE-c multivariable logistic regression models were constructed by entering continuous age, continuous NIHSS, and dummy variables with natural coding for CDS levels of 1, 2, and 3. Good outcome was defined, as before (1–6), as an mRS of 0–2 at 90 days post stroke, and poor outcome was defined as an mRS of 3–6.
816
Vol 10, August 2015, 815–821
Results Lack of interaction with tPA treatment Intravenous tPA administration in the VISTA-Acute dataset used in the present study occurred in 2398/5724 (41·9%), whereas the SITS-MOST study only included intravenous tPA-treated patients (6483/6483, 100%). In order to use the combined VISTA + SITSMOST datasets here, we first confirmed our prior observation that the relationship between THRIVE score and outcome does not interact with the relationship between tPA administration and outcome (3,4). As shown in Table 1, when tPA is added to a multivariable logistic regression model of THRIVE score predicting good outcome, the odds ratio for THRIVE score is not altered, confirming that THRIVE score and tPA are independent predictors of outcome and that there is no statistical interaction between these predictors.
Table 1 Lack of interaction between THRIVE and tPA in prediction of good outcome
Model 1: Good outcome THRIVE score Model 2: Good outcome THRIVE score tPA
Odds ratio
95% CI
P value
0·57
0·55–0·58
cohorts. The THRIVE-c model (area under the curve = 0·785, 95% confidence interval 0·777–0·793) had superior performance when compared with the traditional THRIVE score (area under the curve = 0·746, 95% confidence interval 0·737–0·755) (P < 0·001). Conclusion By computing the logistic equation with patientspecific continuous variables in the THRIVE-c calculation, outcomes at the individual patient level are more accurately estimated. Given the widespread availability of computers and devices at the point of care, such calculations can be easily performed with a simple user interface. Key words: cerebral infarction, ischemic stroke, methodology, stroke, therapy, vascular events
Introduction The Totaled Health Risks in Vascular Events (THRIVE) score is a simple-to-use ischemic stroke outcome prediction tool based on clinical variables available at the time of stroke presentation [National Institutes of Health Stroke Scale (NIHSS), age, and presence of hypertension (HTN), diabetes mellitus (DM), or atrial fibrillation (AF)]. The THRIVE score was originally developed with data from the MERCI and Multi-MERCI trials (1) and has been validated in the Merci Registry (2), the National Institute of Neurological Disorders and Stroke (NINDS) tissue plasminogen activator (tPA) trial (3), the Virtual International Stroke Trials Archive (VISTA) (4), the Safe Implementation of Thrombolysis in Stroke-Monitoring Study (SITS-MOST), the TREVO-2 trial (5), and the SWIFT and STAR trials (6). The THRIVE score trichotomizes age and NIHSS scores, assigning 0, 1, or 2 points for trichotomized age (≤59, 60–79, or ≥80) and 0, 2, or 4 points for trichotomized NIHSS score (≤10, 11–20, or ≥21), and then combines these elements with a Chronic Disease Scale (CDS, 1 point each for presence of HTN, DM, or AF) (1). The composite THRIVE score (0–9 points) predicts increased chances of poor outcome [modified Rankin Scale (mRS) 3–6] or death by 90 days (1–6). The original THRIVE score is thus similar to many other clinical outcome prediction scores, such as the ABCD2 score (7) or the CHADS2 score (8), in that continuous variables such as age are dichotomized or trichotomized to facilitate score calculation. Such simplifications are reasonable if the typical clinician using a predictive score would not be expected to have ready access to a computer-based calculator, but these simplifications come with a potential loss of accuracy. For example, if one patient is 81 years old and another is 99 years old, they could have the same THRIVE score, because age is cut at 80 in the traditional THRIVE score. Therefore, in this example, the trichotomization of age in the THRIVE score would spuriously Vol 10, August 2015, 815–821
815
Research yield a similar estimation of outcome for these two patients with substantially different ages. The widespread availability of computers and Internetconnected devices at the point of care means that clinical prediction tools no longer have to be simplified by cutting continuous predictor variables like age. Instead, binary outcomes may be modeled using multivariable logistic regression using large datasets, and the obtained regression coefficients may be entered into the logistic equation along with individual subject data to yield an estimated probability of outcome (9). The user may enter the same data they would already enter to determine the traditional predictive score, and the computer can generate both the traditional score as well as the more precise estimation of outcome probability using continuous variables and the logistic equation. Here, we use a large cohort of subjects from the VISTA (10) and the SITS-MOST (11), to build and validate a tool for continuous prediction of outcome probability, the THRIVE-c calculation, and compare the performance of this tool to the traditional THRIVE score.
A. C. Flint et al. Statistical analysis Multivariable logistic regression was performed using standard techniques. For the THRIVE-c calculation, model-predicted probabilities were estimated using the logistic equation. Receiver operator characteristics (ROC) analysis for the discrete THRIVE score was performed as previously reported (2,4,6). ROC analysis of THRIVE-c logistic regression models was performed using standard post-estimation techniques. Statistical comparisons of ROC curve area under the curve (AUC, C-statistic) was performed using a two-tailed Chi-squared test as previously described (2,4,6). To perform separate analysis in a development and validation cohort, the overall dataset was randomized into two group of similar size (development cohort n = 6194, validation cohort n = 6013). Randomization at the individual subject level was performed using integer data derived from a true random source (http:// www.random.org). Patients with missing data on THRIVE components or outcomes were excluded from analysis. For determination of the THRIVE score and THRIVE-c across all permutations of age (40–90), NIHSS (0–42), and CDS (0–3), custom code was written in Python 2·7·8 (Python Software Foundation; http://python.org). All statistical analyses were performed using Stata MP version 12·1 (Stata Corp., College Station, TX).
Methods Data source and subjects We obtained demographic data, clinical data, three-month functional outcome on the mRS, and three-month mortality from VISTA (n = 5724) (10) and from the SITS-MOST study (n = 6483) (11). VISTA contains pooled data from multiple clinical trials (10), and SITS-MOST was a large, multicenter, openlabel post-market study of tPA administration for ischemic stroke (11). Performance of the original THRIVE score in VISTA and SITS-MOST has been previously described in detail (4,12). Institutional review board and regulatory approvals for both data sources were obtained, where appropriate, from the originating centers in the original data collection. For the present analysis, deidentified VISTA and SITS-MOST data were transmitted to the investigators via VISTA-plus (10). For confirmatory external validation of THRIVE-c calculation performance, we employed two datasets in which the THRIVE score has previously been validated: the NINDS tPA trial and the combined MERCI Trial, Multi MERCI Trial, and Merci Registry datasets, as previously described in detail (1–3). Measurements The traditional THRIVE score is calculated from age, initial stroke severity on the NIHSS score, and CDS (the presence or absence of HTN, DM, or AF). The THRIVE score assigns 1 point for age 60–79 years, 2 points for age ≥80 years, 2 points for NIHSS score 11–20, 4 points for NIHSS score ≥21, and 1 point for each CDS component (HTN, DM, and AF) (1). The THRIVE-c multivariable logistic regression models were constructed by entering continuous age, continuous NIHSS, and dummy variables with natural coding for CDS levels of 1, 2, and 3. Good outcome was defined, as before (1–6), as an mRS of 0–2 at 90 days post stroke, and poor outcome was defined as an mRS of 3–6.
816
Vol 10, August 2015, 815–821
Results Lack of interaction with tPA treatment Intravenous tPA administration in the VISTA-Acute dataset used in the present study occurred in 2398/5724 (41·9%), whereas the SITS-MOST study only included intravenous tPA-treated patients (6483/6483, 100%). In order to use the combined VISTA + SITSMOST datasets here, we first confirmed our prior observation that the relationship between THRIVE score and outcome does not interact with the relationship between tPA administration and outcome (3,4). As shown in Table 1, when tPA is added to a multivariable logistic regression model of THRIVE score predicting good outcome, the odds ratio for THRIVE score is not altered, confirming that THRIVE score and tPA are independent predictors of outcome and that there is no statistical interaction between these predictors.
Table 1 Lack of interaction between THRIVE and tPA in prediction of good outcome
Model 1: Good outcome THRIVE score Model 2: Good outcome THRIVE score tPA
Odds ratio
95% CI
P value
0·57
0·55–0·58
