http://www.jhltonline.org
FEATURED PAPERS
Incremental and independent value of cardiopulmonary exercise test measures and the Seattle Heart Failure Model for prediction of risk in patients with heart failure Todd Dardas, MD, MS,a Yanhong Li, MD, MS,b Shelby D. Reed, PhD,b Christopher M. O’Connor, MD,b David J. Whellan, MD,c Stephen J. Ellis, PhD,b Kevin A. Schulman, MD,b William E. Kraus, MD,b Daniel E. Forman, MD,d and Wayne C. Levy, MDa From the aDivision of Cardiology, Department of Internal Medicine, University of Washington Seattle, Washington; bDuke Clinical Research Institute, Durham, North Carolina; cDivision of Cardiology, Department of Medicine, Jefferson University Hospitals, Philadelphia, Pennsylvania; and the dCardiovascular Division, Brigham and Women’s Hospital, Boston, Massachusetts.
KEYWORDS: heart failure; risk factors; exercise; Seattle Heart Failure Model; risk prediction
BACKGROUND: Multivariable risk scores and exercise measures are well-validated risk prediction methods. Combining information from a functional evaluation and a risk model may improve accuracy of risk predictions. We analyzed whether adding exercise measures to the Seattle Heart Failure Model (SHFM) improves risk prediction accuracy in systolic heart failure. METHODS: We used a sample of patients from the Heart Failure and A Controlled Trial Investigating Outcomes of Exercise TraiNing (HF-ACTION) study (http://www.clinicaltrials.gov; unique identifier: NCT00047437) to examine the addition of peak oxygen consumption, expired volume per unit time/ volume of carbon dioxide slope, 6-minute walk distance, or cardiopulmonary exercise duration to the SHFM. Multivariable Cox proportional hazards models were used to test the association between the combined end point (death, left ventricular assist device, or cardiac transplantation) and the addition of exercise variables to the SHFM. RESULTS: The sample included 2,152 patients. The SHFM and all exercise measures were associated with events (all p o 0.0001) in proportional hazards models. There was statistically significant improvement in risk estimation when exercise measures were added to the SHFM. However, the improvement in the C index for the addition of peak volume of oxygen consumption (þ0.01), expired volume per unit time/volume of carbon dioxide slope (þ0.02), 6-minute walk distance (–0.001), and cardiopulmonary exercise duration (þ0. 001) to the SHFM was small or slightly worse than the SHFM alone. Changes in risk assignment with the addition of exercise variables were minimal for patients above or below a 15% 1-year mortality. COnclusions: Exercise performance measures and the SHFM are independently useful for predicting risk in systolic heart failure. Adding cardiopulmonary exercise testing measures and 6MWD to the SHFM offers only minimal improvement in risk reassignment at clinically meaningful cut points. J Heart Lung Transplant 2015;34:1017–1023 r 2015 International Society for Heart and Lung Transplantation. All rights reserved.
Reprint requests: Todd Dardas, MD, MS, University of Washington Seattle, Division of Cardiology, 1959 NE Pacific St, Box 356422, Seattle, WA 98195-6422. Telephone: 206-997-8448. Fax: 206-616-4847. E-mail address: [email protected]
Risk models remain useful for normalizing risk among groups of patients in trials, risk-adjusting performance metrics, or as part of broader discussions of prognosis.1
1053-2498/$ - see front matter r 2015 International Society for Heart and Lung Transplantation. All rights reserved. http://dx.doi.org/10.1016/j.healun.2015.03.017
1018
The Journal of Heart and Lung Transplantation, Vol 34, No 8, August 2015
These applications demand that models are well validated, updated to modern samples, and contain variables considered relevant to patients and physicians. A wide array of options exist for determining prognosis in patients with systolic heart failure (HF). Transplant candidacy selection guidelines recommend assessment of each patient’s functional capacity with respect to activities of daily living and formal cardiopulmonary exercise testing (CPET).2 CPET measures have the advantage of being inexpensive and extensively validated and are important objective measures of patient-reported limitations that advanced heart failure therapies seek to remedy. However, CPET measures are not widely available, do not account for certain therapies explicitly (e.g., defibrillators), and cannot be performed by all patients. The 6-minute walk distance (6MWD) has been shown to be associated with prognosis and may be useful if a full CPET is not available.3,4 Guidelines and experts also recommend the use of the Heart Failure Survival Score, which includes peak oxygen consumption (VO2), in cases where exercise test results may be inconclusive.2,5,6 Risk models, such as the Seattle Heart Failure Model (SHFM), also offer utility by accounting explicitly for modern HF therapies and can be calculated for patients who are nonambulatory.7 We studied whether integrating CPET variables with the SHFM improved the accuracy of risk predictions in the ambulatory sample of patients enrolled in the Heart Failure and A Controlled Trial Investigating Outcomes of Exercise TraiNing (HF-ACTION) randomized trial (http://www. clinicaltrials.gov; unique identifier: NCT00047437). This analysis was motivated by the need to determine whether a multivariable risk score is equivalent to single measures of exercise tolerance and to determine whether adding exercise information to the SHFM might improve risk assessment at clinically meaningful cutoffs.
Methods The results of the HF-ACTION trial and the original derivation and validation of the SHFM have been previously reported.7,8 Briefly, the HF-ACTION trial enrolled ambulatory patients with systolic HF who could comply with exercise training. Patients were randomized 1:1 to usual care with an intensive, supervised exercise training program, followed by home-based training, or to usual care with printed information describing the benefits of exercise. The primary end point of time to death or all-cause hospitalization was not significantly reduced in the group assigned to exercise training (hazard ratio, 0.93; 95% confidence interval, 0.84–1.02). CPET was performed using a modified Naughton protocol.9 Peak VO2 was defined as the highest oxygen consumption for a given 15- or 20-second interval within the last 90 seconds of exercise or the first 30 seconds of recovery.10 The expired volume per unit time/volume of carbon dioxide slope (VE/VCO2) was recorded as the slope across the entire course of the exercise effort.10 The 6MWD was performed as previously described.11
SHFM time-to-event estimates The HF-ACTION trial collected all the variables necessary to derive the SHFM scores with the exception of allopurinol use, uric
acid, and percentage of lymphocytes. To handle these missing data, we assumed that patients were not receiving treatment with allopurinol and derived predicted estimates of uric acid levels and percentage of lymphocytes from multivariable linear regression models derived from the original SHFM cohort. Missing values for hemoglobin (24%), cholesterol (35%), and sodium (11%) were imputed using group means derived from HF-ACTION. SHFM scores were then calculated using the originally published β coefficients.7 Assessment of the SHFM model fit in the HFACTION sample was evaluated by comparing the observed vs predicted survival at specific time points based on the originallypublished equation: eðSHFM ScoreÞ
Survival ðt Þ¼ eðλtÞ
where λ ¼ 0.0405 is the baseline hazard function. The combined end point of death, left ventricular assist device (LVAD), or cardiac transplantation, regardless of transplant urgency, was used for all analyses. This end point differed from the main HF-ACTION trial end point of all-cause mortality or hospitalization.
Additive effect of exercise measures with methods accounting for censoring To evaluate the contributions of CPET measures and 6MWD to the SHFM, Cox proportional hazards models were used as the primary analytic method. The SHFM score, peak VO2, VE/VCO2, cardiopulmonary exercise (CPE) duration, and 6MWD were modeled as continuous variables. The SHFM scores and exercise measures were evaluated individually in separate univariable models. Cox models were also used to test for statistical interactions among the SHFM, CPET measures, and 6MWD to detect differential associations over ranges of the covariates. We report standard measures of model significance and the Akaike information criterion. Reclassification tables were created to describe the effect of risk reassignment for patients with and without events. Groups for reclassification tables were created according to SHFMpredicted 1-year mortality: 0% to 5%, 45% to 10%, 410% to 15%, and 4 15%, which provide clinically meaningful changes in risk assignment. Patients were then reassigned to groups based on 1-year mortality estimates derived from Cox models with the SHFM and the additional exercise measures. The tables assume that reclassification of patients with events to a higher risk group is appropriate. Conversely, reclassification of a patient with an event to a low risk is inappropriate. Participants were excluded from the tables if censored before the specified follow-up time. Net reclassification index (NRI) statistics and p-values were not reported based on published work, suggesting issues with dual hypothesis testing and p-value estimation.12–16 The effect of adding CPET variables or 6MWD to the SHFM was estimated using Harrell’s C-index, a measure of concordance probability in the presence of censoring.17 Analyses were performed using SAS 9.2 software (SAS Institute Inc, Cary, NC) and R software. Two of the authors (Y.L. and S.R.) had full access to the complete data. The Duke University Institutional Review Board approved the analyses.
Results HF-ACTION contained 2,331 patients. The current sample excluded 179 patients for the absence of minimal information required to calculate the SHFM score or when not all CPETderived variables were available. There were 2,152 patients in
Dardas et al.
Adding Exercise Test Measures to the SHFM
1019
the final sample, with 418 events during a median of 2.5 years of follow-up. At 1 year, 2.8% of the sample was censored, and at 3 years, 44.0% of the sample was censored. Demographic characteristics and outcomes are described in Table 1.
(p o 0.0001), and 6MWD (p o 0.0001) remained significant predictors of the composite end point, as did the SHFM (p o 0.0001 for all models). There were non-significant interactions between the SHFM and peak VO2 (p ¼ 0.79), CPE duration (p ¼ 0.42), and the 6MWD (p ¼ 0.35), suggesting that the association between events and the SHFM does not change over values of these exercise measures (i.e., even at low values of peak VO2, the SHFM retains an association with events). There was a significant interaction between the SHFM and VE/ VCO2 slope (p ¼ 0.04), suggesting a non-linear association between events, SHFM-predicted risk, and VE/VCO2 slopepredicted risk (interaction model coefficient estimates: SHFM VE/VCO2 ¼ –0.01, VE/VCO2 slope ¼ 0.04, SHFM ¼ 1.1). The Akaike information criteria for the univariable models and the combined models are reported in Table 2. The best model was the SHFM þ peak VO2. Reclassification tables are shown in Tables 3 and 4 for peak VO2 and VE/VCO2 and exercise duration. The C-index at 1 year was 0.72 for the SHFM, 0.67 for peak VO2, 0.66 for VE/VCO2 slope, 0.67 for CPE duration, and 0.63 for the 6MWD (Table 5). When combined with the SHFM, the 1-year C-indices of peak VO2 (0.73), VE/VCO2 slope (0.74), CPE duration (0.72) and the 6MWD (0.72) were minimally increased (Table 5, Figure 2).
SHFM fit in the HF-ACTION sample The Kaplan-Meier survival in the sample was 94.1% at 1 year and 78.8% at 3 years. Predicted survival for the entire sample using the SHFM was 93.8% at 1 year and 83.2% at 3 years (Figure 1).
Effect of adding exercise variables to the SHFM CPET measures of peak VO2 (1 ml/kg/min; hazard ratio [HR], 0.87; 95% confidence interval [CI] 0.85–0.89), VE/ VCO2 (1 unit slope; HR, 1.05; 95% CI 1.04–1.06), exercise duration (1 minute; HR, 0.86; 95% CI, 0.83–0.88), and the 6MWD (100 meters; HR, 0.67; 95% CI 0.61–0.73) were associated with occurrence of the composite outcome over the course of all available follow-up in separate univariable models (Table 2). When each CPET variable and 6MWD were added to the SHFM in separate Cox models, peak VO2 (p o 0.0001), VE/VCO2 slope (p o 0.0001), CPE duration Table 1
Demographic Characteristics and Outcomes
Variable
Mean
SD
Age NYHA Functional Classification Peak VO2, ml/kg/min VE/VCO2, slope CPE duration, min 6-minute walk distance, 100 m SHFM, sum of β coefficients
59.1 2.4 14.9 34.3 9.8 3.65 0.24
12.7 0.5 4.7 9 4 1.04 0.65
No.
%
Total sample Sex Female Male NYHA Functional Classification II III IV ACE inhibitor Angiotensin receptor blockade β-Blocker Aldosterone antagonist Biventricular pacemaker Implantable cardioverter-defibrillator Outcomes Death Transplant or LVAD
2,152 608 1,544
28 72
1,383 749 20 1,604 497 2,030 949 383 855
64 35 1 75 23 94 44 18 40
362 73
17 3
ACE, angiotensin-converting enzyme; CPE, cardiopulmonary exercise; LVAD, left ventricular assist device; NYHA, New York Heart Association; VO2, maximum volume of oxygen consumption for at least 15 seconds taken in the last 90 seconds of exercise or the first 30 seconds of recovery; SHFM, Seattle Heart Failure Model; VE/VCO2, expired volume per unit time/volume of carbon dioxide measured as the slope taken over the entire exercise effort.
Figure 1 Predicted Seattle Heart Failure Model vs observed Kaplan-Meier survival plotted by quintiles of Seattle Heart Failure Model survival at (Top) 1-year and (Bottom) 3 years.
1020 Table 2
The Journal of Heart and Lung Transplantation, Vol 34, No 8, August 2015 Cox Proportional Hazards Analysis Including All Available Follow-up
Univariable models
Univariate HR
95% CI
p-value
Model AIC
SHFM Peak VO2 VE/VCO2 CPE duration 6MWD
2.2 0.87 1.05 0.86 0.67
1.93–2.48 0.85–0.89 1.04–1.06 0.83–0.88 0.61–0.73
o0.0001 o0.0001 o0.0001 o0.0001 o0.0001
5,958 5,957 5,982 5,960 6,017
Multivariable models with the SHFM
Exercise variable p-value
Interaction p-value
Model AIC
o0.0001 o0.0001 o0.0001 o0.0001
0.79 0.04 0.42 0.35
5,890 5,913 5,900 5,931
Peak VO2 VE/VCO2 CPE duration 6MWD SHFM þ Peak VO2 þ VE/VCO2 þ CPE duration All CPET variables þ 6MWD
5,873 5,875
6MWD, 6-minute walk distance; AIC, Akaike information criteria; CI, confidence interval; CPE, cardiopulmonary exercise CPET, cardiopulmonary exercise test; HR, hazard ratio; SHFM, Seattle Heart Failure Model; VE/VCO2, expired volume per unit time/volume of carbon dioxide measured as the slope taken over the entire exercise effort; VO2, volume of oxygen consumption.
Discussion This analysis demonstrates that adding CPET-derived variables or the 6MWD to the SHFM minimally improves assignment of risk predicted by the SHFM. These results and those of other investigators suggest that it is reasonable to interpret the results of the SHFM and a test of exercise capacity independently and that a small amount of additional prognostic information results from integrating the 2 results. This study substantially adds to the body of riskprediction literature in HF by using the largest sample of HF patients capable of exercise who were enrolled at multiple centers following uniform research protocols. The analysis also benefits from simultaneous recording of CPET results, the 6MWD, and the SHFM.
Table 3
Identifying patients at high risk for adverse events is an important component of optimizing outcomes with LVAD therapy and making the best use of the limited number of hearts available for transplantation. Ideally, patients could be identified for advanced HF therapies with enough lead time to offer treatments that significantly alter the risk of death and improve quality of life at a time before risk is expected to increase sharply in the short-term. A multitude of data points could be used to make these decisions, and clinical assessment will always be chief among the variables necessary to counsel patients and recommend therapy. However, biomarkers or test results that can generate a numeric estimate of risk are particularly attractive for their potential to (1) reduce practice variation in timing of advanced therapies and (2) describe the efficacy of treatment
Reclassification Table for Peak Volume of Oxygen Consumption at 1 Year
Variables
SFHM þ peak VO2-predicted mortality
Patients with eventsa
0–0.05
SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total Patients without eventsa SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total
40.05–0.1
40.1–0.15
40.15
Total
25 8 1 0 34
7 24 2 0 33
1 12 5 5 23
0 2 15 19 36
33 46 23 24 126
1014 166 3 0 1,183
130 353 37 7 527
1 71 61 11 144
0 6 38 68 112
1,145 596 139 86 1,966
SHFM, Seattle Heart Failure Model; VO2, volume of oxygen consumption. a Combined outcome of death, left ventricular assist device, or transplant.
Dardas et al. Table 4
Adding Exercise Test Measures to the SHFM
1021
Reclassification Table for Peak Expired Volume per Unit Time/Volume of Carbon Dioxide at 1 Year SHFM þ VE/VCO2-predicted mortality
Variable Patients with eventsa SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total Patients without eventsa SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total
40.05–0.1
0–0.05
40.1–0.15
40.15
Total
23 8 0 0 31
9 29 5 0 43
1 7 13 7 28
0 2 5 17 24
33 46 23 102 126
1073 135 0 0 1,208
69 416 68 1 554
3 36 47 30 116
0 9 24 55 88
1,145 596 139 1,880 1,966
SHFM, Seattle Heart Failure Model; VE/VCO2, expired volume per unit time/volume of carbon dioxide. a Combined outcome of death, left ventricular assist device, or transplant.
along a spectrum of risk in a clinical trial. For these reasons, risk assignment based on novel biomarkers, imaging, exercise testing, and multivariable models remain of great interest.
Implications for choice of risk stratification strategies in outpatients The SHFM was derived from and has been well-validated in samples of ambulatory patients from trials and registries.7 The SHFM has a tendency to underestimate risk when applied to samples of patients with a higher acuity of illness or with high use of LVAD or transplant.18,19 Using a second prognostic test to appropriately select high-risk patients misclassified as low risk is attractive. Thus, combining the SHFM calculation with a cardiopulmonary exercise study or Table 5 Area Under the Receiver Operating Characteristic Curve Measure Model
1-year
95% CI
SHFM Peak VO2 VE/VCO2 CPE duration 6MWD SHFM þ Peak VO2 VE/VCO2 CPE duration 6MWD All CPET variables All CPET variables þ 6MWD
0.72 0.67 0.66 0.67 0.63
0.67–0.76 0.62–0.72 0.60–0.71 0.62–0.72 0.58–0.68
0.73 0.74 0.72 0.72 0.74 0.74
0.68–0.77 0.69–0.78 0.68–0.77 0.67–0.77 0.69–0.78 0.69–0.78
6MWD, 6-minute walk distance; CPE, cardiopulmonary exercise; CPET, cardiopulmonary exercise test; CI, confidence interval; VE/VCO2, expired volume per unit time/volume of carbon dioxide; VO2, volume of oxygen consumption.
exercise tolerance test may assist in improving identification of high-risk patients likely to benefit from advanced heart failure therapies. In the HF-ACTION sample, the combination of the SHFM with CPET measures or the 6MWD only slightly increased the accuracy of risk assignment. These results are commonly encountered when attempting to improve on an existing model. Previous reports suggest that a very strong association between a predictor and outcomes is necessary (odds ratio of 35) to significantly improve an existing model’s area under the curve (AUC).15,20 The improvement in association between markers and outcomes as measured by the Cox model may not be equivalent to an improvement in risk classification. At the cutoff of a 1-year mortality of 15%, few meaningful reassignments are evident based on the reclassification table method. Consistent with this observation was a small change in the C-index with the addition of CPET measures or the 6MWD to the SHFM compared with the SHFM alone. Previous reports of peak VO2 and VE/VCO2 demonstrate the ability of these measures to divide patients into high-risk and low-risk groups.21–24 Guidelines specifically recommend the use of stable peak VO2 values for risk stratification, although test-retest variability is low.2,10,25 A debate continues about the ideal thresholds for each measure and which exercise variable, if either, is superior.23,26 A peak VO2 between 10 and 14 ml/kg/min or a VE/VCO2 4 34 taken over the entirety of the exercise effort or at the anaerobic threshold define a group of patients with less than 80% survival at 1 year.22,23 This poor prognosis warrants further consideration for advanced therapies. The non–time-dependent AUCs of peak VO2 and VE/ VCO2 in contemporary samples range between 0.61 and 0.75, which is consistent with a mild to moderate improvement in risk assignment compared with chance.27–29 Current recommendations favor sequential use of risk model assessment when exercise test results are inconclusive, although no studies have described the accuracy of this approach.2,5,6
1022
The Journal of Heart and Lung Transplantation, Vol 34, No 8, August 2015
Figure 2 Area under the receiver operator characteristic (ROC) curves (Harrell’s C) for the Seattle Heart Failure Model (SHFM) survival over 1 year of follow-up (Top) with the addition of individual cardiopulmonary exercise testing (CPET) variables and (Bottom) with the addition of the 6-minute walk distance (6MWD) and multivariable models. CPXDUR, cardiopulmonary exercise duration; VE/VO2, expired volume per unit time/volume of carbon dioxide slope.
Previous evaluation of SHFM model changes with added exercise variables This investigation adds evidence to previously published reports of the minimal incremental value of adding CPET variables to existing multivariable models by validation in a large sample of patients from multiple centers, standardized exercise measures, and event adjudication. Levy et al27,28 demonstrated varying improvement in risk assignment with the addition of peak VO2 and VE/VCO2 to the SHFM in
samples of patients referred for transplantation or enrolled in a large randomized controlled trial of biventricular pacing. Peak VO2 improved the NRI (0.14, p o 0.0001) but not the AUC (0.76 to 0.77, p ¼ 0.16) when added to the SHFM.28 Contrary to the findings of the present investigation, the VE/ VCO2 slope, when added to the SHFM, resulted in a large improvement in NRI of 0.4 (p ¼ 0.0023).27 There were also varying improvements in the Cox model significance for the addition of peak VO2 (p o 0.0001 to p ¼ 0.3), whereas the single evaluation of VE/VCO2 (p o 0.0001) showed consistent improvement.27,28 When combined with the current results, most of the evidence suggests that the SHFM and CPET measures are accurate prognostic tools, but only a small benefit (1% improvement) occurs after integrating the SHFM model with CPET results. This study has some limitations. We studied a large group of patients enrolled in a randomized controlled trial of exercise training. Our findings may not extend to patients too ill to participate in an intensive exercise program. Survival in HF-ACTION was 94% at 1 year, and all patients were able to perform an exercise test, which may raise concern regarding the use of this sample to estimate risk for the purpose of providing advanced HF treatments. However, our study demonstrates that groups of patients with observed mortality consistent with the use of advanced HF therapies (o75% survival at 1 year) can be accurately identified within this ambulatory population. Missing data were imputed using group means from the HF-ACTION sample. Thus, patients with advanced HF and missing data may have an underestimated risk. Regardless of this bias, the SHFM accurately identified high-risk patients. The SHFM estimates in this analysis were fitted to the sample, which provides a baseline survival function and β coefficients intrinsic to the HF-ACTION sample, which gives the SHFM a more favorable fit to the data than the originally published model. This type of fitting was necessary to generate estimates for the additive value of CPET measures and 6MWD. However, we compared the Kaplan-Meier observed survival to the non-refitted SHFMcalculated survival and found them to be very close (Figure 1). The use of imputation was necessary to estimate the SHFM model but would not be expected to create a bias in any predictable direction. The choice of risk boundaries used in reclassification tables and the need to use event status (not time to event) results in some loss of information. These are intrinsic limitations to the reclassification table method and motivated the use of the Cox model for testing of the primary hypothesis in this investigation. We chose the clinically relevant boundaries to assess risk reassignment because these boundaries pertain to decisions of timing surrounding LVAD implantation and heart transplantation. In conclusion, the SHFM and exercise performance testing remain well-proven tests that independently add to clinical judgment when stratifying patients with systolic HF. Integrating exercise performance measures and the SHFM resulted in a minimal improvement in risk classification at clinically important boundaries when studied in this large, high-quality, ambulatory patient sample.
Dardas et al.
Adding Exercise Test Measures to the SHFM
Disclosure statement The University of Washington Center 4 Commercialization holds the copyright to the Seattle Heart Failure Model and has received licensing from Epocrates, HeartWare, Thoratec, and General Electric Healthcare for use of the model. There was no licensing charged for use of the Seattle Heart Failure Model in this analysis of HF-ACTION. W.C.L. reports research funding from HeartWare and General Electric. None of the other authors has a financial relationship with a commercial entity that has an interest in the subject of the presented manuscript or other conflicts of interest to disclose. T.D. is funded by the American College of Cardiology/Daiichi Sankyo Career Development Award. S.D.R. Y.L., K.A.S., and W.C.L. received funding from grant 5R01-NR-011873 from the National Institute of Nursing Research. HF-ACTION was funded by grants 5U01-HL-063747, 5U01-HL-066461, 5U01-HL-068973, 5U01-HL-066501, 5U01-HL-066482, 5U01-HL-064250, 5U01-HL066494, 5U01-HL-064257, 5U01-HL-066497, 5U01-HL-068980, 5U01-HL-064265, 5U01-HL-066491, and 5U01-HL-064264 from the National Heart, Lung, and Blood Institute; and grants R37-AG018915 and P60-AG-010484 from the National Institute on Aging.
References 1. Aaronson KD, Cowger J. Heart failure prognostic models: why bother? Circ Heart Fail 2012;5:6-9. 2. Mehra MR, Kobashigawa J, Starling R, et al. Listing criteria for heart transplantation: International Society for Heart and Lung Transplantation guidelines for the care of cardiac transplant candidates— 2006. J Heart Lung Transplant 2006;25:1024-42. 3. Bittner V, Weiner DH, Yusuf S, et al. Prediction of mortality and morbidity with a 6-minute walk test in patients with left ventricular dysfunction. SOLVD Investigators. JAMA 1993;270:1702-7. 4. Forman DE, Fleg JL, Kitzman DW, et al. 6-min walk test provides prognostic utility comparable to cardiopulmonary exercise testing in ambulatory outpatients with systolic heart failure. J Am Coll Cardiol 2012;60:2653-61. 5. Mancini D, Lietz K. Selection of cardiac transplantation candidates in 2010. Circulation 2010;122:173-83. 6. Butler J, Khadim G, Paul KM, et al. Selection of patients for heart transplantation in the current era of heart failure therapy. J Am Coll Cardiol 2004;43:787-93. 7. Levy WC, Mozaffarian D, Linker DT, et al. The Seattle Heart Failure Model: prediction of survival in heart failure. Circulation 2006; 113:1424-33. 8. O’Connor CM, Whellan DJ, Lee KL, et al. HF-ACTION Investigators. Efficacy and safety of exercise training in patients with chronic heart failure: HF-ACTION randomized controlled trial. JAMA 2009; 301:1439-50. 9. Whellan DJ, O’Connor CM, Lee KL, et al. HF-ACTION Trial Investigators. Heart failure and a controlled trial investigating outcomes of exercise training (HF-ACTION): design and rationale. Am Heart J 2007;153:201-11. 10. Bensimhon DR, Leifer ES, Ellis SJ, et al. HF-ACTION Trial Investigators. Reproducibility of peak oxygen uptake and other cardiopulmonary exercise testing parameters in patients with heart failure (from the Heart Failure and A Controlled Trial Investigating Outcomes of exercise traiNing). Am J Cardiol 2008;102:712-7.
1023 11. Guyatt GH, Sullivan MJ, Thompson PJ, et al. The 6-minute walk: a new measure of exercise capacity in patients with chronic heart failure. Can Med Assoc J 1985;132:919-23. 12. Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology (Cambridge, Mass) 2014;25: 114-21. 13. Pepe MS. Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol 2011;173:1327-35. 14. Pepe MS, Janes H, Li CI. Net risk reclassification p values: valid or misleading? J Natl Cancer Inst 2014;106:dju041. 15. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 2004;159:882-90. 16. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med 2014;33:3405-14. 17. Harrell FE Jr, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. JAMA 1982;247:2543-6. 18. Kalogeropoulos AP, Georgiopoulou VV, Giamouzis G, et al. Utility of the Seattle Heart Failure Model in patients with advanced heart failure. J Am Coll Cardiol 2009;53:334-42. 19. Gorodeski EZ, Chu EC, Chow CH, Levy WC, Hsich E, Starling RC. Application of the Seattle Heart Failure Model in ambulatory patients presented to an advanced heart failure therapeutics committee. Circ Heart Fail 2010;3:706-14. 20. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72: discussion 207-12. 21. Mancini D, Goldsmith R, Levin H, et al. Comparison of exercise performance in patients with chronic severe heart failure versus left ventricular assist devices. Circulation 1998;98:1178-83. 22. Mancini DM, Eisen H, Kussmaul W, Mull R, Edmunds LH Jr, Wilson JR. Value of peak exercise oxygen consumption for optimal timing of cardiac transplantation in ambulatory patients with heart failure. Circulation 1991;83:778-86. 23. Arena R, Myers J, Abella J, Pinkstaff S, et al. Defining the optimal prognostic window for cardiopulmonary exercise testing in patients with heart failure. Circ Heart Fail 2010;3:405-11. 24. Cahalin LP, Chase P, Arena R. A meta-analysis of the prognostic significance of cardiopulmonary exercise testing in patients with heart failure. Heart Fail Rev 2013;18:79-94. 25. Scott JM, Haykowsky MJ, Eggebeen J, Morgan TM, Brubaker PH, Kitzman DW. Reliability of peak exercise testing in patients with heart failure with preserved ejection fraction. Am J Cardiol 2012;110: 1809-13. 26. Mancini D, LeJemtel TH. Is ventilatory classification preferable to peak oxygen consumption for risk stratification in heart failure? Circulation 2007;115:2376-8. 27. Levy WC, Arena R, Wagoner LE, Dardas T, Abraham WT. Prognostic impact of the addition of ventilatory efficiency to the Seattle heart failure model in patients with heart failure. J Card Fail 2012;18: 614-619. 28. Levy WC, Aaronson KD, Dardas TF, Williams P, Haythe J, Mancini D. Prognostic impact of the addition of peak oxygen consumption to the Seattle Heart Failure Model in a transplant referral population. J Heart Lung Transplant 2012;31:817-24. 29. Kato TS, Collado E, Khawaja T, et al. Value of peak exercise oxygen consumption combined with B-type natriuretic peptide levels for optimal timing of cardiac transplantation. Circ Heart Fail 2013;6:6-14.
FEATURED PAPERS
Incremental and independent value of cardiopulmonary exercise test measures and the Seattle Heart Failure Model for prediction of risk in patients with heart failure Todd Dardas, MD, MS,a Yanhong Li, MD, MS,b Shelby D. Reed, PhD,b Christopher M. O’Connor, MD,b David J. Whellan, MD,c Stephen J. Ellis, PhD,b Kevin A. Schulman, MD,b William E. Kraus, MD,b Daniel E. Forman, MD,d and Wayne C. Levy, MDa From the aDivision of Cardiology, Department of Internal Medicine, University of Washington Seattle, Washington; bDuke Clinical Research Institute, Durham, North Carolina; cDivision of Cardiology, Department of Medicine, Jefferson University Hospitals, Philadelphia, Pennsylvania; and the dCardiovascular Division, Brigham and Women’s Hospital, Boston, Massachusetts.
KEYWORDS: heart failure; risk factors; exercise; Seattle Heart Failure Model; risk prediction
BACKGROUND: Multivariable risk scores and exercise measures are well-validated risk prediction methods. Combining information from a functional evaluation and a risk model may improve accuracy of risk predictions. We analyzed whether adding exercise measures to the Seattle Heart Failure Model (SHFM) improves risk prediction accuracy in systolic heart failure. METHODS: We used a sample of patients from the Heart Failure and A Controlled Trial Investigating Outcomes of Exercise TraiNing (HF-ACTION) study (http://www.clinicaltrials.gov; unique identifier: NCT00047437) to examine the addition of peak oxygen consumption, expired volume per unit time/ volume of carbon dioxide slope, 6-minute walk distance, or cardiopulmonary exercise duration to the SHFM. Multivariable Cox proportional hazards models were used to test the association between the combined end point (death, left ventricular assist device, or cardiac transplantation) and the addition of exercise variables to the SHFM. RESULTS: The sample included 2,152 patients. The SHFM and all exercise measures were associated with events (all p o 0.0001) in proportional hazards models. There was statistically significant improvement in risk estimation when exercise measures were added to the SHFM. However, the improvement in the C index for the addition of peak volume of oxygen consumption (þ0.01), expired volume per unit time/volume of carbon dioxide slope (þ0.02), 6-minute walk distance (–0.001), and cardiopulmonary exercise duration (þ0. 001) to the SHFM was small or slightly worse than the SHFM alone. Changes in risk assignment with the addition of exercise variables were minimal for patients above or below a 15% 1-year mortality. COnclusions: Exercise performance measures and the SHFM are independently useful for predicting risk in systolic heart failure. Adding cardiopulmonary exercise testing measures and 6MWD to the SHFM offers only minimal improvement in risk reassignment at clinically meaningful cut points. J Heart Lung Transplant 2015;34:1017–1023 r 2015 International Society for Heart and Lung Transplantation. All rights reserved.
Reprint requests: Todd Dardas, MD, MS, University of Washington Seattle, Division of Cardiology, 1959 NE Pacific St, Box 356422, Seattle, WA 98195-6422. Telephone: 206-997-8448. Fax: 206-616-4847. E-mail address: [email protected]
Risk models remain useful for normalizing risk among groups of patients in trials, risk-adjusting performance metrics, or as part of broader discussions of prognosis.1
1053-2498/$ - see front matter r 2015 International Society for Heart and Lung Transplantation. All rights reserved. http://dx.doi.org/10.1016/j.healun.2015.03.017
1018
The Journal of Heart and Lung Transplantation, Vol 34, No 8, August 2015
These applications demand that models are well validated, updated to modern samples, and contain variables considered relevant to patients and physicians. A wide array of options exist for determining prognosis in patients with systolic heart failure (HF). Transplant candidacy selection guidelines recommend assessment of each patient’s functional capacity with respect to activities of daily living and formal cardiopulmonary exercise testing (CPET).2 CPET measures have the advantage of being inexpensive and extensively validated and are important objective measures of patient-reported limitations that advanced heart failure therapies seek to remedy. However, CPET measures are not widely available, do not account for certain therapies explicitly (e.g., defibrillators), and cannot be performed by all patients. The 6-minute walk distance (6MWD) has been shown to be associated with prognosis and may be useful if a full CPET is not available.3,4 Guidelines and experts also recommend the use of the Heart Failure Survival Score, which includes peak oxygen consumption (VO2), in cases where exercise test results may be inconclusive.2,5,6 Risk models, such as the Seattle Heart Failure Model (SHFM), also offer utility by accounting explicitly for modern HF therapies and can be calculated for patients who are nonambulatory.7 We studied whether integrating CPET variables with the SHFM improved the accuracy of risk predictions in the ambulatory sample of patients enrolled in the Heart Failure and A Controlled Trial Investigating Outcomes of Exercise TraiNing (HF-ACTION) randomized trial (http://www. clinicaltrials.gov; unique identifier: NCT00047437). This analysis was motivated by the need to determine whether a multivariable risk score is equivalent to single measures of exercise tolerance and to determine whether adding exercise information to the SHFM might improve risk assessment at clinically meaningful cutoffs.
Methods The results of the HF-ACTION trial and the original derivation and validation of the SHFM have been previously reported.7,8 Briefly, the HF-ACTION trial enrolled ambulatory patients with systolic HF who could comply with exercise training. Patients were randomized 1:1 to usual care with an intensive, supervised exercise training program, followed by home-based training, or to usual care with printed information describing the benefits of exercise. The primary end point of time to death or all-cause hospitalization was not significantly reduced in the group assigned to exercise training (hazard ratio, 0.93; 95% confidence interval, 0.84–1.02). CPET was performed using a modified Naughton protocol.9 Peak VO2 was defined as the highest oxygen consumption for a given 15- or 20-second interval within the last 90 seconds of exercise or the first 30 seconds of recovery.10 The expired volume per unit time/volume of carbon dioxide slope (VE/VCO2) was recorded as the slope across the entire course of the exercise effort.10 The 6MWD was performed as previously described.11
SHFM time-to-event estimates The HF-ACTION trial collected all the variables necessary to derive the SHFM scores with the exception of allopurinol use, uric
acid, and percentage of lymphocytes. To handle these missing data, we assumed that patients were not receiving treatment with allopurinol and derived predicted estimates of uric acid levels and percentage of lymphocytes from multivariable linear regression models derived from the original SHFM cohort. Missing values for hemoglobin (24%), cholesterol (35%), and sodium (11%) were imputed using group means derived from HF-ACTION. SHFM scores were then calculated using the originally published β coefficients.7 Assessment of the SHFM model fit in the HFACTION sample was evaluated by comparing the observed vs predicted survival at specific time points based on the originallypublished equation: eðSHFM ScoreÞ
Survival ðt Þ¼ eðλtÞ
where λ ¼ 0.0405 is the baseline hazard function. The combined end point of death, left ventricular assist device (LVAD), or cardiac transplantation, regardless of transplant urgency, was used for all analyses. This end point differed from the main HF-ACTION trial end point of all-cause mortality or hospitalization.
Additive effect of exercise measures with methods accounting for censoring To evaluate the contributions of CPET measures and 6MWD to the SHFM, Cox proportional hazards models were used as the primary analytic method. The SHFM score, peak VO2, VE/VCO2, cardiopulmonary exercise (CPE) duration, and 6MWD were modeled as continuous variables. The SHFM scores and exercise measures were evaluated individually in separate univariable models. Cox models were also used to test for statistical interactions among the SHFM, CPET measures, and 6MWD to detect differential associations over ranges of the covariates. We report standard measures of model significance and the Akaike information criterion. Reclassification tables were created to describe the effect of risk reassignment for patients with and without events. Groups for reclassification tables were created according to SHFMpredicted 1-year mortality: 0% to 5%, 45% to 10%, 410% to 15%, and 4 15%, which provide clinically meaningful changes in risk assignment. Patients were then reassigned to groups based on 1-year mortality estimates derived from Cox models with the SHFM and the additional exercise measures. The tables assume that reclassification of patients with events to a higher risk group is appropriate. Conversely, reclassification of a patient with an event to a low risk is inappropriate. Participants were excluded from the tables if censored before the specified follow-up time. Net reclassification index (NRI) statistics and p-values were not reported based on published work, suggesting issues with dual hypothesis testing and p-value estimation.12–16 The effect of adding CPET variables or 6MWD to the SHFM was estimated using Harrell’s C-index, a measure of concordance probability in the presence of censoring.17 Analyses were performed using SAS 9.2 software (SAS Institute Inc, Cary, NC) and R software. Two of the authors (Y.L. and S.R.) had full access to the complete data. The Duke University Institutional Review Board approved the analyses.
Results HF-ACTION contained 2,331 patients. The current sample excluded 179 patients for the absence of minimal information required to calculate the SHFM score or when not all CPETderived variables were available. There were 2,152 patients in
Dardas et al.
Adding Exercise Test Measures to the SHFM
1019
the final sample, with 418 events during a median of 2.5 years of follow-up. At 1 year, 2.8% of the sample was censored, and at 3 years, 44.0% of the sample was censored. Demographic characteristics and outcomes are described in Table 1.
(p o 0.0001), and 6MWD (p o 0.0001) remained significant predictors of the composite end point, as did the SHFM (p o 0.0001 for all models). There were non-significant interactions between the SHFM and peak VO2 (p ¼ 0.79), CPE duration (p ¼ 0.42), and the 6MWD (p ¼ 0.35), suggesting that the association between events and the SHFM does not change over values of these exercise measures (i.e., even at low values of peak VO2, the SHFM retains an association with events). There was a significant interaction between the SHFM and VE/ VCO2 slope (p ¼ 0.04), suggesting a non-linear association between events, SHFM-predicted risk, and VE/VCO2 slopepredicted risk (interaction model coefficient estimates: SHFM VE/VCO2 ¼ –0.01, VE/VCO2 slope ¼ 0.04, SHFM ¼ 1.1). The Akaike information criteria for the univariable models and the combined models are reported in Table 2. The best model was the SHFM þ peak VO2. Reclassification tables are shown in Tables 3 and 4 for peak VO2 and VE/VCO2 and exercise duration. The C-index at 1 year was 0.72 for the SHFM, 0.67 for peak VO2, 0.66 for VE/VCO2 slope, 0.67 for CPE duration, and 0.63 for the 6MWD (Table 5). When combined with the SHFM, the 1-year C-indices of peak VO2 (0.73), VE/VCO2 slope (0.74), CPE duration (0.72) and the 6MWD (0.72) were minimally increased (Table 5, Figure 2).
SHFM fit in the HF-ACTION sample The Kaplan-Meier survival in the sample was 94.1% at 1 year and 78.8% at 3 years. Predicted survival for the entire sample using the SHFM was 93.8% at 1 year and 83.2% at 3 years (Figure 1).
Effect of adding exercise variables to the SHFM CPET measures of peak VO2 (1 ml/kg/min; hazard ratio [HR], 0.87; 95% confidence interval [CI] 0.85–0.89), VE/ VCO2 (1 unit slope; HR, 1.05; 95% CI 1.04–1.06), exercise duration (1 minute; HR, 0.86; 95% CI, 0.83–0.88), and the 6MWD (100 meters; HR, 0.67; 95% CI 0.61–0.73) were associated with occurrence of the composite outcome over the course of all available follow-up in separate univariable models (Table 2). When each CPET variable and 6MWD were added to the SHFM in separate Cox models, peak VO2 (p o 0.0001), VE/VCO2 slope (p o 0.0001), CPE duration Table 1
Demographic Characteristics and Outcomes
Variable
Mean
SD
Age NYHA Functional Classification Peak VO2, ml/kg/min VE/VCO2, slope CPE duration, min 6-minute walk distance, 100 m SHFM, sum of β coefficients
59.1 2.4 14.9 34.3 9.8 3.65 0.24
12.7 0.5 4.7 9 4 1.04 0.65
No.
%
Total sample Sex Female Male NYHA Functional Classification II III IV ACE inhibitor Angiotensin receptor blockade β-Blocker Aldosterone antagonist Biventricular pacemaker Implantable cardioverter-defibrillator Outcomes Death Transplant or LVAD
2,152 608 1,544
28 72
1,383 749 20 1,604 497 2,030 949 383 855
64 35 1 75 23 94 44 18 40
362 73
17 3
ACE, angiotensin-converting enzyme; CPE, cardiopulmonary exercise; LVAD, left ventricular assist device; NYHA, New York Heart Association; VO2, maximum volume of oxygen consumption for at least 15 seconds taken in the last 90 seconds of exercise or the first 30 seconds of recovery; SHFM, Seattle Heart Failure Model; VE/VCO2, expired volume per unit time/volume of carbon dioxide measured as the slope taken over the entire exercise effort.
Figure 1 Predicted Seattle Heart Failure Model vs observed Kaplan-Meier survival plotted by quintiles of Seattle Heart Failure Model survival at (Top) 1-year and (Bottom) 3 years.
1020 Table 2
The Journal of Heart and Lung Transplantation, Vol 34, No 8, August 2015 Cox Proportional Hazards Analysis Including All Available Follow-up
Univariable models
Univariate HR
95% CI
p-value
Model AIC
SHFM Peak VO2 VE/VCO2 CPE duration 6MWD
2.2 0.87 1.05 0.86 0.67
1.93–2.48 0.85–0.89 1.04–1.06 0.83–0.88 0.61–0.73
o0.0001 o0.0001 o0.0001 o0.0001 o0.0001
5,958 5,957 5,982 5,960 6,017
Multivariable models with the SHFM
Exercise variable p-value
Interaction p-value
Model AIC
o0.0001 o0.0001 o0.0001 o0.0001
0.79 0.04 0.42 0.35
5,890 5,913 5,900 5,931
Peak VO2 VE/VCO2 CPE duration 6MWD SHFM þ Peak VO2 þ VE/VCO2 þ CPE duration All CPET variables þ 6MWD
5,873 5,875
6MWD, 6-minute walk distance; AIC, Akaike information criteria; CI, confidence interval; CPE, cardiopulmonary exercise CPET, cardiopulmonary exercise test; HR, hazard ratio; SHFM, Seattle Heart Failure Model; VE/VCO2, expired volume per unit time/volume of carbon dioxide measured as the slope taken over the entire exercise effort; VO2, volume of oxygen consumption.
Discussion This analysis demonstrates that adding CPET-derived variables or the 6MWD to the SHFM minimally improves assignment of risk predicted by the SHFM. These results and those of other investigators suggest that it is reasonable to interpret the results of the SHFM and a test of exercise capacity independently and that a small amount of additional prognostic information results from integrating the 2 results. This study substantially adds to the body of riskprediction literature in HF by using the largest sample of HF patients capable of exercise who were enrolled at multiple centers following uniform research protocols. The analysis also benefits from simultaneous recording of CPET results, the 6MWD, and the SHFM.
Table 3
Identifying patients at high risk for adverse events is an important component of optimizing outcomes with LVAD therapy and making the best use of the limited number of hearts available for transplantation. Ideally, patients could be identified for advanced HF therapies with enough lead time to offer treatments that significantly alter the risk of death and improve quality of life at a time before risk is expected to increase sharply in the short-term. A multitude of data points could be used to make these decisions, and clinical assessment will always be chief among the variables necessary to counsel patients and recommend therapy. However, biomarkers or test results that can generate a numeric estimate of risk are particularly attractive for their potential to (1) reduce practice variation in timing of advanced therapies and (2) describe the efficacy of treatment
Reclassification Table for Peak Volume of Oxygen Consumption at 1 Year
Variables
SFHM þ peak VO2-predicted mortality
Patients with eventsa
0–0.05
SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total Patients without eventsa SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total
40.05–0.1
40.1–0.15
40.15
Total
25 8 1 0 34
7 24 2 0 33
1 12 5 5 23
0 2 15 19 36
33 46 23 24 126
1014 166 3 0 1,183
130 353 37 7 527
1 71 61 11 144
0 6 38 68 112
1,145 596 139 86 1,966
SHFM, Seattle Heart Failure Model; VO2, volume of oxygen consumption. a Combined outcome of death, left ventricular assist device, or transplant.
Dardas et al. Table 4
Adding Exercise Test Measures to the SHFM
1021
Reclassification Table for Peak Expired Volume per Unit Time/Volume of Carbon Dioxide at 1 Year SHFM þ VE/VCO2-predicted mortality
Variable Patients with eventsa SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total Patients without eventsa SHFM predicted risk 0–0.05 40.05–0.1 40.1–0.15 40.15 Total
40.05–0.1
0–0.05
40.1–0.15
40.15
Total
23 8 0 0 31
9 29 5 0 43
1 7 13 7 28
0 2 5 17 24
33 46 23 102 126
1073 135 0 0 1,208
69 416 68 1 554
3 36 47 30 116
0 9 24 55 88
1,145 596 139 1,880 1,966
SHFM, Seattle Heart Failure Model; VE/VCO2, expired volume per unit time/volume of carbon dioxide. a Combined outcome of death, left ventricular assist device, or transplant.
along a spectrum of risk in a clinical trial. For these reasons, risk assignment based on novel biomarkers, imaging, exercise testing, and multivariable models remain of great interest.
Implications for choice of risk stratification strategies in outpatients The SHFM was derived from and has been well-validated in samples of ambulatory patients from trials and registries.7 The SHFM has a tendency to underestimate risk when applied to samples of patients with a higher acuity of illness or with high use of LVAD or transplant.18,19 Using a second prognostic test to appropriately select high-risk patients misclassified as low risk is attractive. Thus, combining the SHFM calculation with a cardiopulmonary exercise study or Table 5 Area Under the Receiver Operating Characteristic Curve Measure Model
1-year
95% CI
SHFM Peak VO2 VE/VCO2 CPE duration 6MWD SHFM þ Peak VO2 VE/VCO2 CPE duration 6MWD All CPET variables All CPET variables þ 6MWD
0.72 0.67 0.66 0.67 0.63
0.67–0.76 0.62–0.72 0.60–0.71 0.62–0.72 0.58–0.68
0.73 0.74 0.72 0.72 0.74 0.74
0.68–0.77 0.69–0.78 0.68–0.77 0.67–0.77 0.69–0.78 0.69–0.78
6MWD, 6-minute walk distance; CPE, cardiopulmonary exercise; CPET, cardiopulmonary exercise test; CI, confidence interval; VE/VCO2, expired volume per unit time/volume of carbon dioxide; VO2, volume of oxygen consumption.
exercise tolerance test may assist in improving identification of high-risk patients likely to benefit from advanced heart failure therapies. In the HF-ACTION sample, the combination of the SHFM with CPET measures or the 6MWD only slightly increased the accuracy of risk assignment. These results are commonly encountered when attempting to improve on an existing model. Previous reports suggest that a very strong association between a predictor and outcomes is necessary (odds ratio of 35) to significantly improve an existing model’s area under the curve (AUC).15,20 The improvement in association between markers and outcomes as measured by the Cox model may not be equivalent to an improvement in risk classification. At the cutoff of a 1-year mortality of 15%, few meaningful reassignments are evident based on the reclassification table method. Consistent with this observation was a small change in the C-index with the addition of CPET measures or the 6MWD to the SHFM compared with the SHFM alone. Previous reports of peak VO2 and VE/VCO2 demonstrate the ability of these measures to divide patients into high-risk and low-risk groups.21–24 Guidelines specifically recommend the use of stable peak VO2 values for risk stratification, although test-retest variability is low.2,10,25 A debate continues about the ideal thresholds for each measure and which exercise variable, if either, is superior.23,26 A peak VO2 between 10 and 14 ml/kg/min or a VE/VCO2 4 34 taken over the entirety of the exercise effort or at the anaerobic threshold define a group of patients with less than 80% survival at 1 year.22,23 This poor prognosis warrants further consideration for advanced therapies. The non–time-dependent AUCs of peak VO2 and VE/ VCO2 in contemporary samples range between 0.61 and 0.75, which is consistent with a mild to moderate improvement in risk assignment compared with chance.27–29 Current recommendations favor sequential use of risk model assessment when exercise test results are inconclusive, although no studies have described the accuracy of this approach.2,5,6
1022
The Journal of Heart and Lung Transplantation, Vol 34, No 8, August 2015
Figure 2 Area under the receiver operator characteristic (ROC) curves (Harrell’s C) for the Seattle Heart Failure Model (SHFM) survival over 1 year of follow-up (Top) with the addition of individual cardiopulmonary exercise testing (CPET) variables and (Bottom) with the addition of the 6-minute walk distance (6MWD) and multivariable models. CPXDUR, cardiopulmonary exercise duration; VE/VO2, expired volume per unit time/volume of carbon dioxide slope.
Previous evaluation of SHFM model changes with added exercise variables This investigation adds evidence to previously published reports of the minimal incremental value of adding CPET variables to existing multivariable models by validation in a large sample of patients from multiple centers, standardized exercise measures, and event adjudication. Levy et al27,28 demonstrated varying improvement in risk assignment with the addition of peak VO2 and VE/VCO2 to the SHFM in
samples of patients referred for transplantation or enrolled in a large randomized controlled trial of biventricular pacing. Peak VO2 improved the NRI (0.14, p o 0.0001) but not the AUC (0.76 to 0.77, p ¼ 0.16) when added to the SHFM.28 Contrary to the findings of the present investigation, the VE/ VCO2 slope, when added to the SHFM, resulted in a large improvement in NRI of 0.4 (p ¼ 0.0023).27 There were also varying improvements in the Cox model significance for the addition of peak VO2 (p o 0.0001 to p ¼ 0.3), whereas the single evaluation of VE/VCO2 (p o 0.0001) showed consistent improvement.27,28 When combined with the current results, most of the evidence suggests that the SHFM and CPET measures are accurate prognostic tools, but only a small benefit (1% improvement) occurs after integrating the SHFM model with CPET results. This study has some limitations. We studied a large group of patients enrolled in a randomized controlled trial of exercise training. Our findings may not extend to patients too ill to participate in an intensive exercise program. Survival in HF-ACTION was 94% at 1 year, and all patients were able to perform an exercise test, which may raise concern regarding the use of this sample to estimate risk for the purpose of providing advanced HF treatments. However, our study demonstrates that groups of patients with observed mortality consistent with the use of advanced HF therapies (o75% survival at 1 year) can be accurately identified within this ambulatory population. Missing data were imputed using group means from the HF-ACTION sample. Thus, patients with advanced HF and missing data may have an underestimated risk. Regardless of this bias, the SHFM accurately identified high-risk patients. The SHFM estimates in this analysis were fitted to the sample, which provides a baseline survival function and β coefficients intrinsic to the HF-ACTION sample, which gives the SHFM a more favorable fit to the data than the originally published model. This type of fitting was necessary to generate estimates for the additive value of CPET measures and 6MWD. However, we compared the Kaplan-Meier observed survival to the non-refitted SHFMcalculated survival and found them to be very close (Figure 1). The use of imputation was necessary to estimate the SHFM model but would not be expected to create a bias in any predictable direction. The choice of risk boundaries used in reclassification tables and the need to use event status (not time to event) results in some loss of information. These are intrinsic limitations to the reclassification table method and motivated the use of the Cox model for testing of the primary hypothesis in this investigation. We chose the clinically relevant boundaries to assess risk reassignment because these boundaries pertain to decisions of timing surrounding LVAD implantation and heart transplantation. In conclusion, the SHFM and exercise performance testing remain well-proven tests that independently add to clinical judgment when stratifying patients with systolic HF. Integrating exercise performance measures and the SHFM resulted in a minimal improvement in risk classification at clinically important boundaries when studied in this large, high-quality, ambulatory patient sample.
Dardas et al.
Adding Exercise Test Measures to the SHFM
Disclosure statement The University of Washington Center 4 Commercialization holds the copyright to the Seattle Heart Failure Model and has received licensing from Epocrates, HeartWare, Thoratec, and General Electric Healthcare for use of the model. There was no licensing charged for use of the Seattle Heart Failure Model in this analysis of HF-ACTION. W.C.L. reports research funding from HeartWare and General Electric. None of the other authors has a financial relationship with a commercial entity that has an interest in the subject of the presented manuscript or other conflicts of interest to disclose. T.D. is funded by the American College of Cardiology/Daiichi Sankyo Career Development Award. S.D.R. Y.L., K.A.S., and W.C.L. received funding from grant 5R01-NR-011873 from the National Institute of Nursing Research. HF-ACTION was funded by grants 5U01-HL-063747, 5U01-HL-066461, 5U01-HL-068973, 5U01-HL-066501, 5U01-HL-066482, 5U01-HL-064250, 5U01-HL066494, 5U01-HL-064257, 5U01-HL-066497, 5U01-HL-068980, 5U01-HL-064265, 5U01-HL-066491, and 5U01-HL-064264 from the National Heart, Lung, and Blood Institute; and grants R37-AG018915 and P60-AG-010484 from the National Institute on Aging.
References 1. Aaronson KD, Cowger J. Heart failure prognostic models: why bother? Circ Heart Fail 2012;5:6-9. 2. Mehra MR, Kobashigawa J, Starling R, et al. Listing criteria for heart transplantation: International Society for Heart and Lung Transplantation guidelines for the care of cardiac transplant candidates— 2006. J Heart Lung Transplant 2006;25:1024-42. 3. Bittner V, Weiner DH, Yusuf S, et al. Prediction of mortality and morbidity with a 6-minute walk test in patients with left ventricular dysfunction. SOLVD Investigators. JAMA 1993;270:1702-7. 4. Forman DE, Fleg JL, Kitzman DW, et al. 6-min walk test provides prognostic utility comparable to cardiopulmonary exercise testing in ambulatory outpatients with systolic heart failure. J Am Coll Cardiol 2012;60:2653-61. 5. Mancini D, Lietz K. Selection of cardiac transplantation candidates in 2010. Circulation 2010;122:173-83. 6. Butler J, Khadim G, Paul KM, et al. Selection of patients for heart transplantation in the current era of heart failure therapy. J Am Coll Cardiol 2004;43:787-93. 7. Levy WC, Mozaffarian D, Linker DT, et al. The Seattle Heart Failure Model: prediction of survival in heart failure. Circulation 2006; 113:1424-33. 8. O’Connor CM, Whellan DJ, Lee KL, et al. HF-ACTION Investigators. Efficacy and safety of exercise training in patients with chronic heart failure: HF-ACTION randomized controlled trial. JAMA 2009; 301:1439-50. 9. Whellan DJ, O’Connor CM, Lee KL, et al. HF-ACTION Trial Investigators. Heart failure and a controlled trial investigating outcomes of exercise training (HF-ACTION): design and rationale. Am Heart J 2007;153:201-11. 10. Bensimhon DR, Leifer ES, Ellis SJ, et al. HF-ACTION Trial Investigators. Reproducibility of peak oxygen uptake and other cardiopulmonary exercise testing parameters in patients with heart failure (from the Heart Failure and A Controlled Trial Investigating Outcomes of exercise traiNing). Am J Cardiol 2008;102:712-7.
1023 11. Guyatt GH, Sullivan MJ, Thompson PJ, et al. The 6-minute walk: a new measure of exercise capacity in patients with chronic heart failure. Can Med Assoc J 1985;132:919-23. 12. Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology (Cambridge, Mass) 2014;25: 114-21. 13. Pepe MS. Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol 2011;173:1327-35. 14. Pepe MS, Janes H, Li CI. Net risk reclassification p values: valid or misleading? J Natl Cancer Inst 2014;106:dju041. 15. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 2004;159:882-90. 16. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med 2014;33:3405-14. 17. Harrell FE Jr, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. JAMA 1982;247:2543-6. 18. Kalogeropoulos AP, Georgiopoulou VV, Giamouzis G, et al. Utility of the Seattle Heart Failure Model in patients with advanced heart failure. J Am Coll Cardiol 2009;53:334-42. 19. Gorodeski EZ, Chu EC, Chow CH, Levy WC, Hsich E, Starling RC. Application of the Seattle Heart Failure Model in ambulatory patients presented to an advanced heart failure therapeutics committee. Circ Heart Fail 2010;3:706-14. 20. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72: discussion 207-12. 21. Mancini D, Goldsmith R, Levin H, et al. Comparison of exercise performance in patients with chronic severe heart failure versus left ventricular assist devices. Circulation 1998;98:1178-83. 22. Mancini DM, Eisen H, Kussmaul W, Mull R, Edmunds LH Jr, Wilson JR. Value of peak exercise oxygen consumption for optimal timing of cardiac transplantation in ambulatory patients with heart failure. Circulation 1991;83:778-86. 23. Arena R, Myers J, Abella J, Pinkstaff S, et al. Defining the optimal prognostic window for cardiopulmonary exercise testing in patients with heart failure. Circ Heart Fail 2010;3:405-11. 24. Cahalin LP, Chase P, Arena R. A meta-analysis of the prognostic significance of cardiopulmonary exercise testing in patients with heart failure. Heart Fail Rev 2013;18:79-94. 25. Scott JM, Haykowsky MJ, Eggebeen J, Morgan TM, Brubaker PH, Kitzman DW. Reliability of peak exercise testing in patients with heart failure with preserved ejection fraction. Am J Cardiol 2012;110: 1809-13. 26. Mancini D, LeJemtel TH. Is ventilatory classification preferable to peak oxygen consumption for risk stratification in heart failure? Circulation 2007;115:2376-8. 27. Levy WC, Arena R, Wagoner LE, Dardas T, Abraham WT. Prognostic impact of the addition of ventilatory efficiency to the Seattle heart failure model in patients with heart failure. J Card Fail 2012;18: 614-619. 28. Levy WC, Aaronson KD, Dardas TF, Williams P, Haythe J, Mancini D. Prognostic impact of the addition of peak oxygen consumption to the Seattle Heart Failure Model in a transplant referral population. J Heart Lung Transplant 2012;31:817-24. 29. Kato TS, Collado E, Khawaja T, et al. Value of peak exercise oxygen consumption combined with B-type natriuretic peptide levels for optimal timing of cardiac transplantation. Circ Heart Fail 2013;6:6-14.
