International Journal of Cardiology 171 (2014) 265–269
Contents lists available at ScienceDirect
International Journal of Cardiology journal homepage: www.elsevier.com/locate/ijcard
A neural network approach to predicting outcomes in heart failure using cardiopulmonary exercise testing Jonathan Myers a,⁎, Cesar Roberto de Souza b, Audrey Borghi-Silva b, Marco Guazzi c, Paul Chase d, Daniel Bensimhon d, Mary Ann Peberdy e, Euan Ashley f, Erin West g, Lawrence P. Cahalin g, Daniel Forman h, Ross Arena i a
Division of Cardiology, VA Palo Alto Healthcare System/Stanford University, United States Department of Physical Therapy, Federal University Sao Carlos, Brazil c Division of Cardiology, University of Milano, Milano, Italy d Lebauer Cardiovascular Research Foundation, Greensboro, NC, United States e Department of Internal Medicine, Virginia Commonwealth University, Richmond, VA, United States f Cardiovascular Medicine, Stanford University, Palo Alto, CA, United States g Department of Physical Therapy, Leonard M. Miller School of Medicine, University of Miami, Miami, FL, United States h Division of Cardiovascular Medicine, Brigham and Women's Hospital, Boston, MA, United States i Department of Physical Therapy, College of Applied Health Sciences, University of Illinois Chicago, Chicago, IL, United States b
a r t i c l e
i n f o
Article history: Received 25 August 2013 Received in revised form 26 October 2013 Accepted 14 December 2013 Available online 22 December 2013 Keywords: Heart failure Cardiopulmonary exercise testing Mortality Oxygen uptake
a b s t r a c t Objectives: To determine the utility of an artificial neural network (ANN) in predicting cardiovascular (CV) death in patients with heart failure (HF). Background: ANNs use weighted inputs in multiple layers of mathematical connections in order to predict outcomes from multiple risk markers. This approach has not been applied in the context of cardiopulmonary exercise testing (CPX) to predict risk in patients with HF. Methods: 2635 patients with HF underwent CPX and were followed for a mean of 29 ± 30 months. The sample was divided randomly into ANN training and testing sets to predict CV mortality. Peak VO2, VE/VCO2 slope, heart rate recovery, oxygen uptake efficiency slope, and end-tidal CO2 pressure were included in the model. The predictive accuracy of the ANN was compared to logistic regression (LR) and a Cox proportional hazards (PH) score. A multi-layer feed-forward ANN was used and was tested with a single hidden layer containing a varying number of hidden neurons. Results: There were 291 CV deaths during the follow-up. An abnormal VE/VCO2 slope was the strongest predictor of CV mortality using conventional PH analysis (hazard ratio 3.04; 95% CI 2.2–4.2, p b 0.001). After training, the ANN was more accurate in predicting CV mortality compared to LR and PH; ROC areas for the ANN, LR, and PH models were 0.72, 0.70, and 0.69, respectively. Age and BMI-adjusted odds ratios were 4.2, 2.6, and 2.9, for ANN, LR, and PH, respectively. Conclusion: An ANN model slightly improves upon conventional methods for estimating CV mortality risk using established CPX responses. Published by Elsevier Ireland Ltd.
1. Introduction Over the last two decades, the cardiopulmonary exercise test (CPX) has become an important procedure for quantifying the degree of exercise intolerance in patients with chronic heart failure (HF). Many recent studies have demonstrated that CPX responses powerfully stratify risk for mortality, hospitalization, and other adverse outcomes in patients with HF [1–3]. While the focus of the CPX has historically centered on ⁎ Corresponding author at: VA Palo Alto Health Care System, Cardiology Division (111C), 3801 Miranda Avenue, Palo Alto, CA 94304, United States. Tel.: + 1 650 493 5000x64661; fax: +1 650 852 3473. E-mail address: [email protected] (J. Myers). 0167-5273/$ – see front matter. Published by Elsevier Ireland Ltd. http://dx.doi.org/10.1016/j.ijcard.2013.12.031
peak VO2 [4,5], more recently ventilatory abnormalities, heart rate recovery (HRR), and other responses have been demonstrated to provide clinically significant and independent information for estimating prognosis in these patients [1–3]. In particular, the recognition that indices of ventilatory inefficiency, alone or in combination with peak VO2, powerfully predict outcomes in HF has been a major advancement for the clinical utility of this procedure [1–3,6]. Other approaches designed to optimize risk stratification have included the application of algorithms that combine peak VO2 with clinical and hemodynamic data [7–9], the combination of several CPX indices [2,10–13] or the application of composite multivariate scores [7,10]. All of these methods have been shown to more accurately estimate risk for adverse events in patients with HF when compared to peak VO2 alone [2,7–13].
266
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
A great deal of effort in recent years has been directed toward the optimal application of CPX variables for estimating risk of adverse outcomes in patients with HF. Integrating several clinical and CPX responses into a single estimate of overall risk is advantageous [2,7–13], but is an obvious challenge for clinicians who likely do not have the time to perform complex, multivariate calculations. In addition, conventional statistical methods for assessing survival, such as proportional hazards analysis, are limited because they assume that a risk marker is associated with an outcome in a linear fashion (e.g. relative risks between subjects are constant over time), which is rarely the case in epidemiological studies. In recent years, artificial neural networks (ANNs), a form of artificial intelligence which processes large amounts of information and considers complex non-linear relationships that inevitably exist in epidemiologic data, have been applied in various domains of medicine [14–16]. ANNs process information by means of networks of interconnected processing elements, similar to neurons. ANNs can detect non-linear relationships between dependent and independent variables and uncover all possible interactions between predictors, and have the capacity to produce more accurate estimations of risk than conventional methods [14–17]. Although ANNs have been successfully used to optimize diagnostic and prognostic performance of numerous tests in medicine [16,18], to our knowledge this technique has not been applied in the context of the CPX for the estimation of risk among patients with HF. In the current study, we hypothesized that by training and testing an ANN using multiple CPX variables based on a validated score [10,19], prediction of risk for adverse events in patients with HF would be improved relative to conventional survival analyses. 2. Methods The study was performed as part of a HF consortium; a multicenter, retrospective analysis including HF patients from the exercise laboratories at the VA Palo Alto Health Care System and Stanford University, Palo Alto, California, San Paolo Hospital, Milan, Italy, Virginia Commonwealth University, Richmond, Virginia, Brigham and Women's Hospital, Boston, Massachusetts, and the LeBauer Cardiovascular Research Foundation, Greensboro, North Carolina. A total of 2635 patients with chronic HF, tested between 1993 and 2010, were included. The sample included 1974 males and 651 females, with a mean age of 56 ± 14 years. Inclusion criteria consisted of a diagnosis of HF [20] and evidence of left ventricular systolic (ejection fraction [EF] b40%) and/or HF with preserved EF by twodimensional echocardiography obtained within one month of exercise testing. HF with preserved systolic function was considered to be present if the ejection fraction was normal (N45%) and the subject had a history of decompensated heart failure. Subjects received routine follow-up care at the five institutions included in the study. All subjects were stable and receiving optimal medical therapy at the time of testing. The subjects completed a written informed consent and institutional review board approval was obtained at each institution.
and/or hospital and outpatient medical chart review. Follow-up was performed by the HF program at each respective institution, providing a high likelihood that all major events were captured. Individuals conducting the CPX were not involved in decisions regarding cause of death or heart transplant/LVAD implantation. 2.3. Statistical analysis NCSS (Kaysville, UT), software was used to perform unpaired t-tests to compare continuous variables, and chi-square tests were used to compare categorical variables between those who experienced a cardiac event and those who did not. Receiver operating characteristic (ROC) curve analysis was used to define optimal threshold values for each CPX response. Optimal thresholds for each of the CPX variables were as follows: VE/VCO2 slope (≥34) abnormal HRR (≤6 beats at 1 min), OUES (≤1.4), PetCO2 (b33 mm Hg), and peak VO2 (≤14 ml·kg−1·min−1). Three sequential approaches were used to evaluate prediction of cardiovascular events. All survival analyses were adjusted for age, BMI and gender. Initially, Cox proportional hazards analysis was used to determine age-adjusted hazard ratios for the 5 CPX variables included in the model, each expressed dichotomously using the threshold value. For comparison with ANN and logistic regression, a composite CPX score was used as described previously [10,19], in which each variable was assigned a weight according to the hazard ratios, and summed to calculate the composite score. Stepwise logistic regression was applied using each of the CPX variables, and odds ratios were calculated along with their 95% confidence intervals. Receiver operating characteristic (ROC) curves were constructed to compare the predictive value of each method for predicting cardiovascular mortality. Z-tests were used to compare the areas under the ROC curves. ANN analyses were performed using Accord.NET Framework (Sao Carlos, Brazil). The ANN employed is commonly viewed as a mathematical model that parallels neurotransmission in biology, in which computer technology is used to model complex relationships to find patterns in data. An ANN consists of a highly interconnected network of processing units (analogous to neurons in biology) including inputs (predictor variables), hidden layers (data processing elements), and outputs (outcomes). The ANN is initially presented with input variables for which the outcome (in the current study, cardiovascular death) is known. At each input node, the inputs are weighted numerically; these weights are randomly derived initially before learning takes place. The weighted sum of the input data is determined and presented to each node in the hidden layer. A transfer function then takes the weighted sum and calculates a value that numerically scales the strength of each node's output. The output of each succeeding layer then becomes the input for the next layer. A major distinction between an ANN and conventional methods is its ability to “learn”. Learning occurs when an output is calculated; it is compared with the known output from the training set, and differences between the two are propagated through the ANN and the weights are adjusted in a repetitive manner. The ANN used a feed-forward multilayer learning strategy with a back-propagation training method. It was created using age, BMI, ejection fraction and the 5 CPX variables included in the proportional hazards model described above. Of the 2635 patients in the sample, 550 randomly selected tests were set aside for testing and were not included in the training set. A feed-forward neural network with a sigmoid activation function was used. The feed-forward ANN is a multi-layer network in which no loops occur in the network path. The Levenberg–Marquardt algorithm was used during the learning phase of the ANN to optimize fitting [27]. The Bayesian regularization method was employed to control over-fitting and enforce capacity control through regularization [28]. The ANN was tested with the random 550 tests using a single hidden layer containing a varying number of hidden neurons.
2.1. CPX procedure and data collection Symptom-limited CPX was performed on all patients using treadmill or cycle ergometer ramping protocols [21]. A cycle ergometer was used at Brigham and Women's Hospital and in the European center, while a treadmill was used at the other centers. We previously observed that optimal peak VO2 and VE/VCO2 slope threshold values for estimating prognosis were similar irrespective of mode of exercise in patients with HF [22]. Ventilatory expired gas analysis was performed using a metabolic cart at all five centers (Medgraphics CPX-D or ULTIMA PFX, Minneapolis, MN, Orca Diagnostics, Santa Barbara, CA, Parvo Medics TrueOne 2400, Sandy, UT or CareFusion Oxycon Pro, San Diego, CA). Before each test, the equipment was calibrated in standard fashion using reference gasses. A standard 12-lead electrocardiogram was obtained at rest, each minute during exercise, and for at least 5 min during the recovery phase; blood pressure was measured using a standard cuff sphygmomanometer. Minute ventilation (VE, BTPS), oxygen uptake (VO2, STPD), carbon dioxide production (VCO2, STPD) and other CPX variables were acquired breath-by-breath and averaged over 10- or 15-second intervals. VE and VCO2 responses throughout exercise were used to calculate the VE/VCO2 slope via least squares linear regression (y = mx + b, m = slope). Previous work by our group and others has shown this method of calculating the VE/VCO2 slope to be optimal for estimating prognosis [23,24]. The OUES was calculated using [(VO2 (l/min) = m (log10VE) + b, where m = OUES)] [25]. Heart rate recovery was defined as [maximal heart rate minus heart rate at 1 min in recovery] [11]. Resting end-tidal CO2 pressure (PetCO2) was derived from the average of a two minute sitting resting period prior to the test [26]. 2.2. Endpoints The endpoint was cardiac-related mortality. Subjects were followed for major cardiacrelated events for three years after their exercise test using the Social Security Death Index
3. Results During a mean follow-up of 2.4 ± 2.5 years, 291 subjects died from cardiovascular causes. Clinical characteristics and exercise test responses between survivors and those who experienced cardiovascular death are shown in Table 1. Subjects who died were older, had a lower ejection fraction and a higher New York Heart Association class. Among CPX variables, peak VO2 (18.7 ± 8.5 vs. 14.1 ± 5.5 ml·kg−1·min−1), peak heart rate, HRR, OUES, and PetCO2 were higher among those with no events. Conversely, resting heart rate, the VE/VCO2 slope and the CPX weighted summed score were lower among survivors vs. those who died from cardiac causes. Age-adjusted CPX predictors of cardiovascular mortality are presented in Table 2. Each component of the CPX score was significantly associated with cardiovascular mortality. The VE/VCO2 slope generated the highest hazard ratio and an ROC area similar to that of peak VO2 (0.72), while an abnormal OUES had an ROC area of 0.75 (all p b 0.01). Test performance characteristics of the CPX variables from logistic regression analysis are presented in Table 3. All the CPX variables except the OUES were significant predictors of cardiovascular mortality. The highest odds ratio for predicting mortality was generated by an abnormal VE/VCO2 slope (OR 2.83, 95% CI 2.4–3.3, p b 0.001). A comparison
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
267
Table 1 Clinical characteristics and exercise test responses between survivors and non-survivors (mean ± SD). Variables
All Subjects (n = 2635)
Survivors (n = 2344)
Cardiovascular death (n = 291)
p-Value⁎
Age (years) BMI (kg/m2) Ejection fraction (%) NYHA class
55.78 ± 14.29 28.52 ± 6.01 35.46 ± 15.77 2.42 ± 0.84
55.1 ± 14.3 28.6 ± 6.1 36.03 ± 15.9 2.38 ± 0.84
60.9 27.8 31.1 2.69
b0.0001 0.024 b0.0001 0.001
Medications, n(%) Beta blocker Ace inhibitor Diuretics
1611 (61.1) 1469 (55.7) 1248 (47.4)
1450 (61.8) 1301 (55.4) 1104 (47.1)
161 (55.3) 168 (57.7) 144 (49.5)
0.04 0.47 0.004
75 ± 14 129 ± 26 18.15 ± 8.32
75 ± 14 130 ± 27 18.70 ± 8.5
79 ± 15 119 ± 22 14.1 ± 5.5
b0.0001 b0.0001 b0.0001
33.35 ± 9.23 19.29 ± 13.2 2.03 ± 0.88 33.69 ± 4.61 5.59 ± 4.41
32.70 ± 8.8 20.26 ± 13.2 2.08 ± 0.89 33.73 ± 4.87 5.31 ± 4.27
38.3 ± 11 13.06 ± 11.04 1.58 ± 0.64 32.43 ± 4.22 7.89 ± 4.84
b0.0001 b0.0001 b0.0001 0.21 b0.0001
Exercise test responses Resting heart rate (b/min) Peak heart rate (b/min) Peak SBP (mm Hg) Peak VO2 (ml·kg−1·min−1) VE/VCO2 slope HRR (beats) OUES PetCO2 (mm Hg) Weighted summed score
± ± ± ±
13 5.6 14.4 0.75
NYHA = New York Heart Association; HRR = heart rate recovery 1-minute post exercise; OUES = oxygen uptake efficiency slope; PetCO2 = end-tidal CO2 pressure. ⁎ p-Value reflects comparison between survivors and non-survivors.
Table 2 Test performance characteristics for predicting cardiovascular mortality from CPX variables included in the neural network analysis. Variables
Hazard ratio
95% CI
ROC area
Sensitivity
Specificity
Wald p value
Likelihood ratio p value
Peak VO2 VE/VCO2 slope HRR OUES PetCO2
1.72 3.04 2.17 1.59 1.34
1.3–2.3 2.2–4.2 1.4–3.3 1.1–2.3 0.99–1.8
0.72 0.72 0.68 0.75 0.63
76 72 73 75 71
55 62 61 56 47
b0.0001 b0.0001 0.0001 0.008 0.06
b0.001 b0.001 0.001 0.008 0.06
Hazard ratios from proportional hazards analysis, adjusted for age, BMI, and ejection fraction; HRR —heart rate recovery at 1-minute; OUES —oxygen uptake efficiency slope; PetCO2 —endtidal CO2 pressure.
of the performance characteristics for predicting mortality using ANN (from the testing set), logistic regression, and proportional hazards analysis is presented in Table 4. The ANN had a slightly higher ROC area, sensitivity, and odds ratio compared to logistic regression and proportional hazards analysis (ROC areas 0.72, 0.70, and 0.69, respectively, Fig. 1). Differences in ROC areas between the three methods were not significant. Fig. 2 illustrates ROC areas produced by the number of hidden neurons; performance of the ANN peaked at 5 hidden neurons, after which a reduction in the ROC occurred due to overfitting. 4. Discussion The appeal of ANNs is evident from their escalating use in various fields over the last two decades [14,16]. The advantages of ANNs include their ability to explore multiple hidden patterns to find non-linear interactions between independent variables, their freedom from statistical assumptions, and their ability to “learn” [17,18,29–31]. In cardiovascular medicine, these techniques have been applied to improve prediction of mortality risk from resting ECGs [32], to predict acute coronary syndrome or myocardial infarction [33,34], to associate conventional risk factors with 25-year mortality [35], and other applications [36]. To our
Table 3 Test performance characteristics from logistic regression analysis. Variables
Odds ratio
95% CI
Wald test p value
Peak VO2 VE/VCO2 slope HRR OUES PetCO2
1.72 2.83 2.30 1.12 1.20
1.3–1.8 2.4–3.3 1.7–3.1 0.71–1.1 0.71–0.99
b0.001 b0.001 b0.001 0.29 0.04
knowledge, ANNs have not been applied in the context of CPX to predict outcomes in patients with HF. In the current study, we determined the predictive accuracy of established CPX variables using an ANN, and compared it to more conventional methods to estimate risk for cardiovascular mortality. In training the ANN, we employed 5 easily obtained CPX variables that have been demonstrated to be strong predictors of mortality risk in patients with HF [1,2,10]; we recently reported that a multivariate score derived from these variables provided an incremental improvement in the estimation of risk over clinical information and individual CPX responses [19]. Risk estimates using this CPX score and proportional hazards analysis were compared to logistic regression and ANN. Proportional hazards analysis is the most commonly used method for estimating survival, with extensive applications in medicine over many decades. Logistic regression is a statistical fitting model that is highly effective in creating estimates for the likelihood that an event will occur. It is a well-established method used to model medical problems and provides coefficients that have intuitive clinical interpretations [37]. By improving upon these methods, an ANN approach could potentially enhance clinical decision making among patients with HF. Moreover, even by providing similar test characteristics to the more established
Table 4 Performance characteristics for predicting mortality for ANN, logistic regression, and proportional hazards analysis using a CPX score. Model
ROC Area
ROC, SE
Sensitivity
Specificity
Odds Ratio
ANN Logistic Regression Proportional Hazards
0.72 0.70 0.69
0.032 0.033 0.033
0.79 0.71 0.74
0.63 0.65 0.53
4.2 2.6 2.9
ANN = artificial neural network.
268
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
Fig. 1. Receiver operator characteristic (ROC) curves for cardiovascular death using artificial neural network (ANN), logistic regression, and Cox regression analyses (ROC areas 0.72, 0.70.and 0.69, respectively; statistic p = NS).
techniques, ANNs could complement conventional methods by providing a measure of validation in a particular data set [30,38–40]. In addition, the use of ANNs may be justified in the many instances in which the assumptions required by proportional hazards analysis are not met or cannot be verified, a common problem in epidemiological research [41]. Our initial analysis involved the more conventional logistic regression and proportional hazards models. Unlike the ANN, these permit the determination of the most predictive explanatory variables. As we [2,10–12] and others [1,13,24] have observed previously, the VE/VCO2 slope was the strongest predictor of risk, exhibiting the best performance characteristics in the test set (Tables 2–4). This was followed by training and testing the ANN in an independent subset of patients. The ANN, using 3 clinical variables (age, ejection fraction and BMI) along with the 5 CPX variables, exhibited a slightly higher predictive accuracy when compared to proportional hazards and logistic regression. The superior performance of the ANN was evidenced by a higher ROC area, a more optimal balance of sensitivity and specificity, and a higher odds ratio versus the other methods (Table 4). While this suggests a potential role for ANNs in estimating risk based on the CPX, incorporating the ANN model would require
structured reporting software that is currently unavailable in metabolic systems. Such programs would permit the clinician to apply the probability calculations generated by the software and potentially aid in clinical decision making. There have been a number of comparisons between ANN and either logistic regression or proportional hazards analysis for predicting outcomes in various chronic conditions [30,35,38–40,42–44]. Some of these studies have demonstrated better risk estimation from ANNs, but these differences have generally been small; others have shown no differences between ANN and conventional linear-based methods. Although the performance characteristics of the ANN in the current study were not strikingly different from conventional methods, each technique provides particular advantages. ANNs represent a more elegant approach, although inputs and outputs are subtle and non-intuitive. While this better reflects the complexity of clinical dynamics, it may seem disconcerting to clinicians who prefer the relatively more quantifiable and concrete relationships provided by traditional methods. However, logistic regression and proportional hazards are limited in that they are dependent upon assumptions that can distort relationships and result in biases that provide inaccurate estimates of risk. Their appeal is related to the fact that they provide direct estimates of variables that predict outcomes based on coefficients and corresponding odds or hazard ratios which are intuitively attractive to clinicians. While additional studies are needed to explore the utility of ANNs in estimating prognosis from CPX responses, an optimal approach might be the combination of linear and non-linear statistics since each has elements that account for limitations of the other [30,39,40,42,43]. The current results confirm the many studies performed over the last two decades demonstrating the value of CPX responses for estimating mortality risk in patients with HF. Importantly, the VE/VCO2 slope was the most powerful predictor of risk (Tables 2 and 3), which is in agreement with numerous studies over the last decade [1,2,23]. Our results also provide further validation of a multivariate CPX score, which has been shown to provide significant net reclassification improvement compared with individual CPX and clinical variables [19]. Notably, the ANN provided superior risk stratification relative to these well-established predictors in our sample. It should also be noted that the test sample, chosen at random from our larger data set, had a comparatively low mortality rate relative to other studies from our group [2,8,10–12] and others [2,5,13,24]. Since predictive value is strongly related to prevalence, this may explain the comparatively modest prognostic characteristics of the 3 approaches (Table 4). While it would be of value to replicate these findings in an independent sample of patients with HF, the results nevertheless provide a valid comparison between ANN, proportional hazards analysis, and logistic regression.
4.1. Limitations Our main interest in the current study was limited to assessing the performance of an ANN using CPX data; there are many other variables and biomarkers that have been used to define risk in HF, and superior prediction of cardiovascular mortality or other events might be achieved by including additional clinical variables. We only included 5 CPX variables that were part of a validated score [10]; there are other CPX responses that predict risk, particularly oscillatory breathing [1,2], which were not included. In addition, the sample was 75% male, and the results may not be as applicable to women.
4.2. Conclusion
Fig. 2. Receiver–operator characteristic (ROC) areas under the curve (AUC) as the number of hidden neurons increases in the neural network.
These findings suggest that an ANN employing established CPX responses improves modestly upon conventional survival analyses for estimating CV mortality risk in patients with HF.
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
References [1] Balady GJ, Arena R, Sietsema K, et al. Milani RV, on behalf of the American Heart Association Exercise, Cardiac Rehabilitation, and Prevention Committee of the Council on Clinical Cardiology; Council on Epidemiology and Prevention; Council on Peripheral vascular disease; and Interdisciplinary Council on Quality of Care and Outcomes Research. Clinician's guide to cardiopulmonary exercise testing in adults. Circulation 2010;122:191–225. [2] Cahalin LP, Chase P, Arena R, et al. A meta-analysis of the prognostic significance of cardiopulmonary exercise testing in patients with heart failure. Heart Fail Rev 2013;18:79–94. [3] Ingle L. Theoretical rationale and practical recommendations for cardiopulmonary exercise testing in patients with chronic heart failure. Heart Fail Rev 2007;12:12–22. [4] Mancini D, LeJemtel TH. Is ventilatory classification preferable to peak oxygen consumption for risk stratification in heart failure? Circulation 2007;115:2376–8. [5] Mancini DM, Eisen H, Kussmaul W, Mull R, Edmunds Jr LH, Wilson JR. Value of peak exercise oxygen consumption for optimal timing of cardiac transplantation in ambulatory patients with heart failure. Circulation 1991;83:778–86. [6] Tumminello G, Guazzi M, Lancellotti P, Piérard LA. Exercise ventilation inefficiency in heart failure: pathophysiological and clinical significance. Eur Heart J 2007;28: 673–8. [7] Aaronson KD, Schwartz SJ, Chen TM, Wong KL, Goin JE, Mancini DM. Development and prospective validation of a clinical index to predict survival in ambulatory patients referred for cardiac transplant evaluation. Circulation 1997;95: 2660–7. [8] Franco V. Cardiopulmonary exercise test in chronic heart failure: beyond peak oxygen consumption. Curr Heart Fail Rep 2011;8:45–50. [9] Myers J, Wong M, Adhikarla C, et al. Cardiopulmonary and non-invasive hemodynamic responses to exercise predict outcomes in heart failure. J Cardiac Failure 2013;19:101–7. [10] Myers J, Arena R, Dewey F, et al. A cardiopulmonary exercise testing score for predicting outcomes in patients with heart failure. Am Heart J 2008;156:1177–83. [11] Bilsel T, Terzi S, Akbulut T, Sayar N, Hobikoglu G, Yesilcimen K. Abnormal heart rate recovery immediately after cardiopulmonary exercise testing in heart failure patients. Int Heart J 2006;47:431–40. [12] Myers J, Arena R, Oliveira RB, et al. The lowest VE/VCO2 ratio during exercise as a predictor of outcomes in patients with heart failure. J Card Fail 2009;15:756–62. [13] Francis DP, Shamim W, Davies LC, et al. Cardiopulmonary exercise testing for prognosis in chronic heart failure: continuous and independent prognostic value from VE/VCO2 slope and peak VO2. Eur Heart J 2000;21:154–61. [14] Zou J, Han Y, So SS. Overview of artificial neural networks. Methods Mol Biol 2008;458:15–23. [15] Krogh A. What are artificial neural networks? Nat Biotechnol 2008;2:195–7. [16] Patel JL, Goyal RK. Applications of artificial neural networks in medical science. Curr Clin Pharmacol 2007;2:217–26. [17] Rodvold DM, McLeod DG, Brandt JM, Snow PB, Murphy GP. Introduction to artificial neural networks for physicians: taking the lid off the black box. Prostate 2001;46: 39–44. [18] Papik K, Molnar B, Schaefer R, Dombovari Z, Tulassay Z, Feher J. Application of neural networks in medicine: a review. Med Sci Monit 1998;4:538–46. [19] Myers J, Oliveira R, Dewey R, et al. Validation of a cardiopulmonary exercise test score in heart failure. Circ Heart Fail 2013;6:211–8. [20] Radford MJ, Arnold JM, Bennett SJ, et al. ACC/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with chronic heart failure: a report of the American College of Cardiology/American Heart Association Task Force on clinical data standards. Circulation 2005;112:1888–916. [21] Myers J, Buchanan N, Walsh D, et al. Comparison of the ramp versus standard exercise protocols. J Am Coll Cardiol 1991;17:1334–42.
269
[22] Arena R, Guazzi M, Myers J, Peberdy AM. Prognostic characteristics of cardiopulmonary exercise testing in heart failure: comparing American and European models. Eur J Cardiovasc Prev Rehabil 2005;12:562–7. [23] Arena R, Myers J, Aslam S, Varughese EB, Peberdy MA. Technical considerations related to the minute ventilation/carbon dioxide output slope in patients with heart failure. Chest 2003;124:720–7. [24] Bard RL, Gillespie BW, Clarke NS, Egan TG, Nicklas JM. Determining the best ventilatory efficiency measure to predict mortality in patients with heart failure. J Heart Lung Transplant 2006;25:589–95. [25] Baba R, Nagashima M, Goto M, et al. Oxygen uptake efficiency slope: a new index of cardiorespiratory functional reserve derived from the relation between oxygen uptake and minute ventilation during incremental exercise. J Am Coll Cardiol 1996;28:1567–72. [26] Arena R, Guazzi M, Myers J. Prognostic value of end-tidal carbon dioxide during exercise testing in heart failure. Int J Cardiol 2007;117:103–8. [27] Hagan MT, Menhaj MB. Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw 1994;5:989–93. [28] Pujol J. The solution of nonlinear inverse problems and the Levenberg–Marquardt method. Geophys (SEG) 2007;72:W1–W16. [29] Burden F, Winkler D. Bayesian regularization of neural networks. Methods Mol Biol 2008;458:25–44. [30] Tu JV. Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. J Clin Epidemiol 1996;49:1225–31. [31] Ramlall I. Artificial intelligence: neural networks simplified. Int Res J Fin Econ 2010;39:105–20. [32] Perez MV, Dewey FE, Yan SY, Myers J, Froelicher VF. Added value of a resting ECG neural network that predicts cardiovascular mortality. Ann Noninvasive Electrocardiol 2009;14:26–34. [33] Kennedy RL, Harrison RF, Burton AM, et al. An artificial neural network system for diagnosis of acute myocardial infarction (AMI) in the accident and emergency department: evaluation and comparison with serum myoglobin measurements. Comput Methods Prog Biomed 1997;52:93–103. [34] Harrison RF, Kennedy RL. Artificial neural network models for prediction of acute coronary syndromes using clinical data from the time of presentation. Ann Emerg Med 2005;46:431–9. [35] Puddu PE, Menotti A. Artificial neural network versus multiple logistic function to predict 25-year coronary heart disease mortality in the seven countries study. Eur J Cardiovasc Prev Rehabil 2009;16:583–91. [36] Itchhaporia D, Snow PB, Almassy RJ, Oetgen WJ. Artificial neural networks: current status in cardiovascular medicine. J Am Coll Cardiol 1996;28:515–21. [37] Hosmer DW, Lemeshow S. Applied logistic regression. New York, NY: Wiley; 1989. [38] Ayer T, Chhatwal J, Alagoz O, Kahn CE, Woods RW, Burnside ES. Comparison of logistic regression of artificial neural network models in breast cancer risk estimation. RadioGraphics 2010;30:13–22. [39] Freeman RV, Eagle KA, Bates ER, et al. Comparison of artificial neural networks with logistic regression in prediction of in-hospital death after percutaneous transluminal coronary angioplasty. Am Heart J 2000;140:511–20. [40] Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform 2002;35:352–9. [41] Fletcher RH, Fletcher SW. Clinical epidemiology: the essentials. Baltimore: Lippincott, Williams & Wilkins; 2005. [42] Suka M, Oeda S, Ichimura T, Yoshida K, Takezawa J. Comparison of proportional hazard model and neural network models in a real data set of intensive care unit patients. Stud Health Technol Inform 2004;107:741–5. [43] Ohno-Machado L. A comparison of cox proportional hazards and artificial neural network models for medical prognosis. Comput Biol Med 1997;27:55–65. [44] Jacob AN, Khuder S, Malhotra N, et al. Neural network analysis to predict mortality in end-stage renal disease: application to United States renal data system. Nephron Clin Pract 2010;116:c148–58.
Contents lists available at ScienceDirect
International Journal of Cardiology journal homepage: www.elsevier.com/locate/ijcard
A neural network approach to predicting outcomes in heart failure using cardiopulmonary exercise testing Jonathan Myers a,⁎, Cesar Roberto de Souza b, Audrey Borghi-Silva b, Marco Guazzi c, Paul Chase d, Daniel Bensimhon d, Mary Ann Peberdy e, Euan Ashley f, Erin West g, Lawrence P. Cahalin g, Daniel Forman h, Ross Arena i a
Division of Cardiology, VA Palo Alto Healthcare System/Stanford University, United States Department of Physical Therapy, Federal University Sao Carlos, Brazil c Division of Cardiology, University of Milano, Milano, Italy d Lebauer Cardiovascular Research Foundation, Greensboro, NC, United States e Department of Internal Medicine, Virginia Commonwealth University, Richmond, VA, United States f Cardiovascular Medicine, Stanford University, Palo Alto, CA, United States g Department of Physical Therapy, Leonard M. Miller School of Medicine, University of Miami, Miami, FL, United States h Division of Cardiovascular Medicine, Brigham and Women's Hospital, Boston, MA, United States i Department of Physical Therapy, College of Applied Health Sciences, University of Illinois Chicago, Chicago, IL, United States b
a r t i c l e
i n f o
Article history: Received 25 August 2013 Received in revised form 26 October 2013 Accepted 14 December 2013 Available online 22 December 2013 Keywords: Heart failure Cardiopulmonary exercise testing Mortality Oxygen uptake
a b s t r a c t Objectives: To determine the utility of an artificial neural network (ANN) in predicting cardiovascular (CV) death in patients with heart failure (HF). Background: ANNs use weighted inputs in multiple layers of mathematical connections in order to predict outcomes from multiple risk markers. This approach has not been applied in the context of cardiopulmonary exercise testing (CPX) to predict risk in patients with HF. Methods: 2635 patients with HF underwent CPX and were followed for a mean of 29 ± 30 months. The sample was divided randomly into ANN training and testing sets to predict CV mortality. Peak VO2, VE/VCO2 slope, heart rate recovery, oxygen uptake efficiency slope, and end-tidal CO2 pressure were included in the model. The predictive accuracy of the ANN was compared to logistic regression (LR) and a Cox proportional hazards (PH) score. A multi-layer feed-forward ANN was used and was tested with a single hidden layer containing a varying number of hidden neurons. Results: There were 291 CV deaths during the follow-up. An abnormal VE/VCO2 slope was the strongest predictor of CV mortality using conventional PH analysis (hazard ratio 3.04; 95% CI 2.2–4.2, p b 0.001). After training, the ANN was more accurate in predicting CV mortality compared to LR and PH; ROC areas for the ANN, LR, and PH models were 0.72, 0.70, and 0.69, respectively. Age and BMI-adjusted odds ratios were 4.2, 2.6, and 2.9, for ANN, LR, and PH, respectively. Conclusion: An ANN model slightly improves upon conventional methods for estimating CV mortality risk using established CPX responses. Published by Elsevier Ireland Ltd.
1. Introduction Over the last two decades, the cardiopulmonary exercise test (CPX) has become an important procedure for quantifying the degree of exercise intolerance in patients with chronic heart failure (HF). Many recent studies have demonstrated that CPX responses powerfully stratify risk for mortality, hospitalization, and other adverse outcomes in patients with HF [1–3]. While the focus of the CPX has historically centered on ⁎ Corresponding author at: VA Palo Alto Health Care System, Cardiology Division (111C), 3801 Miranda Avenue, Palo Alto, CA 94304, United States. Tel.: + 1 650 493 5000x64661; fax: +1 650 852 3473. E-mail address: [email protected] (J. Myers). 0167-5273/$ – see front matter. Published by Elsevier Ireland Ltd. http://dx.doi.org/10.1016/j.ijcard.2013.12.031
peak VO2 [4,5], more recently ventilatory abnormalities, heart rate recovery (HRR), and other responses have been demonstrated to provide clinically significant and independent information for estimating prognosis in these patients [1–3]. In particular, the recognition that indices of ventilatory inefficiency, alone or in combination with peak VO2, powerfully predict outcomes in HF has been a major advancement for the clinical utility of this procedure [1–3,6]. Other approaches designed to optimize risk stratification have included the application of algorithms that combine peak VO2 with clinical and hemodynamic data [7–9], the combination of several CPX indices [2,10–13] or the application of composite multivariate scores [7,10]. All of these methods have been shown to more accurately estimate risk for adverse events in patients with HF when compared to peak VO2 alone [2,7–13].
266
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
A great deal of effort in recent years has been directed toward the optimal application of CPX variables for estimating risk of adverse outcomes in patients with HF. Integrating several clinical and CPX responses into a single estimate of overall risk is advantageous [2,7–13], but is an obvious challenge for clinicians who likely do not have the time to perform complex, multivariate calculations. In addition, conventional statistical methods for assessing survival, such as proportional hazards analysis, are limited because they assume that a risk marker is associated with an outcome in a linear fashion (e.g. relative risks between subjects are constant over time), which is rarely the case in epidemiological studies. In recent years, artificial neural networks (ANNs), a form of artificial intelligence which processes large amounts of information and considers complex non-linear relationships that inevitably exist in epidemiologic data, have been applied in various domains of medicine [14–16]. ANNs process information by means of networks of interconnected processing elements, similar to neurons. ANNs can detect non-linear relationships between dependent and independent variables and uncover all possible interactions between predictors, and have the capacity to produce more accurate estimations of risk than conventional methods [14–17]. Although ANNs have been successfully used to optimize diagnostic and prognostic performance of numerous tests in medicine [16,18], to our knowledge this technique has not been applied in the context of the CPX for the estimation of risk among patients with HF. In the current study, we hypothesized that by training and testing an ANN using multiple CPX variables based on a validated score [10,19], prediction of risk for adverse events in patients with HF would be improved relative to conventional survival analyses. 2. Methods The study was performed as part of a HF consortium; a multicenter, retrospective analysis including HF patients from the exercise laboratories at the VA Palo Alto Health Care System and Stanford University, Palo Alto, California, San Paolo Hospital, Milan, Italy, Virginia Commonwealth University, Richmond, Virginia, Brigham and Women's Hospital, Boston, Massachusetts, and the LeBauer Cardiovascular Research Foundation, Greensboro, North Carolina. A total of 2635 patients with chronic HF, tested between 1993 and 2010, were included. The sample included 1974 males and 651 females, with a mean age of 56 ± 14 years. Inclusion criteria consisted of a diagnosis of HF [20] and evidence of left ventricular systolic (ejection fraction [EF] b40%) and/or HF with preserved EF by twodimensional echocardiography obtained within one month of exercise testing. HF with preserved systolic function was considered to be present if the ejection fraction was normal (N45%) and the subject had a history of decompensated heart failure. Subjects received routine follow-up care at the five institutions included in the study. All subjects were stable and receiving optimal medical therapy at the time of testing. The subjects completed a written informed consent and institutional review board approval was obtained at each institution.
and/or hospital and outpatient medical chart review. Follow-up was performed by the HF program at each respective institution, providing a high likelihood that all major events were captured. Individuals conducting the CPX were not involved in decisions regarding cause of death or heart transplant/LVAD implantation. 2.3. Statistical analysis NCSS (Kaysville, UT), software was used to perform unpaired t-tests to compare continuous variables, and chi-square tests were used to compare categorical variables between those who experienced a cardiac event and those who did not. Receiver operating characteristic (ROC) curve analysis was used to define optimal threshold values for each CPX response. Optimal thresholds for each of the CPX variables were as follows: VE/VCO2 slope (≥34) abnormal HRR (≤6 beats at 1 min), OUES (≤1.4), PetCO2 (b33 mm Hg), and peak VO2 (≤14 ml·kg−1·min−1). Three sequential approaches were used to evaluate prediction of cardiovascular events. All survival analyses were adjusted for age, BMI and gender. Initially, Cox proportional hazards analysis was used to determine age-adjusted hazard ratios for the 5 CPX variables included in the model, each expressed dichotomously using the threshold value. For comparison with ANN and logistic regression, a composite CPX score was used as described previously [10,19], in which each variable was assigned a weight according to the hazard ratios, and summed to calculate the composite score. Stepwise logistic regression was applied using each of the CPX variables, and odds ratios were calculated along with their 95% confidence intervals. Receiver operating characteristic (ROC) curves were constructed to compare the predictive value of each method for predicting cardiovascular mortality. Z-tests were used to compare the areas under the ROC curves. ANN analyses were performed using Accord.NET Framework (Sao Carlos, Brazil). The ANN employed is commonly viewed as a mathematical model that parallels neurotransmission in biology, in which computer technology is used to model complex relationships to find patterns in data. An ANN consists of a highly interconnected network of processing units (analogous to neurons in biology) including inputs (predictor variables), hidden layers (data processing elements), and outputs (outcomes). The ANN is initially presented with input variables for which the outcome (in the current study, cardiovascular death) is known. At each input node, the inputs are weighted numerically; these weights are randomly derived initially before learning takes place. The weighted sum of the input data is determined and presented to each node in the hidden layer. A transfer function then takes the weighted sum and calculates a value that numerically scales the strength of each node's output. The output of each succeeding layer then becomes the input for the next layer. A major distinction between an ANN and conventional methods is its ability to “learn”. Learning occurs when an output is calculated; it is compared with the known output from the training set, and differences between the two are propagated through the ANN and the weights are adjusted in a repetitive manner. The ANN used a feed-forward multilayer learning strategy with a back-propagation training method. It was created using age, BMI, ejection fraction and the 5 CPX variables included in the proportional hazards model described above. Of the 2635 patients in the sample, 550 randomly selected tests were set aside for testing and were not included in the training set. A feed-forward neural network with a sigmoid activation function was used. The feed-forward ANN is a multi-layer network in which no loops occur in the network path. The Levenberg–Marquardt algorithm was used during the learning phase of the ANN to optimize fitting [27]. The Bayesian regularization method was employed to control over-fitting and enforce capacity control through regularization [28]. The ANN was tested with the random 550 tests using a single hidden layer containing a varying number of hidden neurons.
2.1. CPX procedure and data collection Symptom-limited CPX was performed on all patients using treadmill or cycle ergometer ramping protocols [21]. A cycle ergometer was used at Brigham and Women's Hospital and in the European center, while a treadmill was used at the other centers. We previously observed that optimal peak VO2 and VE/VCO2 slope threshold values for estimating prognosis were similar irrespective of mode of exercise in patients with HF [22]. Ventilatory expired gas analysis was performed using a metabolic cart at all five centers (Medgraphics CPX-D or ULTIMA PFX, Minneapolis, MN, Orca Diagnostics, Santa Barbara, CA, Parvo Medics TrueOne 2400, Sandy, UT or CareFusion Oxycon Pro, San Diego, CA). Before each test, the equipment was calibrated in standard fashion using reference gasses. A standard 12-lead electrocardiogram was obtained at rest, each minute during exercise, and for at least 5 min during the recovery phase; blood pressure was measured using a standard cuff sphygmomanometer. Minute ventilation (VE, BTPS), oxygen uptake (VO2, STPD), carbon dioxide production (VCO2, STPD) and other CPX variables were acquired breath-by-breath and averaged over 10- or 15-second intervals. VE and VCO2 responses throughout exercise were used to calculate the VE/VCO2 slope via least squares linear regression (y = mx + b, m = slope). Previous work by our group and others has shown this method of calculating the VE/VCO2 slope to be optimal for estimating prognosis [23,24]. The OUES was calculated using [(VO2 (l/min) = m (log10VE) + b, where m = OUES)] [25]. Heart rate recovery was defined as [maximal heart rate minus heart rate at 1 min in recovery] [11]. Resting end-tidal CO2 pressure (PetCO2) was derived from the average of a two minute sitting resting period prior to the test [26]. 2.2. Endpoints The endpoint was cardiac-related mortality. Subjects were followed for major cardiacrelated events for three years after their exercise test using the Social Security Death Index
3. Results During a mean follow-up of 2.4 ± 2.5 years, 291 subjects died from cardiovascular causes. Clinical characteristics and exercise test responses between survivors and those who experienced cardiovascular death are shown in Table 1. Subjects who died were older, had a lower ejection fraction and a higher New York Heart Association class. Among CPX variables, peak VO2 (18.7 ± 8.5 vs. 14.1 ± 5.5 ml·kg−1·min−1), peak heart rate, HRR, OUES, and PetCO2 were higher among those with no events. Conversely, resting heart rate, the VE/VCO2 slope and the CPX weighted summed score were lower among survivors vs. those who died from cardiac causes. Age-adjusted CPX predictors of cardiovascular mortality are presented in Table 2. Each component of the CPX score was significantly associated with cardiovascular mortality. The VE/VCO2 slope generated the highest hazard ratio and an ROC area similar to that of peak VO2 (0.72), while an abnormal OUES had an ROC area of 0.75 (all p b 0.01). Test performance characteristics of the CPX variables from logistic regression analysis are presented in Table 3. All the CPX variables except the OUES were significant predictors of cardiovascular mortality. The highest odds ratio for predicting mortality was generated by an abnormal VE/VCO2 slope (OR 2.83, 95% CI 2.4–3.3, p b 0.001). A comparison
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
267
Table 1 Clinical characteristics and exercise test responses between survivors and non-survivors (mean ± SD). Variables
All Subjects (n = 2635)
Survivors (n = 2344)
Cardiovascular death (n = 291)
p-Value⁎
Age (years) BMI (kg/m2) Ejection fraction (%) NYHA class
55.78 ± 14.29 28.52 ± 6.01 35.46 ± 15.77 2.42 ± 0.84
55.1 ± 14.3 28.6 ± 6.1 36.03 ± 15.9 2.38 ± 0.84
60.9 27.8 31.1 2.69
b0.0001 0.024 b0.0001 0.001
Medications, n(%) Beta blocker Ace inhibitor Diuretics
1611 (61.1) 1469 (55.7) 1248 (47.4)
1450 (61.8) 1301 (55.4) 1104 (47.1)
161 (55.3) 168 (57.7) 144 (49.5)
0.04 0.47 0.004
75 ± 14 129 ± 26 18.15 ± 8.32
75 ± 14 130 ± 27 18.70 ± 8.5
79 ± 15 119 ± 22 14.1 ± 5.5
b0.0001 b0.0001 b0.0001
33.35 ± 9.23 19.29 ± 13.2 2.03 ± 0.88 33.69 ± 4.61 5.59 ± 4.41
32.70 ± 8.8 20.26 ± 13.2 2.08 ± 0.89 33.73 ± 4.87 5.31 ± 4.27
38.3 ± 11 13.06 ± 11.04 1.58 ± 0.64 32.43 ± 4.22 7.89 ± 4.84
b0.0001 b0.0001 b0.0001 0.21 b0.0001
Exercise test responses Resting heart rate (b/min) Peak heart rate (b/min) Peak SBP (mm Hg) Peak VO2 (ml·kg−1·min−1) VE/VCO2 slope HRR (beats) OUES PetCO2 (mm Hg) Weighted summed score
± ± ± ±
13 5.6 14.4 0.75
NYHA = New York Heart Association; HRR = heart rate recovery 1-minute post exercise; OUES = oxygen uptake efficiency slope; PetCO2 = end-tidal CO2 pressure. ⁎ p-Value reflects comparison between survivors and non-survivors.
Table 2 Test performance characteristics for predicting cardiovascular mortality from CPX variables included in the neural network analysis. Variables
Hazard ratio
95% CI
ROC area
Sensitivity
Specificity
Wald p value
Likelihood ratio p value
Peak VO2 VE/VCO2 slope HRR OUES PetCO2
1.72 3.04 2.17 1.59 1.34
1.3–2.3 2.2–4.2 1.4–3.3 1.1–2.3 0.99–1.8
0.72 0.72 0.68 0.75 0.63
76 72 73 75 71
55 62 61 56 47
b0.0001 b0.0001 0.0001 0.008 0.06
b0.001 b0.001 0.001 0.008 0.06
Hazard ratios from proportional hazards analysis, adjusted for age, BMI, and ejection fraction; HRR —heart rate recovery at 1-minute; OUES —oxygen uptake efficiency slope; PetCO2 —endtidal CO2 pressure.
of the performance characteristics for predicting mortality using ANN (from the testing set), logistic regression, and proportional hazards analysis is presented in Table 4. The ANN had a slightly higher ROC area, sensitivity, and odds ratio compared to logistic regression and proportional hazards analysis (ROC areas 0.72, 0.70, and 0.69, respectively, Fig. 1). Differences in ROC areas between the three methods were not significant. Fig. 2 illustrates ROC areas produced by the number of hidden neurons; performance of the ANN peaked at 5 hidden neurons, after which a reduction in the ROC occurred due to overfitting. 4. Discussion The appeal of ANNs is evident from their escalating use in various fields over the last two decades [14,16]. The advantages of ANNs include their ability to explore multiple hidden patterns to find non-linear interactions between independent variables, their freedom from statistical assumptions, and their ability to “learn” [17,18,29–31]. In cardiovascular medicine, these techniques have been applied to improve prediction of mortality risk from resting ECGs [32], to predict acute coronary syndrome or myocardial infarction [33,34], to associate conventional risk factors with 25-year mortality [35], and other applications [36]. To our
Table 3 Test performance characteristics from logistic regression analysis. Variables
Odds ratio
95% CI
Wald test p value
Peak VO2 VE/VCO2 slope HRR OUES PetCO2
1.72 2.83 2.30 1.12 1.20
1.3–1.8 2.4–3.3 1.7–3.1 0.71–1.1 0.71–0.99
b0.001 b0.001 b0.001 0.29 0.04
knowledge, ANNs have not been applied in the context of CPX to predict outcomes in patients with HF. In the current study, we determined the predictive accuracy of established CPX variables using an ANN, and compared it to more conventional methods to estimate risk for cardiovascular mortality. In training the ANN, we employed 5 easily obtained CPX variables that have been demonstrated to be strong predictors of mortality risk in patients with HF [1,2,10]; we recently reported that a multivariate score derived from these variables provided an incremental improvement in the estimation of risk over clinical information and individual CPX responses [19]. Risk estimates using this CPX score and proportional hazards analysis were compared to logistic regression and ANN. Proportional hazards analysis is the most commonly used method for estimating survival, with extensive applications in medicine over many decades. Logistic regression is a statistical fitting model that is highly effective in creating estimates for the likelihood that an event will occur. It is a well-established method used to model medical problems and provides coefficients that have intuitive clinical interpretations [37]. By improving upon these methods, an ANN approach could potentially enhance clinical decision making among patients with HF. Moreover, even by providing similar test characteristics to the more established
Table 4 Performance characteristics for predicting mortality for ANN, logistic regression, and proportional hazards analysis using a CPX score. Model
ROC Area
ROC, SE
Sensitivity
Specificity
Odds Ratio
ANN Logistic Regression Proportional Hazards
0.72 0.70 0.69
0.032 0.033 0.033
0.79 0.71 0.74
0.63 0.65 0.53
4.2 2.6 2.9
ANN = artificial neural network.
268
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
Fig. 1. Receiver operator characteristic (ROC) curves for cardiovascular death using artificial neural network (ANN), logistic regression, and Cox regression analyses (ROC areas 0.72, 0.70.and 0.69, respectively; statistic p = NS).
techniques, ANNs could complement conventional methods by providing a measure of validation in a particular data set [30,38–40]. In addition, the use of ANNs may be justified in the many instances in which the assumptions required by proportional hazards analysis are not met or cannot be verified, a common problem in epidemiological research [41]. Our initial analysis involved the more conventional logistic regression and proportional hazards models. Unlike the ANN, these permit the determination of the most predictive explanatory variables. As we [2,10–12] and others [1,13,24] have observed previously, the VE/VCO2 slope was the strongest predictor of risk, exhibiting the best performance characteristics in the test set (Tables 2–4). This was followed by training and testing the ANN in an independent subset of patients. The ANN, using 3 clinical variables (age, ejection fraction and BMI) along with the 5 CPX variables, exhibited a slightly higher predictive accuracy when compared to proportional hazards and logistic regression. The superior performance of the ANN was evidenced by a higher ROC area, a more optimal balance of sensitivity and specificity, and a higher odds ratio versus the other methods (Table 4). While this suggests a potential role for ANNs in estimating risk based on the CPX, incorporating the ANN model would require
structured reporting software that is currently unavailable in metabolic systems. Such programs would permit the clinician to apply the probability calculations generated by the software and potentially aid in clinical decision making. There have been a number of comparisons between ANN and either logistic regression or proportional hazards analysis for predicting outcomes in various chronic conditions [30,35,38–40,42–44]. Some of these studies have demonstrated better risk estimation from ANNs, but these differences have generally been small; others have shown no differences between ANN and conventional linear-based methods. Although the performance characteristics of the ANN in the current study were not strikingly different from conventional methods, each technique provides particular advantages. ANNs represent a more elegant approach, although inputs and outputs are subtle and non-intuitive. While this better reflects the complexity of clinical dynamics, it may seem disconcerting to clinicians who prefer the relatively more quantifiable and concrete relationships provided by traditional methods. However, logistic regression and proportional hazards are limited in that they are dependent upon assumptions that can distort relationships and result in biases that provide inaccurate estimates of risk. Their appeal is related to the fact that they provide direct estimates of variables that predict outcomes based on coefficients and corresponding odds or hazard ratios which are intuitively attractive to clinicians. While additional studies are needed to explore the utility of ANNs in estimating prognosis from CPX responses, an optimal approach might be the combination of linear and non-linear statistics since each has elements that account for limitations of the other [30,39,40,42,43]. The current results confirm the many studies performed over the last two decades demonstrating the value of CPX responses for estimating mortality risk in patients with HF. Importantly, the VE/VCO2 slope was the most powerful predictor of risk (Tables 2 and 3), which is in agreement with numerous studies over the last decade [1,2,23]. Our results also provide further validation of a multivariate CPX score, which has been shown to provide significant net reclassification improvement compared with individual CPX and clinical variables [19]. Notably, the ANN provided superior risk stratification relative to these well-established predictors in our sample. It should also be noted that the test sample, chosen at random from our larger data set, had a comparatively low mortality rate relative to other studies from our group [2,8,10–12] and others [2,5,13,24]. Since predictive value is strongly related to prevalence, this may explain the comparatively modest prognostic characteristics of the 3 approaches (Table 4). While it would be of value to replicate these findings in an independent sample of patients with HF, the results nevertheless provide a valid comparison between ANN, proportional hazards analysis, and logistic regression.
4.1. Limitations Our main interest in the current study was limited to assessing the performance of an ANN using CPX data; there are many other variables and biomarkers that have been used to define risk in HF, and superior prediction of cardiovascular mortality or other events might be achieved by including additional clinical variables. We only included 5 CPX variables that were part of a validated score [10]; there are other CPX responses that predict risk, particularly oscillatory breathing [1,2], which were not included. In addition, the sample was 75% male, and the results may not be as applicable to women.
4.2. Conclusion
Fig. 2. Receiver–operator characteristic (ROC) areas under the curve (AUC) as the number of hidden neurons increases in the neural network.
These findings suggest that an ANN employing established CPX responses improves modestly upon conventional survival analyses for estimating CV mortality risk in patients with HF.
J. Myers et al. / International Journal of Cardiology 171 (2014) 265–269
References [1] Balady GJ, Arena R, Sietsema K, et al. Milani RV, on behalf of the American Heart Association Exercise, Cardiac Rehabilitation, and Prevention Committee of the Council on Clinical Cardiology; Council on Epidemiology and Prevention; Council on Peripheral vascular disease; and Interdisciplinary Council on Quality of Care and Outcomes Research. Clinician's guide to cardiopulmonary exercise testing in adults. Circulation 2010;122:191–225. [2] Cahalin LP, Chase P, Arena R, et al. A meta-analysis of the prognostic significance of cardiopulmonary exercise testing in patients with heart failure. Heart Fail Rev 2013;18:79–94. [3] Ingle L. Theoretical rationale and practical recommendations for cardiopulmonary exercise testing in patients with chronic heart failure. Heart Fail Rev 2007;12:12–22. [4] Mancini D, LeJemtel TH. Is ventilatory classification preferable to peak oxygen consumption for risk stratification in heart failure? Circulation 2007;115:2376–8. [5] Mancini DM, Eisen H, Kussmaul W, Mull R, Edmunds Jr LH, Wilson JR. Value of peak exercise oxygen consumption for optimal timing of cardiac transplantation in ambulatory patients with heart failure. Circulation 1991;83:778–86. [6] Tumminello G, Guazzi M, Lancellotti P, Piérard LA. Exercise ventilation inefficiency in heart failure: pathophysiological and clinical significance. Eur Heart J 2007;28: 673–8. [7] Aaronson KD, Schwartz SJ, Chen TM, Wong KL, Goin JE, Mancini DM. Development and prospective validation of a clinical index to predict survival in ambulatory patients referred for cardiac transplant evaluation. Circulation 1997;95: 2660–7. [8] Franco V. Cardiopulmonary exercise test in chronic heart failure: beyond peak oxygen consumption. Curr Heart Fail Rep 2011;8:45–50. [9] Myers J, Wong M, Adhikarla C, et al. Cardiopulmonary and non-invasive hemodynamic responses to exercise predict outcomes in heart failure. J Cardiac Failure 2013;19:101–7. [10] Myers J, Arena R, Dewey F, et al. A cardiopulmonary exercise testing score for predicting outcomes in patients with heart failure. Am Heart J 2008;156:1177–83. [11] Bilsel T, Terzi S, Akbulut T, Sayar N, Hobikoglu G, Yesilcimen K. Abnormal heart rate recovery immediately after cardiopulmonary exercise testing in heart failure patients. Int Heart J 2006;47:431–40. [12] Myers J, Arena R, Oliveira RB, et al. The lowest VE/VCO2 ratio during exercise as a predictor of outcomes in patients with heart failure. J Card Fail 2009;15:756–62. [13] Francis DP, Shamim W, Davies LC, et al. Cardiopulmonary exercise testing for prognosis in chronic heart failure: continuous and independent prognostic value from VE/VCO2 slope and peak VO2. Eur Heart J 2000;21:154–61. [14] Zou J, Han Y, So SS. Overview of artificial neural networks. Methods Mol Biol 2008;458:15–23. [15] Krogh A. What are artificial neural networks? Nat Biotechnol 2008;2:195–7. [16] Patel JL, Goyal RK. Applications of artificial neural networks in medical science. Curr Clin Pharmacol 2007;2:217–26. [17] Rodvold DM, McLeod DG, Brandt JM, Snow PB, Murphy GP. Introduction to artificial neural networks for physicians: taking the lid off the black box. Prostate 2001;46: 39–44. [18] Papik K, Molnar B, Schaefer R, Dombovari Z, Tulassay Z, Feher J. Application of neural networks in medicine: a review. Med Sci Monit 1998;4:538–46. [19] Myers J, Oliveira R, Dewey R, et al. Validation of a cardiopulmonary exercise test score in heart failure. Circ Heart Fail 2013;6:211–8. [20] Radford MJ, Arnold JM, Bennett SJ, et al. ACC/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with chronic heart failure: a report of the American College of Cardiology/American Heart Association Task Force on clinical data standards. Circulation 2005;112:1888–916. [21] Myers J, Buchanan N, Walsh D, et al. Comparison of the ramp versus standard exercise protocols. J Am Coll Cardiol 1991;17:1334–42.
269
[22] Arena R, Guazzi M, Myers J, Peberdy AM. Prognostic characteristics of cardiopulmonary exercise testing in heart failure: comparing American and European models. Eur J Cardiovasc Prev Rehabil 2005;12:562–7. [23] Arena R, Myers J, Aslam S, Varughese EB, Peberdy MA. Technical considerations related to the minute ventilation/carbon dioxide output slope in patients with heart failure. Chest 2003;124:720–7. [24] Bard RL, Gillespie BW, Clarke NS, Egan TG, Nicklas JM. Determining the best ventilatory efficiency measure to predict mortality in patients with heart failure. J Heart Lung Transplant 2006;25:589–95. [25] Baba R, Nagashima M, Goto M, et al. Oxygen uptake efficiency slope: a new index of cardiorespiratory functional reserve derived from the relation between oxygen uptake and minute ventilation during incremental exercise. J Am Coll Cardiol 1996;28:1567–72. [26] Arena R, Guazzi M, Myers J. Prognostic value of end-tidal carbon dioxide during exercise testing in heart failure. Int J Cardiol 2007;117:103–8. [27] Hagan MT, Menhaj MB. Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw 1994;5:989–93. [28] Pujol J. The solution of nonlinear inverse problems and the Levenberg–Marquardt method. Geophys (SEG) 2007;72:W1–W16. [29] Burden F, Winkler D. Bayesian regularization of neural networks. Methods Mol Biol 2008;458:25–44. [30] Tu JV. Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. J Clin Epidemiol 1996;49:1225–31. [31] Ramlall I. Artificial intelligence: neural networks simplified. Int Res J Fin Econ 2010;39:105–20. [32] Perez MV, Dewey FE, Yan SY, Myers J, Froelicher VF. Added value of a resting ECG neural network that predicts cardiovascular mortality. Ann Noninvasive Electrocardiol 2009;14:26–34. [33] Kennedy RL, Harrison RF, Burton AM, et al. An artificial neural network system for diagnosis of acute myocardial infarction (AMI) in the accident and emergency department: evaluation and comparison with serum myoglobin measurements. Comput Methods Prog Biomed 1997;52:93–103. [34] Harrison RF, Kennedy RL. Artificial neural network models for prediction of acute coronary syndromes using clinical data from the time of presentation. Ann Emerg Med 2005;46:431–9. [35] Puddu PE, Menotti A. Artificial neural network versus multiple logistic function to predict 25-year coronary heart disease mortality in the seven countries study. Eur J Cardiovasc Prev Rehabil 2009;16:583–91. [36] Itchhaporia D, Snow PB, Almassy RJ, Oetgen WJ. Artificial neural networks: current status in cardiovascular medicine. J Am Coll Cardiol 1996;28:515–21. [37] Hosmer DW, Lemeshow S. Applied logistic regression. New York, NY: Wiley; 1989. [38] Ayer T, Chhatwal J, Alagoz O, Kahn CE, Woods RW, Burnside ES. Comparison of logistic regression of artificial neural network models in breast cancer risk estimation. RadioGraphics 2010;30:13–22. [39] Freeman RV, Eagle KA, Bates ER, et al. Comparison of artificial neural networks with logistic regression in prediction of in-hospital death after percutaneous transluminal coronary angioplasty. Am Heart J 2000;140:511–20. [40] Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform 2002;35:352–9. [41] Fletcher RH, Fletcher SW. Clinical epidemiology: the essentials. Baltimore: Lippincott, Williams & Wilkins; 2005. [42] Suka M, Oeda S, Ichimura T, Yoshida K, Takezawa J. Comparison of proportional hazard model and neural network models in a real data set of intensive care unit patients. Stud Health Technol Inform 2004;107:741–5. [43] Ohno-Machado L. A comparison of cox proportional hazards and artificial neural network models for medical prognosis. Comput Biol Med 1997;27:55–65. [44] Jacob AN, Khuder S, Malhotra N, et al. Neural network analysis to predict mortality in end-stage renal disease: application to United States renal data system. Nephron Clin Pract 2010;116:c148–58.
