American Journal of Emergency Medicine xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
American Journal of Emergency Medicine journal homepage: www.elsevier.com/locate/ajem
Original Contribution
A retrospective analysis of the utility of an artificial neural network to predict ED volume☆,☆☆,★,★★ Nathan Benjamin Menke, MD, PhD a, Nicholas Caputo, MD, MSc b,⁎, Robert Fraser, MD b, Jordana Haber, MD b, Christopher Shields, MD b, Marie Nam Menke, MD, MPH c a b c
Division of Medical Toxicology, Department of Emergency Medicine, University of Pittsburgh Medical Center, Pittsburgh, PA Department of Emergency Medicine, Lincoln Medical and Mental Health Center, Bronx, NY Division of Reproductive, Endocrinology, and Infertility, Department of Obstetrics, Gynecology, and Reproductive Sciences, University of Pittsburgh Medical Center, Pittsburgh, PA
a r t i c l e
i n f o
Article history: Received 18 February 2014 Accepted 11 March 2014 Available online xxxx
a b s t r a c t Objective: The objectives of this study are to design an artificial neural network (ANN) and to test it retrospectively to determine if it may be used to predict emergency department (ED) volume. Methods: We conducted a retrospective review of patient registry data from February 4, 2007, to December 31, 2009, from an inner city, tertiary care hospital. We harvested data regarding weather, days of week, air quality, and special events to train the ANN. The ANN belongs to a class of neural networks called multilayer perceptrons. We designed an ANN composed of 37 input neurons, 22 hidden neurons, and 1 output neuron designed to predict the daily number of ED visits. The training method is a supervised backpropagation algorithm that uses mean squared error to minimize the average squared error between the ANN's output and the number of ED visits over all the example pairs. Results: A linear regression between the predicted and actual ED visits demonstrated an R2 of 0.957 with a slope of 0.997. Ninety-five percent of the time, the ANN was within 20 visits. Conclusion: The results of this study show that a properly designed ANN is an effective tool that may be used to predict ED volume. The scatterplot demonstrates that the ANN is least predictive at the extreme ends of the spectrum suggesting that the ANN may be missing important variables. A properly calibrated ANN may have the potential to allow ED administrators to staff their units more appropriately in an effort to reduce patient wait times, decrease ED physician burnout rates, and increase the ability of caregivers to provide quality patient care. A prospective is needed to validate the utility of the ANN. © 2014 Elsevier Inc. All rights reserved.
1. Introduction 1.1. Background Emergency department (ED) overcrowding is a problem that is stressing the nation's safety net. From 1995 through 2005, the annual number of ED visits increased by 20%, from 96.5 million to 115.3 million visits [1,2]. This represents an average increase of more than 1.7 million visits per year. Concomitantly, the number of hospital EDs decreased from 4176 to 3795, thereby increasing the annual number
☆ Grant: None. ☆☆ Conflicts of interest: None. ★ Author contributions: NBM and MNM conceived and designed the study. NBM, CS, NC, RF, and JH participated in the data collection. NBM, CS, NC, RF, and JH managed the data, including quality control. MNM provided statistical advice on study design and analyzed the data. NBM drafted the article, and all authors contributed substantially to its revision. NBM takes responsibility for the article as a whole. ★★ This research was presented as a platform presentation at the 2010 Society for Academic Emergency Medicine Annual Meeting. ⁎ Corresponding author. E-mail address: [email protected] (N. Caputo).
of visits per ED from 23 119 in 1995 to 30 388 in 2005 [1,2]. The rise in ED utilization has effectively saturated the capacity of EDs and the emergency medical services in many communities [3]. Emergency department overcrowding, defined as patient demand outstripping the ED's capacity to provide services, has significant and deleterious effects on both patients and ED staff [4]. Emergency departments must maintain a constant amount of human and technical resources. The aim is to offer patients consistent medical care 24 hours a day and 7 days a week without deterioration in quality or effectiveness [5]. Delays in care translate into prolonged pain and needless suffering for patients. Long waiting times pose an even greater risk if an overwhelmed triage system results in underassessment of patient acuity or if a patient's condition significantly deteriorates while awaiting medical care [6]. The resulting long wait times also promote patient dissatisfaction. The impact on health care workers is more subtle but equally detrimental. The high-stress practice environment of an overcrowded ED contributes to staff burnout, higher turnover rates, and worsening deficiencies in clinical staffing. Physical expansion alone will not be sufficient to meet the needs of ED volume. Patient flow is a dynamic process requiring access to inpatient beds, adequate nursing resources, and timely services from
http://dx.doi.org/10.1016/j.ajem.2014.03.011 0735-6757/© 2014 Elsevier Inc. All rights reserved.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
2
N.B. Menke et al. / American Journal of Emergency Medicine xxx (2014) xxx–xxx
on-call specialists. Therefore, an appropriate alternative to ED expansion is to maximize resource use. Maximization of capacity and patient throughout drives the impetus for use of demand management systems and operational improvements as method to anticipate normal surges and demand for ED services [7,8]. Prediction of demand requires adequate assessment of factors affecting patient flow. To date, studies have demonstrated patterns that result from time of day, climate changes, and air pollution levels. However, accurate methods for predicting demand (ie, patient arrivals to the ED) have eluded investigators thus far. Clinical researchers commonly use univariate analysis as a method of hypothesis testing. In its most common form, univariate analysis consists of computing a P value for each variable of interest; variables are defined as statistically significant if they have a P value lower than an arbitrary threshold. Univariate tests are designed to identify variables that provide a significant amount of information about the output variable in isolation from the other variables. Biological processes are complex systems and oftentimes involve nonlinear interactions between variables; therefore, univariate tests may miss some clinically important information [9]. In this context, machine-based learning and other nonlinear analytical tools pose a rational alternative to current statistical approaches. Technological and theoretical advances in computer science and mathematics offer new options to complement traditional statistical analysis. Through a process known as machine learning, computer algorithms may use empirical data to create probability distributions [10]. A major focus of machine learning research is to automatically learn to recognize complex patterns and make intelligent decisions based on data. In the past decade, machine learning algorithms have revealed previously undetected trends in historical data [11,12]. Such tools include decision tree learning, association rule learning, artificial neural networks (ANNs), genetic programming, and support vector machines. 1.2. Importance of the study Technological advances offer a tool that ED administrators and physicians may adapt to adjust to ever-increasing ED volumes and to help improve patient care. The ANN may be used by health care administrators to determine staffing needs. An ANN is a machine-based learning algorithm that is composed of a large number of interconnected processing elements working together to solve specific problems [13-15]. An ANN's ability to learn by example has attracted the most interest in neural networks. It is an adaptive system that changes its structure based on the information that flows through the network during the learning phase. Ultimately, meaning is derived from complicated or imprecise data. Artificial neural networks are complex and flexible nonlinear systems with properties not found in other modeling systems. These properties include robust performance in dealing with noisy or incomplete input patterns, high fault tolerance, and the ability to generalize from the input data. The properties allow the creation of prediction models from computationally derived patterns based on data that are too complex for humans to appreciate. Artificial neural networks excel at applications where pattern recognition is important and precise computational answers are not required, such as forecasting weather, stock predicting, or speech recognition. Biomedical applications of ANN include diagnosing diseases such as myocardial infarction or pulmonary emboli, radiographic image analysis, and waveform analysis of electrocardiograms [16-20].
2. Methods 2.1. Study design and setting We conducted a retrospective review of patient registry data from February 4, 2007, to December 31, 2009, from an inner city, tertiary care hospital. 2.2. Methods and measurements We harvested data regarding weather, days of week, air quality, and special events to train the ANN. Our ANN belongs to a class of neural networks called multilayer perceptron. The ANN is composed of 37 input neurons, 22 hidden neurons, and 1 output neuron. The training method is supervised backpropagation that uses mean squared error to minimize the average squared error between the ANN's output and the number of ED visits over all the example pairs. The network weights were initiated with random numbers. During a training process, the connection weights between the neurons were adjusted by use of the backpropagation updating algorithm. The learning rate (η) had a start value of 0.5. During the training, η was decreased geometrically every seventh epoch using the following equation: η = kη, with k = 0.99. The initial momentum (α) was set to 0.7. During the training, α was decreased geometrically every 1000th epoch using the following equation: α = kα, with k = 0.99. The stop criteria were defined as when the root mean squared error was less than 0.01. The graphics processing unit (GPU) implementation of the multiple backpropagation algorithm developed by Lopes and Ribeiro [21] was used to implement the ANN. The study was approved by the Lincoln Medical and Mental Health Center Institutional Review Board. 2.3. Outcomes Accurate prediction of ED volume by the ANN from distinct variables was the primary outcome of the study. 2.4. Analysis The statistical package R v2.7.0, Vienna, Austria was used for all statistical calculations [22]. All confidence intervals in the study are at
1.3. Goals of the investigation We describe the creation of an ANN designed to predict ED volume. The primary goal of the study was to determine if the ANN could accurately train itself to predict daily patient census from retrospective data harvesting.
Fig. 1. A histogram of the visits per day.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
N.B. Menke et al. / American Journal of Emergency Medicine xxx (2014) xxx–xxx
3
Table The frequency that the ANN predicted the number of visits within a given number % Within Within Within Within Within
1 visit 2 visits 5 visits 10 visits 20 visits
53 66 83 91 95
3.2. Main outcome Fig. 3 provides a scatterplot of the ANN-predicted visits vs the actual ED visits including a line of best fit calculated using ordinary least squares regression. The adjusted R 2 of 0.957 demonstrated the strong correlation between the predicted number of visits as compared with the observed visits (y = 0.977 * x + 6.39). The Table demonstrates the percentage of time that the ANN was within a given number of visits per day; 95% of the time, the ANN was within 20 visits. The largest prediction error was 88 (predicted 447, actual 359) that overestimated the number of visits by 24.5%. Fig. 2. A boxplot of the ED visits per day. There are several outliers at the upper range of the data.
the 95% level. Supporting histogram and boxplot were provided for a general overview of the distribution of the data. The correlation between the model and the ED visit data was determined using linear regression. 3. Results
3.3. Limitations The primary limitation of this study was its use of data abstracted from a single urban center. The ANN takes into consideration and sets weights to certain variables as determined by the supervised learning algorithm. Not all institutions are busy urban tertiary care hospitals. This must be taken into consideration; however, the network input variables may be tailored to the specific hospital environment. The model also requires validation with data that was not used to train the ANN.
3.1. Characteristics of the study The mean number of visits is 304 (301-307; 95% confidence interval). Figs. 1 and 2 demonstrate a histogram and boxplot of ED visits, respectively. As seen in Fig. 2, a large amount of outliers are noted at the higher end of visits.
4. Discussion Concerns regarding ED overcrowding has led to time series analysis and formation of prediction models. An effective prediction model has important implications for the ED. First, these findings could help reduce business costs for the hospital by allowing for ED staff adjustments according to the daily patient patterns. Second, optimization of inpatient staffing based on predicted volume may reduce the time from admission to patient arrival on the floor thereby increasing throughput in the ED. Third, nonemergent surgeries may be scheduled in such a manner to keep the maximum number of inpatient beds available for days with predicted heavy ED volume. Despite numerous efforts, an accurate model for the prediction of ED volume has yet to be validated. Here, we describe a multilayer perceptron ANN with the ability to accurately predict ED volume. As such, it represents a significant improvement over current models; however, further work is required to maximize its potential and determine its validity. Although the ANN is an effective prediction tool, the scatterplot demonstrates that the ANN is least predictive at the extreme ends of the spectrum suggesting that the ANN may be missing important variables. The next step requires prospective validation of the ANN's ability to predict patient volume. A properly calibrated ANN will allow EDs to staff their units more appropriately in an effort to reduce patient wait times, decreased ED health care workers' burnout rates, and increase the ability of caregivers to provide quality patient care. Consideration should be given to the ANN as a valid model to address staffing needs. References
Fig. 3. The linear correlation between the ANN-predicted visits and the number of observed visits.
[1] Nawar EW, Niska RW, Xu J. National Hospital Ambulatory Medical Care Survey: 2005 emergency department summary. Adv Data 2007;29(386):1–32.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
4
N.B. Menke et al. / American Journal of Emergency Medicine xxx (2014) xxx–xxx
[2] Middleton K, Hing E. National Hospital Ambulatory Medical Care Survey: 2003 outpatient department summary. Adv Data 2005;14(366):1–36. [3] Mitka M. Emergency department overcrowding gives ambulances the runaround. JAMA [News] 2006;295(13):1504–5. [4] Medicine CoPE. Overcrowding crisis in our nation's emergency departments: is our safety net unraveling? Pediatrics 2004;114(3):878–88. [5] Miro O, Sanchez M, Espinosa G, Milla J. Quality and effectiveness of an emergency department during weekends. Emerg Med J 2004;21(5):573–4. [6] Trzeciak S, Rivers EP. Emergency department overcrowding in the United States: an emerging threat to patient safety and public health. Emerg Med J 2003;20 (5):402–5 [Review]. [7] McCarthy ML, Zeger SL, Ding R, Aronsky D, Hoot NR, Kelen GD. The challenge of predicting demand for emergency department services. Acad Emerg Med 2008;15 (4):337–46 [Research support, US Gov't, non-PHS]. [8] Flottemesch TJ, Gordon BD, Jones SS. Advanced statistics: developing a formal model of emergency department census and defining operational efficiency. Acad Emerg Med 2007;14(9):799–809. [9] Huynh-Thu VA, Saeys Y, Wehenkel L, Geurts P. Statistical interpretation of machine learning–based feature importance scores for biomarker discovery. Bioinformatics 2012;28(13):1766–74. [10] Mitchell TM. Mining our reality. Science 2009;326:1644–5. [11] Waltz D, Buchanan BG. Computer science: automating science. Science 2009;324 (5923):43–4. [12] Schmidt M, Lipson H. Distilling free-form natural laws from experimental data. Science 2009;324(5923):81–5.
[13] Schultz E. Artificial neural networks: laboratory aid or sorcerer's apprentice? Clin Chem 1996;42(4):496–7. [14] Baxt W. Application of artificial neural networks to clinical medicine. Lancet 1995;346(8983):1135–8. [15] Griffith J. Artificial neural networks: are they ready for use as clinical decision aids? Med Decis Making 2000;20(2):243–4. [16] Baxt WG. Use of an artificial neural network for the diagnosis of myocardial infarction. Ann Intern Med 1991;115(11):843–8. [17] Devine B, Macfarlane PW. Detection of electrocardiographic “left ventricular strain” using neural nets. Med Biol Eng Comput 1993 Jul;31(4):343–8. [18] Wu Y, Giger ML, Doi K, Vyborny CJ, Schmidt RA, Metz CE. Artificial neural networks in mammography: application to decision making in the diagnosis of breast cancer. Radiology 1993 Apr;187(1):81–7. [19] Dietzel M, Baltzer PA, Dietzel A, et al. Artificial neural networks for differential diagnosis of breast lesions in MR-mammography: a systematic approach addressing the influence of network architecture on diagnostic performance using a large clinical database. Eur J Radiol 2011;81(7):1508–13. [20] Heden B, Ohlin H, Rittner R, Edenbrandt L. Acute myocardial infarction detected in the 12-lead ECG by artificial neural networks. Circulation 1997;96 (6):1798–802. [21] Lopes N, Ribeiro B. An evaluation of multiple feed-forward networks on GPUs. Int J Neural Syst 2011;21(1):31–47. [22] Team RDC. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2009.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
Contents lists available at ScienceDirect
American Journal of Emergency Medicine journal homepage: www.elsevier.com/locate/ajem
Original Contribution
A retrospective analysis of the utility of an artificial neural network to predict ED volume☆,☆☆,★,★★ Nathan Benjamin Menke, MD, PhD a, Nicholas Caputo, MD, MSc b,⁎, Robert Fraser, MD b, Jordana Haber, MD b, Christopher Shields, MD b, Marie Nam Menke, MD, MPH c a b c
Division of Medical Toxicology, Department of Emergency Medicine, University of Pittsburgh Medical Center, Pittsburgh, PA Department of Emergency Medicine, Lincoln Medical and Mental Health Center, Bronx, NY Division of Reproductive, Endocrinology, and Infertility, Department of Obstetrics, Gynecology, and Reproductive Sciences, University of Pittsburgh Medical Center, Pittsburgh, PA
a r t i c l e
i n f o
Article history: Received 18 February 2014 Accepted 11 March 2014 Available online xxxx
a b s t r a c t Objective: The objectives of this study are to design an artificial neural network (ANN) and to test it retrospectively to determine if it may be used to predict emergency department (ED) volume. Methods: We conducted a retrospective review of patient registry data from February 4, 2007, to December 31, 2009, from an inner city, tertiary care hospital. We harvested data regarding weather, days of week, air quality, and special events to train the ANN. The ANN belongs to a class of neural networks called multilayer perceptrons. We designed an ANN composed of 37 input neurons, 22 hidden neurons, and 1 output neuron designed to predict the daily number of ED visits. The training method is a supervised backpropagation algorithm that uses mean squared error to minimize the average squared error between the ANN's output and the number of ED visits over all the example pairs. Results: A linear regression between the predicted and actual ED visits demonstrated an R2 of 0.957 with a slope of 0.997. Ninety-five percent of the time, the ANN was within 20 visits. Conclusion: The results of this study show that a properly designed ANN is an effective tool that may be used to predict ED volume. The scatterplot demonstrates that the ANN is least predictive at the extreme ends of the spectrum suggesting that the ANN may be missing important variables. A properly calibrated ANN may have the potential to allow ED administrators to staff their units more appropriately in an effort to reduce patient wait times, decrease ED physician burnout rates, and increase the ability of caregivers to provide quality patient care. A prospective is needed to validate the utility of the ANN. © 2014 Elsevier Inc. All rights reserved.
1. Introduction 1.1. Background Emergency department (ED) overcrowding is a problem that is stressing the nation's safety net. From 1995 through 2005, the annual number of ED visits increased by 20%, from 96.5 million to 115.3 million visits [1,2]. This represents an average increase of more than 1.7 million visits per year. Concomitantly, the number of hospital EDs decreased from 4176 to 3795, thereby increasing the annual number
☆ Grant: None. ☆☆ Conflicts of interest: None. ★ Author contributions: NBM and MNM conceived and designed the study. NBM, CS, NC, RF, and JH participated in the data collection. NBM, CS, NC, RF, and JH managed the data, including quality control. MNM provided statistical advice on study design and analyzed the data. NBM drafted the article, and all authors contributed substantially to its revision. NBM takes responsibility for the article as a whole. ★★ This research was presented as a platform presentation at the 2010 Society for Academic Emergency Medicine Annual Meeting. ⁎ Corresponding author. E-mail address: [email protected] (N. Caputo).
of visits per ED from 23 119 in 1995 to 30 388 in 2005 [1,2]. The rise in ED utilization has effectively saturated the capacity of EDs and the emergency medical services in many communities [3]. Emergency department overcrowding, defined as patient demand outstripping the ED's capacity to provide services, has significant and deleterious effects on both patients and ED staff [4]. Emergency departments must maintain a constant amount of human and technical resources. The aim is to offer patients consistent medical care 24 hours a day and 7 days a week without deterioration in quality or effectiveness [5]. Delays in care translate into prolonged pain and needless suffering for patients. Long waiting times pose an even greater risk if an overwhelmed triage system results in underassessment of patient acuity or if a patient's condition significantly deteriorates while awaiting medical care [6]. The resulting long wait times also promote patient dissatisfaction. The impact on health care workers is more subtle but equally detrimental. The high-stress practice environment of an overcrowded ED contributes to staff burnout, higher turnover rates, and worsening deficiencies in clinical staffing. Physical expansion alone will not be sufficient to meet the needs of ED volume. Patient flow is a dynamic process requiring access to inpatient beds, adequate nursing resources, and timely services from
http://dx.doi.org/10.1016/j.ajem.2014.03.011 0735-6757/© 2014 Elsevier Inc. All rights reserved.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
2
N.B. Menke et al. / American Journal of Emergency Medicine xxx (2014) xxx–xxx
on-call specialists. Therefore, an appropriate alternative to ED expansion is to maximize resource use. Maximization of capacity and patient throughout drives the impetus for use of demand management systems and operational improvements as method to anticipate normal surges and demand for ED services [7,8]. Prediction of demand requires adequate assessment of factors affecting patient flow. To date, studies have demonstrated patterns that result from time of day, climate changes, and air pollution levels. However, accurate methods for predicting demand (ie, patient arrivals to the ED) have eluded investigators thus far. Clinical researchers commonly use univariate analysis as a method of hypothesis testing. In its most common form, univariate analysis consists of computing a P value for each variable of interest; variables are defined as statistically significant if they have a P value lower than an arbitrary threshold. Univariate tests are designed to identify variables that provide a significant amount of information about the output variable in isolation from the other variables. Biological processes are complex systems and oftentimes involve nonlinear interactions between variables; therefore, univariate tests may miss some clinically important information [9]. In this context, machine-based learning and other nonlinear analytical tools pose a rational alternative to current statistical approaches. Technological and theoretical advances in computer science and mathematics offer new options to complement traditional statistical analysis. Through a process known as machine learning, computer algorithms may use empirical data to create probability distributions [10]. A major focus of machine learning research is to automatically learn to recognize complex patterns and make intelligent decisions based on data. In the past decade, machine learning algorithms have revealed previously undetected trends in historical data [11,12]. Such tools include decision tree learning, association rule learning, artificial neural networks (ANNs), genetic programming, and support vector machines. 1.2. Importance of the study Technological advances offer a tool that ED administrators and physicians may adapt to adjust to ever-increasing ED volumes and to help improve patient care. The ANN may be used by health care administrators to determine staffing needs. An ANN is a machine-based learning algorithm that is composed of a large number of interconnected processing elements working together to solve specific problems [13-15]. An ANN's ability to learn by example has attracted the most interest in neural networks. It is an adaptive system that changes its structure based on the information that flows through the network during the learning phase. Ultimately, meaning is derived from complicated or imprecise data. Artificial neural networks are complex and flexible nonlinear systems with properties not found in other modeling systems. These properties include robust performance in dealing with noisy or incomplete input patterns, high fault tolerance, and the ability to generalize from the input data. The properties allow the creation of prediction models from computationally derived patterns based on data that are too complex for humans to appreciate. Artificial neural networks excel at applications where pattern recognition is important and precise computational answers are not required, such as forecasting weather, stock predicting, or speech recognition. Biomedical applications of ANN include diagnosing diseases such as myocardial infarction or pulmonary emboli, radiographic image analysis, and waveform analysis of electrocardiograms [16-20].
2. Methods 2.1. Study design and setting We conducted a retrospective review of patient registry data from February 4, 2007, to December 31, 2009, from an inner city, tertiary care hospital. 2.2. Methods and measurements We harvested data regarding weather, days of week, air quality, and special events to train the ANN. Our ANN belongs to a class of neural networks called multilayer perceptron. The ANN is composed of 37 input neurons, 22 hidden neurons, and 1 output neuron. The training method is supervised backpropagation that uses mean squared error to minimize the average squared error between the ANN's output and the number of ED visits over all the example pairs. The network weights were initiated with random numbers. During a training process, the connection weights between the neurons were adjusted by use of the backpropagation updating algorithm. The learning rate (η) had a start value of 0.5. During the training, η was decreased geometrically every seventh epoch using the following equation: η = kη, with k = 0.99. The initial momentum (α) was set to 0.7. During the training, α was decreased geometrically every 1000th epoch using the following equation: α = kα, with k = 0.99. The stop criteria were defined as when the root mean squared error was less than 0.01. The graphics processing unit (GPU) implementation of the multiple backpropagation algorithm developed by Lopes and Ribeiro [21] was used to implement the ANN. The study was approved by the Lincoln Medical and Mental Health Center Institutional Review Board. 2.3. Outcomes Accurate prediction of ED volume by the ANN from distinct variables was the primary outcome of the study. 2.4. Analysis The statistical package R v2.7.0, Vienna, Austria was used for all statistical calculations [22]. All confidence intervals in the study are at
1.3. Goals of the investigation We describe the creation of an ANN designed to predict ED volume. The primary goal of the study was to determine if the ANN could accurately train itself to predict daily patient census from retrospective data harvesting.
Fig. 1. A histogram of the visits per day.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
N.B. Menke et al. / American Journal of Emergency Medicine xxx (2014) xxx–xxx
3
Table The frequency that the ANN predicted the number of visits within a given number % Within Within Within Within Within
1 visit 2 visits 5 visits 10 visits 20 visits
53 66 83 91 95
3.2. Main outcome Fig. 3 provides a scatterplot of the ANN-predicted visits vs the actual ED visits including a line of best fit calculated using ordinary least squares regression. The adjusted R 2 of 0.957 demonstrated the strong correlation between the predicted number of visits as compared with the observed visits (y = 0.977 * x + 6.39). The Table demonstrates the percentage of time that the ANN was within a given number of visits per day; 95% of the time, the ANN was within 20 visits. The largest prediction error was 88 (predicted 447, actual 359) that overestimated the number of visits by 24.5%. Fig. 2. A boxplot of the ED visits per day. There are several outliers at the upper range of the data.
the 95% level. Supporting histogram and boxplot were provided for a general overview of the distribution of the data. The correlation between the model and the ED visit data was determined using linear regression. 3. Results
3.3. Limitations The primary limitation of this study was its use of data abstracted from a single urban center. The ANN takes into consideration and sets weights to certain variables as determined by the supervised learning algorithm. Not all institutions are busy urban tertiary care hospitals. This must be taken into consideration; however, the network input variables may be tailored to the specific hospital environment. The model also requires validation with data that was not used to train the ANN.
3.1. Characteristics of the study The mean number of visits is 304 (301-307; 95% confidence interval). Figs. 1 and 2 demonstrate a histogram and boxplot of ED visits, respectively. As seen in Fig. 2, a large amount of outliers are noted at the higher end of visits.
4. Discussion Concerns regarding ED overcrowding has led to time series analysis and formation of prediction models. An effective prediction model has important implications for the ED. First, these findings could help reduce business costs for the hospital by allowing for ED staff adjustments according to the daily patient patterns. Second, optimization of inpatient staffing based on predicted volume may reduce the time from admission to patient arrival on the floor thereby increasing throughput in the ED. Third, nonemergent surgeries may be scheduled in such a manner to keep the maximum number of inpatient beds available for days with predicted heavy ED volume. Despite numerous efforts, an accurate model for the prediction of ED volume has yet to be validated. Here, we describe a multilayer perceptron ANN with the ability to accurately predict ED volume. As such, it represents a significant improvement over current models; however, further work is required to maximize its potential and determine its validity. Although the ANN is an effective prediction tool, the scatterplot demonstrates that the ANN is least predictive at the extreme ends of the spectrum suggesting that the ANN may be missing important variables. The next step requires prospective validation of the ANN's ability to predict patient volume. A properly calibrated ANN will allow EDs to staff their units more appropriately in an effort to reduce patient wait times, decreased ED health care workers' burnout rates, and increase the ability of caregivers to provide quality patient care. Consideration should be given to the ANN as a valid model to address staffing needs. References
Fig. 3. The linear correlation between the ANN-predicted visits and the number of observed visits.
[1] Nawar EW, Niska RW, Xu J. National Hospital Ambulatory Medical Care Survey: 2005 emergency department summary. Adv Data 2007;29(386):1–32.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
4
N.B. Menke et al. / American Journal of Emergency Medicine xxx (2014) xxx–xxx
[2] Middleton K, Hing E. National Hospital Ambulatory Medical Care Survey: 2003 outpatient department summary. Adv Data 2005;14(366):1–36. [3] Mitka M. Emergency department overcrowding gives ambulances the runaround. JAMA [News] 2006;295(13):1504–5. [4] Medicine CoPE. Overcrowding crisis in our nation's emergency departments: is our safety net unraveling? Pediatrics 2004;114(3):878–88. [5] Miro O, Sanchez M, Espinosa G, Milla J. Quality and effectiveness of an emergency department during weekends. Emerg Med J 2004;21(5):573–4. [6] Trzeciak S, Rivers EP. Emergency department overcrowding in the United States: an emerging threat to patient safety and public health. Emerg Med J 2003;20 (5):402–5 [Review]. [7] McCarthy ML, Zeger SL, Ding R, Aronsky D, Hoot NR, Kelen GD. The challenge of predicting demand for emergency department services. Acad Emerg Med 2008;15 (4):337–46 [Research support, US Gov't, non-PHS]. [8] Flottemesch TJ, Gordon BD, Jones SS. Advanced statistics: developing a formal model of emergency department census and defining operational efficiency. Acad Emerg Med 2007;14(9):799–809. [9] Huynh-Thu VA, Saeys Y, Wehenkel L, Geurts P. Statistical interpretation of machine learning–based feature importance scores for biomarker discovery. Bioinformatics 2012;28(13):1766–74. [10] Mitchell TM. Mining our reality. Science 2009;326:1644–5. [11] Waltz D, Buchanan BG. Computer science: automating science. Science 2009;324 (5923):43–4. [12] Schmidt M, Lipson H. Distilling free-form natural laws from experimental data. Science 2009;324(5923):81–5.
[13] Schultz E. Artificial neural networks: laboratory aid or sorcerer's apprentice? Clin Chem 1996;42(4):496–7. [14] Baxt W. Application of artificial neural networks to clinical medicine. Lancet 1995;346(8983):1135–8. [15] Griffith J. Artificial neural networks: are they ready for use as clinical decision aids? Med Decis Making 2000;20(2):243–4. [16] Baxt WG. Use of an artificial neural network for the diagnosis of myocardial infarction. Ann Intern Med 1991;115(11):843–8. [17] Devine B, Macfarlane PW. Detection of electrocardiographic “left ventricular strain” using neural nets. Med Biol Eng Comput 1993 Jul;31(4):343–8. [18] Wu Y, Giger ML, Doi K, Vyborny CJ, Schmidt RA, Metz CE. Artificial neural networks in mammography: application to decision making in the diagnosis of breast cancer. Radiology 1993 Apr;187(1):81–7. [19] Dietzel M, Baltzer PA, Dietzel A, et al. Artificial neural networks for differential diagnosis of breast lesions in MR-mammography: a systematic approach addressing the influence of network architecture on diagnostic performance using a large clinical database. Eur J Radiol 2011;81(7):1508–13. [20] Heden B, Ohlin H, Rittner R, Edenbrandt L. Acute myocardial infarction detected in the 12-lead ECG by artificial neural networks. Circulation 1997;96 (6):1798–802. [21] Lopes N, Ribeiro B. An evaluation of multiple feed-forward networks on GPUs. Int J Neural Syst 2011;21(1):31–47. [22] Team RDC. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2009.
Please cite this article as: Menke NB, et al, A retrospective analysis of the utility of an artificial neural network to predict ED volume, Am J Emerg Med (2014), http://dx.doi.org/10.1016/j.ajem.2014.03.011
