Journal of Biomedical Informatics xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
Journal of Biomedical Informatics journal homepage: www.elsevier.com/locate/yjbin
Predicting patient acuity from electronic patient records Elina Kontio a,b,⇑, Antti Airola c, Tapio Pahikkala c, Heljä Lundgren-Laine a,d, Kristiina Junttila e, Heikki Korvenranta d, Tapio Salakoski c, Sanna Salanterä a,d a
University of Turku, Department of Nursing Science, Finland Turku University of Applied Sciences, Finland c University of Turku, Department of Information Technology, Finland d Turku University Hospital, Finland e Hospital District of Helsinki and Uusimaa, Finland b
a r t i c l e
i n f o
Article history: Received 6 October 2013 Accepted 1 April 2014 Available online xxxx Keywords: Patient acuity Patient classification system Electronic patient record Machine learning
a b s t r a c t Background: The ability to predict acuity (patients’ care needs), would provide a powerful tool for health care managers to allocate resources. Such estimations and predictions for the care process can be produced from the vast amounts of healthcare data using information technology and computational intelligence techniques. Tactical decision-making and resource allocation may also be supported with different mathematical optimization models. Methods: This study was conducted with a data set comprising electronic nursing narratives and the associated Oulu Patient Classification (OPCq) acuity. A mathematical model for the automated assignment of patient acuity scores was utilized and evaluated with the pre-processed data from 23,528 electronic patient records. The methods to predict patient’s acuity were based on linguistic pre-processing, vector-space text modeling, and regularized least-squares regression. Results: The experimental results show that it is possible to obtain accurate predictions about patient acuity scores for the coming day based on the assigned scores and nursing notes from the previous day. Making same-day predictions leads to even better results, as access to the nursing notes for the same day boosts the predictive performance. Furthermore, textual nursing notes allow for more accurate predictions than previous acuity scores. The best results are achieved by combining both of these information sources. The developed model achieves a concordance index of 0.821 when predicting the patient acuity scores for the following day, given the scores and text recorded on the previous day. Conclusions: By applying language technology to electronic patient documents it is possible to accurately predict the value of the acuity scores of the coming day based on the previous day´s assigned scores and nursing notes. Ó 2014 Published by Elsevier Inc.
1. Introduction One key question in a care process continuum is how to predict the next steps within it. Crucially, being able to predict the care needs of patients would help to reduce the rising costs of healthcare. The ability to predict acuity (patients’ care needs), would provide a powerful tool for health-care managers to allocate resources [1,2]. Tactical decision-making and resource planning in hospitals includes short- and medium-range plans, schedules and budgets. Tactical decision-making also includes monitoring the ⇑ Corresponding author at: University of Turku, Department of Nursing Science, FI-20014 Turku, Finland. Fax: +358 10 5536179. E-mail address: elina.kontio@turkuamk.fi (E. Kontio).
performance of organizational subunits, including departments, divisions, process teams, project teams and other workgroups [3,4]. Tactical resource planning in hospitals focuses on elective patient admission planning and the intermediate-term allocation of resource capacities [5,6]. The main objectives of this planning are equitable access for patients, meeting production targets and/or serving the strategically agreed number of patients, and efficiently using resources [7]. For example, in a perioperative unit this might include the decisions to hire more staff to extend hours, expand the operating room capacity, purchase equipment, increase block time for a surgical group or to build a free-standing facility [8]. The objectives of capacity allocation are to balance surgical and postsurgical resources [9–11] to maximize the contribution margin per hour of surgical time [12].
http://dx.doi.org/10.1016/j.jbi.2014.04.001 1532-0464/Ó 2014 Published by Elsevier Inc.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
2
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
2. Background
3. Contributions
2.1. Patient classification systems
The aim of this research is to study to what degree the clinical information in the EPRs of cardiac patients can be used to predict their OPCq acuity scores for the following day. Our hypothesis is that textual nursing notes and previously assigned acuity scores can be utilized to predict the different sub-categories of a patient’s acuity for the next day through the application of machinelearning techniques. When evaluating our models, we consider two distinct settings: one with the aim of accurately predicting acuity scores, and another where patients can be ranked from those needing the most care to those needing the least. In recent years there has been significant interest in developing and applying text mining techniques based on machine learning to the analysis of EPRs, leading to applications such as automated diagnostic systems [29–31], text segmentation tools for nursing narratives [32,33], and quality-of-life-prediction for patients [32]. For a more thorough overview of research on text mining EPRs, we refer to [33]. To the best of our knowledge, the present study is the first to address the problem of predicting patient acuity scores.
One tool that can provide detailed clinical data for forecasting and real-time human resource allocation is the patient classification system (PCS). Patient classification can be determined by methods and processes that are used to identify, validate and monitor the needs of an individual patient [13–15]. Furthermore, the PCS provides information for human resource administration, accounting, budgeting and other functions of management [13,16]. A PCS assesses and classifies patients according to their acuity, their need of care, as well as the nursing activities that are necessary to fulfill those care needs during a certain time period [13]. PCSs play an important role in supporting nurse managers’ decision-making in organizing the care process and required resources. They also provide information on resource consumption, and assist in the budget planning for nursing services and care quality evaluation [13,16]. In addition, they are used to optimize available resources and provide estimations of nurse-to-patient ratios [13,17].
4. Methods 2.2. Oulu Patient Classification system Countless PCSs are currently in use globally. In Finland, the most widely used classification in inpatient units is the Oulu Patient Classification (OPCq), which was developed on the basis of the HSSG Hospital Systems Study Group classification (HSSG) from 1991 to 1993 in Oulu University Hospital. The reliability and validity of the OPCq has been previously tested [18–20]. Using the OPCq and a nurse resource registry, it is possible to calculate the nursing intensity per nurse: the nursing intensity indicates the nursing workload caused by the patients’ care needs (acuity). The OPCq is based on the principles of nursing presented in the quality control program and on Roper’s model of nursing [21]. The OPCq scoring does not describe what specific tasks nurses have completed during the day, but how they responded to the patients’ care needs with different processes and nursing interventions [22–24]. The OPCq [21,24] consists of six nursing care subsections: (1) planning and co-ordination of care; (2) breathing, blood circulation and symptoms of disease; (3) nutrition and medication; (4) personal hygiene and excretion; (5) activity, movement, sleep and rest; (6) teaching, guidance during care and follow-up care, and emotional support. Each subsection is graded by the nurse daily on a scale from A to D according to the patient’s care needs: A = 1 point, B = 2 points, C = 3 points and D = 4 points, resulting in a possible range of summarized OPCq scores from 6 to 24 points. The higher the score, the more demanding the need for care [23]. Based on their OPCq score, patients are classified into five different acuity categories: category I (6–8 points), category II (9–12 points), category III (13–15 points), category IV (16–20 points) and category V (21–24 points) [18,25,26]. A classification manual is available to support this scoring process. Data concerning patient acuity in Finland is one of the core nursing data belonging to the electronic patient record (EPR) [27]. According to the definition by the Organization for Standardization (ISO), the EPR is a repository of patient data in digital form, stored and exchanged securely that is accessible for multiple authorized users. It contains retrospective, concurrent, and prospective information and its primary purpose is to support continuing, efficient and quality integrated health care. [28] However, as a method of data storage the EPR is complex, longitudinal and challenging.
4.1. Data The original data consists of 23,528 electronic patient records of patients with any type of heart problem that were admitted to a university hospital between 2005 and 2009. The data is collected from six different information systems: text data from electronic patient records; patients’ administrative data such as admission, discharge, transfer, patient acuity scores; text data from the radiology system and text data from the pathology system. The inclusion criteria were: – – – –
Diagnosis: ICD 10: I20-I25, I27, I30-I52, R00, R01. Date of birth: 09/23/1901 – 10/21/2009. Admitted into hospital between 2005 and 2009. Length of hospital stay >1 day.
The data were further processed in order to connect the daily acuity scores and the corresponding nursing documentation, since these data came from different information systems. For each two consecutive days of a hospital stay for the patient, a data point consisting of the text for the previous and following days, and the associated acuity scores, was formed. After this pre-processing, the dataset consisted of 132,053 data points. The research was conducted according to established ethical guidelines.1 4.2. Data representation The textual records were lowercased, tokenized, and the words were reduced to their base forms using the FinTWOL morphological analyzer. A number of common Finnish stop words were removed from the texts; after this, the 10,000 most commonly occurring tokens were retained and the remaining tokens filtered out. Finally, the textual records were mapped into numerical vectors using the vector space model, defined as follows: Let us define the term frequency of term t in textual record d, denoted as tft,d, as the number of occurrences of the term in the record. Further, let us 1 Approval of the ethical committee of the hospital district (number 12/2009). The ethical discussion in this study is centered on the process of obtaining the necessary permissions to carry out the research and to use the electronic documents. In addition, this study followed the security procedures designed for accessing patient data [34].
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
define the document frequency dft for a given term t as the number of records in our collection containing the term, and the corresponding inverse document frequency as idft ¼ log dfNt , where N denotes the overall number of records. Then, a term t in record d is assigned the weight as tf idft,d = tft,d idft. This approach assigns the more weight to a word the more frequently it appears in a record, while at the same time assigning reduced importance to very common words that are shared by a large number of records (see Manning et al. [35] for a detailed description of vector space model and tf-idf weighting). Finally, each vector is normalized to Euclidean unit length. By combining the vector space representation of the text and previous acuity scores, we recovered the following feature sets considered in the experiments: ‘‘PScore’’, the acuity scores assigned to the patient on the previous day. ‘‘PText’’, the free text nursing notes written about the patient on the previous day. ‘‘Text’’, the textual nursing notes written about the patient on the same day as the acuity scores are to be assigned. The experiments where ‘‘PText’’ and/or ‘‘PScore’’ were used for making predictions correspond to the setting where the aim was to predict the acuity scores for the following day. In the setting where the ‘‘Text’’ features were also used, we were no longer predicting future scores, but instead simply assuming that the model might be used to automatically input acuity scores to free nurses from this task.
RLS from the RLScore software package,2 which solves the linear system using conjugate gradient optimization, as described by Rifkin et al. [37]. 4.4. Evaluation measures Recalling the one-to-five scale of the acuity scores and one-tofour scale of the subcategory scores, given a real valued prediction for a record made by the model, we round the prediction to its closest score value among the possible scores. Consequently, the prediction error is measured as the absolute difference between the rounded prediction and the true score, indicating that for a record with a true score of 1, it is more wrong to predict 3 than to predict 2. This reflects the need for predicting the absolute number of resources required. Given that the amount of certain available resources is fixed, and thus needs to be allocated among patients, relative acuity is a more interesting quantity than absolute acuity. Therefore, we also measure the model’s ranking performance. In the simplest case of two patients under consideration, the question would be which of them is likely to require more resources. For this purpose, we use the concordance index (CI)[38] [39]. The concordance index over a set of data is the probability that the predictions for two records with different scores randomly drawn from the set are in a correct order. That is, the prediction for the record with the larger score is larger than that for the record with the smaller score:
CI ¼
yi ¼
d X wj xi;j þ ei j¼1
where xi,j denotes the jth vector entry of the ith record, wj are the d unknown real-valued parameters of the model, and ei are noise terms that do not depend linearly on the ith record. Accordingly, we formulate the problem of inferring the model parameters from the set of labeled records as finding the minimizer of the following penalized regression problem: m X
d X yi wj xi;j
i¼1
j¼1
!2
1X dðfi fj Þ; Z y >y i
4.3. Predictive model Next, we describe a method known in the machine learning and statistics literature as the regularized least-squares (RLS) or the Ridge regression method [36]. This method is one of the most widely applied algorithms in the area of machine learning and has been applied in previous research, for example on the automated analysis of electronic patient records or the automated classification of intensive care nursing narratives [30]. Let d be the dimension of the vector space into which each record is transformed, and let m denote the number of records about which the scores are known at the time of the model construction. We assume that the score yi of the ith record obeys the following linear relationship with the entries of the corresponding vector entries:
d X þ k w2j j¼1
Here, the first term measures the regression error on the training data, the second term is the so-called Ridge penalty, and k > 0 is a hyper-parameter controlling the trade-off between the two terms. The role of the second term is to penalize the model complexity to avoid overfitting the model to the training data. The minimizer of the objective function can be found by solving a linear system of equations. In our experiments, we use an implementation of
3
j
where fi and fj denote the predictions made for the ith and jth record, and Z is a normalization constant equal to the number of record pairs with different scores, and d is the Heaviside step function, with d(z) = 1, d(z) = 0.5, and d(z) = 0, for z > 0, z = 0, and, z < 0, respectively. The CI for a random model is 0.5, and for a perfect predictor 1.0. CI can be interpreted as an extension of the area under the ROC curve (AUC) [40] to tasks that have multiple ordered categories, and in cases where there are only two categories, they are equivalent. CI corresponds to an estimate of the probability that given two randomly chosen patients with different acuity scores, the model is able to predict which one should be assigned the higher score. The distribution of the OPCq acuity scores in the data is visualized in Fig. 1. It shows that most patients are assigned to categories 3 and 4 for the overall score. For the 6 subcategories rated on a scale of A–D, scores B and C are the most prevalent. 4.5. Evaluation The same experimental procedure is performed separately for each of the considered feature sets, and for each acuity category. We considered both the regression setting, where the aim is to predict the scores exactly, and the ranking setting, where our aim is to order the patients from those needing the most care to those needing the least. In addition to the RLS models, we also present results for two simple baseline approaches. The previous day baseline always assigns the same acuity score for each category that was assigned to that category on the previous day. The majority voter baseline always predicts, for each category, the class that appears most commonly in the training data. 2
Available at http://staff.cs.utu.fi/~aatapa/software/RLScore/.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
4
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
both the previous scores and the text gives much lower errors than using either of them separately. If we have access to the ‘‘Text’’ features recording the nursing notes for the same day on which the score is to be given, the errors are even lower, but not dramatically so. Of the nursing care subsections, subsection 1 (planning and coordination of care) proves to be the most difficult to predict in each case, while subsection 2 (breathing, blood circulation and symptoms of disease) proves to be the easiest to predict. In Table 2, we present the C-index for each subsection (we do not consider the majority voter baseline here, since by definition it would always result in a random concordance index of 0.5). In all cases, the results are more accurate than the random performance of 0.5, demonstrating that the models do have predictive power. Further, the models clearly outperform the previous day baseline. The overall trends are the same as for the regression experiment - the text proves to be more informative than the previous scores, combining them leads to the best models, and using the text from the same day allows for better prediction than the texts recorded on the previous day. Subsection 1 is again the most difficult to predict, also in terms of the C-index. However, the easiest in this case is subsection 4 (personal hygiene and excretion).
Fig. 1. Histogram of the OPCq acuity scores in the data set.
The experiments were performed using 5-fold cross-validation. The division was made at the patient level, meaning that there is no overlap between the patients in the different folds. The regularization parameter was chosen from an exponential grid. On each round of cross-validation, models corresponding to different choices of the regularization parameter were trained on three training folds, and the parameter chosen was the one that provided best performance on the fourth training fold. Finally, the model was re-trained on the four training folds, and predictions made on the test fold. The final performance was computed by comparing the predicted and true intensity scores on the test folds. For the cross-validated results, 95% confidence intervals were computed. Since the concordance index is computed over pairs of predictions, we randomly chose a subsample of pairs from the test folds where each data point appears only once, in order not to violate the independence assumptions necessary to compute the confidence interval.
5. Results The results for the regression experiments, where we try to predict the scores exactly, are presented in Table 1. These results are presented separately for the different combinations of features, and for the overall acuity score and acuity scores for six nursing care subsections. All the feature sets allow for significant outperforming of the two simple baseline approaches. For all non-baseline results, the mean absolute error is below 0.5. First, we consider the problem in which our aim is to predict the scores for tomorrow, in which case we only have access to the ‘‘PScore’’ and ‘‘PText’’ features. Interestingly, the texts recorded for the previous day prove to be more informative than the recorded scores, as in each case they enable more accurate predictions. These two information sources complement each other in all cases; using
6. Discussion The aim of this research was to study to what degree the clinical information in the EPRs of cardiac patients can be used to predict their OPCq acuity scores for the following day. Our hypothesis was that textual nursing notes and previously assigned acuity scores can be utilized to predict the different subsections of patients’ acuity for the next day by applying machine learning techniques. We tested our hypothesis by training and evaluating a mathematical model to automatically assign patient acuity scores on a data set consisting of nursing documentation and related acuity scores from 23,528 electronic patient records. The methods to predict a patient’s acuity were based on linguistic pre-processing, vector -space modeling of the text, and regularized least-squares regression. The experimental results show that it is possible to accurately predict patients’ acuity scores. We considered settings in which the aim was to predict the scores exactly, as well as those where the aim was to rank patients in order from those needing the most care to those requiring the least. From the results, we can see a number of interesting and consistent trends throughout all of the experiments: The machine learning based approach to predicting acuity scores allows significantly better predictions than naïve approaches such as predicting the score from the previous day, or using a majority score. The texts and previous scores contain important and complementary information, with the best results being achieved by combining both information sources. Having access to notes from the same day that the prediction is made further boosts predictive accuracy.
Table 1 Regression results (mean absolute error). The regression errors for the different considered feature sets, as well as the two baseline methods. The errors are reported separately for predicting the overall acuity score as well as for each of the six OPCq nursing care subsections (see Section 2 for description of the OPCq system). Features
Acuity
Subsec.1
Subsec.2
Subsec.3
Subsec.4
Subsec.5
Subsec.6
Previous day baseline Majority voter baseline PScore PText PScore & PText Text PScore & Text
0.527 ± 0.003 0.715 ± 0.004 0.491 ± 0.003 0.475 ± 0.003 0.443 ± 0.003 0.467 ± 0.003 0.432 ± 0.003
0.518 ± 0.003 0.530 ± 0.003 0.475 ± 0.003 0.471 ± 0.003 0.451 ± 0.003 0.452 ± 0.003 0.429 ± 0.003
0.438 ± 0.003 0.491 ± 0.003 0.415 ± 0.003 0.399 ± 0.003 0.373 ± 0.003 0.396 ± 0.003 0.367 ± 0.003
0.454 ± 0.003 0.598 ± 0.003 0.438 ± 0.003 0.433 ± 0.003 0.395 ± 0.003 0.427 ± 0.003 0.388 ± 0.003
0.470 ± 0.003 0.703 ± 0.003 0.449 ± 0.003 0.437 ± 0.003 0.406 ± 0.003 0.430 ± 0.003 0.394 ± 0.003
0.477 ± 0.003 0.748 ± 0.004 0.462 ± 0.003 0.437 ± 0.003 0.410 ± 0.003 0.430 ± 0.003 0.398 ± 0.003
0.4463 ± 0.003 0.457 ± 0.003 0.428 ± 0.003 0.423 ± 0.003 0.400 ± 0.003 0.413 ± 0.003 0.390 ± 0.003
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
5
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
Table 2 Ranking results (C-index%). The ranking accuracies for the different considered feature sets, as well as for the baseline method. The accuracies are reported separately for predicting the overall acuity score, as well as for each of the six OPCq nursing care subsections (see Section 2 for description of the OPCq system). Features
Acuity
Subsec.1
Subsec.2
Subsec.3
Subsec.4
Subsec.5
Subsec.6
Prev. day baseline PScore PText PScore & PText Text PScore & Text
75.6 ± 0.3 79.2 ± 0.4 79.9 ± 0.4 82.1 ± 0.4 80.8 ± 0.4 83.7 ± 0.3
65.5 ± 0.4 68.2 ± 0.5 68.2 ± 0.5 70.2 ± 0.4 70.0 ± 0.4 72.6 ± 0.4
71.3 ± 0.3 75.3 ± 0.4 75.9 ± 0.4 78.4 ± 0.4 76.8 ± 0.4 79.5 ± 0.4
73.8 ± 0.3 77.2 ± 0.4 77.8 ± 0.4 80.5 ± 0.4 78.2 ± 0.4 81.5 ± 0.4
78.3 ± 0.3 80.7 ± 0.4 82.6 ± 0.3 84.8 ± 0.3 83.3 ± 0.3 85.9 ± 0.3
76.8 ± 0.3 79.5 ± 0.4 81.3 ± 0.4 83.6 ± 0.3 81.9 ± 0.3 84.7 ± 0.3
67.8 ± 0.3 73.2 ± 0.4 71.8 ± 0.5 74.7 ± 0.4 73.1 ± 0.4 76.2 ± 0.4
Of the OPCq subsections, the predictions are the least accurate for subcategory 1 (planning and co-ordination of care). Currently, nurses define their patients’ acuity scores manually in the patient classification system once a day. This task is quite time-consuming and a reliable patient acuity predicting system would thus decrease the nurses’ workload, freeing them for patient care. The availability of this kind of predicting tool would also transform patient classification into a real-time process instead of a monthly reporting task. Real-time data analysis and multi-user access to data also increases the ability to respond quickly and appropriately to changes in human resource allocation. The effective use of patient classification provides improved resource utilization and the management of variances from planned resource use. The possibilities to utilize free-form textual patient documentation are currently weak. Our results suggest that there is significant potential to improve this utilization. The models and tools used in this study succeeded in using free-form textual patient documentation together with other information sources. Indeed, our findings show one example of the possible benefits of an integrated hospital information system. Thus, integrating information from patient classification systems, human resource systems and electronic patient record systems could support decision-making in cardiac care. This could help organizations plan, manage, evaluate and support their tactical decision-making [41]. The predictive accuracy of the developed model was evaluated using cross-validation which demonstrated that the models were indeed able to make accurate predictions. This suggests that the trained mathematical models may be directly applicable for Finnish hospitals using the same kinds of electronic patient record systems and recording practices, and treating similar types of patients as those contained in our data. The results, however, cannot be directly generalized to other hospitals using different types of recording practices. In settings where the type of data differs significantly the trained models might either perform worse than was observed in our experiments, or not work at all if the differences are substantial. Here, we considered the problem of predicting acuity scores based on electronic patient records containing free-form nursing documentation and previous acuity scores. In these systems, other sources of information are also present, such as lists of medications administered, diagnostic codes and notes recorded by doctors. Such information could be further leveraged to enable even more accurate predictions. These additional information sources combined with the positive results achieved with the available data confirm the need for an integrated hospital information system. Having even more accurate predictions would further support tactical decision-making, and could further improve the care process. 7. Conclusion The results of this study confirm that it is possible to use electronic textual nursing notes and previously assigned acuity scores
to predict a patient’s acuity for the next day through the application of machine learning techniques. In addition, there is room for a real-time integrated information system targeted for tactical decision-making. Authors’ contributions E.K. and S.S. designed the study. E.K., A.A., T.P., H.L.-L., T.S. and S.S. contributed to the data collection. The data was analyzed and interpreted by A.A. and T.P. The other co-authors had the opportunity to comment on the statistical methods and analysis. E.K., A.A. and T.P. prepared the manuscript. H.L.-L., K.J., H.K., T.S. and S.S. contributed to revising and improving the manuscript. Conflict of Interest There were no conflicts of interest in the construction of this study, or in the reporting of it. Acknowledgments The data was gathered as a part of larger research project ‘‘Clinical language management for improvement of utilization and usability of patient journals’’ funded by the Academy of Finland. We would like to thank Juho Heimonen for his help in preparing the data, and Lingsoft Ltd for providing tools for language processing. References [1] Fagerstrom L. Evidence-based human resource management: a study of nurse leaders’ resource allocation. J Nurs Manag 2009;17:415–25. [2] Frilund M, Fagerstrom L. Managing the optimal workload by the PAONCIL method – a challenge for nursing leadership in care of older people. J Nurs Manag 2009;17:426–34. [3] Heller FP, Drenth P, Koopman P, Rus V. Decisions in organizations: a three country comparative study. London: Sage; 1988. [4] Winter AF, Ammenwerth E, Bott OJ, Brigl B, Buchauer A, Gräber S, et al. Strategic information management plans: the basis for systematic information management in hospitals. Int J Med Informatics 2001;64:99–109. [5] Kontio E, Lundgren-Laine H, Kontio J, Korvenranta H, Salantera S. Information utilization in tactical decision making of middle management health managers. Comput Inform Nurs 2013;31:9–16. [6] Hulshof PJ, Boucherie RJ, Hans EW, Hurink JL. Tactical resource allocation and elective patient admission planning in care processes. Health Care Manag Sci 2013. [7] Hulshof P, Boucherie R, Hans E, Hurink J. Tactical resource allocation and elective patient admission planning in care processes. Health Care Manag Sci 2013. [8] Dexter F, Ledolter J, Wachtel RE. Tactical decision making for selective expansion of operating room resources incorporating financial criteria and uncertainty in subspecialties’ future workloads. Anesth Analg 2005;100:1425–32. [9] Dexter F, Traub RD. How to schedule elective surgical cases into specific operating rooms to maximize the efficiency of use of operating room time. Anesth Analg 2002;94:933–42 [table of contents]. [10] Guerriero F, Guido R. Operational research in the management of the operating theatre: a survey. Health Care Manag Sci 2011;14:89–114. [11] Testi A, Tanfani E. Tactical and operational decisions for operating room planning: efficiency and welfare implications. Health Care Manag Sci 2009;12:363–73.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
6
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
[12] Cardoen B, Demeulemeester E, Beliën J. Operating room planning and scheduling: a literature review. Eur J Oper Res 2010;201:921–32. [13] De Groot H. Patient classification system evaluation. Part 1: Essential system elements. J Nurs Adminstr 1989;19:30–5. [14] De Groot H. Patient classification system evaluation: Part Two, System selection and implementation. J Nurs Adminstr 1989;19:24–30. [15] Huhkabay L, Skonieeczny R. Patient classification systems: the problems faced. Nurs Health Care 1981:89–103. [16] Alward RR. Patient classification systems: the ideal vs. reality. J Nurs Adminstr 1983;13:14–9. [17] Giovannetti P. Staffing methods—implications for nurse staffing. In: Willis MLE L, editor. Measuring the quality of care Livingstone, London; 1984. p. 123–50. [18] Fagerstrom L, Rainio AK, Rauhala A, Nojonen K. Validation of a new method for patient classification, the Oulu Patient Classification. J Adv Nurs 2000;31:481–90. [19] Fagerstrom L. Benchmarking by the RAFAELA Patient Classification system – a descriptive study of optimal nursing intensity levels. Stud Health Technol Inform 2009;146:25–9. [20] Fagerstrom L, Engberg IB. Measuring the unmeasurable: a caring science perspective on patient classification. J Nurs Manag 1998;6:165–72. [21] Oulun yliopistollinen keskussairaala. Hoitoisuusluokituksen kehittäminen osaksi potilasryhmäkohtaista kustannuslaskentaa Oulun yliopistollisessa keskussairaalassa. Tutkimus- ja kehittämishankkeen loppuraportti. Oulu, Finland; 1994. [22] Hoffman F. Nursing productivity assesment and costing out nursing services. Philadelphia: J.B. Lippincott Company; 1989. [23] Rainio A-K, Ohinmaa AE. Assessment of nursing management and utilization of nursing resources with the RAFAELA patient classification system – case study from the general wards of one central hospital. J Clin Nurs 2005;14:674–84. [24] Kaustinen T. Hoitoisuusluokituksen kehittäminen ja arviointi. Oulu: Hoitotieteen laitos, Oulun yliopisto; 1995. [25] Fagerstrom L. The patient’s perceived caring needs as a message of suffering. J Adv Nurs 1998;28:978–87. [26] Fagerstrom L, Eriksson K, Engberg IB. The patient’s perceived caring needs: measuring the unmeasurable. Int J Nurs Pract 1999;5:199–208. [27] Liljamo P, Kaakinen P. Cross-mapping the finnish classification of nursing diagnosis, nursing interventions and the Oulu Patient Classification. Stud Health Technol Inform 2009;146:774–5.
[28] Hayrinen K, Saranto K, Nykanen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008;77:291–304. [29] Wang Z, Shah AD, Tate AR, Denaxas S, Shawe-Taylor J, Hemingway H. Extracting diagnoses and investigation results from unstructured text in electronic health records by semi-supervised machine learning. PLoS ONE 2012;7:e30412. [30] Suominen H, Ginter F, Pyysalo S, Airola A, Pahikkala T, Salanterä S, et al. Machine learning to automate the assignment of diagnosis codes to free-text radiology reports: a method description. In: Schuurmans Dale, Hauskrecht Milos, editors. Proceedings of the ICML/UAI/COLT 2008 workshop on machine learning for health-care applications. Helsinki, Finland; 2008. [31] Aronow D, Soderland S, Ponte J, Feng F, Croft W, Lehnert W. Automated classification of encounter notes in a computer based medical record. Medinfo 1995:8–12. [32] Hiissa M, Pahikkala T, Suominen H, Lehtikunnas T, Back B, Karsten H, et al. Towards automated classification of intensive care nursing narratives. Int J Med Informatics 2007;76. p. S362-S8. [33] Ginter F, Suominen H, Pyysalo S, Salakoski T. Combining hidden Markov models and latent semantic analysis for topic segmentation and labeling: method and clinical application. Int J Med Informatics 2009;78. p. e1–6. [34] Berman JJ. Confidentiality issues for medical data miners. Artif Intell Med 2002;26:25–36. [35] Raghavan Prabhakar DMC, Hinrich S. Introduction to information retrieval. Cambridge University Press; 2008. [36] Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 1970;12:55–67. [37] Rifkin R, Yeo G, Poggio T. Regularized least-squares classification. Amsterdam: IOS Press; 2003. [38] Harrell FE, Lee KLDBM. Tutorial in Biostatistics: multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361–87. [39] Gönen M, Heller G. Concordance probability and discriminatory power in proportional hazards regression. Biometrika 2005;92:965–70. [40] Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29–36. [41] Canela-Soler J, Elvira-Martinez D, Labordena-Barcelo MJ, Loyola-Elizondo E. Information systems in health and health indicators: an integrating perspective. Med Clin (Barc) 2010;134(Suppl 1):3–9.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
Contents lists available at ScienceDirect
Journal of Biomedical Informatics journal homepage: www.elsevier.com/locate/yjbin
Predicting patient acuity from electronic patient records Elina Kontio a,b,⇑, Antti Airola c, Tapio Pahikkala c, Heljä Lundgren-Laine a,d, Kristiina Junttila e, Heikki Korvenranta d, Tapio Salakoski c, Sanna Salanterä a,d a
University of Turku, Department of Nursing Science, Finland Turku University of Applied Sciences, Finland c University of Turku, Department of Information Technology, Finland d Turku University Hospital, Finland e Hospital District of Helsinki and Uusimaa, Finland b
a r t i c l e
i n f o
Article history: Received 6 October 2013 Accepted 1 April 2014 Available online xxxx Keywords: Patient acuity Patient classification system Electronic patient record Machine learning
a b s t r a c t Background: The ability to predict acuity (patients’ care needs), would provide a powerful tool for health care managers to allocate resources. Such estimations and predictions for the care process can be produced from the vast amounts of healthcare data using information technology and computational intelligence techniques. Tactical decision-making and resource allocation may also be supported with different mathematical optimization models. Methods: This study was conducted with a data set comprising electronic nursing narratives and the associated Oulu Patient Classification (OPCq) acuity. A mathematical model for the automated assignment of patient acuity scores was utilized and evaluated with the pre-processed data from 23,528 electronic patient records. The methods to predict patient’s acuity were based on linguistic pre-processing, vector-space text modeling, and regularized least-squares regression. Results: The experimental results show that it is possible to obtain accurate predictions about patient acuity scores for the coming day based on the assigned scores and nursing notes from the previous day. Making same-day predictions leads to even better results, as access to the nursing notes for the same day boosts the predictive performance. Furthermore, textual nursing notes allow for more accurate predictions than previous acuity scores. The best results are achieved by combining both of these information sources. The developed model achieves a concordance index of 0.821 when predicting the patient acuity scores for the following day, given the scores and text recorded on the previous day. Conclusions: By applying language technology to electronic patient documents it is possible to accurately predict the value of the acuity scores of the coming day based on the previous day´s assigned scores and nursing notes. Ó 2014 Published by Elsevier Inc.
1. Introduction One key question in a care process continuum is how to predict the next steps within it. Crucially, being able to predict the care needs of patients would help to reduce the rising costs of healthcare. The ability to predict acuity (patients’ care needs), would provide a powerful tool for health-care managers to allocate resources [1,2]. Tactical decision-making and resource planning in hospitals includes short- and medium-range plans, schedules and budgets. Tactical decision-making also includes monitoring the ⇑ Corresponding author at: University of Turku, Department of Nursing Science, FI-20014 Turku, Finland. Fax: +358 10 5536179. E-mail address: elina.kontio@turkuamk.fi (E. Kontio).
performance of organizational subunits, including departments, divisions, process teams, project teams and other workgroups [3,4]. Tactical resource planning in hospitals focuses on elective patient admission planning and the intermediate-term allocation of resource capacities [5,6]. The main objectives of this planning are equitable access for patients, meeting production targets and/or serving the strategically agreed number of patients, and efficiently using resources [7]. For example, in a perioperative unit this might include the decisions to hire more staff to extend hours, expand the operating room capacity, purchase equipment, increase block time for a surgical group or to build a free-standing facility [8]. The objectives of capacity allocation are to balance surgical and postsurgical resources [9–11] to maximize the contribution margin per hour of surgical time [12].
http://dx.doi.org/10.1016/j.jbi.2014.04.001 1532-0464/Ó 2014 Published by Elsevier Inc.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
2
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
2. Background
3. Contributions
2.1. Patient classification systems
The aim of this research is to study to what degree the clinical information in the EPRs of cardiac patients can be used to predict their OPCq acuity scores for the following day. Our hypothesis is that textual nursing notes and previously assigned acuity scores can be utilized to predict the different sub-categories of a patient’s acuity for the next day through the application of machinelearning techniques. When evaluating our models, we consider two distinct settings: one with the aim of accurately predicting acuity scores, and another where patients can be ranked from those needing the most care to those needing the least. In recent years there has been significant interest in developing and applying text mining techniques based on machine learning to the analysis of EPRs, leading to applications such as automated diagnostic systems [29–31], text segmentation tools for nursing narratives [32,33], and quality-of-life-prediction for patients [32]. For a more thorough overview of research on text mining EPRs, we refer to [33]. To the best of our knowledge, the present study is the first to address the problem of predicting patient acuity scores.
One tool that can provide detailed clinical data for forecasting and real-time human resource allocation is the patient classification system (PCS). Patient classification can be determined by methods and processes that are used to identify, validate and monitor the needs of an individual patient [13–15]. Furthermore, the PCS provides information for human resource administration, accounting, budgeting and other functions of management [13,16]. A PCS assesses and classifies patients according to their acuity, their need of care, as well as the nursing activities that are necessary to fulfill those care needs during a certain time period [13]. PCSs play an important role in supporting nurse managers’ decision-making in organizing the care process and required resources. They also provide information on resource consumption, and assist in the budget planning for nursing services and care quality evaluation [13,16]. In addition, they are used to optimize available resources and provide estimations of nurse-to-patient ratios [13,17].
4. Methods 2.2. Oulu Patient Classification system Countless PCSs are currently in use globally. In Finland, the most widely used classification in inpatient units is the Oulu Patient Classification (OPCq), which was developed on the basis of the HSSG Hospital Systems Study Group classification (HSSG) from 1991 to 1993 in Oulu University Hospital. The reliability and validity of the OPCq has been previously tested [18–20]. Using the OPCq and a nurse resource registry, it is possible to calculate the nursing intensity per nurse: the nursing intensity indicates the nursing workload caused by the patients’ care needs (acuity). The OPCq is based on the principles of nursing presented in the quality control program and on Roper’s model of nursing [21]. The OPCq scoring does not describe what specific tasks nurses have completed during the day, but how they responded to the patients’ care needs with different processes and nursing interventions [22–24]. The OPCq [21,24] consists of six nursing care subsections: (1) planning and co-ordination of care; (2) breathing, blood circulation and symptoms of disease; (3) nutrition and medication; (4) personal hygiene and excretion; (5) activity, movement, sleep and rest; (6) teaching, guidance during care and follow-up care, and emotional support. Each subsection is graded by the nurse daily on a scale from A to D according to the patient’s care needs: A = 1 point, B = 2 points, C = 3 points and D = 4 points, resulting in a possible range of summarized OPCq scores from 6 to 24 points. The higher the score, the more demanding the need for care [23]. Based on their OPCq score, patients are classified into five different acuity categories: category I (6–8 points), category II (9–12 points), category III (13–15 points), category IV (16–20 points) and category V (21–24 points) [18,25,26]. A classification manual is available to support this scoring process. Data concerning patient acuity in Finland is one of the core nursing data belonging to the electronic patient record (EPR) [27]. According to the definition by the Organization for Standardization (ISO), the EPR is a repository of patient data in digital form, stored and exchanged securely that is accessible for multiple authorized users. It contains retrospective, concurrent, and prospective information and its primary purpose is to support continuing, efficient and quality integrated health care. [28] However, as a method of data storage the EPR is complex, longitudinal and challenging.
4.1. Data The original data consists of 23,528 electronic patient records of patients with any type of heart problem that were admitted to a university hospital between 2005 and 2009. The data is collected from six different information systems: text data from electronic patient records; patients’ administrative data such as admission, discharge, transfer, patient acuity scores; text data from the radiology system and text data from the pathology system. The inclusion criteria were: – – – –
Diagnosis: ICD 10: I20-I25, I27, I30-I52, R00, R01. Date of birth: 09/23/1901 – 10/21/2009. Admitted into hospital between 2005 and 2009. Length of hospital stay >1 day.
The data were further processed in order to connect the daily acuity scores and the corresponding nursing documentation, since these data came from different information systems. For each two consecutive days of a hospital stay for the patient, a data point consisting of the text for the previous and following days, and the associated acuity scores, was formed. After this pre-processing, the dataset consisted of 132,053 data points. The research was conducted according to established ethical guidelines.1 4.2. Data representation The textual records were lowercased, tokenized, and the words were reduced to their base forms using the FinTWOL morphological analyzer. A number of common Finnish stop words were removed from the texts; after this, the 10,000 most commonly occurring tokens were retained and the remaining tokens filtered out. Finally, the textual records were mapped into numerical vectors using the vector space model, defined as follows: Let us define the term frequency of term t in textual record d, denoted as tft,d, as the number of occurrences of the term in the record. Further, let us 1 Approval of the ethical committee of the hospital district (number 12/2009). The ethical discussion in this study is centered on the process of obtaining the necessary permissions to carry out the research and to use the electronic documents. In addition, this study followed the security procedures designed for accessing patient data [34].
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
define the document frequency dft for a given term t as the number of records in our collection containing the term, and the corresponding inverse document frequency as idft ¼ log dfNt , where N denotes the overall number of records. Then, a term t in record d is assigned the weight as tf idft,d = tft,d idft. This approach assigns the more weight to a word the more frequently it appears in a record, while at the same time assigning reduced importance to very common words that are shared by a large number of records (see Manning et al. [35] for a detailed description of vector space model and tf-idf weighting). Finally, each vector is normalized to Euclidean unit length. By combining the vector space representation of the text and previous acuity scores, we recovered the following feature sets considered in the experiments: ‘‘PScore’’, the acuity scores assigned to the patient on the previous day. ‘‘PText’’, the free text nursing notes written about the patient on the previous day. ‘‘Text’’, the textual nursing notes written about the patient on the same day as the acuity scores are to be assigned. The experiments where ‘‘PText’’ and/or ‘‘PScore’’ were used for making predictions correspond to the setting where the aim was to predict the acuity scores for the following day. In the setting where the ‘‘Text’’ features were also used, we were no longer predicting future scores, but instead simply assuming that the model might be used to automatically input acuity scores to free nurses from this task.
RLS from the RLScore software package,2 which solves the linear system using conjugate gradient optimization, as described by Rifkin et al. [37]. 4.4. Evaluation measures Recalling the one-to-five scale of the acuity scores and one-tofour scale of the subcategory scores, given a real valued prediction for a record made by the model, we round the prediction to its closest score value among the possible scores. Consequently, the prediction error is measured as the absolute difference between the rounded prediction and the true score, indicating that for a record with a true score of 1, it is more wrong to predict 3 than to predict 2. This reflects the need for predicting the absolute number of resources required. Given that the amount of certain available resources is fixed, and thus needs to be allocated among patients, relative acuity is a more interesting quantity than absolute acuity. Therefore, we also measure the model’s ranking performance. In the simplest case of two patients under consideration, the question would be which of them is likely to require more resources. For this purpose, we use the concordance index (CI)[38] [39]. The concordance index over a set of data is the probability that the predictions for two records with different scores randomly drawn from the set are in a correct order. That is, the prediction for the record with the larger score is larger than that for the record with the smaller score:
CI ¼
yi ¼
d X wj xi;j þ ei j¼1
where xi,j denotes the jth vector entry of the ith record, wj are the d unknown real-valued parameters of the model, and ei are noise terms that do not depend linearly on the ith record. Accordingly, we formulate the problem of inferring the model parameters from the set of labeled records as finding the minimizer of the following penalized regression problem: m X
d X yi wj xi;j
i¼1
j¼1
!2
1X dðfi fj Þ; Z y >y i
4.3. Predictive model Next, we describe a method known in the machine learning and statistics literature as the regularized least-squares (RLS) or the Ridge regression method [36]. This method is one of the most widely applied algorithms in the area of machine learning and has been applied in previous research, for example on the automated analysis of electronic patient records or the automated classification of intensive care nursing narratives [30]. Let d be the dimension of the vector space into which each record is transformed, and let m denote the number of records about which the scores are known at the time of the model construction. We assume that the score yi of the ith record obeys the following linear relationship with the entries of the corresponding vector entries:
d X þ k w2j j¼1
Here, the first term measures the regression error on the training data, the second term is the so-called Ridge penalty, and k > 0 is a hyper-parameter controlling the trade-off between the two terms. The role of the second term is to penalize the model complexity to avoid overfitting the model to the training data. The minimizer of the objective function can be found by solving a linear system of equations. In our experiments, we use an implementation of
3
j
where fi and fj denote the predictions made for the ith and jth record, and Z is a normalization constant equal to the number of record pairs with different scores, and d is the Heaviside step function, with d(z) = 1, d(z) = 0.5, and d(z) = 0, for z > 0, z = 0, and, z < 0, respectively. The CI for a random model is 0.5, and for a perfect predictor 1.0. CI can be interpreted as an extension of the area under the ROC curve (AUC) [40] to tasks that have multiple ordered categories, and in cases where there are only two categories, they are equivalent. CI corresponds to an estimate of the probability that given two randomly chosen patients with different acuity scores, the model is able to predict which one should be assigned the higher score. The distribution of the OPCq acuity scores in the data is visualized in Fig. 1. It shows that most patients are assigned to categories 3 and 4 for the overall score. For the 6 subcategories rated on a scale of A–D, scores B and C are the most prevalent. 4.5. Evaluation The same experimental procedure is performed separately for each of the considered feature sets, and for each acuity category. We considered both the regression setting, where the aim is to predict the scores exactly, and the ranking setting, where our aim is to order the patients from those needing the most care to those needing the least. In addition to the RLS models, we also present results for two simple baseline approaches. The previous day baseline always assigns the same acuity score for each category that was assigned to that category on the previous day. The majority voter baseline always predicts, for each category, the class that appears most commonly in the training data. 2
Available at http://staff.cs.utu.fi/~aatapa/software/RLScore/.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
4
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
both the previous scores and the text gives much lower errors than using either of them separately. If we have access to the ‘‘Text’’ features recording the nursing notes for the same day on which the score is to be given, the errors are even lower, but not dramatically so. Of the nursing care subsections, subsection 1 (planning and coordination of care) proves to be the most difficult to predict in each case, while subsection 2 (breathing, blood circulation and symptoms of disease) proves to be the easiest to predict. In Table 2, we present the C-index for each subsection (we do not consider the majority voter baseline here, since by definition it would always result in a random concordance index of 0.5). In all cases, the results are more accurate than the random performance of 0.5, demonstrating that the models do have predictive power. Further, the models clearly outperform the previous day baseline. The overall trends are the same as for the regression experiment - the text proves to be more informative than the previous scores, combining them leads to the best models, and using the text from the same day allows for better prediction than the texts recorded on the previous day. Subsection 1 is again the most difficult to predict, also in terms of the C-index. However, the easiest in this case is subsection 4 (personal hygiene and excretion).
Fig. 1. Histogram of the OPCq acuity scores in the data set.
The experiments were performed using 5-fold cross-validation. The division was made at the patient level, meaning that there is no overlap between the patients in the different folds. The regularization parameter was chosen from an exponential grid. On each round of cross-validation, models corresponding to different choices of the regularization parameter were trained on three training folds, and the parameter chosen was the one that provided best performance on the fourth training fold. Finally, the model was re-trained on the four training folds, and predictions made on the test fold. The final performance was computed by comparing the predicted and true intensity scores on the test folds. For the cross-validated results, 95% confidence intervals were computed. Since the concordance index is computed over pairs of predictions, we randomly chose a subsample of pairs from the test folds where each data point appears only once, in order not to violate the independence assumptions necessary to compute the confidence interval.
5. Results The results for the regression experiments, where we try to predict the scores exactly, are presented in Table 1. These results are presented separately for the different combinations of features, and for the overall acuity score and acuity scores for six nursing care subsections. All the feature sets allow for significant outperforming of the two simple baseline approaches. For all non-baseline results, the mean absolute error is below 0.5. First, we consider the problem in which our aim is to predict the scores for tomorrow, in which case we only have access to the ‘‘PScore’’ and ‘‘PText’’ features. Interestingly, the texts recorded for the previous day prove to be more informative than the recorded scores, as in each case they enable more accurate predictions. These two information sources complement each other in all cases; using
6. Discussion The aim of this research was to study to what degree the clinical information in the EPRs of cardiac patients can be used to predict their OPCq acuity scores for the following day. Our hypothesis was that textual nursing notes and previously assigned acuity scores can be utilized to predict the different subsections of patients’ acuity for the next day by applying machine learning techniques. We tested our hypothesis by training and evaluating a mathematical model to automatically assign patient acuity scores on a data set consisting of nursing documentation and related acuity scores from 23,528 electronic patient records. The methods to predict a patient’s acuity were based on linguistic pre-processing, vector -space modeling of the text, and regularized least-squares regression. The experimental results show that it is possible to accurately predict patients’ acuity scores. We considered settings in which the aim was to predict the scores exactly, as well as those where the aim was to rank patients in order from those needing the most care to those requiring the least. From the results, we can see a number of interesting and consistent trends throughout all of the experiments: The machine learning based approach to predicting acuity scores allows significantly better predictions than naïve approaches such as predicting the score from the previous day, or using a majority score. The texts and previous scores contain important and complementary information, with the best results being achieved by combining both information sources. Having access to notes from the same day that the prediction is made further boosts predictive accuracy.
Table 1 Regression results (mean absolute error). The regression errors for the different considered feature sets, as well as the two baseline methods. The errors are reported separately for predicting the overall acuity score as well as for each of the six OPCq nursing care subsections (see Section 2 for description of the OPCq system). Features
Acuity
Subsec.1
Subsec.2
Subsec.3
Subsec.4
Subsec.5
Subsec.6
Previous day baseline Majority voter baseline PScore PText PScore & PText Text PScore & Text
0.527 ± 0.003 0.715 ± 0.004 0.491 ± 0.003 0.475 ± 0.003 0.443 ± 0.003 0.467 ± 0.003 0.432 ± 0.003
0.518 ± 0.003 0.530 ± 0.003 0.475 ± 0.003 0.471 ± 0.003 0.451 ± 0.003 0.452 ± 0.003 0.429 ± 0.003
0.438 ± 0.003 0.491 ± 0.003 0.415 ± 0.003 0.399 ± 0.003 0.373 ± 0.003 0.396 ± 0.003 0.367 ± 0.003
0.454 ± 0.003 0.598 ± 0.003 0.438 ± 0.003 0.433 ± 0.003 0.395 ± 0.003 0.427 ± 0.003 0.388 ± 0.003
0.470 ± 0.003 0.703 ± 0.003 0.449 ± 0.003 0.437 ± 0.003 0.406 ± 0.003 0.430 ± 0.003 0.394 ± 0.003
0.477 ± 0.003 0.748 ± 0.004 0.462 ± 0.003 0.437 ± 0.003 0.410 ± 0.003 0.430 ± 0.003 0.398 ± 0.003
0.4463 ± 0.003 0.457 ± 0.003 0.428 ± 0.003 0.423 ± 0.003 0.400 ± 0.003 0.413 ± 0.003 0.390 ± 0.003
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
5
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
Table 2 Ranking results (C-index%). The ranking accuracies for the different considered feature sets, as well as for the baseline method. The accuracies are reported separately for predicting the overall acuity score, as well as for each of the six OPCq nursing care subsections (see Section 2 for description of the OPCq system). Features
Acuity
Subsec.1
Subsec.2
Subsec.3
Subsec.4
Subsec.5
Subsec.6
Prev. day baseline PScore PText PScore & PText Text PScore & Text
75.6 ± 0.3 79.2 ± 0.4 79.9 ± 0.4 82.1 ± 0.4 80.8 ± 0.4 83.7 ± 0.3
65.5 ± 0.4 68.2 ± 0.5 68.2 ± 0.5 70.2 ± 0.4 70.0 ± 0.4 72.6 ± 0.4
71.3 ± 0.3 75.3 ± 0.4 75.9 ± 0.4 78.4 ± 0.4 76.8 ± 0.4 79.5 ± 0.4
73.8 ± 0.3 77.2 ± 0.4 77.8 ± 0.4 80.5 ± 0.4 78.2 ± 0.4 81.5 ± 0.4
78.3 ± 0.3 80.7 ± 0.4 82.6 ± 0.3 84.8 ± 0.3 83.3 ± 0.3 85.9 ± 0.3
76.8 ± 0.3 79.5 ± 0.4 81.3 ± 0.4 83.6 ± 0.3 81.9 ± 0.3 84.7 ± 0.3
67.8 ± 0.3 73.2 ± 0.4 71.8 ± 0.5 74.7 ± 0.4 73.1 ± 0.4 76.2 ± 0.4
Of the OPCq subsections, the predictions are the least accurate for subcategory 1 (planning and co-ordination of care). Currently, nurses define their patients’ acuity scores manually in the patient classification system once a day. This task is quite time-consuming and a reliable patient acuity predicting system would thus decrease the nurses’ workload, freeing them for patient care. The availability of this kind of predicting tool would also transform patient classification into a real-time process instead of a monthly reporting task. Real-time data analysis and multi-user access to data also increases the ability to respond quickly and appropriately to changes in human resource allocation. The effective use of patient classification provides improved resource utilization and the management of variances from planned resource use. The possibilities to utilize free-form textual patient documentation are currently weak. Our results suggest that there is significant potential to improve this utilization. The models and tools used in this study succeeded in using free-form textual patient documentation together with other information sources. Indeed, our findings show one example of the possible benefits of an integrated hospital information system. Thus, integrating information from patient classification systems, human resource systems and electronic patient record systems could support decision-making in cardiac care. This could help organizations plan, manage, evaluate and support their tactical decision-making [41]. The predictive accuracy of the developed model was evaluated using cross-validation which demonstrated that the models were indeed able to make accurate predictions. This suggests that the trained mathematical models may be directly applicable for Finnish hospitals using the same kinds of electronic patient record systems and recording practices, and treating similar types of patients as those contained in our data. The results, however, cannot be directly generalized to other hospitals using different types of recording practices. In settings where the type of data differs significantly the trained models might either perform worse than was observed in our experiments, or not work at all if the differences are substantial. Here, we considered the problem of predicting acuity scores based on electronic patient records containing free-form nursing documentation and previous acuity scores. In these systems, other sources of information are also present, such as lists of medications administered, diagnostic codes and notes recorded by doctors. Such information could be further leveraged to enable even more accurate predictions. These additional information sources combined with the positive results achieved with the available data confirm the need for an integrated hospital information system. Having even more accurate predictions would further support tactical decision-making, and could further improve the care process. 7. Conclusion The results of this study confirm that it is possible to use electronic textual nursing notes and previously assigned acuity scores
to predict a patient’s acuity for the next day through the application of machine learning techniques. In addition, there is room for a real-time integrated information system targeted for tactical decision-making. Authors’ contributions E.K. and S.S. designed the study. E.K., A.A., T.P., H.L.-L., T.S. and S.S. contributed to the data collection. The data was analyzed and interpreted by A.A. and T.P. The other co-authors had the opportunity to comment on the statistical methods and analysis. E.K., A.A. and T.P. prepared the manuscript. H.L.-L., K.J., H.K., T.S. and S.S. contributed to revising and improving the manuscript. Conflict of Interest There were no conflicts of interest in the construction of this study, or in the reporting of it. Acknowledgments The data was gathered as a part of larger research project ‘‘Clinical language management for improvement of utilization and usability of patient journals’’ funded by the Academy of Finland. We would like to thank Juho Heimonen for his help in preparing the data, and Lingsoft Ltd for providing tools for language processing. References [1] Fagerstrom L. Evidence-based human resource management: a study of nurse leaders’ resource allocation. J Nurs Manag 2009;17:415–25. [2] Frilund M, Fagerstrom L. Managing the optimal workload by the PAONCIL method – a challenge for nursing leadership in care of older people. J Nurs Manag 2009;17:426–34. [3] Heller FP, Drenth P, Koopman P, Rus V. Decisions in organizations: a three country comparative study. London: Sage; 1988. [4] Winter AF, Ammenwerth E, Bott OJ, Brigl B, Buchauer A, Gräber S, et al. Strategic information management plans: the basis for systematic information management in hospitals. Int J Med Informatics 2001;64:99–109. [5] Kontio E, Lundgren-Laine H, Kontio J, Korvenranta H, Salantera S. Information utilization in tactical decision making of middle management health managers. Comput Inform Nurs 2013;31:9–16. [6] Hulshof PJ, Boucherie RJ, Hans EW, Hurink JL. Tactical resource allocation and elective patient admission planning in care processes. Health Care Manag Sci 2013. [7] Hulshof P, Boucherie R, Hans E, Hurink J. Tactical resource allocation and elective patient admission planning in care processes. Health Care Manag Sci 2013. [8] Dexter F, Ledolter J, Wachtel RE. Tactical decision making for selective expansion of operating room resources incorporating financial criteria and uncertainty in subspecialties’ future workloads. Anesth Analg 2005;100:1425–32. [9] Dexter F, Traub RD. How to schedule elective surgical cases into specific operating rooms to maximize the efficiency of use of operating room time. Anesth Analg 2002;94:933–42 [table of contents]. [10] Guerriero F, Guido R. Operational research in the management of the operating theatre: a survey. Health Care Manag Sci 2011;14:89–114. [11] Testi A, Tanfani E. Tactical and operational decisions for operating room planning: efficiency and welfare implications. Health Care Manag Sci 2009;12:363–73.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
6
E. Kontio et al. / Journal of Biomedical Informatics xxx (2014) xxx–xxx
[12] Cardoen B, Demeulemeester E, Beliën J. Operating room planning and scheduling: a literature review. Eur J Oper Res 2010;201:921–32. [13] De Groot H. Patient classification system evaluation. Part 1: Essential system elements. J Nurs Adminstr 1989;19:30–5. [14] De Groot H. Patient classification system evaluation: Part Two, System selection and implementation. J Nurs Adminstr 1989;19:24–30. [15] Huhkabay L, Skonieeczny R. Patient classification systems: the problems faced. Nurs Health Care 1981:89–103. [16] Alward RR. Patient classification systems: the ideal vs. reality. J Nurs Adminstr 1983;13:14–9. [17] Giovannetti P. Staffing methods—implications for nurse staffing. In: Willis MLE L, editor. Measuring the quality of care Livingstone, London; 1984. p. 123–50. [18] Fagerstrom L, Rainio AK, Rauhala A, Nojonen K. Validation of a new method for patient classification, the Oulu Patient Classification. J Adv Nurs 2000;31:481–90. [19] Fagerstrom L. Benchmarking by the RAFAELA Patient Classification system – a descriptive study of optimal nursing intensity levels. Stud Health Technol Inform 2009;146:25–9. [20] Fagerstrom L, Engberg IB. Measuring the unmeasurable: a caring science perspective on patient classification. J Nurs Manag 1998;6:165–72. [21] Oulun yliopistollinen keskussairaala. Hoitoisuusluokituksen kehittäminen osaksi potilasryhmäkohtaista kustannuslaskentaa Oulun yliopistollisessa keskussairaalassa. Tutkimus- ja kehittämishankkeen loppuraportti. Oulu, Finland; 1994. [22] Hoffman F. Nursing productivity assesment and costing out nursing services. Philadelphia: J.B. Lippincott Company; 1989. [23] Rainio A-K, Ohinmaa AE. Assessment of nursing management and utilization of nursing resources with the RAFAELA patient classification system – case study from the general wards of one central hospital. J Clin Nurs 2005;14:674–84. [24] Kaustinen T. Hoitoisuusluokituksen kehittäminen ja arviointi. Oulu: Hoitotieteen laitos, Oulun yliopisto; 1995. [25] Fagerstrom L. The patient’s perceived caring needs as a message of suffering. J Adv Nurs 1998;28:978–87. [26] Fagerstrom L, Eriksson K, Engberg IB. The patient’s perceived caring needs: measuring the unmeasurable. Int J Nurs Pract 1999;5:199–208. [27] Liljamo P, Kaakinen P. Cross-mapping the finnish classification of nursing diagnosis, nursing interventions and the Oulu Patient Classification. Stud Health Technol Inform 2009;146:774–5.
[28] Hayrinen K, Saranto K, Nykanen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008;77:291–304. [29] Wang Z, Shah AD, Tate AR, Denaxas S, Shawe-Taylor J, Hemingway H. Extracting diagnoses and investigation results from unstructured text in electronic health records by semi-supervised machine learning. PLoS ONE 2012;7:e30412. [30] Suominen H, Ginter F, Pyysalo S, Airola A, Pahikkala T, Salanterä S, et al. Machine learning to automate the assignment of diagnosis codes to free-text radiology reports: a method description. In: Schuurmans Dale, Hauskrecht Milos, editors. Proceedings of the ICML/UAI/COLT 2008 workshop on machine learning for health-care applications. Helsinki, Finland; 2008. [31] Aronow D, Soderland S, Ponte J, Feng F, Croft W, Lehnert W. Automated classification of encounter notes in a computer based medical record. Medinfo 1995:8–12. [32] Hiissa M, Pahikkala T, Suominen H, Lehtikunnas T, Back B, Karsten H, et al. Towards automated classification of intensive care nursing narratives. Int J Med Informatics 2007;76. p. S362-S8. [33] Ginter F, Suominen H, Pyysalo S, Salakoski T. Combining hidden Markov models and latent semantic analysis for topic segmentation and labeling: method and clinical application. Int J Med Informatics 2009;78. p. e1–6. [34] Berman JJ. Confidentiality issues for medical data miners. Artif Intell Med 2002;26:25–36. [35] Raghavan Prabhakar DMC, Hinrich S. Introduction to information retrieval. Cambridge University Press; 2008. [36] Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 1970;12:55–67. [37] Rifkin R, Yeo G, Poggio T. Regularized least-squares classification. Amsterdam: IOS Press; 2003. [38] Harrell FE, Lee KLDBM. Tutorial in Biostatistics: multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361–87. [39] Gönen M, Heller G. Concordance probability and discriminatory power in proportional hazards regression. Biometrika 2005;92:965–70. [40] Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29–36. [41] Canela-Soler J, Elvira-Martinez D, Labordena-Barcelo MJ, Loyola-Elizondo E. Information systems in health and health indicators: an integrating perspective. Med Clin (Barc) 2010;134(Suppl 1):3–9.
Please cite this article in press as: Kontio E et al. Predicting patient acuity from electronic patient records. J Biomed Inform (2014), http://dx.doi.org/ 10.1016/j.jbi.2014.04.001
