Research
Original Investigation
Using Electronic Health Records for Surgical Quality Improvement in the Era of Big Data Jamie E. Anderson, MD, MPH; David C. Chang, PhD, MPH, MBA
IMPORTANCE Risk adjustment is an important component of quality assessment in surgical health care. However, data collection places an additional burden on physicians. There is also concern that outcomes can be gamed depending on the information recorded for each patient. OBJECTIVE To determine whether a number of machine-collected data elements could perform as well as a traditional full-risk adjustment model that includes other physician-assessed and physician-recorded data elements. DESIGN, SETTINGS, AND PARTICIPANTS All general surgery patients from the National Surgical Quality Improvement Program database from January 1, 2005, to December 31, 2010, were included. Separate multivariate logistic regressions were performed using either all 66 preoperative risk variables or only 25 objective variables. The area under the receiver operating characteristic curve (AUC) of each regression using objective preoperative risk variables was compared with its corresponding regression with all preoperative variables. Subset analyses were performed among patients who received certain operations. MAIN OUTCOMES AND MEASURES Mortality or any surgical complication captured by the National Surgical Quality Improvement Program, both inpatient and within 30 days postoperatively. RESULTS Data from a total of 745 053 patients were included. More than 15.8% of patients
had at least 1 complication and the mortality rate was 2.8%. When examining inpatient mortality, the AUC was 0.9104 with all 66 variables vs 0.8918 with all 25 objective variables. The difference in AUC comparing models with all variables with objective variables ranged from −0.0073 to 0.1944 for mortality and 0.0198 to 0.0687 for complications. In models predicting mortality, the difference in AUC was less than 0.05 among all patients and subsets of patients with abdominal aortic aneurysm repair, pancreatic resection, colectomy, and appendectomy. In models predicting complications, the difference in AUC was less than 0.05 among all patients and subsets of patients with pancreatic resection, laparoscopic cholecystectomy, colectomy, and appendectomy. CONCLUSIONS AND RELEVANCE Rigorous risk-adjusted surgical quality assessment can be performed solely with objective variables. By leveraging data already routinely collected for patient care, this approach allows for wider adoption of quality assessment systems in health care. Identifying data elements that can be automatically collected can make future improvements to surgical outcomes and quality analyses.
Author Affiliations: Department of Surgery, University of California, San Diego.
JAMA Surg. 2015;150(1):24-29. doi:10.1001/jamasurg.2014.947 Published online November 5, 2014. 24
Corresponding Author: Jamie E. Anderson, MD, MPH, Center for Surgical Systems and Public Health, Department of Surgery, University of California, San Diego, 200 W Arbor Dr, San Diego, CA 92103 ([email protected]). jamasurgery.com
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
Electronic Health Records and Surgical Quality Improvement
R
isk adjustment is an important component of outcomes and quality analysis in surgical health care. However, there are 2 major concerns regarding the collection of risk-adjustment data in practice today. First, data collection places an additional burden on physicians, who feel that they have become data-entry personnel and are spending more time in front of the computer than with the patient. This concern could be partially addressed by studies that aim to identify minimum data sets that require only a handful of variables. For example, previous work has determined that the ability of the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) database to risk adjust may be adequate using as few as 4 variables.1-4 However, the more serious concern is that risk-adjusted quality benchmarking systems can be gamed because they include data elements that require subjective interpretation by hospital personnel, such as patient history of comorbidities. There may be subtle financial incentives to overcode these data elements to increase the risk scores for a hospital and, thus, improve a hospital’s risk-adjusted outcomes profile. These concerns can be addressed if risk-adjustment models avoid subjective data elements, such as history of comorbidities, and rely on objective data, such as laboratory values or other machine-collected variables that do not require the subjective interpretation and input of hospital personnel. The adoption of electronic health records has doubled from 2009 to 2011, partly a result of funding provided by the Health Information Technology for Economic and Health Act of 2009.5 This has contributed to so-called big data, multiple massive data sets that include patient demographics, medical history, disease course with corresponding treatment, and transaction information, such as device usage and medication administration. However, much of these data are currently perceived as by-products of health care delivery rather than a central asset to improve its efficiency.5 Therefore, the aim of this study was to determine whether a number of machinecollected data elements could perform as well as a traditional, full risk-adjustment model that includes other physician-assessed and physician-recorded data elements. We tested this hypothesis with an analysis of the NSQIP database.
Methods This research uses all available NSQIP data from January 1, 2005, to December 31, 2010. This nationally validated program measures more than 135 variables on each patient and follows up each patient for 30 days postoperatively. The 2005 to 2006 database included information from 121 hospitals while data from 2010 included information from 237 hospitals.6 This data set was chosen for its breadth of preoperative and postoperative variables collected for each patient. Owing to its retrospective nature, this study was exempt from institutional review board approval and participants did not provide consent. The primary analysis included all patients in the database who were categorized as having had an operation performed by a general surgeons or surgeons in the following subspecialties: plastic surgery, cardiothoracic surgery, or vascular
Original Investigation Research
surgery. Subset analyses were performed in patients who underwent abdominal aortic aneurysm repair, aortic valve replacement, coronary artery bypass graft, esophagectomy, pancreatic resection, laparoscopic cholecystectomy, colectomy (total or partial, laparoscopic or open), appendectomy (open or laparoscopic), and ventral hernia repair. The first 5 procedures listed were selected because of presence of Leapfrog quality standards.7 The last 4 procedures were selected because they are common general surgery procedures. Patients were determined to have an adverse event after surgery if they experienced death or at least 1 of the following complications as captured by NSQIP before discharge: superficial surgical site infection; deep incisional surgical site infection; organ space surgical site infection; wound disruption; pneumonia; unplanned intubation; pulmonary embolism; ventilator for more than 48 hours; progressive renal insufficiency; acute renal failure; urinary tract infection; stroke/ cerebrovascular accident with neurological deficit; coma of more than 24 hours; peripheral nerve injury; cardiac arrest requiring cardiopulmonary resuscitation; myocardial infarction; bleeding requiring transfusions; graft, prosthesis, or flap failure; deep vein thrombosis or thrombophlebitis; sepsis; or septic shock. Multivariate logistic regression models were created to predict either mortality or any complication in the inpatient setting or within 30 days of surgery. In one set of models we included all preoperative risk variables included in NSQIP, a total of 66 variables. In a second set of models we only included preoperative variables that were either laboratory data or able to be automatically extracted from a patient medical record without additional data input. This was a total of 25 variables, which are listed in the Box. All continuous variables were kept as such except for age, which was grouped into 10-year categories. We compared the area under the receiver operating characteristic curve (AUC) of each regression using objective preoperative risk variables to its corresponding regression with all variables. The AUC is a discriminative measure to identify how well a model separates 2 groups (ie, patients with vs without adverse events). An AUC value of 0.5 indicates that the model separates the 2 groups no better than chance whereas an AUC value of 1.0 indicates that the model completely separates the 2 groups. The AUC statistic is actually the percentage of randomly selected pairs correctly predicted by the model. Thus, the AUC allows us to determine which model can more accurately discriminate between the 2 groups of interest.8-11 Statistical analysis was performed using Stata 64-bit special edition, version 11.2 (StataCorp).
Results Data from a total of 745 053 patients from 2005 to 2010 from the NSQIP database were included (Table 1). More than 15.8% of patients had at least 1 complication and the mortality rate was 2.8%. Mean age was higher among those with complications (61.5 years) and highest among those who died (71.1 years). Among the subset procedures, colectomy had the highest per-
jamasurgery.com
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
25
Research Original Investigation
Electronic Health Records and Surgical Quality Improvement
Box. Preoperative National Surgical Quality Improvement Program Variables Classified According to Objectivity Objective variables included in the analysis: Age in 10-y categories
Current smoker within 1 y Currently receiving dialysis (prior to operation)
Body mass index
Cerebrovascular accident/stroke with neurological deficit
Pregnancy
Cerebrovascular accident/stroke with no neurological deficit
Alkaline phosphatase
Diabetes mellitus with oral agents or insulin
Blood urea nitrogen
Disseminated cancer
Hematocrit
Do not resuscitate status
International normalized ratio of PT values
Dyspnea
Platelet count
Emergency case
PT
Esophageal varices
PTT
Functional health status prior to current illness
Serum albumin
Functional health status prior to surgery
Serum creatinine
Hemiplegia
Serum sodium
History of angina in 1 mo prior to surgery
SGOT
History of myocardial infarction 6 mo prior to surgery
Systemic sepsis Total bilirubin
History of revascularization/amputation for peripheral vascular disease
White blood cell count
History of severe COPD
Principal anesthesia technique
History of transient ischemic attacks
Prior operation within 30 d
Hypertension requiring medication
Race/ethnicity (white, black, Hispanic, Asian or Pacific Islander, American Indian, Alaska Native, or other)
Impaired sensorium
Sex
Paraplegia
Surgical specialty of surgeon performing procedure
Previous cardiac surgery
Transfusion >4 U PRBCs in 72 h before surgery
Previous percutaneous coronary intervention
Subjective variables not included in the analysis: >10% Loss body weight in last 6 mo Acute renal failure
Mallampati scale
Quadriplegia Radiotherapy for malignancy in last 90 d Rest pain/gangrene
Airway trauma Alcohol, >2 drinks/d in 2 wk before admission ASA classification
Steroid use for chronic condition Tumor involving CNS Wound classification
Ascites
Wound infection
Bleeding disorder Chemotherapy for malignancy in ⱕ30 d prior to surgery Coma >24 h Congestive heart failure in 30 d prior to surgery
Abbreviations: ASA, American Society of Anesthesiologists; CNS, central nervous system; COPD, chronic obstructive pulmonary disease; PRBCs, packed red blood cells; PT, prothrombin time; PTT, partial thromboplastin time; SGOT, serum glutamic oxaloacetic transaminase.
Current pneumonia
centage of patients (12.7%). The fewest number of patients received aortic valve replacement (0.2%) and esophagectomy (0.3%). Overall complication rates were highest in esophagectomy (49.5%) and lowest in laparoscopic cholecystectomy (5.2%) and appendectomy (6.6%). Mortality rates were highest in colectomy (4.4%) and aortic valve replacement (4.3%). The AUC was slightly higher in models that included all variables compared with models with only objective variables. Among all patients, the AUC was 0.9005 with all variables vs 0.8774 with objective variables for 30-day mortality and 0.9078 with all variables vs 0.8881 with objective variables for inpatient mortality (Table 2). When examining com26
plications among all patients, the AUC was 0.7401 with all variables vs 0.7137 w ith objec tive variables for 30-day complications and 0.7859 with all variables vs 0.7609 with objective variables for inpatient complications (Table 2). The difference in AUC ranged from −0.0073 to 0.1944 for mortality and 0.0198 to 0.0687 for complications. In models predicting mortality, the difference in AUC was less than 0.05 among all patients and subsets of patients with abdominal aortic aneurysm repair, pancreatic resection, colectomy, and appendectomy. The difference in AUC was greatest among patients with esophagectomy (0.1937 to 0.1944) and smallest among patients with appendectomy (−0.0073 to 0.0171). In
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
jamasurgery.com
Electronic Health Records and Surgical Quality Improvement
Original Investigation Research
Table 1. Patient Demographics and Outcomes Within a 30-Day Postoperative Period
Characteristic Age, mean (SD), y
All (n = 745 053)
No Complications
Complications
Died
n = 606 468 (81.4%)
n = 117 673 (15.8%)
n = 20 912 (2.8%)
57.4 (17.3)
56.1 (17.4)
61.5 (15.8)
71.1 (13.5)
Race/ethnicity, No. (%) White
410 907 (79.0)
337 559 (78.9)
61 688 (79.3)
11 620 (80.6)
African American
58 804 (11.3)
47 064 (11.0)
9927 (12.8)
1813 (12.6)
Hispanic
34 479 (6.6)
29 779 (7.0)
4068 (5.2)
632 (4.4)
Asian or Pacific Islander
12 037 (2.3)
10 283 (2.4)
1491 (1.9)
263 (1.8)
3894 (0.8)
3224 (0.8)
580 (0.8)
Other or unknown Female, No. (%) Characteristic Abdominal aortic aneurysm repair Aortic valve replacement Pancreatic resection Esophagectomy
414 386 (55.8)
344 997 (57.0)
90 (0.6)
59 407 (50.6)
All Patients, No. (%)
9982 (47.9)
Patients With Procedure, No. (%)
14 042 (1.9)
11 537 (82.2)
2047 (14.6)
1551 (0.2)
943 (60.8)
542 (35.0)
458 (3.3) 66 (4.3)
13 530 (1.8)
8560 (63.3)
4619 (34.1)
351 (2.6)
2089 (0.3)
1055 (50.5)
Laparoscopic cholecystectomy
44 645 (6.0)
42 353 (94.9)
2040 (4.6)
957 (45.8)
252 (0.6)
77 (3.7)
Colectomy
94 554 (12.7)
66 872 (70.7)
23 522 (24.9)
4160 (4.4)
Appendectomy
63 855 (8.6)
59 597 (93.3)
4104 (6.4)
154 (0.2)
Table 2. AUC of Models Predicting Mortality and Complicationsa AUC, % Mortality
Complications
All Variables
Objective Variables
Difference
All Variables
Objective Variables
Difference
All patients
0.901
0.877
0.023
0.740
0.714
0.026
Abdominal aortic aneurysm repair
0.875
0.828
0.048
0.750
0.712
0.038
Aortic valve replacement
0.976
0.883
0.093
0.796
0.740
0.055
Outcomes 30-d
Pancreatic resection
0.774
0.739
0.035
0.641
0.621
0.020
Esophagectomy
0.949
0.755
0.194
0.701
0.645
0.057
Laparoscopic cholecystectomy
0.932
0.871
0.061
0.763
0.722
0.041
Colectomy
0.899
0.881
0.019
0.729
0.708
0.021
Appendectomy
0.981
0.964
0.017
0.763
0.730
0.033
0.908
0.888
0.020
0.786
0.761
0.025
Inpatient All patients Abdominal aortic aneurysm repair
0.868
0.821
0.047
0.808
0.751
0.057
Aortic valve replacement
0.974
0.902
0.073
0.829
0.760
0.069
Pancreatic resection
0.773
0.759
0.014
0.658
0.637
0.020
Esophagectomy
0.955
0.762
0.194
0.722
0.658
0.064
Laparoscopic cholecystectomy
0.950
0.878
0.072
0.847
0.808
0.039
Colectomy
0.907
0.891
0.016
0.777
0.756
0.021
Appendectomy
0.947
0.954
-0.007
0.852
0.812
0.040
Abbreviation: AUC, area under the receiver operating characteristic curve. a
Comparing models that include all variables with models that only include objective variables.
models predicting complications, the difference in AUC was less than 0.05 among all patients and subsets of patients with pancreatic resection, laparoscopic cholecystectomy, colectomy, and appendectomy. The difference in AUC was greatest among patients with aortic valve replacement (0.0552 to 0.0687) and smallest among patients with pancreatic resection (0.0198 to 0.0204).
Discussion These data suggest that it is possible to create a riskadjustment system with high discriminatory value based only on objective variables. In this study, there was generally minimal difference in AUC when using all preoperative risk
jamasurgery.com
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
27
Research Original Investigation
Electronic Health Records and Surgical Quality Improvement
variables compared with using only objective variables. The difference in AUC was lower when examining complications than mortality, although the range of difference in AUC values differed by procedure. Thus, this method may be best to use when wanting to perform risk-adjusted analyses for complications, although the implications of this research are broad. By restricting data collection to objective data, we can reduce concerns about reliability and validity as well as threats of gaming the system from attempting to increase the risk score of patients through subjective variables. Restricting riskadjusted analyses to objective variables also improves efficiency and reduces cost of data collection. Clinicians would no longer have to deliberate and decide on patient functional status or the surgical wound classification and then take the time to enter this information in the medical record. Data collectors would also not have to scour the medical record to identify various comorbidities. Instead, basic laboratory values already ordered for direct patient care could be extracted from electronic medical records. For example, instead of having to input a separate code for a diagnosis of diabetes, a hemoglobin A1C value combined with prescriptions for insulin or oral hypoglycemic agents would provide enough information on this comorbidity. This research also generates opportunities for wider participation in surgical quality assessment and improvement programs. Currently, 461 US hospitals and 36 hospitals in Australia, Austria, Canada, Lebanon, the United Kingdom, and Saudi Arabia participate in the American College of Surgeons NSQIP.12 Yet, these are mostly high-volume hospitals, with very limited participation by smaller community hospitals. With the advent of electronic health records in all hospitals, it is not unrealistic to propose complete participation of all US hospitals in a national risk-adjusted program managed through an automated system. This concept has even wider implications. As the amount of data and number of databases continue to expand both within health care and the private sector, new opportunities for sophisticated and novel uses of information abound. In addition to the growing amount of data collected by electronic health records, the private sector has a plethora of databases that could be linked to patient data in innovative ways, from tracking purchasing patterns of healthy food or over-the-counter medicines to using global positioning system technology (now embedded in many personal handheld devices) to record patient exercise regimens or geographical disease hot spots. In our data-driven envi-
ARTICLE INFORMATION Accepted for Publication: April 3, 2014. Published Online: November 5, 2014. doi:10.1001/jamasurg.2014.947. Author Contributions: Drs Chang and Anderson had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Both authors. Acquisition, analysis, or interpretation of data: Chang. 28
ronment in which technology continues to become an integral part of our daily lives, the possibilities are endless. In the future, it is not inconceivable that we will be able to perform retrospective analyses of complex databases with the same rigor as prospective trials. Hiring additional personnel for the sole purpose of data collection may become outdated in the era of big data. This research was not without limitations. We narrowed down all NSQIP preoperative variables to a subset of objective variables but this list was not without flaws. Despite our attempts to be as strict as possible, we recognize that this list is still open to interpretation. For example, we classified more than 10% loss of body weight in the last 6 months as subjective because this may not always be accurately measured and recorded. Other NSQIP variables record clearly defined medical history or recent treatment given, such as chemotherapy for malignancy in the last 30 days prior to operation, but these variables may not always be easy to obtain or completely accurate because they may depend on outside records or history per the patient. Furthermore, many patients do not receive all preoperative laboratory tests, and thus, these missing data were excluded from these analyses. We accounted for this by using a large sample size including subset analyses of several different operations. Our findings are also consistent with the literature. For example, the predictive value of albumin in surgical outcomes has been well studied.13 Many of these variables have also found to be significant in other studies examining particular operations and correspond highly with our own previous research.1,2
Conclusions Rigorous risk-adjusted surgical quality assessment can be performed relying solely on objective, automated variables. By leveraging data that are already regularly collected for patient care, this approach is a more efficient use of the massive amounts of information available in the age of big data. This approach also addresses common concerns about the burden of data collection and the validity and reliability of data elements and can lead to wider adoption of quality assessment systems in health care. Identifying new ways to automate data collection using existing technology can make further improvements to the fields of surgical outcomes research and quality assessment and improvement.
Drafting of the manuscript: Anderson. Critical revision of the manuscript for important intellectual content: Both authors. Statistical analysis: Both authors. Administrative, technical, or material support: Chang. Study supervision: Chang. Conflict of Interest Disclosures: None reported. Disclaimer: The American College of Surgeons National Surgical Quality Improvement Program and its participating hospitals are the source of data
used in this research; they have not verified and are not responsible for the statistical validity of the data analysis or conclusions of the authors. REFERENCES 1. Anderson JE, Lassiter R, Bickler SW, Talamini MA, Chang DC. Brief tool to measure risk-adjusted surgical outcomes in resource-limited hospitals. Arch Surg. 2012;147(9):798-803. 2. Dimick JB, Osborne NH, Hall BL, Ko CY, Birkmeyer JD. Risk adjustment for comparing
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
jamasurgery.com
Electronic Health Records and Surgical Quality Improvement
3. Rubinfeld I, Farooq M, Velanovich V, et al. Predicting surgical risk: how much data is enough? AMIA Annu Symp Proc. 2010;777-781.
7. The Leapfrog Hospital Survey Reference Book: Supporting Documentation for the 2013 Leapfrog Hospital Survey. https://leapfroghospitalsurvey.org /web/wp-content/uploads/reference.pdf. Accessed May 14, 2013.
4. Birkmeyer JD, Shahian DM, Dimick JB, et al. Blueprint for a new American College of Surgeons: National Surgical Quality Improvement Program. J Am Coll Surg. 2008;207(5):777-782.
8. Healey C, Osler TM, Rogers FB, et al. Improving the Glasgow Coma Scale score: motor score alone is a better predictor. J Trauma. 2003;54(4):671-678, discussion 678-680.
5. Murdoch TB, Detsky AS. The inevitable application of big data to health care. JAMA. 2013; 309(13):1351-1352.
9. Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care. 2004;8(6):508-512.
6. American College of Surgeons National Surgical Quality Improvement Program. User Guide for the 2010 Participant Use Data File. American College of Surgeons; Chicago, IL: 2012.
10. Meredith JW, Evans G, Kilgo PD, et al. A comparison of the abilities of nine scoring algorithms in predicting mortality. J Trauma. 2002;53(4):621-628, discussion 628-629.
hospital quality with surgery: how many variables are needed? J Am Coll Surg. 2010;210(4):503-508.
jamasurgery.com
Original Investigation Research
11. Connell FA, Koepsell TD. Measures of gain in certainty from a diagnostic test. Am J Epidemiol. 1985;121(5):744-753. 12. American College of Surgeons National Surgical Quality Improvement Program. http://site.acsnsqip .org/participants/. Accessed September 23, 2013. 13. Gibbs J, Cull W, Henderson W, Daley J, Hur K, Khuri SF. Preoperative serum albumin level as a predictor of operative mortality and morbidity: results from the National VA Surgical Risk Study. Arch Surg. 1999;134(1):36-42.
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
29
Original Investigation
Using Electronic Health Records for Surgical Quality Improvement in the Era of Big Data Jamie E. Anderson, MD, MPH; David C. Chang, PhD, MPH, MBA
IMPORTANCE Risk adjustment is an important component of quality assessment in surgical health care. However, data collection places an additional burden on physicians. There is also concern that outcomes can be gamed depending on the information recorded for each patient. OBJECTIVE To determine whether a number of machine-collected data elements could perform as well as a traditional full-risk adjustment model that includes other physician-assessed and physician-recorded data elements. DESIGN, SETTINGS, AND PARTICIPANTS All general surgery patients from the National Surgical Quality Improvement Program database from January 1, 2005, to December 31, 2010, were included. Separate multivariate logistic regressions were performed using either all 66 preoperative risk variables or only 25 objective variables. The area under the receiver operating characteristic curve (AUC) of each regression using objective preoperative risk variables was compared with its corresponding regression with all preoperative variables. Subset analyses were performed among patients who received certain operations. MAIN OUTCOMES AND MEASURES Mortality or any surgical complication captured by the National Surgical Quality Improvement Program, both inpatient and within 30 days postoperatively. RESULTS Data from a total of 745 053 patients were included. More than 15.8% of patients
had at least 1 complication and the mortality rate was 2.8%. When examining inpatient mortality, the AUC was 0.9104 with all 66 variables vs 0.8918 with all 25 objective variables. The difference in AUC comparing models with all variables with objective variables ranged from −0.0073 to 0.1944 for mortality and 0.0198 to 0.0687 for complications. In models predicting mortality, the difference in AUC was less than 0.05 among all patients and subsets of patients with abdominal aortic aneurysm repair, pancreatic resection, colectomy, and appendectomy. In models predicting complications, the difference in AUC was less than 0.05 among all patients and subsets of patients with pancreatic resection, laparoscopic cholecystectomy, colectomy, and appendectomy. CONCLUSIONS AND RELEVANCE Rigorous risk-adjusted surgical quality assessment can be performed solely with objective variables. By leveraging data already routinely collected for patient care, this approach allows for wider adoption of quality assessment systems in health care. Identifying data elements that can be automatically collected can make future improvements to surgical outcomes and quality analyses.
Author Affiliations: Department of Surgery, University of California, San Diego.
JAMA Surg. 2015;150(1):24-29. doi:10.1001/jamasurg.2014.947 Published online November 5, 2014. 24
Corresponding Author: Jamie E. Anderson, MD, MPH, Center for Surgical Systems and Public Health, Department of Surgery, University of California, San Diego, 200 W Arbor Dr, San Diego, CA 92103 ([email protected]). jamasurgery.com
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
Electronic Health Records and Surgical Quality Improvement
R
isk adjustment is an important component of outcomes and quality analysis in surgical health care. However, there are 2 major concerns regarding the collection of risk-adjustment data in practice today. First, data collection places an additional burden on physicians, who feel that they have become data-entry personnel and are spending more time in front of the computer than with the patient. This concern could be partially addressed by studies that aim to identify minimum data sets that require only a handful of variables. For example, previous work has determined that the ability of the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) database to risk adjust may be adequate using as few as 4 variables.1-4 However, the more serious concern is that risk-adjusted quality benchmarking systems can be gamed because they include data elements that require subjective interpretation by hospital personnel, such as patient history of comorbidities. There may be subtle financial incentives to overcode these data elements to increase the risk scores for a hospital and, thus, improve a hospital’s risk-adjusted outcomes profile. These concerns can be addressed if risk-adjustment models avoid subjective data elements, such as history of comorbidities, and rely on objective data, such as laboratory values or other machine-collected variables that do not require the subjective interpretation and input of hospital personnel. The adoption of electronic health records has doubled from 2009 to 2011, partly a result of funding provided by the Health Information Technology for Economic and Health Act of 2009.5 This has contributed to so-called big data, multiple massive data sets that include patient demographics, medical history, disease course with corresponding treatment, and transaction information, such as device usage and medication administration. However, much of these data are currently perceived as by-products of health care delivery rather than a central asset to improve its efficiency.5 Therefore, the aim of this study was to determine whether a number of machinecollected data elements could perform as well as a traditional, full risk-adjustment model that includes other physician-assessed and physician-recorded data elements. We tested this hypothesis with an analysis of the NSQIP database.
Methods This research uses all available NSQIP data from January 1, 2005, to December 31, 2010. This nationally validated program measures more than 135 variables on each patient and follows up each patient for 30 days postoperatively. The 2005 to 2006 database included information from 121 hospitals while data from 2010 included information from 237 hospitals.6 This data set was chosen for its breadth of preoperative and postoperative variables collected for each patient. Owing to its retrospective nature, this study was exempt from institutional review board approval and participants did not provide consent. The primary analysis included all patients in the database who were categorized as having had an operation performed by a general surgeons or surgeons in the following subspecialties: plastic surgery, cardiothoracic surgery, or vascular
Original Investigation Research
surgery. Subset analyses were performed in patients who underwent abdominal aortic aneurysm repair, aortic valve replacement, coronary artery bypass graft, esophagectomy, pancreatic resection, laparoscopic cholecystectomy, colectomy (total or partial, laparoscopic or open), appendectomy (open or laparoscopic), and ventral hernia repair. The first 5 procedures listed were selected because of presence of Leapfrog quality standards.7 The last 4 procedures were selected because they are common general surgery procedures. Patients were determined to have an adverse event after surgery if they experienced death or at least 1 of the following complications as captured by NSQIP before discharge: superficial surgical site infection; deep incisional surgical site infection; organ space surgical site infection; wound disruption; pneumonia; unplanned intubation; pulmonary embolism; ventilator for more than 48 hours; progressive renal insufficiency; acute renal failure; urinary tract infection; stroke/ cerebrovascular accident with neurological deficit; coma of more than 24 hours; peripheral nerve injury; cardiac arrest requiring cardiopulmonary resuscitation; myocardial infarction; bleeding requiring transfusions; graft, prosthesis, or flap failure; deep vein thrombosis or thrombophlebitis; sepsis; or septic shock. Multivariate logistic regression models were created to predict either mortality or any complication in the inpatient setting or within 30 days of surgery. In one set of models we included all preoperative risk variables included in NSQIP, a total of 66 variables. In a second set of models we only included preoperative variables that were either laboratory data or able to be automatically extracted from a patient medical record without additional data input. This was a total of 25 variables, which are listed in the Box. All continuous variables were kept as such except for age, which was grouped into 10-year categories. We compared the area under the receiver operating characteristic curve (AUC) of each regression using objective preoperative risk variables to its corresponding regression with all variables. The AUC is a discriminative measure to identify how well a model separates 2 groups (ie, patients with vs without adverse events). An AUC value of 0.5 indicates that the model separates the 2 groups no better than chance whereas an AUC value of 1.0 indicates that the model completely separates the 2 groups. The AUC statistic is actually the percentage of randomly selected pairs correctly predicted by the model. Thus, the AUC allows us to determine which model can more accurately discriminate between the 2 groups of interest.8-11 Statistical analysis was performed using Stata 64-bit special edition, version 11.2 (StataCorp).
Results Data from a total of 745 053 patients from 2005 to 2010 from the NSQIP database were included (Table 1). More than 15.8% of patients had at least 1 complication and the mortality rate was 2.8%. Mean age was higher among those with complications (61.5 years) and highest among those who died (71.1 years). Among the subset procedures, colectomy had the highest per-
jamasurgery.com
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
25
Research Original Investigation
Electronic Health Records and Surgical Quality Improvement
Box. Preoperative National Surgical Quality Improvement Program Variables Classified According to Objectivity Objective variables included in the analysis: Age in 10-y categories
Current smoker within 1 y Currently receiving dialysis (prior to operation)
Body mass index
Cerebrovascular accident/stroke with neurological deficit
Pregnancy
Cerebrovascular accident/stroke with no neurological deficit
Alkaline phosphatase
Diabetes mellitus with oral agents or insulin
Blood urea nitrogen
Disseminated cancer
Hematocrit
Do not resuscitate status
International normalized ratio of PT values
Dyspnea
Platelet count
Emergency case
PT
Esophageal varices
PTT
Functional health status prior to current illness
Serum albumin
Functional health status prior to surgery
Serum creatinine
Hemiplegia
Serum sodium
History of angina in 1 mo prior to surgery
SGOT
History of myocardial infarction 6 mo prior to surgery
Systemic sepsis Total bilirubin
History of revascularization/amputation for peripheral vascular disease
White blood cell count
History of severe COPD
Principal anesthesia technique
History of transient ischemic attacks
Prior operation within 30 d
Hypertension requiring medication
Race/ethnicity (white, black, Hispanic, Asian or Pacific Islander, American Indian, Alaska Native, or other)
Impaired sensorium
Sex
Paraplegia
Surgical specialty of surgeon performing procedure
Previous cardiac surgery
Transfusion >4 U PRBCs in 72 h before surgery
Previous percutaneous coronary intervention
Subjective variables not included in the analysis: >10% Loss body weight in last 6 mo Acute renal failure
Mallampati scale
Quadriplegia Radiotherapy for malignancy in last 90 d Rest pain/gangrene
Airway trauma Alcohol, >2 drinks/d in 2 wk before admission ASA classification
Steroid use for chronic condition Tumor involving CNS Wound classification
Ascites
Wound infection
Bleeding disorder Chemotherapy for malignancy in ⱕ30 d prior to surgery Coma >24 h Congestive heart failure in 30 d prior to surgery
Abbreviations: ASA, American Society of Anesthesiologists; CNS, central nervous system; COPD, chronic obstructive pulmonary disease; PRBCs, packed red blood cells; PT, prothrombin time; PTT, partial thromboplastin time; SGOT, serum glutamic oxaloacetic transaminase.
Current pneumonia
centage of patients (12.7%). The fewest number of patients received aortic valve replacement (0.2%) and esophagectomy (0.3%). Overall complication rates were highest in esophagectomy (49.5%) and lowest in laparoscopic cholecystectomy (5.2%) and appendectomy (6.6%). Mortality rates were highest in colectomy (4.4%) and aortic valve replacement (4.3%). The AUC was slightly higher in models that included all variables compared with models with only objective variables. Among all patients, the AUC was 0.9005 with all variables vs 0.8774 with objective variables for 30-day mortality and 0.9078 with all variables vs 0.8881 with objective variables for inpatient mortality (Table 2). When examining com26
plications among all patients, the AUC was 0.7401 with all variables vs 0.7137 w ith objec tive variables for 30-day complications and 0.7859 with all variables vs 0.7609 with objective variables for inpatient complications (Table 2). The difference in AUC ranged from −0.0073 to 0.1944 for mortality and 0.0198 to 0.0687 for complications. In models predicting mortality, the difference in AUC was less than 0.05 among all patients and subsets of patients with abdominal aortic aneurysm repair, pancreatic resection, colectomy, and appendectomy. The difference in AUC was greatest among patients with esophagectomy (0.1937 to 0.1944) and smallest among patients with appendectomy (−0.0073 to 0.0171). In
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
jamasurgery.com
Electronic Health Records and Surgical Quality Improvement
Original Investigation Research
Table 1. Patient Demographics and Outcomes Within a 30-Day Postoperative Period
Characteristic Age, mean (SD), y
All (n = 745 053)
No Complications
Complications
Died
n = 606 468 (81.4%)
n = 117 673 (15.8%)
n = 20 912 (2.8%)
57.4 (17.3)
56.1 (17.4)
61.5 (15.8)
71.1 (13.5)
Race/ethnicity, No. (%) White
410 907 (79.0)
337 559 (78.9)
61 688 (79.3)
11 620 (80.6)
African American
58 804 (11.3)
47 064 (11.0)
9927 (12.8)
1813 (12.6)
Hispanic
34 479 (6.6)
29 779 (7.0)
4068 (5.2)
632 (4.4)
Asian or Pacific Islander
12 037 (2.3)
10 283 (2.4)
1491 (1.9)
263 (1.8)
3894 (0.8)
3224 (0.8)
580 (0.8)
Other or unknown Female, No. (%) Characteristic Abdominal aortic aneurysm repair Aortic valve replacement Pancreatic resection Esophagectomy
414 386 (55.8)
344 997 (57.0)
90 (0.6)
59 407 (50.6)
All Patients, No. (%)
9982 (47.9)
Patients With Procedure, No. (%)
14 042 (1.9)
11 537 (82.2)
2047 (14.6)
1551 (0.2)
943 (60.8)
542 (35.0)
458 (3.3) 66 (4.3)
13 530 (1.8)
8560 (63.3)
4619 (34.1)
351 (2.6)
2089 (0.3)
1055 (50.5)
Laparoscopic cholecystectomy
44 645 (6.0)
42 353 (94.9)
2040 (4.6)
957 (45.8)
252 (0.6)
77 (3.7)
Colectomy
94 554 (12.7)
66 872 (70.7)
23 522 (24.9)
4160 (4.4)
Appendectomy
63 855 (8.6)
59 597 (93.3)
4104 (6.4)
154 (0.2)
Table 2. AUC of Models Predicting Mortality and Complicationsa AUC, % Mortality
Complications
All Variables
Objective Variables
Difference
All Variables
Objective Variables
Difference
All patients
0.901
0.877
0.023
0.740
0.714
0.026
Abdominal aortic aneurysm repair
0.875
0.828
0.048
0.750
0.712
0.038
Aortic valve replacement
0.976
0.883
0.093
0.796
0.740
0.055
Outcomes 30-d
Pancreatic resection
0.774
0.739
0.035
0.641
0.621
0.020
Esophagectomy
0.949
0.755
0.194
0.701
0.645
0.057
Laparoscopic cholecystectomy
0.932
0.871
0.061
0.763
0.722
0.041
Colectomy
0.899
0.881
0.019
0.729
0.708
0.021
Appendectomy
0.981
0.964
0.017
0.763
0.730
0.033
0.908
0.888
0.020
0.786
0.761
0.025
Inpatient All patients Abdominal aortic aneurysm repair
0.868
0.821
0.047
0.808
0.751
0.057
Aortic valve replacement
0.974
0.902
0.073
0.829
0.760
0.069
Pancreatic resection
0.773
0.759
0.014
0.658
0.637
0.020
Esophagectomy
0.955
0.762
0.194
0.722
0.658
0.064
Laparoscopic cholecystectomy
0.950
0.878
0.072
0.847
0.808
0.039
Colectomy
0.907
0.891
0.016
0.777
0.756
0.021
Appendectomy
0.947
0.954
-0.007
0.852
0.812
0.040
Abbreviation: AUC, area under the receiver operating characteristic curve. a
Comparing models that include all variables with models that only include objective variables.
models predicting complications, the difference in AUC was less than 0.05 among all patients and subsets of patients with pancreatic resection, laparoscopic cholecystectomy, colectomy, and appendectomy. The difference in AUC was greatest among patients with aortic valve replacement (0.0552 to 0.0687) and smallest among patients with pancreatic resection (0.0198 to 0.0204).
Discussion These data suggest that it is possible to create a riskadjustment system with high discriminatory value based only on objective variables. In this study, there was generally minimal difference in AUC when using all preoperative risk
jamasurgery.com
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
27
Research Original Investigation
Electronic Health Records and Surgical Quality Improvement
variables compared with using only objective variables. The difference in AUC was lower when examining complications than mortality, although the range of difference in AUC values differed by procedure. Thus, this method may be best to use when wanting to perform risk-adjusted analyses for complications, although the implications of this research are broad. By restricting data collection to objective data, we can reduce concerns about reliability and validity as well as threats of gaming the system from attempting to increase the risk score of patients through subjective variables. Restricting riskadjusted analyses to objective variables also improves efficiency and reduces cost of data collection. Clinicians would no longer have to deliberate and decide on patient functional status or the surgical wound classification and then take the time to enter this information in the medical record. Data collectors would also not have to scour the medical record to identify various comorbidities. Instead, basic laboratory values already ordered for direct patient care could be extracted from electronic medical records. For example, instead of having to input a separate code for a diagnosis of diabetes, a hemoglobin A1C value combined with prescriptions for insulin or oral hypoglycemic agents would provide enough information on this comorbidity. This research also generates opportunities for wider participation in surgical quality assessment and improvement programs. Currently, 461 US hospitals and 36 hospitals in Australia, Austria, Canada, Lebanon, the United Kingdom, and Saudi Arabia participate in the American College of Surgeons NSQIP.12 Yet, these are mostly high-volume hospitals, with very limited participation by smaller community hospitals. With the advent of electronic health records in all hospitals, it is not unrealistic to propose complete participation of all US hospitals in a national risk-adjusted program managed through an automated system. This concept has even wider implications. As the amount of data and number of databases continue to expand both within health care and the private sector, new opportunities for sophisticated and novel uses of information abound. In addition to the growing amount of data collected by electronic health records, the private sector has a plethora of databases that could be linked to patient data in innovative ways, from tracking purchasing patterns of healthy food or over-the-counter medicines to using global positioning system technology (now embedded in many personal handheld devices) to record patient exercise regimens or geographical disease hot spots. In our data-driven envi-
ARTICLE INFORMATION Accepted for Publication: April 3, 2014. Published Online: November 5, 2014. doi:10.1001/jamasurg.2014.947. Author Contributions: Drs Chang and Anderson had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Both authors. Acquisition, analysis, or interpretation of data: Chang. 28
ronment in which technology continues to become an integral part of our daily lives, the possibilities are endless. In the future, it is not inconceivable that we will be able to perform retrospective analyses of complex databases with the same rigor as prospective trials. Hiring additional personnel for the sole purpose of data collection may become outdated in the era of big data. This research was not without limitations. We narrowed down all NSQIP preoperative variables to a subset of objective variables but this list was not without flaws. Despite our attempts to be as strict as possible, we recognize that this list is still open to interpretation. For example, we classified more than 10% loss of body weight in the last 6 months as subjective because this may not always be accurately measured and recorded. Other NSQIP variables record clearly defined medical history or recent treatment given, such as chemotherapy for malignancy in the last 30 days prior to operation, but these variables may not always be easy to obtain or completely accurate because they may depend on outside records or history per the patient. Furthermore, many patients do not receive all preoperative laboratory tests, and thus, these missing data were excluded from these analyses. We accounted for this by using a large sample size including subset analyses of several different operations. Our findings are also consistent with the literature. For example, the predictive value of albumin in surgical outcomes has been well studied.13 Many of these variables have also found to be significant in other studies examining particular operations and correspond highly with our own previous research.1,2
Conclusions Rigorous risk-adjusted surgical quality assessment can be performed relying solely on objective, automated variables. By leveraging data that are already regularly collected for patient care, this approach is a more efficient use of the massive amounts of information available in the age of big data. This approach also addresses common concerns about the burden of data collection and the validity and reliability of data elements and can lead to wider adoption of quality assessment systems in health care. Identifying new ways to automate data collection using existing technology can make further improvements to the fields of surgical outcomes research and quality assessment and improvement.
Drafting of the manuscript: Anderson. Critical revision of the manuscript for important intellectual content: Both authors. Statistical analysis: Both authors. Administrative, technical, or material support: Chang. Study supervision: Chang. Conflict of Interest Disclosures: None reported. Disclaimer: The American College of Surgeons National Surgical Quality Improvement Program and its participating hospitals are the source of data
used in this research; they have not verified and are not responsible for the statistical validity of the data analysis or conclusions of the authors. REFERENCES 1. Anderson JE, Lassiter R, Bickler SW, Talamini MA, Chang DC. Brief tool to measure risk-adjusted surgical outcomes in resource-limited hospitals. Arch Surg. 2012;147(9):798-803. 2. Dimick JB, Osborne NH, Hall BL, Ko CY, Birkmeyer JD. Risk adjustment for comparing
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
jamasurgery.com
Electronic Health Records and Surgical Quality Improvement
3. Rubinfeld I, Farooq M, Velanovich V, et al. Predicting surgical risk: how much data is enough? AMIA Annu Symp Proc. 2010;777-781.
7. The Leapfrog Hospital Survey Reference Book: Supporting Documentation for the 2013 Leapfrog Hospital Survey. https://leapfroghospitalsurvey.org /web/wp-content/uploads/reference.pdf. Accessed May 14, 2013.
4. Birkmeyer JD, Shahian DM, Dimick JB, et al. Blueprint for a new American College of Surgeons: National Surgical Quality Improvement Program. J Am Coll Surg. 2008;207(5):777-782.
8. Healey C, Osler TM, Rogers FB, et al. Improving the Glasgow Coma Scale score: motor score alone is a better predictor. J Trauma. 2003;54(4):671-678, discussion 678-680.
5. Murdoch TB, Detsky AS. The inevitable application of big data to health care. JAMA. 2013; 309(13):1351-1352.
9. Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care. 2004;8(6):508-512.
6. American College of Surgeons National Surgical Quality Improvement Program. User Guide for the 2010 Participant Use Data File. American College of Surgeons; Chicago, IL: 2012.
10. Meredith JW, Evans G, Kilgo PD, et al. A comparison of the abilities of nine scoring algorithms in predicting mortality. J Trauma. 2002;53(4):621-628, discussion 628-629.
hospital quality with surgery: how many variables are needed? J Am Coll Surg. 2010;210(4):503-508.
jamasurgery.com
Original Investigation Research
11. Connell FA, Koepsell TD. Measures of gain in certainty from a diagnostic test. Am J Epidemiol. 1985;121(5):744-753. 12. American College of Surgeons National Surgical Quality Improvement Program. http://site.acsnsqip .org/participants/. Accessed September 23, 2013. 13. Gibbs J, Cull W, Henderson W, Daley J, Hur K, Khuri SF. Preoperative serum albumin level as a predictor of operative mortality and morbidity: results from the National VA Surgical Risk Study. Arch Surg. 1999;134(1):36-42.
JAMA Surgery January 2015 Volume 150, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archsurg.jamanetwork.com/ by a Oakland University User on 06/07/2015
29
