ORIGINAL ARTICLE
Misclassification of Pelvic Ring Injuries in the National Trauma Data Bank Brittany E. Haws, BS,* Scott Wuertzer, MD,† Leon Lenchik, MD,† and Anna N. Miller, MD‡
Background: The American College of Surgeons has advocated for all trauma centers to participate in the National Trauma Data Bank (NTDB); however, no previous study has evaluated the accuracy of coding included in this data bank. The purpose of this study was to determine whether pelvic ring injuries are coded accurately in the NTDB and, if not, how they were misclassified.
Methods: A retrospective review of all pelvic ring injuries based on Abbreviated Injury Scale (AIS) codes was performed at a single level I academic trauma center from July 2010 to June 2013. Thin-section computed tomography (CT) scans in all patients were reviewed and classified using AIS codes: posterior arch intact, incomplete posterior arch, or complete posterior arch. The surgeon was blinded to the AIS code from the registry. These CT-based classifications were then compared with the pelvic ring injury codes designated in the trauma registry for each patient to evaluate agreement.
Results: Two hundred thirty-five patients with a mean age of 42 years had pelvic ring injuries in our registry. The agreement between trauma registry codes and CT reclassification was 24% in the intact group, 43% in the incomplete group, and 59% in the complete group. Using only the trauma registry codes, injuries were underclassified in 48% of the incomplete group and 76% of the intact group. Conclusions: Many pelvic ring injuries are miscoded and misclassified in the NTDB. The etiology of this misclassification is unclear, but any research data mined from these databases should be regarded cautiously. Key Words: misclassification, National Trauma Data Bank, pelvic ring, Abbreviated Injury Scale, pelvic ring injury, trauma (J Orthop Trauma 2015;29:460–464)
INTRODUCTION The American College of Surgeons has advocated participation in the National Trauma Data Bank (NTDB) for all trauma centers. Increasingly used for research, the NTDB comprised trauma records from over 800 institutions in the Accepted for publication March 30, 2015. From the *Wake Forest School of Medicine, Winston-Salem, NC; and Departments of †Radiology, and ‡Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC. A. N. Miller has received consultancy fees from Eli Lilly. The remaining authors report no conflict of interest. This study has been approved by our institutional review board. Reprints: Anna N. Miller, MD, Department of Orthopaedic Surgery, Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC 27157 (e-mail: [email protected]). Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
460
| www.jorthotrauma.com
United States as of 2012.1 The NTDB has been used in multiple studies: comparing disaster triage methods,2 evaluating mortality prediction abilities of different severity indices,3–7 and predicting cases of ventilator-assisted pneumonia.8 Some of the limitations of the NTDB have been evaluated, specifically, missing data. Moore et al used an algorithm to fill in some of the missing data and showed a 3.6% median difference in the odds ratios of mortality between the algorithm-produced data and the initial NTDB data.9 No previous study has evaluated the accuracy of the coding included in the NTDB. Pelvic ring and acetabular fractures are relatively common, occurring in 5%–16% of blunt trauma patients.10 These injuries have a high morbidity, with a short-term complication rate of 50%–80%,11 and a high mortality over 8%.12 Additionally, these injuries have historically been difficult to classify.13 However, accurate classification is essential for patient care and for clinical research. The purpose of this study was to determine whether pelvic ring injuries are coded accurately in the NTDB and, if not, how they were misclassified.
METHODS A retrospective review of all pelvic ring injuries at a single academic center from July 2010 to June 2013 was performed. The pelvic ring injuries were extracted from our trauma registry using the following Abbreviated Injury Scale (AIS) codes: 856100, 856101, 856151, 856152, 856161, 856162, 856163, 85616, 856171, 856172, and 856174. A single fellowship-trained orthopaedic trauma surgeon reviewed computed tomography (CT) scans in all patients. All CT scans were obtained on a 64-slice scanner (LightSpeed Plus, General Electric Medical Systems, Milwaukee, WI) located in the emergency department. The scan coverage included the entire bony pelvis at 2-mm slice thickness and 0.625-mm secondary reconstruction. Reconstructions in the sagittal and coronal planes were also obtained. Based on CT scans, the surgeon classified the injuries using the AIS system as: posterior arch intact, incomplete posterior arch, or complete posterior arch. This system was chosen as it is the most commonly used for the NTDB and is one of the simplest classifications with the highest interobserver reliability.14 The surgeon was blinded to the AIS code from the NTDB registry. These CT-based classifications were then compared with the AIS codes designated in the registry.
RESULTS
Two hundred fifty patients in our registry had pelvic ring injuries based on the codes listed. The mean age was 42 J Orthop Trauma Volume 29, Number 10, October 2015
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
J Orthop Trauma Volume 29, Number 10, October 2015
Misclassification of Pelvic Ring Injuries
TABLE 1. Comparison of National Trauma Databank (NTDB) Classification as Originally Entered in Database With Reclassification After Re-examination of Data.
Classification in NTDB Reclassification
Intact Posterior Ring, n (%)
Partial Posterior Ring, n (%)
Complete Posterior Ring, n (%)
NFS, n (%)
Total
122 (52) 44 (19)
75 (32) 110 (47)
17 (7) 81 (34)
21 (9) 0 (0)
235 235
(range 15–90) years. There were 104 women (42%) and 146 men (58%). Fifteen of 250 (6%) had 2 or more conflicting pelvic injury codes in the trauma registry, and these were excluded. Of the remaining 235 patients, the original codes specified in the registry indicated an intact posterior ring in 122 (52%), partial posterior ring injury in 75 (32%), and complete posterior ring injury in 17 (7%) of patients. After reclassification by the orthopaedic trauma–trained surgeon, the comparison showed disagreement 76% of the time in the intact posterior ring groups, 57% of the time in the incomplete posterior ring injury groups, and 41% in the complete posterior ring injury groups. Looking at the trauma registry codes, injuries were underclassified in 48% of the incomplete group and 76% of the intact group, compared with our reclassification. In contrast, the trauma registry codes overclassified 9% of the incomplete group and 41% of the complete group, compared with our reclassification. Finally, the trauma registry coded 21 of 235 (9%) as “not further specified” (NFS). In the NFS group on reclassification, CT showed an intact posterior ring in 5 of 21 (24%), partial posterior ring injury in 12 of 21 (57%), and complete posterior ring injury in 4 of 21 (24%) patients (Table 1).
DISCUSSION Pelvic ring injuries are common, occurring in 3%–8.2% of all trauma patients, and have a strong correlation with complications and poor long-term outcomes.15 In a 3260patient study, complex pelvic injuries had a mortality rate of 31.1%, whereas for pelvic fractures without concomitant soft-tissue injury, the rate was 10.8%.16 Yoshihara and Yoneoka17 identified patients with unstable pelvic fractures from 2000 to 2009 using ICD-9 codes and reported an in-hospital mortality rate of 8.3%, with open-fracture mortality rate 3 times higher than closed fractures. We found that many pelvic ring injuries are misclassified in the NTDB. Compared with our reclassifications, only 24% of the intact, 43% of the incomplete, and 59% of the complete posterior ring injury groups were in agreement with our classifications. This shows that over half of the pelvis fractures that entered the data bank were not in agreement with our classifications, which has important implications for further research with the data. Specifically, the degree of injury (which is often used for mortality prediction research) may not be as accurate as previously assumed. The reasons for the misclassifications are not known; most likely, there are multiple causes. Misclassification may be because of the high difficulty of classifying pelvic ring injuries. Gabbe et al compared the classifications made by 3 orthopaedic surgeons on 187 severe pelvic ring fractures Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
using the Tile/AO and Young–Burgess systems. They determined that there was low interobserver agreement (k = 0.10 to 0.17 and k = 0.09 to 0.21 for Tile/AO and Young–Burgess, respectively) between these surgeons and concluded that the classification of severe pelvic ring fractures is extremely difficult.13 The AIS has previously been evaluated and may be unreliable. Ringdal et al studied the AIS codes assigned to a variety of trauma cases by physicians, nurses, and other health care personnel. They reported that 61.5% of the assigned codes agreed with their reference standard and 13.3% of the AIS codes were misclassified regarding level of severity.18 A similar study had 15 raters classify 375 trauma patients using the AIS codes. They found that interobserver agreement was moderate to substantial (k = 0.66–0.81).19 Alternatively, there may have been human error in transferring the codes into the registry, leading to miscoding. In addition, if the individuals entering data are not medical professionals, they may be incorrectly classifying injuries based on radiology reports or other things they read in the medical record, which may or may not be accurately interpreted. Mackenzie et al showed that physicians and nurses were more accurate than other health care personnel with this coding.19 In our study, injuries were often underclassified, with 48% of the incomplete group and 76% of the intact group having more severe fractures than coded in the data bank. Furthermore, 9% of the injuries were coded as “NFS” and were not included in our calculation of the misclassification percentage. These were all reclassified and had they been included, our misclassification number may have been even higher. All of these NFS codings would have been included with the underclassified injuries, which would increase the severity of the diagnoses. Correct classification of injuries is extremely important for patient care and for research. With an improper diagnosis, improper treatment options may be chosen leading to adverse outcomes. Because of significant underclassification, it is difficult to estimate how this could bias research on pelvic ring injuries and patient outcomes. In conclusion, pelvic injury research that is based on the NTDB should be performed with caution. Retrospective evaluation of imaging studies may be necessary before using the NTDB for research purposes. REFERENCES 1. NTDB Data Center Home. 2014. Available at: https://ntdbdatacenter. com. Accessed June 24, 2014. 2. Cross KP, Cicero MX. Head-to-head comparison of disaster triage methods in pediatric, adult, and geriatric patients. Ann Emerg Med. 2013;61: 668–676.e7.
www.jorthotrauma.com |
461
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
J Orthop Trauma Volume 29, Number 10, October 2015
Haws et al
3. Haider AH, Villegas CV, Saleem T, et al. Should the IDC-9 trauma mortality prediction model become the new paradigm for benchmarking trauma outcomes? J Trauma Acute Care Surg. 2012;72: 1695–1701. 4. Millham FH, LaMorte WW. Factors associated with mortality in trauma: re-evaluation of the TRISS method using the national trauma data bank. J Trauma. 2004;56:1090–1096. 5. Moore L, Lavoie A, Turgeon AF, et al. Improving trauma mortality prediction modeling for blunt trauma. J Trauma. 2010;68:698–705. 6. Rogers FB, Osler T, Krasne M, et al. Has TRISS become an anachronism? A comparison of mortality between the national trauma data bank and major trauma outcome study databases. J Trauma Acute Care Surg. 2012;73:326–331. 7. Schetinin V, Jakaite L, Jakaitis J, et al. Bayesian decision trees for predicting survival of patients: a study on the us national trauma data bank. Comput Methods Programs Biomed. 2013;111:602–612. 8. Pearl A, Bar-Or D. Decision support in trauma management: predicting potential cases of ventilator associated pneumonia. Stud Health Technol Inform. 2012;180:305–309. 9. Moore L, Hanley JA, Lavoie A, et al. Evaluating the validity of multiple imputation for missing physiological data in the national trauma data bank. J Emerg Trauma Shock. 2009;2:73–79. 10. Chesser TJ, Cross AM, Ward AJ. The use of pelvic binders in the emergent management of potential pelvic trauma. Injury. 2012;43: 667–669.
11. Dunet B, Tournier C, Billaud A, et al. Acetabular fracture: long-term follow-up and factors associated with secondary implantation of total hip arthroplasty. Orthop Traumatol Surg Res. 2013;99:281–290. 12. Poole GV, Ward EF. Causes of mortality in patients with pelvic fractures. Orthopedics. 1994;17:691–696. 13. Gabbe BJ, Esser M, Bucknill A, et al. The imaging and classification of severe pelvic ring fractures: experiences from two level 1 trauma centres. Bone Joint J. 2013;95-B:1396–1401. 14. Mitchell CS, Cloeren M, Schwartz BS. Application of an injury surveillance system to injuries at an industrial facility. Accid Anal Prev. 1993; 25:453–458. 15. Mardanpour K, Rahbar M. The outcome of surgically treated traumatic unstable pelvic fractures by open reduction and internal fixation. J Inj Violence Res. 2013;5:77–83. 16. Gänsslen A, Pohlemann T, Paul C, et al. Epidemiology of pelvic ring injuries. Injury. 1996;27(suppl 1):S-A13–S-A20. 17. Yoshihara H, Yoneoka D. Demographic epidemiology of unstable pelvic fracture in the United States from 2000 to 2009: trends and in-hospital mortality. J Trauma Acute Care Surg. 2014;76:380–385. 18. Ringdal KG, Skaga NO, Hestnes M, et al. Abbreviated Injury Scale: not a reliable basis for summation of injury severity in trauma facilities? Injury. 2013;44:691–699. 19. MacKenzie EJ, Shapiro S, Eastham JN. The Abbreviated Injury Scale and Injury Severity Score. Levels of inter- and intrarater reliability. Med Care. 1985;23:823–835.
Invited Commentaries aws et al in their article “Misclassification of Pelvic Ring Injuries in the National Trauma Data Bank” review 235 pelvic ring injuries entered into the National Trauma Data Bank (NTDB) using the Abbreviated Injury Scale (AIS) over a 3-year period in an effort to explore the “accuracy” and “reliability” of each classification. The authors report a high degree of miscoding and misclassification within the database at a single institution and argue that as a result the reader should consider any pelvic ring injury studies from the NTDB with caution. They advocate for a retrospective review of all pelvic imaging before utilization of the NTDB for research. The authors are to be applauded for this effort—as we enter into the era of “big data” in orthopaedic trauma, it is critical that we bring attention to the data on which we are basing our research and therefore potential future clinical decision making. However, the article does not offer the reader far-reaching insights but may confirm some things that we already know; the current state of pelvic ring classification leaves something to be desired, and individual centers may have issues with data bank entry. As Ringdal et al1 have recently shown, the AIS does not have a high degree of reliability and the same holds true for the remaining systems of pelvic ring injury classification.2 Although the article does effectively communicate a clear problem with the classification of pelvic ring injuries in the NTDB, the methodology used is problematic as a single fellowship-trained orthopaedic trauma surgeon reviews the CT scans of each pelvis on one occasion,
H
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
leaving no room for interobserver or intraobserver analysis. The process by which the data was initially entered into the NTDB from their institution is not fully revealed. Shedding more light on the exact process of initial NTDB data entry at this institution and the potential pit falls would have offered a powerful contribution to help guide possible changes needed at their center, as well as others where the problem may (or may not) exist. Furthermore, terms such as accuracy and agreement are used interchangeably but are very different in nature. Accuracy implies that there is some agreed on classification of each injury in this patient population, and agreement suggests multiple observers or even the same observer reviewed the CT scans of these 235 patients more than once. The authors are on point to bring our attention to the NTDB as this database and others such at the American College of Surgeons National Surgical Quality Improvement Program will help to determine quality and outcome measures in the coming years in orthopaedic trauma surgery. However, although Haws et al bring into question the utilization of the AIS for pelvic ring injuries, this scoring system with regard to pelvic fractures has not been shown to be predictive of morbidity and mortality like other classification systems such as the Tile and Young and Burgess classification systems, respectively.3 Given this fact, even with reconsidering the AIS scores of patients with pelvic ring injuries, would the findings of the NTDB with regard to mortality following pelvic fractures dramatically change? When considered in a broader context, this study leads to critical questions on the utilization of a large database like the NTDB. At each institution participating in the NTDB program, perhaps we should bring more attention to the
462
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
| www.jorthotrauma.com
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Misclassification of Pelvic Ring Injuries in the National Trauma Data Bank Brittany E. Haws, BS,* Scott Wuertzer, MD,† Leon Lenchik, MD,† and Anna N. Miller, MD‡
Background: The American College of Surgeons has advocated for all trauma centers to participate in the National Trauma Data Bank (NTDB); however, no previous study has evaluated the accuracy of coding included in this data bank. The purpose of this study was to determine whether pelvic ring injuries are coded accurately in the NTDB and, if not, how they were misclassified.
Methods: A retrospective review of all pelvic ring injuries based on Abbreviated Injury Scale (AIS) codes was performed at a single level I academic trauma center from July 2010 to June 2013. Thin-section computed tomography (CT) scans in all patients were reviewed and classified using AIS codes: posterior arch intact, incomplete posterior arch, or complete posterior arch. The surgeon was blinded to the AIS code from the registry. These CT-based classifications were then compared with the pelvic ring injury codes designated in the trauma registry for each patient to evaluate agreement.
Results: Two hundred thirty-five patients with a mean age of 42 years had pelvic ring injuries in our registry. The agreement between trauma registry codes and CT reclassification was 24% in the intact group, 43% in the incomplete group, and 59% in the complete group. Using only the trauma registry codes, injuries were underclassified in 48% of the incomplete group and 76% of the intact group. Conclusions: Many pelvic ring injuries are miscoded and misclassified in the NTDB. The etiology of this misclassification is unclear, but any research data mined from these databases should be regarded cautiously. Key Words: misclassification, National Trauma Data Bank, pelvic ring, Abbreviated Injury Scale, pelvic ring injury, trauma (J Orthop Trauma 2015;29:460–464)
INTRODUCTION The American College of Surgeons has advocated participation in the National Trauma Data Bank (NTDB) for all trauma centers. Increasingly used for research, the NTDB comprised trauma records from over 800 institutions in the Accepted for publication March 30, 2015. From the *Wake Forest School of Medicine, Winston-Salem, NC; and Departments of †Radiology, and ‡Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC. A. N. Miller has received consultancy fees from Eli Lilly. The remaining authors report no conflict of interest. This study has been approved by our institutional review board. Reprints: Anna N. Miller, MD, Department of Orthopaedic Surgery, Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC 27157 (e-mail: [email protected]). Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
460
| www.jorthotrauma.com
United States as of 2012.1 The NTDB has been used in multiple studies: comparing disaster triage methods,2 evaluating mortality prediction abilities of different severity indices,3–7 and predicting cases of ventilator-assisted pneumonia.8 Some of the limitations of the NTDB have been evaluated, specifically, missing data. Moore et al used an algorithm to fill in some of the missing data and showed a 3.6% median difference in the odds ratios of mortality between the algorithm-produced data and the initial NTDB data.9 No previous study has evaluated the accuracy of the coding included in the NTDB. Pelvic ring and acetabular fractures are relatively common, occurring in 5%–16% of blunt trauma patients.10 These injuries have a high morbidity, with a short-term complication rate of 50%–80%,11 and a high mortality over 8%.12 Additionally, these injuries have historically been difficult to classify.13 However, accurate classification is essential for patient care and for clinical research. The purpose of this study was to determine whether pelvic ring injuries are coded accurately in the NTDB and, if not, how they were misclassified.
METHODS A retrospective review of all pelvic ring injuries at a single academic center from July 2010 to June 2013 was performed. The pelvic ring injuries were extracted from our trauma registry using the following Abbreviated Injury Scale (AIS) codes: 856100, 856101, 856151, 856152, 856161, 856162, 856163, 85616, 856171, 856172, and 856174. A single fellowship-trained orthopaedic trauma surgeon reviewed computed tomography (CT) scans in all patients. All CT scans were obtained on a 64-slice scanner (LightSpeed Plus, General Electric Medical Systems, Milwaukee, WI) located in the emergency department. The scan coverage included the entire bony pelvis at 2-mm slice thickness and 0.625-mm secondary reconstruction. Reconstructions in the sagittal and coronal planes were also obtained. Based on CT scans, the surgeon classified the injuries using the AIS system as: posterior arch intact, incomplete posterior arch, or complete posterior arch. This system was chosen as it is the most commonly used for the NTDB and is one of the simplest classifications with the highest interobserver reliability.14 The surgeon was blinded to the AIS code from the NTDB registry. These CT-based classifications were then compared with the AIS codes designated in the registry.
RESULTS
Two hundred fifty patients in our registry had pelvic ring injuries based on the codes listed. The mean age was 42 J Orthop Trauma Volume 29, Number 10, October 2015
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
J Orthop Trauma Volume 29, Number 10, October 2015
Misclassification of Pelvic Ring Injuries
TABLE 1. Comparison of National Trauma Databank (NTDB) Classification as Originally Entered in Database With Reclassification After Re-examination of Data.
Classification in NTDB Reclassification
Intact Posterior Ring, n (%)
Partial Posterior Ring, n (%)
Complete Posterior Ring, n (%)
NFS, n (%)
Total
122 (52) 44 (19)
75 (32) 110 (47)
17 (7) 81 (34)
21 (9) 0 (0)
235 235
(range 15–90) years. There were 104 women (42%) and 146 men (58%). Fifteen of 250 (6%) had 2 or more conflicting pelvic injury codes in the trauma registry, and these were excluded. Of the remaining 235 patients, the original codes specified in the registry indicated an intact posterior ring in 122 (52%), partial posterior ring injury in 75 (32%), and complete posterior ring injury in 17 (7%) of patients. After reclassification by the orthopaedic trauma–trained surgeon, the comparison showed disagreement 76% of the time in the intact posterior ring groups, 57% of the time in the incomplete posterior ring injury groups, and 41% in the complete posterior ring injury groups. Looking at the trauma registry codes, injuries were underclassified in 48% of the incomplete group and 76% of the intact group, compared with our reclassification. In contrast, the trauma registry codes overclassified 9% of the incomplete group and 41% of the complete group, compared with our reclassification. Finally, the trauma registry coded 21 of 235 (9%) as “not further specified” (NFS). In the NFS group on reclassification, CT showed an intact posterior ring in 5 of 21 (24%), partial posterior ring injury in 12 of 21 (57%), and complete posterior ring injury in 4 of 21 (24%) patients (Table 1).
DISCUSSION Pelvic ring injuries are common, occurring in 3%–8.2% of all trauma patients, and have a strong correlation with complications and poor long-term outcomes.15 In a 3260patient study, complex pelvic injuries had a mortality rate of 31.1%, whereas for pelvic fractures without concomitant soft-tissue injury, the rate was 10.8%.16 Yoshihara and Yoneoka17 identified patients with unstable pelvic fractures from 2000 to 2009 using ICD-9 codes and reported an in-hospital mortality rate of 8.3%, with open-fracture mortality rate 3 times higher than closed fractures. We found that many pelvic ring injuries are misclassified in the NTDB. Compared with our reclassifications, only 24% of the intact, 43% of the incomplete, and 59% of the complete posterior ring injury groups were in agreement with our classifications. This shows that over half of the pelvis fractures that entered the data bank were not in agreement with our classifications, which has important implications for further research with the data. Specifically, the degree of injury (which is often used for mortality prediction research) may not be as accurate as previously assumed. The reasons for the misclassifications are not known; most likely, there are multiple causes. Misclassification may be because of the high difficulty of classifying pelvic ring injuries. Gabbe et al compared the classifications made by 3 orthopaedic surgeons on 187 severe pelvic ring fractures Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
using the Tile/AO and Young–Burgess systems. They determined that there was low interobserver agreement (k = 0.10 to 0.17 and k = 0.09 to 0.21 for Tile/AO and Young–Burgess, respectively) between these surgeons and concluded that the classification of severe pelvic ring fractures is extremely difficult.13 The AIS has previously been evaluated and may be unreliable. Ringdal et al studied the AIS codes assigned to a variety of trauma cases by physicians, nurses, and other health care personnel. They reported that 61.5% of the assigned codes agreed with their reference standard and 13.3% of the AIS codes were misclassified regarding level of severity.18 A similar study had 15 raters classify 375 trauma patients using the AIS codes. They found that interobserver agreement was moderate to substantial (k = 0.66–0.81).19 Alternatively, there may have been human error in transferring the codes into the registry, leading to miscoding. In addition, if the individuals entering data are not medical professionals, they may be incorrectly classifying injuries based on radiology reports or other things they read in the medical record, which may or may not be accurately interpreted. Mackenzie et al showed that physicians and nurses were more accurate than other health care personnel with this coding.19 In our study, injuries were often underclassified, with 48% of the incomplete group and 76% of the intact group having more severe fractures than coded in the data bank. Furthermore, 9% of the injuries were coded as “NFS” and were not included in our calculation of the misclassification percentage. These were all reclassified and had they been included, our misclassification number may have been even higher. All of these NFS codings would have been included with the underclassified injuries, which would increase the severity of the diagnoses. Correct classification of injuries is extremely important for patient care and for research. With an improper diagnosis, improper treatment options may be chosen leading to adverse outcomes. Because of significant underclassification, it is difficult to estimate how this could bias research on pelvic ring injuries and patient outcomes. In conclusion, pelvic injury research that is based on the NTDB should be performed with caution. Retrospective evaluation of imaging studies may be necessary before using the NTDB for research purposes. REFERENCES 1. NTDB Data Center Home. 2014. Available at: https://ntdbdatacenter. com. Accessed June 24, 2014. 2. Cross KP, Cicero MX. Head-to-head comparison of disaster triage methods in pediatric, adult, and geriatric patients. Ann Emerg Med. 2013;61: 668–676.e7.
www.jorthotrauma.com |
461
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
J Orthop Trauma Volume 29, Number 10, October 2015
Haws et al
3. Haider AH, Villegas CV, Saleem T, et al. Should the IDC-9 trauma mortality prediction model become the new paradigm for benchmarking trauma outcomes? J Trauma Acute Care Surg. 2012;72: 1695–1701. 4. Millham FH, LaMorte WW. Factors associated with mortality in trauma: re-evaluation of the TRISS method using the national trauma data bank. J Trauma. 2004;56:1090–1096. 5. Moore L, Lavoie A, Turgeon AF, et al. Improving trauma mortality prediction modeling for blunt trauma. J Trauma. 2010;68:698–705. 6. Rogers FB, Osler T, Krasne M, et al. Has TRISS become an anachronism? A comparison of mortality between the national trauma data bank and major trauma outcome study databases. J Trauma Acute Care Surg. 2012;73:326–331. 7. Schetinin V, Jakaite L, Jakaitis J, et al. Bayesian decision trees for predicting survival of patients: a study on the us national trauma data bank. Comput Methods Programs Biomed. 2013;111:602–612. 8. Pearl A, Bar-Or D. Decision support in trauma management: predicting potential cases of ventilator associated pneumonia. Stud Health Technol Inform. 2012;180:305–309. 9. Moore L, Hanley JA, Lavoie A, et al. Evaluating the validity of multiple imputation for missing physiological data in the national trauma data bank. J Emerg Trauma Shock. 2009;2:73–79. 10. Chesser TJ, Cross AM, Ward AJ. The use of pelvic binders in the emergent management of potential pelvic trauma. Injury. 2012;43: 667–669.
11. Dunet B, Tournier C, Billaud A, et al. Acetabular fracture: long-term follow-up and factors associated with secondary implantation of total hip arthroplasty. Orthop Traumatol Surg Res. 2013;99:281–290. 12. Poole GV, Ward EF. Causes of mortality in patients with pelvic fractures. Orthopedics. 1994;17:691–696. 13. Gabbe BJ, Esser M, Bucknill A, et al. The imaging and classification of severe pelvic ring fractures: experiences from two level 1 trauma centres. Bone Joint J. 2013;95-B:1396–1401. 14. Mitchell CS, Cloeren M, Schwartz BS. Application of an injury surveillance system to injuries at an industrial facility. Accid Anal Prev. 1993; 25:453–458. 15. Mardanpour K, Rahbar M. The outcome of surgically treated traumatic unstable pelvic fractures by open reduction and internal fixation. J Inj Violence Res. 2013;5:77–83. 16. Gänsslen A, Pohlemann T, Paul C, et al. Epidemiology of pelvic ring injuries. Injury. 1996;27(suppl 1):S-A13–S-A20. 17. Yoshihara H, Yoneoka D. Demographic epidemiology of unstable pelvic fracture in the United States from 2000 to 2009: trends and in-hospital mortality. J Trauma Acute Care Surg. 2014;76:380–385. 18. Ringdal KG, Skaga NO, Hestnes M, et al. Abbreviated Injury Scale: not a reliable basis for summation of injury severity in trauma facilities? Injury. 2013;44:691–699. 19. MacKenzie EJ, Shapiro S, Eastham JN. The Abbreviated Injury Scale and Injury Severity Score. Levels of inter- and intrarater reliability. Med Care. 1985;23:823–835.
Invited Commentaries aws et al in their article “Misclassification of Pelvic Ring Injuries in the National Trauma Data Bank” review 235 pelvic ring injuries entered into the National Trauma Data Bank (NTDB) using the Abbreviated Injury Scale (AIS) over a 3-year period in an effort to explore the “accuracy” and “reliability” of each classification. The authors report a high degree of miscoding and misclassification within the database at a single institution and argue that as a result the reader should consider any pelvic ring injury studies from the NTDB with caution. They advocate for a retrospective review of all pelvic imaging before utilization of the NTDB for research. The authors are to be applauded for this effort—as we enter into the era of “big data” in orthopaedic trauma, it is critical that we bring attention to the data on which we are basing our research and therefore potential future clinical decision making. However, the article does not offer the reader far-reaching insights but may confirm some things that we already know; the current state of pelvic ring classification leaves something to be desired, and individual centers may have issues with data bank entry. As Ringdal et al1 have recently shown, the AIS does not have a high degree of reliability and the same holds true for the remaining systems of pelvic ring injury classification.2 Although the article does effectively communicate a clear problem with the classification of pelvic ring injuries in the NTDB, the methodology used is problematic as a single fellowship-trained orthopaedic trauma surgeon reviews the CT scans of each pelvis on one occasion,
H
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
leaving no room for interobserver or intraobserver analysis. The process by which the data was initially entered into the NTDB from their institution is not fully revealed. Shedding more light on the exact process of initial NTDB data entry at this institution and the potential pit falls would have offered a powerful contribution to help guide possible changes needed at their center, as well as others where the problem may (or may not) exist. Furthermore, terms such as accuracy and agreement are used interchangeably but are very different in nature. Accuracy implies that there is some agreed on classification of each injury in this patient population, and agreement suggests multiple observers or even the same observer reviewed the CT scans of these 235 patients more than once. The authors are on point to bring our attention to the NTDB as this database and others such at the American College of Surgeons National Surgical Quality Improvement Program will help to determine quality and outcome measures in the coming years in orthopaedic trauma surgery. However, although Haws et al bring into question the utilization of the AIS for pelvic ring injuries, this scoring system with regard to pelvic fractures has not been shown to be predictive of morbidity and mortality like other classification systems such as the Tile and Young and Burgess classification systems, respectively.3 Given this fact, even with reconsidering the AIS scores of patients with pelvic ring injuries, would the findings of the NTDB with regard to mortality following pelvic fractures dramatically change? When considered in a broader context, this study leads to critical questions on the utilization of a large database like the NTDB. At each institution participating in the NTDB program, perhaps we should bring more attention to the
462
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
| www.jorthotrauma.com
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
