Improving Clinical Data Integrity by using Data Adjudication Techniques for Data Received through a Health Information Exchange (HIE) Pallavi Ranade-Kharkar, MS1,2, Susan E. Pollock, MS1, Darren K. Mann, BS1, Sidney N. Thornton, PhD1,2 1 Intermountain Healthcare, Murray, UT; 2 Department of Biomedical Informatics, University of Utah, Salt Lake City, UT Abstract Growing participation in Healthcare Information Exchange (HIE) has created opportunities for the seamless integration of external data into an organization’s own EHR and clinical workflows. The process of integrating external data has the potential to detect data integrity issues. Lack of critiquing external data before its incorporation can lead to data unfit for use in the clinical setting. HIE data adjudication, by detecting inconsistencies, physiological and temporal incompatibilities, data completeness and timeliness issues in HIE data, facilitates corrective actions and improves clinical data integrity. Introduction Health Information Exchange (HIE) holds the promise of improving the quality and efficiency of health care1, 2. Hospital-based HIE grew 41% between 2008 and 2012 as six out of ten hospitals exchanged information with providers and hospitals external to their organization3. Another study found three-fold increase in HIE participation from 2010 to 2012 when 10% ambulatory practices participated in 119 operational HIEs 4. As the capability to exchange health information among healthcare organizations is a mandatory requirement for “Meaningful Use” and certification of electronic health records (EHR)5, we can expect increased adoption and utilization of HIE in the future. Growing participation in HIE has created opportunities for integrating data from external sources into an organization’s own EHR. The exchange of data for patients served by more than one organization can also lead to uncovering inconsistencies in data. This may open up opportunities for corrective actions on one’s own data. However, more data does not always mean better care. Kuperman and McGowan note that a flood of data from external sources can have the unintended consequence of overwhelming clinicians6. HIE-related errors can also reduce patient safety7. Incoming external data need to be critiqued for duplication, contradiction and physiological compatibilities. Data incorporation done in conjunction with data adjudication has the potential to improve clinical data integrity in the receiving organization’s EHR. This manuscript addresses the following types of data integrity issues: the consistency, physiological compatibility, completeness, and timeliness of data. In a previous study, we analyzed data from Intermountain Healthcare’s (IH) Enterprise Data Warehouse (EDW) to identify error patterns, extracted requirements for a data adjudication framework, and designed architecture for an inferencing solution8. In this manuscript, we present a study which describes how the data adjudication infrastructure addresses data integrity issues for the use case of a point-to-point HIE between IH and a government trading partner that exchanges death information. Our approach as a partner in the Healtheway collaborative23 (formerly Nationwide Health Information Network Exchange) has been point-to-point HIE connections rather than to an HIE as a consolidator organization. We believe this approach allows for local decision autonomy for issues of clinical data integrity as opposed to HIE organizations which are limited by consensus decisions and potentially by the lowest quality source of data. This study is novel for a number of reasons: 1. It addresses the issue of data integrity for HIE data inbound into an organization; 2. It goes beyond passively accepting HIE data and takes it a step further into incorporating that data into an organization’s own EHR; 3. It takes into consideration 4 axes of data integrity: consistency, compatibility, completeness and timeliness; 4. It applies a decision support infrastructure to adjudicate HIE data. Background Data integrity: EHRs have highly variable levels of data accuracy ranging from 30% to 100% data correctness and completeness9. The integrity of data in healthcare information systems affects decision making processes, both manual and automated. Berner et al.10 report that gaps in patient data can affect the accuracy of the output of the Clinical Decision Support (CDS) tools. Guidance and recommendations provided by CDS systems may not be
1894
trustworthy unless they are based on data with high levels of integrity11. Clinician perception of the effectiveness of CDS is dependent on data quality across settings, organizations and types of EHRs12. Data integrity issues also affect secondary use of health care data for the purposes of quality improvement, public health, and research13. In this study, we address the following types of clinical data integrity: consistency, physiological compatibility, completeness and timeliness of data. Current literature on data integrity in the context of HIE focuses on the data integrity issues surrounding patient identification14, 15. In this manuscript, we focus on death information exchanged in an HIE, its consistency and physiological compatibility with other clinical data in the receiving organization’s EHR, and its completeness and timeliness. Data incorporation: While HIE is about moving data from one organization to the other, data incorporation is about integrating appropriate data into an organization’s own EHR and workflows. Literature on data incorporation in the realm of HIE is sparse. Frisse et al.16 describe data incorporation performed in the Memphis HIE which was done at the display level. In another system, a high-tech approach was applied to medication reconciliation where data from a pharmacy system was incorporated into an EHR17. Cerner, a commercial vendor of EHR and IH’s strategic partner, uses a document-based data model for integration of HIE data. The incoming documents with patient data are stored in a repository and the data in it is transformed for display, transfer, etc., as needed18. Data incorporation becomes more challenging if the EHRs on both sides of the exchange are different. Another way to accomplish data integration at the discrete level is through electronic messaging interfaces between organizations using standards such as HL7 v2.x. However, discrete data incorporation done using document-based exchange is more scalable without incurring the expense of traditional messaging interfaces. We did not find evidence of network exchange based “complete” data incorporation in literature where the incorporated data becomes fully integrated with the EHR and is stored just like other similar data natively captured and stored by the EHR. Setting for Health Information Exchange: This study was conducted at IH in Salt Lake City, UT. IH is currently participating in multiple HIE projects including the Care Connectivity Consortium (CCC) 19 at the national level, the Clinical Health Information Exchange (cHIE)20, the Utah Health Information Network (UHIN) 21 and a private exchange with a government trading partner at the state level. IH has developed an infrastructure that processes different types of standards-based documents and can be universally used for any of the various exchanges. The exchanges have had to face administrative, logistical, policy, and other delays. As a result, the exchanges are at different stages of maturity. We chose the exchange of death-related information between our government trading partner and IH as a convenience sample for this study. This type of data is available for all patients regardless of whether they opted-in or opted-out of HIE. Moreover, death is an important clinical outcome. Having high integrity death-related data is important for secondary uses such as patient outcome, resource utilization and cost analyses. Previous work: The authors (PRK, DKM and SNT) analyzed 2.2 million Hemoglobin A1C (HgbA1C) result records from IH’s enterprise data warehouse (EDW) for potential error patterns8. The HgbA1C results records are generated at IH, at point-of-care and external non-Intermountain laboratories. The authors then extracted requirements and designed architecture for a data adjudication infrastructure that integrates into the overall system architecture to detect data integrity issues and facilitate corrective actions 8. Figure 1 shows the architectural diagram of Intermountain’s HIE data adjudication system along with the document processing system used for HIE. External HIE data in the form of a summary of care document in the HL7 Consolidated Clinical Document Architecture format (CCDA) or an HL7 clinical document for reporting death information, inbound into IH, is initially processed to establish positive patient identity and positive consent. Data that pass these criteria are then transferred to the Inbound Message Orchestrator. This component is responsible for managing the workflow marked 2 through 6 in Figure 1. First, incoming data are filtered for new data at the document level to eliminate duplicative processing. Data is then extracted into internal data models and mapped from standard terminologies such as SNOMED and LOINC codes to Intermountain’s own terminology using Intermountain’s proprietary data dictionary. Data that has been extracted and mapped to local terminology is fed into the HIE Data Adjudicator where a series of JBoss® Drools rules are run on it. Context-independent rules, i.e. rules that can run solely on the external data are run first for maximizing efficiency. These include data completeness and temporal compatibility rules. Data completeness rules check whether the incoming data meets mandatory data requirements as determined by the analysts at IH. Temporal compatibility rules check whether data are current and relevant. If the data passes the
1895
context-independent rules, the rules processing continues to the context-dependent rules, i.e. rules which infer on external as well as internal data. These include data redundancy and data compatibility rules. Data redundancy rules compare incoming data to the data that is stored in IH’s longitudinal patient data repository called the “Clinical Data Repository” (CDR) and determine if the data are duplicative. The data compatibility rules are responsible for detecting inconsistencies and physiological incompatibilities between the external HIE data and data recorded at IH. The HIE Data Adjudicator result statuses include “accept”, “accept_as_new” and “reject”. The “accept_as_new” status results in new records in the database. The “accept” adjudication status indicates that the new data is an update to existing data. Data in either statuses of “accept_as_new” and “accept” are stored to Intermountain’s CDR. Data “rejected” by the HIE Data Adjudicator goes through various data integrity workflows. This architecture is fully implemented and is undergoing verification and validation at the time this manuscript was written.
2
1
Filter for new HIE data Data Extraction and Terminology Mapping
Adjudication results
HIE Data Adjudicator JBoss Drools® Data Completeness Rules 4
Temporal Compatibility Rules Data Redundancy Rules
Service Layer
Inbound Message Orchestrator
Positive Patient Identification and Consent Management
External HIE Data
3
Data Dictionary
Data Compatibility Rules 5
Discrete Data Storage
Clinical Data Repository
6
HIE Data Integrity Workflows
Figure 1. Architecture for HIE Document Processing and Data Adjudication Methods The data source (IH’s government trading partner) receives death certificate data from multiple sources within the state of Utah. IH maintains the death records for patients who pass away at any of its own facilities. We performed a retrospective, descriptive analysis to compare death information recorded at IH facilities to that received from the data source between January 1, 1996 and August 31, 2013. The patient identities were matched prior to this analysis using research-driven technologies that included a combination of algorithmic and manual processes. Algorithms used for determining positive patient identity included Jaro-Winkler and Levenshtein distance algorithms. We measured the inconsistencies, physiological incompatibilities, completeness and timeliness of the death information. We enhanced the document processing system developed at IH for HIE by implementing the data adjudication architecture. We then ran a sample of the data through the data adjudication infrastructure to validate the errors we found in our analysis. Data Inconsistencies/Physiological Incompatibility: We first compared the date/time of death in the data generated at IH to that in the data received from the data source. We then analyzed the data to determine if clinical data was recorded for deceased patients to understand the physiological incompatibility issues. We included billed encounters, vital signs, laboratory results, medication orders and problems in our analysis because these were most frequently used data types to record data for all patients at IH. We excluded data recorded for reasons that were
1896
considered appropriate for deceased patients such as encounters for transplants, equipment, and organ and cadaver donors. We also included an allowance of 24 hours after the earliest recorded death date to accommodate for systems that have different ways to record time of death. We did not include data that was marked canceled or error. In the cases where we found multiple recorded death dates, we considered the earliest recorded death date as the death date for our analysis. Finally, we checked for temporal compatibility by looking at date/time of death and determining if it is in the future. Data Completeness: The analysts at IH determined the criteria for completeness for death information. Death information is considered “complete” at IH if a valid patient identification and date/time of death is recorded. Timeliness: We calculated the latency of recording of death information as the difference between the actual death date/time and the date/time the information was processed by IH workflows. The Utah State Code (Title 26 Chapter 2 Section 13)22 stipulates that death certificates must be filed with the health department within five days of death. Our analysis adjusted for this allowance. We had to simulate the exchange between IH and the government trading partner because of unforeseen administrative and logistical delays. The simulation system translated raw death data from the data source into an HL7 clinical document for the template id: 2.16.840.1.113883.10.20.24.1. This document was then input to and processed by the Inbound Message Orchestrator as shown in Figure 1. Process steps 2 through 6 were identical between the simulated and model systems. We randomly selected 100 death records which were found to have inconsistencies (“error” records) and 100 death records for which we did not find data integrity issues (“good” records). The simulation system created HL7 clinical documents using this data. These records were run through the adjudication process for two iterations. The first iteration was performed to validate all the rules. We then added a dummy cause of death to half of the “good” records and performed a second iteration of the data through the system with the updated records as well as the records that were not changed. The second run was performed to validate the data redundancy rules (that detect duplicates) in addition to the other types of rules. We further created “empty” documents that did not have either date or time of death (“empty” records). We ran these through the system to validate the rejection logic in the data completeness rules. We also created documents with date/time of death in the future (“future” records) to validate the temporal compatibility rules. Results IH received a total of 198,453 death records from the data source between January 1, 1996 and August 31, 2013. A total of 47,175 patient deaths were recorded at IH facilities during the same time period. Table 1 describes data inconsistencies and physiological incompatibilities with the clinical data recorded for deceased patients at least 24 hours after their death date/time. We found a total of 15,721 inconsistent records for different types of clinical observations. A total of 4483 unique patients were affected by these inconsistencies across all types of clinical observations we analyzed. We did not find any temporal compatibility issues with the data. Table 1. Death data related inconsistencies and physiological incompatibilities Type of Clinical Observation
Number of Unique Patients
Number of Inconsistent Records
Comments
Billed encounters
401
819
Data was recorded under 110 different types of billed encounters.
Vital Signs
23
2820
12 types of vital signs were recorded (10 from inpatient and 2 from the outpatient setting).
Laboratory Results
1271
2614
Data was recorded for 704 different test codes.
Medications Orders
2889
9467
Out of all the inconsistent records, 9359 were from the inpatient setting, 96 were from the outpatient setting and 12 were from other settings.
Problems
1
1
1897
We found that all data met the data completeness criteria decided upon by our analysts. However, we found that some records were missing cause of death. Table 2 details the results of the data completeness analysis. Table 2. Data completeness profile for death related data Description
Number of records
Contributes to data completeness
Missing patient id
0
Yes
Missing date/time of death
0
Yes
Missing cause of death
852
No
Missing location of death
0
No
We currently manually process death data from our government trading partner. This is typically done at unpredictable intervals. We had timeliness data for only a subset of the death records (35962 or 18.1% of death records). Table 3 shows the results of the latency calculations for the death data from the data source. Table 3. Data timeliness profile for death related data Latency description
Latency value (days)
Minimum
76
Maximum
1051
Mean
549.6304
Median
544
The data adjudication system validation results were as expected. All the data integrity issues were successfully detected and the related records were “rejected”. The “good” records were “accepted” or “accepted_as_new” and stored to the test CDR. Table 4 gives the validation results. Table 4. Validation of the data adjudicator framework for death related data Iteration Number
Number of records
% “Accepted _as_new”
% “Accepted”
% “Rejected”
“Good”
100
100
0
0
n/a
“Error”
100
0
0
100
“Failed Data Compatibility Rules”
“Good” (with dummy cause of death)
50
0
100
0
n/a
“Good”
50
0
0
100
“Failed Rules”
“Error”
100
0
0
100
“Failed Data Compatibility Rules”
Iteration 3
“Empty”
2
0
0
100
“Failed Data Completeness Rules”
Iteration 4
“Future”
2
0
0
100
“Failed Temporal Compatibility Rules”
Iteration 1
Iteration 2
Type of records
1898
Reason for Rejection
Data
Redundancy
Discussion We uncovered inconsistencies and physiological incompatibilities for 4483 patients marked as deceased (2.259% of the HIE inbound data) even after we accounted for the types of data that are appropriate to be recorded on deceased patients (such as encounters for transplants, equipment, and organ and cadaver donors). We augmented our original time criteria for search of 24 hours after death date/time to 15 and 30 days after the death date/time. Although the number of inconsistencies and physiological incompatibilities decreased as we increased our time criteria for search, there appeared to be ongoing data issues that were not picked up by our data integrity workflows. There could be a variety of reasons, both benign and non-benign, for these data integrity issues and may be attributed to either side of the exchange (IH or the data source). An example of a benign reason could be delayed billing by any of the various systems (excluding equipment rentals) involved in a patient’s care. A non-benign reason could be the possibility of fraudulent activity, where the identity of a deceased patient is used to procure healthcare services. Further investigation is necessary uncover details about the reasons. Data adjudication applied to HIE data gives us the opportunity to detect such errors and brings us one step closer to facilitating corrective actions and thus improving the data integrity of one’s own EHR. Data completeness and temporal incompatibility were found to be a non-issue with the death information use case. This could be attributed to the mandatory requirements imposed by the Utah State Code at the source system. We think that the timeliness issues we found (mean latency of 549.63 days) are more of a process issue rather than a data issue. We believe the automation of the HIE exchange between IH and the government trading partner for death information will take us closer to solving the problem. Handling redundant data at the document as well as discrete data level is important to reduce inefficiencies in processing of HIE data. Without the ability to detect duplicate documents, systems may waste resources by reprocessing previously processed data that provides no new information. Likewise, the ability to compare discrete data and decide whether the incoming HIE data is new, duplicate or an update to already existing data can greatly improve process and database resource utilization. The data adjudication framework implemented in this study is highly generalizable and can be extended to handle other clinical data types by adding appropriate data extraction, terminology mapping and data integrity rules specific to the data type. The framework can also be applied to demographic or administrative data. It can be further enhanced to detect more data integrity issues such as clinical data trend incompatibilities. The framework also suggests an approach to integrate HIE data focusing on episodic care details into patient’s longitudinal data record. This approach is scalable and has the advantage of inter-organizational electronic messaging interfaces without the overhead associated with such interfaces. The approach described in this manuscript can be applied bi-directionally. In other words, it can be applied to improve clinical data integrity of the EHR sending the data as well as the one receiving the data. We envision that the data integrity critique (a report of inconsistencies, physiological and temporal incompatibilities and incompleteness of data) can be fed back to the sending system. We believe that the information gained and lessons learned from our study will inform our national and international collaborators working towards improved exchange profiles, standards-based data adjudication rules and HIE best practices. Limitations: In order to validate the data adjudication infrastructure we had to simulate the data exchange between IH and the data source for death information because the effort to establish a working exchange faced administrative and logistical delays. Although our verification and validation test cases adequately covered variations in data, we understand that the real working exchange may add some data or system conditions that we did not anticipate. However, should any issues arise, we feel that we will be able to rectify them efficiently because the implementation of the data adjudication framework allows for easy and efficient updates. Another limitation of the study is related to the accuracy of patient identity matching algorithms and processes. Any errors with patient matching can adversely affect the analysis and data adjudication results. However, we feel that because this study has used the patient identity matching workflows that have been implemented and incrementally improved at IH for over a decade, the potential negative effects of this are minimal. Finally, we acknowledge that clinically more complex data models exist as compared to the death data model we have used in this study. We feel that because of our modular approach, our framework can be augmented to use resources such as ontologies to handle hierarchical and complex data models.
1899
Future work: Our next step is to connect the data adjudication infrastructure demonstrated in this study to a working HIE between IH and the government trading partner. We believe that the experience will help us fine tune the data adjudication logic. We plan to expand the data adjudication to include problems, medications, allergies, and other data types, that are exchanged in the HIE domain. Additionally, we want to understand, in more detail, the reasons for data integrity issues we found. This may further inform our effort to enhance the data adjudication logic. Finally, we intend to research ways to translate the data adjudication rules into interoperable and shareable knowledge. Conclusion We demonstrated the capabilities of a data adjudication infrastructure integrated into Intermountain Healthcare’s HIE architecture. We showed that it can detect data integrity issues related to inconsistencies, physiological and temporal incompatibilities, and completeness. It also addressed new data integrity issues related to redundancy by detecting duplicates external and internal to the organization’s EHR. Data adjudication reduces the burden of manual review and resolution of data integrity issues by automatically connecting the data adjudication results to HIE data integrity workflows and facilitating corrective actions. Acknowledgements We would like to acknowledge Shannon Hood and Dallin Rogers for providing domain knowledge for the patient identification process at Intermountain Healthcare. We would also like to acknowledge Jeff Duncan for his expertise with death related information. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
Walker J, Pan E, Johnston D, Adler-Milstein J, Bates DW, Middleton B. The value of health care information exchange and interoperability. Health Aff (Millwood). 2005;Suppl Web Exclusives:W5-10-W5-8. Epub 2005/01/22. Fontaine P, Ross SE, Zink T, Schilling LM. Systematic review of health information exchange in primary care practices. Journal of the American Board of Family Medicine : JABFM. 2010;23(5):655-70. Epub 2010/09/09. Furukawa MF, Patel V, Charles D, Swain M, Mostashari F. Hospital electronic health information exchange grew substantially in 2008-12. Health Aff (Millwood). 2013;32(8):1346-54. Epub 2013/08/07. Adler-Milstein J, Bates DW, Jha AK. Operational health information exchanges show substantial growth, but long-term funding remains a concern. Health Aff (Millwood). 2013;32(8):1486-92. Epub 2013/07/11. Health Information Exchange (HIE). [URL]; Available from: http://www.healthit.gov/HIE. (Accessed on March 13, 2014) Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. Journal of general internal medicine. 2013;28(12):1663-6. Epub 2013/05/22. Kaelber DC, Bates DW. Health information exchange and patient safety. Journal of biomedical informatics. 2007;40(6 Suppl):S40-5. Epub 2007/10/24. Ranade-Kharkar P, Mann D, Thornton S. Data adjudication architecture for health information exchange (HIE): a case of adjudicating and storing hemoglobin a1c values. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2013. Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. Journal of the American Medical Informatics Association : JAMIA. 1997;4(5):342-55. Epub 1997/09/18. Berner ES, Kasiraman RK, Yu F, Ray MN, Houston TK. Data quality in the outpatient setting: impact on clinical decision support systems. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2005:41-5. Epub 2006/06/17. Hasan S, Padman R. Analyzing the effect of data quality on the accuracy of clinical decision support systems: a computer simulation approach. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2006:324-8. Epub 2007/01/24. McCormack JL, Ash JS. Clinician perspectives on the quality of patient data used for clinical decision support: a qualitative study. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2012;2012:1302-9. Epub 2013/01/11. Ancker JS, Shih S, Singh MP, Snyder A, Edwards A, Kaushal R. Root causes underlying challenges to secondary use of data. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2011;2011:57-62. Epub 2011/12/24.
1900
14. AHIMA. Ensuring data integrity in health information exchange. Available from: http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_049675.pdf. (Accessed on March 13, 2014) 15. Thornton SN WJ, Russ GE, Westburg LJ, Mann DK, Rasumssen DN. Sharing qualitative matching parameters among master patient indices. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2013. 16. Frisse ME, Tang L, Belsito A, Overhage JM. Development and use of a medication history service associated with a health information exchange: architecture and preliminary findings. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2010;2010:242-5. Epub 2011/02/25. 17. High-tech approach to medication reconciliation saves time, bolsters safety at hospital in northern Virginia. ED management : the monthly update on emergency department management. 2011;23(10):117-9. Epub 2011/10/07. 18. Cerner Corporation. Secure exchange of medical information: clinical exchange platform. Available from: https://www.cerner.com/uploadedFiles/fl03_789_10_v1_CXP_hr[1].pdf. (Accessed on March 13, 2014) 19. Care connectivity consortium. Available from: http://www.careconnectivity.org/. (Accessed on March 13, 2014) 20. My clinical health information exchange. Available from: http://mychie.org/. (Accessed on March 13, 2014) 21. Utah Health Information Network. Available from: http://www.uhin.org/. (Accessed on March 13, 2014) 22. Utah State Code (Title 26 Chapter 2 Section 13). http://le.utah.gov/code/TITLE26/htm/26_02_001300.htm. (Accessed on March 13, 2014) 23. Healtheway. Available from: http://healthewayinc.org/. (Accessed on July 22, 2014)
1901
Available
from:
1894
trustworthy unless they are based on data with high levels of integrity11. Clinician perception of the effectiveness of CDS is dependent on data quality across settings, organizations and types of EHRs12. Data integrity issues also affect secondary use of health care data for the purposes of quality improvement, public health, and research13. In this study, we address the following types of clinical data integrity: consistency, physiological compatibility, completeness and timeliness of data. Current literature on data integrity in the context of HIE focuses on the data integrity issues surrounding patient identification14, 15. In this manuscript, we focus on death information exchanged in an HIE, its consistency and physiological compatibility with other clinical data in the receiving organization’s EHR, and its completeness and timeliness. Data incorporation: While HIE is about moving data from one organization to the other, data incorporation is about integrating appropriate data into an organization’s own EHR and workflows. Literature on data incorporation in the realm of HIE is sparse. Frisse et al.16 describe data incorporation performed in the Memphis HIE which was done at the display level. In another system, a high-tech approach was applied to medication reconciliation where data from a pharmacy system was incorporated into an EHR17. Cerner, a commercial vendor of EHR and IH’s strategic partner, uses a document-based data model for integration of HIE data. The incoming documents with patient data are stored in a repository and the data in it is transformed for display, transfer, etc., as needed18. Data incorporation becomes more challenging if the EHRs on both sides of the exchange are different. Another way to accomplish data integration at the discrete level is through electronic messaging interfaces between organizations using standards such as HL7 v2.x. However, discrete data incorporation done using document-based exchange is more scalable without incurring the expense of traditional messaging interfaces. We did not find evidence of network exchange based “complete” data incorporation in literature where the incorporated data becomes fully integrated with the EHR and is stored just like other similar data natively captured and stored by the EHR. Setting for Health Information Exchange: This study was conducted at IH in Salt Lake City, UT. IH is currently participating in multiple HIE projects including the Care Connectivity Consortium (CCC) 19 at the national level, the Clinical Health Information Exchange (cHIE)20, the Utah Health Information Network (UHIN) 21 and a private exchange with a government trading partner at the state level. IH has developed an infrastructure that processes different types of standards-based documents and can be universally used for any of the various exchanges. The exchanges have had to face administrative, logistical, policy, and other delays. As a result, the exchanges are at different stages of maturity. We chose the exchange of death-related information between our government trading partner and IH as a convenience sample for this study. This type of data is available for all patients regardless of whether they opted-in or opted-out of HIE. Moreover, death is an important clinical outcome. Having high integrity death-related data is important for secondary uses such as patient outcome, resource utilization and cost analyses. Previous work: The authors (PRK, DKM and SNT) analyzed 2.2 million Hemoglobin A1C (HgbA1C) result records from IH’s enterprise data warehouse (EDW) for potential error patterns8. The HgbA1C results records are generated at IH, at point-of-care and external non-Intermountain laboratories. The authors then extracted requirements and designed architecture for a data adjudication infrastructure that integrates into the overall system architecture to detect data integrity issues and facilitate corrective actions 8. Figure 1 shows the architectural diagram of Intermountain’s HIE data adjudication system along with the document processing system used for HIE. External HIE data in the form of a summary of care document in the HL7 Consolidated Clinical Document Architecture format (CCDA) or an HL7 clinical document for reporting death information, inbound into IH, is initially processed to establish positive patient identity and positive consent. Data that pass these criteria are then transferred to the Inbound Message Orchestrator. This component is responsible for managing the workflow marked 2 through 6 in Figure 1. First, incoming data are filtered for new data at the document level to eliminate duplicative processing. Data is then extracted into internal data models and mapped from standard terminologies such as SNOMED and LOINC codes to Intermountain’s own terminology using Intermountain’s proprietary data dictionary. Data that has been extracted and mapped to local terminology is fed into the HIE Data Adjudicator where a series of JBoss® Drools rules are run on it. Context-independent rules, i.e. rules that can run solely on the external data are run first for maximizing efficiency. These include data completeness and temporal compatibility rules. Data completeness rules check whether the incoming data meets mandatory data requirements as determined by the analysts at IH. Temporal compatibility rules check whether data are current and relevant. If the data passes the
1895
context-independent rules, the rules processing continues to the context-dependent rules, i.e. rules which infer on external as well as internal data. These include data redundancy and data compatibility rules. Data redundancy rules compare incoming data to the data that is stored in IH’s longitudinal patient data repository called the “Clinical Data Repository” (CDR) and determine if the data are duplicative. The data compatibility rules are responsible for detecting inconsistencies and physiological incompatibilities between the external HIE data and data recorded at IH. The HIE Data Adjudicator result statuses include “accept”, “accept_as_new” and “reject”. The “accept_as_new” status results in new records in the database. The “accept” adjudication status indicates that the new data is an update to existing data. Data in either statuses of “accept_as_new” and “accept” are stored to Intermountain’s CDR. Data “rejected” by the HIE Data Adjudicator goes through various data integrity workflows. This architecture is fully implemented and is undergoing verification and validation at the time this manuscript was written.
2
1
Filter for new HIE data Data Extraction and Terminology Mapping
Adjudication results
HIE Data Adjudicator JBoss Drools® Data Completeness Rules 4
Temporal Compatibility Rules Data Redundancy Rules
Service Layer
Inbound Message Orchestrator
Positive Patient Identification and Consent Management
External HIE Data
3
Data Dictionary
Data Compatibility Rules 5
Discrete Data Storage
Clinical Data Repository
6
HIE Data Integrity Workflows
Figure 1. Architecture for HIE Document Processing and Data Adjudication Methods The data source (IH’s government trading partner) receives death certificate data from multiple sources within the state of Utah. IH maintains the death records for patients who pass away at any of its own facilities. We performed a retrospective, descriptive analysis to compare death information recorded at IH facilities to that received from the data source between January 1, 1996 and August 31, 2013. The patient identities were matched prior to this analysis using research-driven technologies that included a combination of algorithmic and manual processes. Algorithms used for determining positive patient identity included Jaro-Winkler and Levenshtein distance algorithms. We measured the inconsistencies, physiological incompatibilities, completeness and timeliness of the death information. We enhanced the document processing system developed at IH for HIE by implementing the data adjudication architecture. We then ran a sample of the data through the data adjudication infrastructure to validate the errors we found in our analysis. Data Inconsistencies/Physiological Incompatibility: We first compared the date/time of death in the data generated at IH to that in the data received from the data source. We then analyzed the data to determine if clinical data was recorded for deceased patients to understand the physiological incompatibility issues. We included billed encounters, vital signs, laboratory results, medication orders and problems in our analysis because these were most frequently used data types to record data for all patients at IH. We excluded data recorded for reasons that were
1896
considered appropriate for deceased patients such as encounters for transplants, equipment, and organ and cadaver donors. We also included an allowance of 24 hours after the earliest recorded death date to accommodate for systems that have different ways to record time of death. We did not include data that was marked canceled or error. In the cases where we found multiple recorded death dates, we considered the earliest recorded death date as the death date for our analysis. Finally, we checked for temporal compatibility by looking at date/time of death and determining if it is in the future. Data Completeness: The analysts at IH determined the criteria for completeness for death information. Death information is considered “complete” at IH if a valid patient identification and date/time of death is recorded. Timeliness: We calculated the latency of recording of death information as the difference between the actual death date/time and the date/time the information was processed by IH workflows. The Utah State Code (Title 26 Chapter 2 Section 13)22 stipulates that death certificates must be filed with the health department within five days of death. Our analysis adjusted for this allowance. We had to simulate the exchange between IH and the government trading partner because of unforeseen administrative and logistical delays. The simulation system translated raw death data from the data source into an HL7 clinical document for the template id: 2.16.840.1.113883.10.20.24.1. This document was then input to and processed by the Inbound Message Orchestrator as shown in Figure 1. Process steps 2 through 6 were identical between the simulated and model systems. We randomly selected 100 death records which were found to have inconsistencies (“error” records) and 100 death records for which we did not find data integrity issues (“good” records). The simulation system created HL7 clinical documents using this data. These records were run through the adjudication process for two iterations. The first iteration was performed to validate all the rules. We then added a dummy cause of death to half of the “good” records and performed a second iteration of the data through the system with the updated records as well as the records that were not changed. The second run was performed to validate the data redundancy rules (that detect duplicates) in addition to the other types of rules. We further created “empty” documents that did not have either date or time of death (“empty” records). We ran these through the system to validate the rejection logic in the data completeness rules. We also created documents with date/time of death in the future (“future” records) to validate the temporal compatibility rules. Results IH received a total of 198,453 death records from the data source between January 1, 1996 and August 31, 2013. A total of 47,175 patient deaths were recorded at IH facilities during the same time period. Table 1 describes data inconsistencies and physiological incompatibilities with the clinical data recorded for deceased patients at least 24 hours after their death date/time. We found a total of 15,721 inconsistent records for different types of clinical observations. A total of 4483 unique patients were affected by these inconsistencies across all types of clinical observations we analyzed. We did not find any temporal compatibility issues with the data. Table 1. Death data related inconsistencies and physiological incompatibilities Type of Clinical Observation
Number of Unique Patients
Number of Inconsistent Records
Comments
Billed encounters
401
819
Data was recorded under 110 different types of billed encounters.
Vital Signs
23
2820
12 types of vital signs were recorded (10 from inpatient and 2 from the outpatient setting).
Laboratory Results
1271
2614
Data was recorded for 704 different test codes.
Medications Orders
2889
9467
Out of all the inconsistent records, 9359 were from the inpatient setting, 96 were from the outpatient setting and 12 were from other settings.
Problems
1
1
1897
We found that all data met the data completeness criteria decided upon by our analysts. However, we found that some records were missing cause of death. Table 2 details the results of the data completeness analysis. Table 2. Data completeness profile for death related data Description
Number of records
Contributes to data completeness
Missing patient id
0
Yes
Missing date/time of death
0
Yes
Missing cause of death
852
No
Missing location of death
0
No
We currently manually process death data from our government trading partner. This is typically done at unpredictable intervals. We had timeliness data for only a subset of the death records (35962 or 18.1% of death records). Table 3 shows the results of the latency calculations for the death data from the data source. Table 3. Data timeliness profile for death related data Latency description
Latency value (days)
Minimum
76
Maximum
1051
Mean
549.6304
Median
544
The data adjudication system validation results were as expected. All the data integrity issues were successfully detected and the related records were “rejected”. The “good” records were “accepted” or “accepted_as_new” and stored to the test CDR. Table 4 gives the validation results. Table 4. Validation of the data adjudicator framework for death related data Iteration Number
Number of records
% “Accepted _as_new”
% “Accepted”
% “Rejected”
“Good”
100
100
0
0
n/a
“Error”
100
0
0
100
“Failed Data Compatibility Rules”
“Good” (with dummy cause of death)
50
0
100
0
n/a
“Good”
50
0
0
100
“Failed Rules”
“Error”
100
0
0
100
“Failed Data Compatibility Rules”
Iteration 3
“Empty”
2
0
0
100
“Failed Data Completeness Rules”
Iteration 4
“Future”
2
0
0
100
“Failed Temporal Compatibility Rules”
Iteration 1
Iteration 2
Type of records
1898
Reason for Rejection
Data
Redundancy
Discussion We uncovered inconsistencies and physiological incompatibilities for 4483 patients marked as deceased (2.259% of the HIE inbound data) even after we accounted for the types of data that are appropriate to be recorded on deceased patients (such as encounters for transplants, equipment, and organ and cadaver donors). We augmented our original time criteria for search of 24 hours after death date/time to 15 and 30 days after the death date/time. Although the number of inconsistencies and physiological incompatibilities decreased as we increased our time criteria for search, there appeared to be ongoing data issues that were not picked up by our data integrity workflows. There could be a variety of reasons, both benign and non-benign, for these data integrity issues and may be attributed to either side of the exchange (IH or the data source). An example of a benign reason could be delayed billing by any of the various systems (excluding equipment rentals) involved in a patient’s care. A non-benign reason could be the possibility of fraudulent activity, where the identity of a deceased patient is used to procure healthcare services. Further investigation is necessary uncover details about the reasons. Data adjudication applied to HIE data gives us the opportunity to detect such errors and brings us one step closer to facilitating corrective actions and thus improving the data integrity of one’s own EHR. Data completeness and temporal incompatibility were found to be a non-issue with the death information use case. This could be attributed to the mandatory requirements imposed by the Utah State Code at the source system. We think that the timeliness issues we found (mean latency of 549.63 days) are more of a process issue rather than a data issue. We believe the automation of the HIE exchange between IH and the government trading partner for death information will take us closer to solving the problem. Handling redundant data at the document as well as discrete data level is important to reduce inefficiencies in processing of HIE data. Without the ability to detect duplicate documents, systems may waste resources by reprocessing previously processed data that provides no new information. Likewise, the ability to compare discrete data and decide whether the incoming HIE data is new, duplicate or an update to already existing data can greatly improve process and database resource utilization. The data adjudication framework implemented in this study is highly generalizable and can be extended to handle other clinical data types by adding appropriate data extraction, terminology mapping and data integrity rules specific to the data type. The framework can also be applied to demographic or administrative data. It can be further enhanced to detect more data integrity issues such as clinical data trend incompatibilities. The framework also suggests an approach to integrate HIE data focusing on episodic care details into patient’s longitudinal data record. This approach is scalable and has the advantage of inter-organizational electronic messaging interfaces without the overhead associated with such interfaces. The approach described in this manuscript can be applied bi-directionally. In other words, it can be applied to improve clinical data integrity of the EHR sending the data as well as the one receiving the data. We envision that the data integrity critique (a report of inconsistencies, physiological and temporal incompatibilities and incompleteness of data) can be fed back to the sending system. We believe that the information gained and lessons learned from our study will inform our national and international collaborators working towards improved exchange profiles, standards-based data adjudication rules and HIE best practices. Limitations: In order to validate the data adjudication infrastructure we had to simulate the data exchange between IH and the data source for death information because the effort to establish a working exchange faced administrative and logistical delays. Although our verification and validation test cases adequately covered variations in data, we understand that the real working exchange may add some data or system conditions that we did not anticipate. However, should any issues arise, we feel that we will be able to rectify them efficiently because the implementation of the data adjudication framework allows for easy and efficient updates. Another limitation of the study is related to the accuracy of patient identity matching algorithms and processes. Any errors with patient matching can adversely affect the analysis and data adjudication results. However, we feel that because this study has used the patient identity matching workflows that have been implemented and incrementally improved at IH for over a decade, the potential negative effects of this are minimal. Finally, we acknowledge that clinically more complex data models exist as compared to the death data model we have used in this study. We feel that because of our modular approach, our framework can be augmented to use resources such as ontologies to handle hierarchical and complex data models.
1899
Future work: Our next step is to connect the data adjudication infrastructure demonstrated in this study to a working HIE between IH and the government trading partner. We believe that the experience will help us fine tune the data adjudication logic. We plan to expand the data adjudication to include problems, medications, allergies, and other data types, that are exchanged in the HIE domain. Additionally, we want to understand, in more detail, the reasons for data integrity issues we found. This may further inform our effort to enhance the data adjudication logic. Finally, we intend to research ways to translate the data adjudication rules into interoperable and shareable knowledge. Conclusion We demonstrated the capabilities of a data adjudication infrastructure integrated into Intermountain Healthcare’s HIE architecture. We showed that it can detect data integrity issues related to inconsistencies, physiological and temporal incompatibilities, and completeness. It also addressed new data integrity issues related to redundancy by detecting duplicates external and internal to the organization’s EHR. Data adjudication reduces the burden of manual review and resolution of data integrity issues by automatically connecting the data adjudication results to HIE data integrity workflows and facilitating corrective actions. Acknowledgements We would like to acknowledge Shannon Hood and Dallin Rogers for providing domain knowledge for the patient identification process at Intermountain Healthcare. We would also like to acknowledge Jeff Duncan for his expertise with death related information. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
Walker J, Pan E, Johnston D, Adler-Milstein J, Bates DW, Middleton B. The value of health care information exchange and interoperability. Health Aff (Millwood). 2005;Suppl Web Exclusives:W5-10-W5-8. Epub 2005/01/22. Fontaine P, Ross SE, Zink T, Schilling LM. Systematic review of health information exchange in primary care practices. Journal of the American Board of Family Medicine : JABFM. 2010;23(5):655-70. Epub 2010/09/09. Furukawa MF, Patel V, Charles D, Swain M, Mostashari F. Hospital electronic health information exchange grew substantially in 2008-12. Health Aff (Millwood). 2013;32(8):1346-54. Epub 2013/08/07. Adler-Milstein J, Bates DW, Jha AK. Operational health information exchanges show substantial growth, but long-term funding remains a concern. Health Aff (Millwood). 2013;32(8):1486-92. Epub 2013/07/11. Health Information Exchange (HIE). [URL]; Available from: http://www.healthit.gov/HIE. (Accessed on March 13, 2014) Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. Journal of general internal medicine. 2013;28(12):1663-6. Epub 2013/05/22. Kaelber DC, Bates DW. Health information exchange and patient safety. Journal of biomedical informatics. 2007;40(6 Suppl):S40-5. Epub 2007/10/24. Ranade-Kharkar P, Mann D, Thornton S. Data adjudication architecture for health information exchange (HIE): a case of adjudicating and storing hemoglobin a1c values. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2013. Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. Journal of the American Medical Informatics Association : JAMIA. 1997;4(5):342-55. Epub 1997/09/18. Berner ES, Kasiraman RK, Yu F, Ray MN, Houston TK. Data quality in the outpatient setting: impact on clinical decision support systems. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2005:41-5. Epub 2006/06/17. Hasan S, Padman R. Analyzing the effect of data quality on the accuracy of clinical decision support systems: a computer simulation approach. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2006:324-8. Epub 2007/01/24. McCormack JL, Ash JS. Clinician perspectives on the quality of patient data used for clinical decision support: a qualitative study. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2012;2012:1302-9. Epub 2013/01/11. Ancker JS, Shih S, Singh MP, Snyder A, Edwards A, Kaushal R. Root causes underlying challenges to secondary use of data. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2011;2011:57-62. Epub 2011/12/24.
1900
14. AHIMA. Ensuring data integrity in health information exchange. Available from: http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_049675.pdf. (Accessed on March 13, 2014) 15. Thornton SN WJ, Russ GE, Westburg LJ, Mann DK, Rasumssen DN. Sharing qualitative matching parameters among master patient indices. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2013. 16. Frisse ME, Tang L, Belsito A, Overhage JM. Development and use of a medication history service associated with a health information exchange: architecture and preliminary findings. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2010;2010:242-5. Epub 2011/02/25. 17. High-tech approach to medication reconciliation saves time, bolsters safety at hospital in northern Virginia. ED management : the monthly update on emergency department management. 2011;23(10):117-9. Epub 2011/10/07. 18. Cerner Corporation. Secure exchange of medical information: clinical exchange platform. Available from: https://www.cerner.com/uploadedFiles/fl03_789_10_v1_CXP_hr[1].pdf. (Accessed on March 13, 2014) 19. Care connectivity consortium. Available from: http://www.careconnectivity.org/. (Accessed on March 13, 2014) 20. My clinical health information exchange. Available from: http://mychie.org/. (Accessed on March 13, 2014) 21. Utah Health Information Network. Available from: http://www.uhin.org/. (Accessed on March 13, 2014) 22. Utah State Code (Title 26 Chapter 2 Section 13). http://le.utah.gov/code/TITLE26/htm/26_02_001300.htm. (Accessed on March 13, 2014) 23. Healtheway. Available from: http://healthewayinc.org/. (Accessed on July 22, 2014)
1901
Available
from:
