Enabling Cross-Platform Clinical Decision Support through Web-Based Decision Support in Commercial Electronic Health Record Systems: Proposal and Evaluation of Initial Prototype Implementations Mingyuan Zhang1, Ferdinand T. Velasco, MD2, R. Clayton Musser, MD, MS3, Kensaku Kawamoto, MD, PhD1 1 University of Utah, Salt Lake City, UT; 2Texas Health Resources, Arlington, TX; 3 Duke University Health System, Durham, NC Abstract: Enabling clinical decision support (CDS) across multiple electronic health record (EHR) systems has been a desired but largely unattained aim of clinical informatics, especially in commercial EHR systems. A potential opportunity for enabling such scalable CDS is to leverage vendor-supported, Web-based CDS development platforms along with vendor-supported application programming interfaces (APIs). Here, we propose a potential staged approach for enabling such scalable CDS, starting with the use of custom EHR APIs and moving towards standardized EHR APIs to facilitate interoperability. We analyzed three commercial EHR systems for their capabilities to support the proposed approach, and we implemented prototypes in all three systems. Based on these analyses and prototype implementations, we conclude that the approach proposed is feasible, already supported by several major commercial EHR vendors, and potentially capable of enabling cross-platform CDS at scale. Introduction Clinical decision support (CDS), defined as “providing patients and healthcare providers with patient specific information and knowledge at appropriate times to improve the patient care process,” has been touted as a promising approach for improving health care processes, reducing costs, and improving patient safety.1 Systematic reviews of the effects of CDS systems have demonstrated that they can enhance clinical performance. 2-8 In addition, CDS plays an important role in Meaningful Use stage two regulations.8 Despite significant promise, the availability of advanced CDS capabilities has generally been limited. 1 For example, in a systematic review of CDS systems by Chaudhry et al., fully 25% of CDS studies were found to come from just four institutions (Veterans Health Administration, Partners HealthCare, Regenstrief Institute, and Intermountain Healthcare).9 Moreover, according to a survey by the American Hospital Association (AHA), only 17% of hospitals had adopted electronic health records (EHRs) with advanced CDS by the end of 2012. 10 Although many factors limit the adoption of CDS, one main reason is that CDS capabilities are typically tightly coupled to specific modules within specific clinical information systems, making it difficult to share such machine-executable knowledge across multiple EHR platforms and care settings. A potential opportunity for enabling such scalable CDS is to leverage vendor-supported, Web-based development platforms along with vender-supported application programming interfaces (APIs). In this paper, we propose a potential staged approach for achieving scalable CDS in this manner, starting with the use of custom EHR APIs and moving towards standardized EHR APIs to facilitate interoperability. Background The earliest CDS systems began to appear around the 1960s. These early CDS systems, such as Warner’s congenital heart disease diagnosis system, 11 were mostly stand-alone systems that were separate from EHR systems. These systems were easy to share since they did not integrate with any clinical information systems. However, these systems required users to enter patient- and disease-specific information, making them error-prone and timeconsuming to use. Moreover, because users had to take the initiative to use a separate system outside of their routine clinical workflows, these CDS systems risked not being used. Due to the above limitations, later CDS systems, such as CDS systems built into Intermountain Healthcare’s HELP EHR system and the Veterans Health Administration’s VistA EHR system, began to integrate into clinical information systems. 12, 13 Such CDS systems reduced data entry requirements and were easier to integrate into clinical workflows. However, it was difficult to share these CDS systems across multiple information systems and care settings due to their tightly coupling with specific clinical information systems.

1558

To address the challenge of making CDS both (1) integrated with broader clinical information systems and (2) easy to share, efforts have been made to standardize machine-executable clinical knowledge representation. For example, the Arden Syntax14 was designed to represent clinical rules, and the Guideline Interchange Format (GLIF) 15 was designed to represent clinical guidelines in a shareable format. More recently, there has been a concerted effort by the Office of the National Coordinator for Health IT (ONC) to develop interoperability specifications for CDS through a public-private collaborative effort known as Health eDecisions. 16 As one of its deliverables, Health eDecisions has led to the development of a new Health Level 7 (HL7) draft standard for representing alerts, reminders, order sets, and documentation templates, known as the HL7 CDS Knowledge Artifact Implementation Guide.17 Despite the promise of this approach, no single knowledge representation standard has been adopted on a universal basis in commercial EHR systems.18 A complementary approach to enabling CDS across multiple systems is the use of a service-oriented architecture (SOA).19 In this type of approach, important functional capabilities of a clinical information system are encapsulated in independent, Web-accessible services, and they are used in a coordinated fashion to enable desired CDS capabilities. Much of the focus for this type of approach to CDS has been on what are known as CDS services, or Web services that analyze patient data and provide patient-specific assessments and conclusions.20 Examples of such CDS services that have been investigated include SEBASTIAN, 21 the CDS Consortium’s Enterprise Clinical Rules Service,22 and OpenCDS.23 Moreover, there is an HL7 standard for the interfaces of such services known as the HL7 Decision Support Service standard,24 and Health eDecisions is currently working on creating an interoperability specification for such services.25 In addition to CDS Web services, other services have been identified as being potentially important for enabling a SOA approach to CDS, such as retrieving data, placing orders, and communicating CDS recommendations to endusers.26 Moreover, it has been argued that for true interoperability and scaling, these services would need to have standard interfaces that are adopted across the industry.19,26 Despite holding significant promise, however, a standard, SOA-based approach to scaling CDS is still more a vision rather than a reality within the context of most commercial EHR systems. Thus, there is a strong need for developing a pragmatic strategy for moving from the current situation to the desired end result. Here, we propose an approach for bridging this gap through the opportunistic leveraging of existing EHR APIs and functionality, with an eventual goal of enabling more consistent interoperability using standard APIs. Methods Evaluation of Current Vendor Support for SOA-based Approach to CDS As the first step in investigating the ability to instantiate a standard, SOA-based approach to CDS within commercial EHR systems, we evaluated several commercial EHR systems with the goal of identifying how a SOA approach to CDS could be supported using existing EHR APIs. The investigations included commercial EHR systems from Cerner®, Epic, and McKesson. The approach consisted of a combination of reviewing available API documentation, exploring user community discussion forums, and communicating with local and vendor-based experts of the EHR systems. Proposal Development Based on the information gained from investigating the APIs available in the selected commercial EHR systems, we sought to develop an approach to enabling SOA-based CDS that built upon our prior proposals in this area.19,27 The goal in developing the proposal was to identify approaches that were supportable with current EHR functionality, while allowing for a natural progression to a standards-based approach in the future. Evaluation of Proposed Approach To evaluate the proposed approach to scaling CDS, we developed prototype CDS applications in all three commercial EHR systems. We then assessed the proposed approach in terms of feasibility, challenges, and lessons learned. Pseudo-Anonymization of Vendor Identities The intent of this study was to evaluate the capabilities of EHR systems in general to support a SOA approach to CDS, rather than to make a normative judgment on the relative capabilities of individual EHR systems. Thus, the identities of the EHR systems are purposely omitted from our discussions below.

1559

Results Vendor Support for SOA Approach to CDS In all products, APIs were available for enabling advanced CDS using a SOA approach. However, these APIs differed across systems, and the functionality of the supported APIs depended on the vendor. For example, one vendor supported using an API to place orders in the outpatient setting, but not in the inpatient setting. While not explicitly designed to support a SOA approach to CDS, we found that all three vendors supported a mechanism to develop custom Web resources (e.g., HTML, ASP, and JSP Web tools) that could then be integrated with the EHR systems. In one system, such Web resources could be embedded within the “tabs” of the EHR user interface, as well as invoked by the rules engine. In the second system, Web resources could be invoked from within the order entry system. And in the third system, Web resources could be invoked as a separate “tab” or window in the EHR system. These Web resources can be invoked directly by the end user or automatically by the EHR system’s native rules engine. In effect, by supporting the ability to implement CDS capabilities using traditional Web programming techniques, all three EHR systems support leveraging Web services as well as vendor APIs within their custom Web development platforms. Moreover, in each of the three products, the Web resource can be displayed within the EHR’s native user interface instead of a separate Web browser, so that the CDS tool can be coupled to a specific patient’s chart to provide patient-specific recommendations. For all three systems, there was a supported approach for entering orders directly through the Web resource. Of note, one of the EHR vendors has announced that the next major release of its software will directly support calling CDS services from within its native rules management platform. However, we were unable to identify support for this type of functionality in the versions of the EHR systems that we evaluated. Proposed Approach Our proposed approach for enabling standard, SOA-based CDS is as follows, and is described below.  Phase I. Utilize existing APIs and capabilities to enable custom SOA-based CDS for individual EHR platforms.  Phase II. Develop adapters for existing APIs so that available APIs can be accessed using standard interfaces.  Phase III. Standardize the APIs available in commercial EHR systems. Phase I. Use existing APIs and capabilities In the initial phase, we propose utilizing existing APIs and capabilities. For example, in the one system that has announced imminent support for calling CDS services from within the native rules engine, enabling a SOA approach to CDS will be easy to accomplish. As a similar example, we are aware of a major ambulatory EHR system (eClinicalWorks) which has announced that it will use CDS services for immunization CDS. 28 Despite progress on this front by some vendors, we found that many do not support this type of calling a CDS service directly. At the same time, we found that this type of an approach can be enabled through vendors’ support for Web-based custom programming environments. In Phase I (Figure 1), we take advantage of existing EHR APIs (Components A and B in the figure) as well as many EHR vendors’ support for Web-based custom programming platforms (Component E) that are invoked by userdriven or data-driven actions in the EHR. Because each EHR has different APIs, both in terms of capabilities and interfaces, the Web-based CDS tools must be customized to remain within the constraints of what is allowed within the EHRs. For example, a given EHR (EHR B in the figure) may allow orders to be placed using APIs, while another may not (EHR A in the figure). In this case, a Web-based CDS tool designed for EHR B would need to be customized to remove the direct order placement capability when adapted for EHR A. For example, the CDS tool could be adapted to simply recommend the order in narrative text when intended for deployment in EHR A. Of note, the ability to call EHR APIs, such as those for data retrieval and order placement, is consistent with the vision of a SOA-enabled approach to CDS. Moreover, the Web-based CDS tool can leverage services independent of the EHRs, such as CDS guidance services (Component C) that take in patient data as the input and provide back patient-specific evaluation results. Moreover, these Web-based CDS tools can leverage other services (Component D), such as terminology services, services for formatting data (e.g., to create trend graphs), and services for converting units of measure.

1560

C. CDS Guidance Service Inference Engine

Knowledge Modules

Trigger Mechanism (e.g. userdriven or data-driven events)

D. Other Services (for terminology management, data formatting, unit conversion, etc.)

EHR A

Data Addition and Update Evaluation results

Interact Patient data

Data Retrieval Invoke

A. EHR A APIs

Interact

E. EHR-Invoked, Webbased CDS Tool Interact Data Retrieval Interact with users

Invoke

Order Placement

B. EHR B APIs

Trigger Mechanism (e.g. userdriven or data-driven events)

EHR B

Figure 1. Overview of Phase I of proposed architecture. Letters correspond to descriptions in the text.

C. CDS Guidance Service Inference Engine

Evaluation results

Patient data

EHR A

Data Addition and Update Data Retrieval Order Placement

Invoke

Invoke

A. EHR A APIs

Data Addition and Update Data Retrieval

H. Standard APIs

Interact with users

F. EHR API Adapters

G. Standard APIs

E. EHR-Invoked, Webbased CDS Tool

Trigger Mechanism (e.g. userdriven or data-driven events)

H. Standard APIs

Knowledge Modules

D. Other Services (for terminology management, data formatting, unit conversion, etc.)

Order Placement

B. EHR B APIs

Trigger Mechanism (e.g. userdriven or data-driven events)

EHR B

Figure 2. Overview of Phases II and III of proposed architecture. Phase II is shown in solid lines, and Phase III is shown using dashed lines. Letters correspond to descriptions in the text.

1561

Phase II. Create Standard Adapters to EHR-specific APIs While the Phase I approach proposed above is capable of providing advanced CDS capabilities, and also potentially capable of being transported to other EHR environments, it is limited by the need to be customized to each EHR’s APIs. In the second proposed phase (Figure 2, solid lines), adapters (Component F) are created that enable the Web-based CDS tools (Component E) to interface with standard APIs. These adapters transform standard API calls into EHRspecific API calls, and then transform EHR-specific API results to standard API results (Component G). For example, data retrieval requests are made using standard APIs such as the HL7 Retrieve, Locate, and Update Service (RLUS) standard,29 which are then transformed into vendor-specific data retrieval API calls. Then, data are returned in a vendor-specific data format, which are then transformed into a standard format, such as an HL7 Continuity of Care Document (CCD).30 Phase III. Native Adoption of Standard APIs by EHRs A key challenge with the Phase II approach is that adapters must be created to enable the CDS tools to leverage standard APIs. Moreover, another challenge is that an EHR may not support a critical API functionality altogether (e.g., for placing orders or for entering data into the EHR). Thus, in the third proposed phase (Figure 2, dashed lines), EHRs support a core set of commonly agreed upon service capabilities. Moreover, they support the use of standard APIs (Component H) to access these capabilities. In this case, adapters are no longer needed, and there can be consistency in the services that can be expected to be supported on the part of EHRs. Prototype Development In order to evaluate the feasibility and challenges of the proposed approach, we developed prototype systems in accordance with our proposed Phase I architecture in three EHR systems (Epic, Cerner, and McKesson). One of these systems is currently in production use. Here, we describe each of these systems and their clinical contexts of use. Prototype 1: Risk assessment tool for heparin-induced thrombocytopenia (HIT)

Figure 3. Screenshot for CDS tool for assessing risk for, and managing, heparin-induced thrombocytopenia (HIT) In this use case, a CDS tool was developed at the University of Utah for assessing the risk for, and appropriately managing, heparin-induced thrombocytopenia (HIT). HIT is one of the most common adverse drug reactions caused by heparin. Although there are laboratory tests to confirm the presence of HIT, limitations exist. Since the results cannot be available immediately, the decision to continue, stop, or start a non-heparin anticoagulant can be

1562

problematic. Simply stopping heparin for a patient with HIT can cause thromboses, while starting non-heparin anticoagulants such as argatroban on a patient without HIT can be expensive and may potentially lead to major bleeding.31 Thus, it is important to have a clinical risk assessment tool to predict the presence of HIT. In this prototype, the CDS process begins when the clinician tries to place a HIT antibody test order to confirm whether the patient has HIT. When the clinician adds the order to the scratchpad, an HTML Web page is invoked within the EHR system’s native user interface. This Web page interacts with the clinician to facilitate assessing the patient’s risk of HIT. Based on the user’s input, the Web page makes recommendations to the user. Recommended actions can be placed into the computerized provider order entry (CPOE) system directly through the Web page. On placing the orders, the clinician’s assessment is charted within the EHR for future reference. Figure 3 provides a screenshot of this system, which is in the process of being deployed in the production environment. Architecturally, this system is developed according to our Phase I proposal. The CDS Web tool interacts directly with the order placement API of the EHR system. The tool also calls an API in the EHR system that allows specific rules to be triggered. Using this approach, rules to add specific data to the record are used to document relevant data to the relevant inpatient flowsheet. While not utilized in this particular tool, this EHR system provides a variety of other potentially relevant APIs, such as for data retrieval. Furthermore, while not needed for this prototype (and therefore not deployed), we did validate that this Web tool can call an external CDS guidance service. Prototype 2: Insulin infusion calculator Blood glucose management through the titration of continuous parenteral insulin is both critically important and challenging in the critical care setting.32 Jacobi et al. outlined several recommendations to ensure the safe and effective use of insulin.33 Among these recommendations is the use of a reliable insulin infusion protocol, which guides the clinical staff in choosing an appropriate starting dose and in making dosage adjustments in response to shifting blood glucose levels and patient parameters. To support the clinical workflow, Texas Health Resources developed a Web-based insulin infusion calculator within its EHR system. The tool has been in production use in a clinical setting for two years.

Figure 4. Screenshot for insulin infusion calculator

1563

Within the EHR system, access to the insulin calculator is available to all members of the care team. The primary users of the tool are clinical pharmacists, who determine the initial rate of the insulin infusion, and nursing staff, who make adjustments to the infusion based on changes in the patient’s glucose level. To invoke the tool, the user clicks a customized button embedded in the EHR system, which launches an ASP Web page as a pop-up window within EHR native interface. In addition, logic has been programmed into the tool to prevent the user from accessing the calculator if the appropriate conditions are not met (e.g., if the patient does not have an active order for the protocol). This CDS tool incorporates available patient data, and upon collecting further user input, provides a recommendation for insulin titration (Figure 4). As an additional safety feature, the algorithm of the insulin protocol is available as a PDF reference from within the tool, so that the user can double-check the calculations performed by the system. Architecturally, this system also uses the approach proposed for Phase I. The CDS tool is invoked from within the EHR system. Relevant patient data (e.g., blood glucose levels and body mass index) are obtained from the EHR system and provided to the Web tool as URL parameters. To accomplish this, the EHR system provides a variety of APIs for data retrieval. While the calculator is pre-populated with data from the EHR, the user has the option to override these values. Since the tool is used for reference only, it does not ‘write’ data back to the clinical data repository for the patient. However, another Web-based CDS tool developed and implemented at Texas Health Resources can send data back into the EHR system via an HL7 interface. Prototype 3: CDS tool for genetically-guided warfarin dosing In a third system, a prototype for genetically-guided warfarin dosing was developed at Duke University Health System. Warfarin is an anticoagulant drug that is widely prescribed for preventing and treating thrombosis and reducing the risk of embolism. However, dosing of warfarin is complicated, since a patient’s response to warfarin is influenced by many factors including genotype, drugs, diet, and various co-morbidities. Inappropriate dosing may increase the risk of adverse events such as hemorrhage.34 A Web site known as www.WarfarinDosing.org has implemented well-established algorithms for genetically-guided warfarin dosing. In order to take advantage of this resource within the context of usual clinical workflows, a Webbased CDS tool was developed that can be invoked within a third EHR system and then call a CDS guidance service provided by WarfarinDosing.org.

Figure 5. Screenshot of prototype CDS tool for genetically-guided warfarin dosing

1564

In this prototype system, the CDS tool is invoked when the user selects a “warfarin advisor,” which opens a Webbased tool within the native CPOE user interface. Relevant patient data are collected from users, and then WarfarinDosing.org is invoked to provide genetically-guided dosing recommendations, which are within easy view as the warfarin order are placed through this Web tool (Figure 5). Architecturally, this CDS tool was implemented using the proposed Phase I approach. The tool was implemented as a Web form within a JavaServer Page (JSP). This tool was invoked from within the CPOE system and then displayed within its custom Web browser. While not implemented for this prototype, it was verified that patient data could be sent to the CDS tool as URL parameters. Data, including the orders requested by the user, could then be sent back to the CPOE system as a form submission, the results of which could then be used by the system to place orders and save data. The dosing capabilities of WarfarinDosing.org were invoked through a Web service interface provided by the developers of the Web site, and the Web tool was designed so that the dosing guidance would update in real-time as relevant patient parameters were changed in the tool. Evaluation of Proposed Approach Based on our experiences with implementing the proposed Phase I approach in three commercial EHR systems, we found that the approach is in fact feasible in all three of the systems we evaluated. Indeed, all three EHR systems provided various APIs for accessing needed functionality, such as retrieval of patient data. Moreover, all three EHR systems provided mechanisms for invoking Web-based CDS tools, although triggering mechanisms varied. There were several important challenges and limitations, however. First, in all three cases, the steps that needed to be taken to enable the prototypes described here were not well documented. Enabling these approaches required a concerted effort in each case, including making inquiries in user discussion forums, requesting further information from vendor support personnel, carefully scrutinizing available documentation, and trial and error. Moreover, there was significant heterogeneity among the EHR systems in how they supported this approach to CDS, including variability in browser functionality, available API capabilities, and API interfaces. Clearly, if standard API adapters were available (Phase II), or if EHR systems supported a common set of standard APIs (Phase III), it would be much easier to implement the proposed approach to enabling cross-platform CDS. Discussion Study Summary Based on prior proposals for enabling CDS at scale through SOA-based approaches, and based on an evaluation of available APIs and capabilities in three commercial EHR systems, we proposed a three-phased approach to enabling service-oriented CDS across multiple EHR platforms. Through the implementation of prototype systems pursuant to this approach, we determined that the proposed approach is in fact feasible. However, we identified several important challenges, including limited documentation to support the approach, heterogeneous EHR APIs, and differing mechanisms for invoking Web-based CDS tools. Strengths and Limitations One important strength of this study is that we evaluated the proposed approach to scaling CDS in three commercial EHR systems. A second strength is that we implemented actual CDS systems to evaluate the approach, including one that is currently in production use, and one that is about to be deployed for production use. A final strength of this study is that our proposed approach is both (i) feasible to implement using currently available technologies and (ii) compatible with a step-wise progression to a standards-based approach to enabling CDS at scale. One limitation of this study is that we did not evaluate more commercial EHR systems. Therefore, we believe that further investigation of this approach in other EHR systems is warranted. Second, we did not implement any prototypes evaluating our proposed Phase II approach. While we believe implementing such an approach is both logical and feasible, further research is clearly needed in this area as well. Third, it is possible that an EHR supported a functionality that we were not able to identify, such as additional useful APIs. Finally, it is possible that a functionality that we could not identify in an evaluated EHR system is available in a newer version of the system. Need for Standards and Standards Adoption In this study, we found that EHRs generally utilize their own APIs rather than standard APIs. In a previous study,26 several standards for services desired for CDS were identified, including the RLUS standard, 29 the HL7 Decision

1565

Support Services standard,24 and HL7 version 2 and 3 information model standards. 35,36 However, this analysis also identified several desired services for which appropriate standards are not available, such as for order placement. Clearly, standardization efforts in this area will be critical. While the availability of standards is clearly a pre-requisite for a standards-based SOA approach to CDS, perhaps more important is vendor adoption of these standards. For example, for several of the capabilities supported by vendors using custom APIs, standards are available today, as noted above. Future Directions There is a clear need for more research in this space, for example for evaluating the proposed Phase I approach in other EHR systems, and also for evaluating the proposed Phase II approach. Critical to the Phase II approach is developing a business case for developing adapters to bridge standard APIs with EHR-specific APIs. Given the size of the task, the creation of such adapters will require significant resources, whether pooled from collaborating entities or invested by integration vendors who could spread the costs of developing such capabilities across multiple clients and thereby achieve economies of scale. Looking forward to Phase III, achieving a common set of standard APIs across EHR vendors will be difficult, but it is critical to achieving the vision of truly scalable, crossplatform, service-oriented CDS. One potential avenue to achieving Phase III may include alignment with Meaningful Use regulations. Another potential approach may be to leverage vendor-led efforts to enable enhanced interoperability, such as the recently announced CommonWell Health Alliance. 37 Conclusions Based on these analyses and prototype implementations presented in this manuscript, we conclude that the approach proposed is feasible, already supported by several major commercial EHR vendors, and potentially a promising approach for scaling CDS. Further efforts are needed to further evaluate the proposed approach and to make the approach more standards-based and scalable. Acknowledgements This study was supported by the University of Utah Department of Biomedical Informatics (KK), University of Utah Health Care Information Technology Services (KK and MZ), grant K01HG004645 from the U.S. National Human Genome Research Institute (KK), and Texas Health Resources (FTV). Competing interests KK is currently or recently served as a consultant on CDS to the Office of the National Coordinator for Health IT, ARUP Laboratories, Inflexxion, Inc., Intelligent Automation, Inc., Partners HealthCare, and the RAND Corporation. KK receives royalties for a Duke University-owned CDS technology for infectious disease management known as CustomID that he helped develop. KK was formerly a consultant for Religent, Inc. and a co-owner and consultant for Clinica Software, Inc., both of which provide commercial CDS services, including through use of a CDS technology known as SEBASTIAN that KK developed. KK no longer has a financial relationship with either Religent or Clinica Software. References 1. 2. 3. 4. 5. 6. 7.

Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc. 2007;14(2):141-5. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339-46. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120(2):135-42. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765-8. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223-38. Randell R, Mitchell N, Dowding D, Cullum N, Thompson C. Effects of computerized decision support systems on nursing performance and patient outcomes: a systematic review. J Health Serv Res Policy. 2007;12(4):242-9. Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc. 2011;18(3):327-34.

1566

8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32.

33. 34. 35. 36. 37.

Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evidence Report No. 203. AHRQ Publication No. 12-E001-EF. Rockville, MD: Agency for Healthcare Research and Quality. April 2012. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-52. Charles D, King J, Patel V, Furukawa M. ONC Data Brief No. 9 March 2013: Adoption of Electronic Health Record Systems among U.S. Non-federal Acute Care Hospitals: 2008-2012 [cited 2012 Mar 07]. Available from: http://www.healthit.gov/sites/default/files/oncdatabrief9final.pdf. Warner HR, Toronto AF, Veasey LG, Stephenson R. A mathematical approach to medical diagnosis. Application to congenital heart disease. JAMA. 1961;177:177-83. Pryor TA, Gardner RM, Clayton PD, Warner HR. The HELP system. Journal of medical systems. 1983;7(2):87-102. Brown SH, Lincoln MJ, Groen PJ, Kolodner RM. VistA—US Department of Veterans Affairs national-scale HIS. International journal of medical informatics. 2003;69(2):135-56. Pryor TA, Hripcsak G. The Arden syntax for medical logic modules. Int J Clin Monit Comput. 1993;10(4):215-24. Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, et al. GLIF3: a representation format for sharable computerinterpretable clinical practice guidelines. J Biomed Inform. 2004;37(3):147-61. Health eDecisions [cited 2013 Mar 04]. Available from: http://wiki.siframework.org/Health+eDecisions+Homepage. Health Level 7 CDS Knowledge Sharing Implementation Guide [cited 2013 Mar 04]. Available from: http://www.hl7.org/special/Committees/projman/searchableProjectIndex.cfm?action=edit&ProjectNumber=931. Wright A, Sittig DF. A framework and model for evaluating clinical decision support architectures. J Biomed Inform. 2008;41(6):982-90. Kawamoto K, Lobach DF. Proposal for fulfilling strategic objectives of the U.S. Roadmap for national action on clinical decision support through a service-oriented architecture leveraging HL7 services. J Am Med Inform Assoc. 2007;14(2):14655. Kawamoto K, Del Fiol G, Orton C, Lobach DF. System-agnostic clinical decision support services: benefits and challenges for scalable decision support. Open Med Inform J. 2010;4:245-54. Kawamoto K, Lobach DF. Design, implementation, use, and preliminary evaluation of SEBASTIAN, a standards-based Web service for clinical decision support. AMIA Annu Symp Proc. 2005:380-4. Paterno MD, Goldberg HS, Simonaitis L, Dixon BE, Wright A, Rocha BH, et al. Using a service oriented architecture approach to clinical decision support: performance results from two CDS Consortium demonstrations. AMIA Annu Symp Proc. 2012;2012:690-8. OpenCDS [cited 2013 March 04]. Available from: http://www.opencds.org/. HL7 Decision Support Service (DSS), release 1 [cited 2013 Mar 04]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=12. Health eDecisions CDS Guidance Service [cited 2013 Mar 04]. Available from: http://wiki.siframework.org/UC+2++CDS+Guidance+Service. Kawamoto K, Jacobs J, Welch BM, Huser V, Paterno MD, Del Fiol G, et al. Clinical Information System Services and Capabilities Desired for Scalable, Standards-Based, Service-oriented Decision Support: Consensus Assessment of the Health Level 7 Clinical Decision Support Work Group. AMIA Annu Symp Proc. 2012;2012:446-55. Kawamoto K, Lobach DF, Willard HF, Ginsburg GS. A national clinical decision support infrastructure to enable the widespread and consistent practice of genomic and personalized medicine. BMC Med Inform Decis Mak. 2009;9:17. Immunization Calculation Engine [cited 2013 Mar 05]. Available from: http://www.hln.com/ice/. Health Level 7 Retrieve, Locate, Update Service [cited 2013 Mar 07]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=89. Health Level 7 Continuity of Care Document (CCD) [cited 2013 Mar 07]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=6. Lo GK, Juhl D, Warkentin TE, Sigouin CS, Eichler P, Greinacher A. Evaluation of pretest clinical score (4 T's) for the diagnosis of heparin-induced thrombocytopenia in two clinical settings. J Thromb Haemost. 2006;4(4):759-65. Workbook for improvement: improving glycemic control, preventing hypoglycemia, and optimizing care of the inpatient with hyperglycemia and diabetes. [cited 2013 Mar 12]. Available from: http://www.hospitalmedicine.org/AM/Template.cfm?Section=Home&Template=/CM/ContentDisplay.cfm&ContentID=118 78. Jacobi J, Bircher N, Krinsley J, Agus M, Braithwaite SS, Deutschman C, et al. Guidelines for the use of an insulin infusion for the management of hyperglycemia in critically ill patients. Critical care medicine. 2012;40(12):3251-76. Klein TE, Altman RB, Eriksson N, Gage BF, Kimmel SE, Lee MT, et al. Estimation of the warfarin dose with clinical and pharmacogenetic data. N Engl J Med. 2009;360(8):753-64. Health Level 7 messaging standard version 2.7 [cited 2013 Mar 07]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=146. Health Level 7 version 3 product suite [cited 2013 Mar 07]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=186. Six HIT heavy-hitters announce interoperability organization [cited 2013 Mar 07]. Available from: http://www.healthcareitnews.com/news/six-hit-heavy-hitters-announce-interoperability-organization.

1567