Journal of the American Medical Informatics Association Advance Access published November 13, 2014 Perspective
Re-examining health IT policy: what will it take to derive value from our investment? Loren Riskin,1 Ross Koppel,2 Daniel Riskin3 1
Department of Anesthesiology, Stanford Hospital & Clinics, Stanford, California, USA 2 Department of Sociology, University of Pennsylvania, Philadelphia, Pennsylvania, USA 3 Department of Surgery, Stanford University, Stanford, California, USA Correspondence to Dr Loren Riskin, Department of Anesthesiology, Stanford Hospital & Clinics, 300 Pasteur Drive, H3589, Stanford, CA 94305, USA; [email protected] Received 18 June 2014 Revised 22 August 2014 Accepted 21 September 2014
ABSTRACT Despite substantial investments in health information technology (HIT), the nation’s goals of reducing cost and improving outcomes through HIT remain elusive. This period of transition, with new Office of National Coordinator for HIT leadership, upcoming Meaningful Use Stage III definitions, and increasing congressional oversight, is opportune to consider needed course corrections in HIT strategy. This article describes current problems and recommended changes in HIT policy, including approaches to usability, interoperability, and quality measurement. Recommendations refrain from interim measures, such as electronic health record adoption rates, and instead focus on measurable national value to benefit the economy, to reduce healthcare costs, and to improve clinical efficiency and care quality.
INTRODUCTION
To cite: Riskin L, Koppel R, Riskin D. J Am Med Inform Assoc Published Online First: [please include Day Month Year] doi:10.1136/amiajnl2014-003065
US health care suffers from its inability to provide high quality and consistent care despite massive and increasing spending.1 The value deficit is substantial, with the Congressional Budget Office and Institute of Medicine each estimating that 30–40% of US spending is ‘waste’.2 3 Similarly, our national health information technology (HIT) policy has been accused of costing trillions, burdening clinicians and adding risks without a clear return of increased efficiency, safety or quality.4–7 Electronic health records (EHRs) were ubiquitously promoted to deliver value by decreasing effort and errors.5 6 8 Extensive EHR software was purchased and implemented at significant cost, with 75% of hospitals and 44% of primary care providers now using at least basic electronic records.9 However, increasing evidence suggests EHR implementations alone have not delivered anticipated cost reductions or outcome improvements.4 10–13 Also, experience with use of data through enterprise data warehouses (EDW) and analytics is limited. As of 2014, the benefit of HIT remains controversial.9 14 This discussion first explores ‘value’ in the context of our major national IT investment, and then proposes three recommendations to improve HIT’s return on investment. Recommendations focus on known areas of weakness and confusion: HIT usability, interoperability, and quality. For each, we offer a summary of the current approach, reasons the approach has and will remain inadequate, and recommendations for policy course corrections. 2014 is a time when course corrections are possible. Office of National Coordinator (ONC) leadership transition, specification of Meaningful Use
Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
(MU) Stage III, congressional engagement, and proposed shortened regulatory cycles15 all reflect an opportunity. We suggest course corrections are needed now that tie our substantial HIT investment to enhanced national value to benefit the economy, to reduce healthcare costs, and to improve clinical efficiency and care quality.
UNDERSTANDING VALUE IN THE CONTEXT OF A NATIONAL INVESTMENT Before proposing policy recommendations, there should be alignment between goals and our national investment. Early in the last decade, the USA responded to an alarming report by the Institute of Medicine that medical errors represented a serious public threat.8 National efforts were undertaken, including creation of the ONC for Health IT.16 The healthcare industry rallied in response, highlighting safety and error reduction as critical needs requiring technology solutions such as the EHR. Fifteen years later, the landscape has changed. Payment models are shifting toward risk-sharing and value-based healthcare. Measurement of risk and quality are critical to national health reform. Studies and pilots over the last decade reveal that cost reduction and quality improvement require far more than safety and error reduction. The most successful quality improvement efforts have incorporated clinical data to: (1) benchmark outcomes to detect inconsistency and to align care with best practices,17 18 (2) identify high-risk patients and augment resources before expensive catastrophes,19 20 and (3) support population-based analytics and preventive care.21 22 As experts increasingly used and benefited from clinical data, these efforts have become known as clinical analytics. While clinical analytics can have various meanings, we define it here as those systems beyond the health record that use data to drive operational, financial, or clinical improvement. Health information technology goals have expanded from safety, error reduction, and core measures to include clinical analytics and new workflow throughout the care cycle. With national focus on health reform, value is increasingly defined as improved outcomes and reduced costs.
RECOMMENDATION: REQUIRE CLINICAL SOFTWARE USABILITY OVER FUNCTIONALITY Usability of HIT is the learnability, efficiency and satisfaction with which users achieve a specific set of tasks in a particular environment.23 Usability is required for HIT to be effective and is foundational to deriving benefit from software.24 25 HIT usability is widely acknowledged as poor, generating 1
Perspective frustration, medical errors, and inefficiencies.9 15 26–28 Failure to integrate smoothly with clinical workflow severely exacerbates the lack of usability and, at times, lack of usefulness of the technology.29–32 Just as a perceived lack of usefulness hinders HIT adoption,33 34 a lack of EHR usability undermines patient care, accurate documentation and functional clinical decision support (CDS). A Healthcare Information and Management Systems Society (HIMSS) task force states that poor usability is “possibly the most important factor hindering widespread adoption of EMRs”.23 Documentation and order entry time may increase by up to 300% after EHR implementation,35 and medication errors may increase with computerized physician order entry.30 36 CDS designed to help clinicians adhere to standards of care is supposed to improve quality of care,37–39 but functionality proves cumbersome and often ineffective. With extremely high documented override rates,40 41 poor CDS usability undermines its utility and one of the underpinnings of all HIT. Usability for analytics is even less often discussed than usability of the EHR itself, but is at least as important to achieve care consistency and risk reduction.42 43 Because so much workflow occurs outside of the EHR, EDWs—holding data from one or many institutions—combined with analytic systems, offer additional and largely untapped potential for care improvement with little focus on usable systems that offer an effective workflow to improve clinical, operational, and financial efficiency.
Recommendation for usability Address poorly designed and user-unfriendly software, proliferated through rushed national mandates that limit HIT’s benefits. MU Stage III should reduce requirements on EHR functionality and instead focus on defining and requiring effective usability for both EHR and analytics systems.
RECOMMENDATION: MANDATE ROBUST INTEROPERABILITY To use data, a system must have access to the data. Yet, our national health infrastructure created islands, not highways. Today, data are held in many silos, including laboratory systems, pharmacy, revenue cycle software, EHRs, EDWs, and other independent systems. While interoperability is widely acknowledged as essential,31 there is increasing understanding that it is seldom achieved44 and failure will undermine national goals.45–47 Why is this information not shared between clinical systems that received heavy national subsidization and the systems that offer analytics capabilities? Despite successful sharing in other industries and several data standards in healthcare, the general argument is that data sharing within known formats is too hard. An immediate challenge is assuring information is available for transitions in care. Planned policy changes require sharing of patient summaries within a consolidated clinical document architecture format to address this need.48 But, while sharing summaries addresses a decade-old goal to reduce errors when a patient moves from one healthcare organization to another, it fails to actually improve the practice of care. A patient summary is simply not adequate to perform the clinical analytics that can reduce costs and improve outcomes. Consider a patient summary describing a patient as hypertensive, diabetic, and on two medications. Now consider the full clinical record, which incorporates information on social situation, course of care, medication compliance, recent complications, vital signs, and laboratory values. A summary alone simply does not have enough information to measure quality, understand risk, empower population health management, or support 2
benchmarking. The nation subsidized and gave incentive to healthcare organizations to implement EHRs and to doctors to capture the data, but we have allowed lock-in of the precious data in which we invested. Locking in data stifles innovation in healthcare and severely reduces our chance of meeting national goals.
Recommendation for interoperability Current policy requires sharing patient summaries, but not full clinical content to empower analytics. MU Stage III must require sharing full clinical data in a common format among EHRs, EDWs, and analytics systems. Only with this change can we empower clinical analytics and quality workflow to deliver on HIT’s promised value.
RECOMMENDATION: IMPLEMENT A REFINED APPROACH TO QUALITY Though the HITECH Act of 2009 authorized billions of dollars to support a pathway to meaningful use of data,49 the effects of this national approach to quality measurement remain unclear.50 51 While there is broad agreement that quality is important, national discussions seldom focus on what is actually measured and how those measurements are used. Early efforts in quality measurement, such as those from the National Surgical Quality Improvement Program, were focused exclusively on aligning care with national guidelines proven to improve outcome. This was effective in improving care and reducing costs.52 53 As clinical quality measure (CQM) efforts were expanded, the program changed. Quality measures became centrally defined and standards bodies came to serve two masters: national care guidelines, and ‘feasibility.’ Industry and healthcare organizations found newly imposed measures difficult, and the government worked to be responsive. The change in practice was to no longer solely align measures with what improved care, but to also focus on feasibility—how easy it was to input and extract the data.54 It is not possible to serve two masters equally. Either quality measures align with what is proven to advance national goals or they align with what is easiest for vendors and busy clinicians. Thus, new measures focus heavily on discrete data elements— for example, hemoglobin A1c values rather than poorly controlled diabetes, or process measures such as smoking cessation counseling rather than tobacco use. An example highlights the challenge: Although lowering smoking rates is tied to reduced costs, the government has chosen the ‘feasible’ CQM of smoking cessation counseling. A healthcare organization (HCO) wishing to address this measure has added a line to every discharge summary recommending the patient quit smoking. To avoid missing patients, this is added to all discharge summaries of smokers and non-smokers, buried within pages of text. The result is excellent calculated quality scores, but little or no proven influence on outcomes or cost. The risk, well described by Goodhart’s law, is that the measures become a target in themselves rather than a road toward national goals.55 When the targets (measures) are so distant from the goals (reduced costs and improved outcomes), is it really a surprise that goals are not met? In fact, the problem is even worse. Quality measures were initially piloted within motivated institutions, where clinicians worked hard to assign accurately the correct patients to the correct measures. Now, CQMs are being reported by less motivated hospitals and staff, struggling under constrained resources to meet minimal national reporting requirements. If there were a requirement that measurement needed to be accurate, this Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
Perspective Table 1
Relationships among usability, interoperability and quality with current problems, measures, policy changes and rationales
Issue
Current problem
Policy change
Reason
Usability
Rushed HIT adoption with poor usability; vendors claim limited time to work on usability because of meaningful use requirements Only clinical summaries exchanged in a common format, providing inadequate substrate for analytics CQMs based on feasibility rather than on clinical outcomes and evidence; lack of accurate reporting of CQMs
Slow and reduce EHR and analytics functionality requirements and increase usability requirements in MU3 Shift to require sharing of full clinical data, not just patient summaries, in MU3
With extensive adoption of basic HIT, usability and usefulness have become more important than additional functions National goals require analytics; analytics require full clinical data. If the goal was to only use claims data for analytics, EHR implementation was unnecessary CQMs represent the national target; ignoring accuracy and alignment with improved outcomes and reduced costs puts national goals at risk
Inter-operability
Quality
Restore alignment of CQMs with care guidelines rather than feasibility Institute meaningful accuracy requirements in MU3
CQMs, clinical quality measures; EHR, electronic health record; HIT, health information technology; MU3, Meaningful Use Stage 3.
would be part of the healthcare organization focus. But there is currently no requirement for accuracy; payment is for the report, whether right or not. Are we surprised current quality measurement is shown repeatedly to be wrong?56–58 The foundation of national efforts to improve quality is both inaccurate and incomplete.
Recommendations for quality Current quality measures focus on convenience and on process rather than on clinical outcomes; they lack incentive for accurate capture. Instead, we urge that: (1) measures should be aligned with national care guidelines rather than feasibility, targeting what is meaningful rather than what is easy to hit; (2) MU Stage III should include meaningful requirements on accurate measure collection. It will not matter how successful we are at reporting and improving quality measures if the underlying information infrastructure is misdirected (table 1).
CONCLUSION Healthcare’s inefficiency and mixed quality contrast starkly with national goals of increasing value from HIT. If we are to meet national goals of improving outcomes and reducing costs, our intermediate markers must align with those goals. Aiming toward unproven intermediate markers—such as level of EHR adoption, sharing of a small subset of patient data, and unproven and inaccurate quality measures—places our heavy HIT investment at risk of failure. Significant use of analytics is an essential step. Many analytics software vendors have transitioned from exclusive use of claims data to incorporating additional portions of the clinical record. However, with the government decision to not require interoperability of full clinical data, large vendors are now building systems de novo to access EHR databases directly. These custom interfaces, built uniquely for each software system and its EHR, are extremely expensive, stifling innovation. Not requiring safe and consistent movement of data from the EHR to the EDW is like the government paying for and building trains and tracks, but not demanding the trains fit on the tracks. With MU Stage III not yet concrete, and with acknowledgement that current approaches have not yet significantly reduced costs or improved outcomes, this is the time to act. We suggest adjusting current efforts in usability, usefulness, interoperability, and quality measures to reflect national goals better (table 1.) These concepts have appropriately earned broad support, but current approaches have been inadequate or ineffective. Revised policies must not just support these overarching goals, but must rationally understand and address the underlying challenges in achieving them. Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
We also recognize that health care is a complex system and that technology is only part of the solution. We present technology solutions to address weakness in technology policy. Other factors outside the scope of this paper, especially workflow, health IT safety, and implementation processes, are equally critical to achieving clinical goals, improving patient experience, and maintaining public trust. The national health IT infrastructure investment is staggering, but small compared to the prize of addressing our healthcare woes. In the complex healthcare system, results will not be instantaneous or guaranteed, but progress must be closely watched and measured. Attaining the ambitious and noble goals of improving outcomes and reducing costs in health care will meaningfully influence our economy and national well-being. As a nation, we have already spent hundreds of billions of dollars on HIT.59 60 With reasoned and moderate course corrections, our health IT investment will be far more likely to deliver its promised benefits to our population and economy. Contributors All persons listed above are authors on this manuscript, as defined by The International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals —(a) making substantial contributions to the design of the work, (b) drafting and revising the work, (c) giving final approval on the submitted version, and (d) agreeing to be accountable for the accuracy of said version. There were no additional contributors. Competing interests DR is a Founder of Health Fidelity and supported by National Science Foundation grant IIP-1330136 to study quality reporting within clinical documentation. RK is supported by an FDA grant to study CPOE screens (HHSF2232010000081); by a NSF grant to study smart alarms and safe communications within hospitals (CPS CNS 1035715); and by an Army grant to study workarounds to cyber security (W911NF-13-1-0086 Sub Award: 2013-00347-03). Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES 1
2
3
4 5
6
Jamoom E, Beatty P, Bercovitz A, et al. Physician adoption of electronic health record systems: United States, 2011. NCHS data brief, no. 98. Hyattsville, MD: National Center for Health Statistics, 2012. Orszag PR. Opportunities to increase efficiency in health care [Internet]. Statement at the Health Reform Summit of the Committee on Finance, United States Senate. Washington, DC: CBO, 2008. [cited 10 February 2014]. http://www.cbo.gov/ ftpdocs/93xx/doc9384/06-16-HealthSummit.pdf National Research Council. Core measurement needs for better care, better health, and lower costs: counting what counts: workshop summary. Washington, DC: The National Academies Press, 2013. Kellermann AL, Jones SS. What it will take to achieve the as-yet-unfulfilled promises of health information technology. Health Aff 2013;32:63–8. Institute of Medicine. Health IT and patient safety: building safer systems for better care. Committee on patient safety and health information technology. Washington, DC: The National Academies Press, 2012. HealthIT.gov. SAFER Guides. [Internet]. Undated [cited 20 August 2014]. http:// www.healthit.gov/policy-researchers-implementers/safer
3
Perspective 7 8 9
10
11
12 13
14 15
16
17 18
19 20
21 22 23
24
25
26 27
28
29
30 31 32 33 34
4
Schneider EC, Ridgely MS, Meeker D, et al. Promoting patient safety through effective Health Information Technology risk management. Santa Monica, CA: RAND, 2014. Corrigan JM, Kohn LT, Donaldson MS. eds. To err is human: building a safer health system. Washington: National Academies Press, 1999. Office of the National Coordinator of HIT. Get the facts about electronic health records: advancing America’s health care. [Internet]. Washington, DC: ONC, [cited 22 February 2014]. http://www.healthit.gov/sites/default/files/pdf/fact-sheets/ ehrs-advancing-americas-health-care.pdf DesRoches CM, Audet AM, Painter M, et al. Meeting Meaningful Use criteria and managing patient populations: a national survey of practicing physicians. Ann Intern Med 2013;158:791–9. Guerra A. Healthcare providers voice gripes at EHR usability hearing. [Internet]. InformationWeek; 2011 Apr [cited 18 February 2014]. http://www.informationweek. com/healthcare/electronic-medical-records/healthcare-providers-voice-gripes-at-ehr/ 229402180 Bowman S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 2013;10:1c. Makam AN, Lanham HJ, Batchelor K, et al. Use and satisfaction with key functions of a common commercial electronic health record: a survey of primary care providers. BMC Med Inform Decis Mak 2013;13:1–7. Karsh BT, Weinger MB, Abbott PA, et al. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010;17:617–23. Sebelius K. Voluntary 2015 edition electronic health record (EHR) certification criteria; interoperability updates and regulatory improvements. [Internet]. Washington, DC: ONC, 2014. [cited 23 February 2014]. http://www.ofr.gov/ OFRUpload/OFRData/2014-03959_PI.pdf Bush GW. Incentives for the use of health information technology and establishing the position of the national health information technology coordinator. Executive Order 2004;13335. Ettorchi-Tardy A, Levif M, Michel P. Benchmarking: a method for continuous quality improvement in health. Healthc Policy 2012;7:e101. Miyata H, Motomura N, Murakami A, et al. Effect of benchmarking projects on outcomes of coronary artery bypass graft surgery: challenges and prospects regarding the quality improvement initiative. J Thorac Cardiovasc Surg 2012;143:1364–9. National Research Council. Toward quality measures for population health and the leading health indicators. Washington, DC: The National Academies Press, 2013. Baillie CA, VanZandbergen C, Tait G, et al. The readmission risk flag: using the electronic health record to automatically identify patients at risk for 30-day readmission. J Hosp Med 2013;8:689–95. Birk S. Population health: strategies that deliver value and results. Healthc Exec 2012;28:10–12. Sanford KD. Population health management: a “start small” strategy. Healthc Financ Manage 2013;67:44–7. HIMSS. Defining and testing EMR usability: principles and proposed methods of EMR usability evaluation and rating [Internet]. Chicago: Healthcare Information and Management Systems Society (HIMSS), 2009. [cited 8 February 2014]. http://www. himss.org/files/HIMSSorg/content/files/himss_definingandtestingemrusability.pdf Lowry S, Ramaiah M, Patterson E, et al. Integrating electronic health records into clinical workflow: an application of human factors modeling methods to ambulatory care. Natl Inst Stand Technol 2014. http://dx.doi.org/10.6028/NIST.IR.7988 Redish J, Lowry S, eds. Usability in health IT: technical strategy, implementation and research. Summary of workshop; 13 Jul 2010; Gaithersburg, MD. Natl Inst Stand Technol [Internet]: 2010 [cited 12 Oct 2014]. p 11-14. http://www.nist.gov/ healthcare/usability/upload/U-HIT_Workshop_Report_Publication_Version.pdf DesRoches CM, Campbell EG, Rao SR, et al. Electronic health records in ambulatory care—a national survey of physicians. N Engl J Med 2008;359:50–60. Hsiao CJ, Hing E. Use and characteristics of electronic health record systems among office-based physician practices: United States, 2001–2013. NCHS data brief, no. 143. Hyattsville, MD: Natl Center for Health Statistics, 2014. Reider J. Usability of EHRs remains a priority for ONC [Internet]. Washington, DC: HealthITBuzz, 2014 [cited 19 February 2014]. http://www.healthit.gov/buzz-blog/ electronic-health-and-medical-records/usability-ehrs-remains-priority-onc/ Sinsky C, Hess J, Karsh BT. Comparative user experiences of health IT products: how user experiences would be reported and used. Washington, DC: Institute of Medicine of the National Academies, 2012. Koppel R. Demanding utility from health information technology. Ann Intern Med 2013;158:845–6. Leviss J. HIT or miss: lessons learned from health information technology implementations. American Health Information Management Association Press, 2010. Koppel R. Keynote chapter: is HIT evidenced-based? International Medical Informatics Yearbook. Stuttgart, Germany: Schattauer Publishers, 2013. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989;13:319–40. Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010;43:159–72.
35 36 37
38
39
40 41
42
43 44
45
46 47
48
49 50
51
52 53
54 55
56
57
58
59
60
National Research Council. Health IT and patient safety: building safer systems for better care. Washington, DC: The National Academies Press, 2012. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203. Hicks LS, Sequist TD, Ayanian JZ, et al. Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med 2008;23:429–41. Ridgely MS, Greenberg MD. Too many alerts, too much liability: sorting through the malpractice implications of drug-drug interaction clinical decision support. St Louis UJ Health L & Pol’y 2012;5:257–79. Classen DC, Phansalkar S, Bates DW. Critical drug-drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf 2011;7:61–5. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2013(0):1–5. Van Der Sijs H, Aarts J, Van Gelder T, et al. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008;15:439–48. Stolba N, Nguyen TM, Tjoa AM. Data warehouse facilitating evidence-based medicine. Complex data warehousing and knowledge discovery for advanced retrieval development: innovative methods and applications. IGI Global 2010:174–207. Evans RS, Lloyd JF, Pierce LA. Clinical use of an enterprise data warehouse. AMIA Annu Symp Proc 2012;2012:189–98. Health Affairs. Health policy brief: interoperability [Internet]. Robert Woods Johnson Foundation, 2014. [cited 2014 August 13]. http://healthaffairs.org/ healthpolicybriefs/brief_pdfs/healthpolicybrief_122.pdf The Office of the National Coordinator for Health Information Technology. Connecting Health and Care for the Nation: A 10-year vision to achieve an interoperable Health IT infrastructure [Internet]. [cited 20 August 2014]. http://www. healthit.gov/sites/default/files/ONC10yearInteroperabilityConceptPaper.pdf Bitton A, Flier LA, Jha AK. Health Information Technology in the era of care delivery reform: to what end?. JAMA 2012;307:2593–4. Riskin D. Congressional testimony. Hearing on “21st Century Technology for 21st Century Cures,” Subcommittees on Communications and Technology and Health 2014 [cited 20 August 2014]. http://democrats.energycommerce.house.gov/sites/ default/files/documents/Testimony-Riskin-HE-CAT-Telehealth-2014-7-17.pdf Health Information Technology Policy Committee. Meaningful Use workgroup request for comments regarding Meaningful Use Stage 2. Washington, DC: Office of the National Coordinator for Health Information Technology, 2011. Blumenthal D. Implementation of the federal health information technology initiative. N Engl J Med 2011;365:2426–31. Zhou L, Soran CS, Jenter CA, et al. The relationship between electronic health record use and quality of care over time. J Am Med Inform Assoc 2009;16:457–64. Welch WP, Bazarko D, Ritten K, et al. Electronic health records in four community physician practices: impact on quality and cost of care. J Am Med Inform Assoc 2007;14:320–8. Ozhathil DK, Li Y, Smith JK, et al. Colectomy performance improvement within NSQIP 2005–2008. J Surg Res 2011;171:e9–e13. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organizations in an oncologic setting. BMC Health Serv Res 2013;13:211. Panzer RJ, Gitomer RS, Greene WH, et al. Increasing demands for quality measurement. JAMA 2013;310:1971–80. Goodhart C. Problems of monetary management: the U.K. experience. In: Courakis AS. ed. Inflation, depression, and economic policy in the West. Totowa, NJ: Rowman & Littlefield, 1981:111–46. Jollis JG, Ancukiewicz M, DeLong ER, et al. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med 1993;119:844–50. Tang PC, Ralston M, Arrigotti MF, et al. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc 2007;14:10–15. Parsons A, McCullough C, Wang J, et al. Validity of electronic health record-derived quality measurement for performance monitoring. J Am Med Inform Assoc 2012;19:604–9. Shaw G. The cost-benefit calculation of electronic health records systems [Internet]. FierceHealthIT, 2013. [cited 20 August 2014]. http://www.fiercehealthit.com/story/ cost-benefit-calculation-electronic-health-records-systems/2013-08-19 Lewis N. EHR implementation still costs too much [Internet]. InformationWeek: HealthCare; 2012 Jul [cited 20 August 2014]. http://www.informationweek.com/ healthcare/electronic-health-records/ehr-implementation-still-costs-too-much/d/d-id/ 1105211
Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
Re-examining health IT policy: what will it take to derive value from our investment? Loren Riskin,1 Ross Koppel,2 Daniel Riskin3 1
Department of Anesthesiology, Stanford Hospital & Clinics, Stanford, California, USA 2 Department of Sociology, University of Pennsylvania, Philadelphia, Pennsylvania, USA 3 Department of Surgery, Stanford University, Stanford, California, USA Correspondence to Dr Loren Riskin, Department of Anesthesiology, Stanford Hospital & Clinics, 300 Pasteur Drive, H3589, Stanford, CA 94305, USA; [email protected] Received 18 June 2014 Revised 22 August 2014 Accepted 21 September 2014
ABSTRACT Despite substantial investments in health information technology (HIT), the nation’s goals of reducing cost and improving outcomes through HIT remain elusive. This period of transition, with new Office of National Coordinator for HIT leadership, upcoming Meaningful Use Stage III definitions, and increasing congressional oversight, is opportune to consider needed course corrections in HIT strategy. This article describes current problems and recommended changes in HIT policy, including approaches to usability, interoperability, and quality measurement. Recommendations refrain from interim measures, such as electronic health record adoption rates, and instead focus on measurable national value to benefit the economy, to reduce healthcare costs, and to improve clinical efficiency and care quality.
INTRODUCTION
To cite: Riskin L, Koppel R, Riskin D. J Am Med Inform Assoc Published Online First: [please include Day Month Year] doi:10.1136/amiajnl2014-003065
US health care suffers from its inability to provide high quality and consistent care despite massive and increasing spending.1 The value deficit is substantial, with the Congressional Budget Office and Institute of Medicine each estimating that 30–40% of US spending is ‘waste’.2 3 Similarly, our national health information technology (HIT) policy has been accused of costing trillions, burdening clinicians and adding risks without a clear return of increased efficiency, safety or quality.4–7 Electronic health records (EHRs) were ubiquitously promoted to deliver value by decreasing effort and errors.5 6 8 Extensive EHR software was purchased and implemented at significant cost, with 75% of hospitals and 44% of primary care providers now using at least basic electronic records.9 However, increasing evidence suggests EHR implementations alone have not delivered anticipated cost reductions or outcome improvements.4 10–13 Also, experience with use of data through enterprise data warehouses (EDW) and analytics is limited. As of 2014, the benefit of HIT remains controversial.9 14 This discussion first explores ‘value’ in the context of our major national IT investment, and then proposes three recommendations to improve HIT’s return on investment. Recommendations focus on known areas of weakness and confusion: HIT usability, interoperability, and quality. For each, we offer a summary of the current approach, reasons the approach has and will remain inadequate, and recommendations for policy course corrections. 2014 is a time when course corrections are possible. Office of National Coordinator (ONC) leadership transition, specification of Meaningful Use
Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
(MU) Stage III, congressional engagement, and proposed shortened regulatory cycles15 all reflect an opportunity. We suggest course corrections are needed now that tie our substantial HIT investment to enhanced national value to benefit the economy, to reduce healthcare costs, and to improve clinical efficiency and care quality.
UNDERSTANDING VALUE IN THE CONTEXT OF A NATIONAL INVESTMENT Before proposing policy recommendations, there should be alignment between goals and our national investment. Early in the last decade, the USA responded to an alarming report by the Institute of Medicine that medical errors represented a serious public threat.8 National efforts were undertaken, including creation of the ONC for Health IT.16 The healthcare industry rallied in response, highlighting safety and error reduction as critical needs requiring technology solutions such as the EHR. Fifteen years later, the landscape has changed. Payment models are shifting toward risk-sharing and value-based healthcare. Measurement of risk and quality are critical to national health reform. Studies and pilots over the last decade reveal that cost reduction and quality improvement require far more than safety and error reduction. The most successful quality improvement efforts have incorporated clinical data to: (1) benchmark outcomes to detect inconsistency and to align care with best practices,17 18 (2) identify high-risk patients and augment resources before expensive catastrophes,19 20 and (3) support population-based analytics and preventive care.21 22 As experts increasingly used and benefited from clinical data, these efforts have become known as clinical analytics. While clinical analytics can have various meanings, we define it here as those systems beyond the health record that use data to drive operational, financial, or clinical improvement. Health information technology goals have expanded from safety, error reduction, and core measures to include clinical analytics and new workflow throughout the care cycle. With national focus on health reform, value is increasingly defined as improved outcomes and reduced costs.
RECOMMENDATION: REQUIRE CLINICAL SOFTWARE USABILITY OVER FUNCTIONALITY Usability of HIT is the learnability, efficiency and satisfaction with which users achieve a specific set of tasks in a particular environment.23 Usability is required for HIT to be effective and is foundational to deriving benefit from software.24 25 HIT usability is widely acknowledged as poor, generating 1
Perspective frustration, medical errors, and inefficiencies.9 15 26–28 Failure to integrate smoothly with clinical workflow severely exacerbates the lack of usability and, at times, lack of usefulness of the technology.29–32 Just as a perceived lack of usefulness hinders HIT adoption,33 34 a lack of EHR usability undermines patient care, accurate documentation and functional clinical decision support (CDS). A Healthcare Information and Management Systems Society (HIMSS) task force states that poor usability is “possibly the most important factor hindering widespread adoption of EMRs”.23 Documentation and order entry time may increase by up to 300% after EHR implementation,35 and medication errors may increase with computerized physician order entry.30 36 CDS designed to help clinicians adhere to standards of care is supposed to improve quality of care,37–39 but functionality proves cumbersome and often ineffective. With extremely high documented override rates,40 41 poor CDS usability undermines its utility and one of the underpinnings of all HIT. Usability for analytics is even less often discussed than usability of the EHR itself, but is at least as important to achieve care consistency and risk reduction.42 43 Because so much workflow occurs outside of the EHR, EDWs—holding data from one or many institutions—combined with analytic systems, offer additional and largely untapped potential for care improvement with little focus on usable systems that offer an effective workflow to improve clinical, operational, and financial efficiency.
Recommendation for usability Address poorly designed and user-unfriendly software, proliferated through rushed national mandates that limit HIT’s benefits. MU Stage III should reduce requirements on EHR functionality and instead focus on defining and requiring effective usability for both EHR and analytics systems.
RECOMMENDATION: MANDATE ROBUST INTEROPERABILITY To use data, a system must have access to the data. Yet, our national health infrastructure created islands, not highways. Today, data are held in many silos, including laboratory systems, pharmacy, revenue cycle software, EHRs, EDWs, and other independent systems. While interoperability is widely acknowledged as essential,31 there is increasing understanding that it is seldom achieved44 and failure will undermine national goals.45–47 Why is this information not shared between clinical systems that received heavy national subsidization and the systems that offer analytics capabilities? Despite successful sharing in other industries and several data standards in healthcare, the general argument is that data sharing within known formats is too hard. An immediate challenge is assuring information is available for transitions in care. Planned policy changes require sharing of patient summaries within a consolidated clinical document architecture format to address this need.48 But, while sharing summaries addresses a decade-old goal to reduce errors when a patient moves from one healthcare organization to another, it fails to actually improve the practice of care. A patient summary is simply not adequate to perform the clinical analytics that can reduce costs and improve outcomes. Consider a patient summary describing a patient as hypertensive, diabetic, and on two medications. Now consider the full clinical record, which incorporates information on social situation, course of care, medication compliance, recent complications, vital signs, and laboratory values. A summary alone simply does not have enough information to measure quality, understand risk, empower population health management, or support 2
benchmarking. The nation subsidized and gave incentive to healthcare organizations to implement EHRs and to doctors to capture the data, but we have allowed lock-in of the precious data in which we invested. Locking in data stifles innovation in healthcare and severely reduces our chance of meeting national goals.
Recommendation for interoperability Current policy requires sharing patient summaries, but not full clinical content to empower analytics. MU Stage III must require sharing full clinical data in a common format among EHRs, EDWs, and analytics systems. Only with this change can we empower clinical analytics and quality workflow to deliver on HIT’s promised value.
RECOMMENDATION: IMPLEMENT A REFINED APPROACH TO QUALITY Though the HITECH Act of 2009 authorized billions of dollars to support a pathway to meaningful use of data,49 the effects of this national approach to quality measurement remain unclear.50 51 While there is broad agreement that quality is important, national discussions seldom focus on what is actually measured and how those measurements are used. Early efforts in quality measurement, such as those from the National Surgical Quality Improvement Program, were focused exclusively on aligning care with national guidelines proven to improve outcome. This was effective in improving care and reducing costs.52 53 As clinical quality measure (CQM) efforts were expanded, the program changed. Quality measures became centrally defined and standards bodies came to serve two masters: national care guidelines, and ‘feasibility.’ Industry and healthcare organizations found newly imposed measures difficult, and the government worked to be responsive. The change in practice was to no longer solely align measures with what improved care, but to also focus on feasibility—how easy it was to input and extract the data.54 It is not possible to serve two masters equally. Either quality measures align with what is proven to advance national goals or they align with what is easiest for vendors and busy clinicians. Thus, new measures focus heavily on discrete data elements— for example, hemoglobin A1c values rather than poorly controlled diabetes, or process measures such as smoking cessation counseling rather than tobacco use. An example highlights the challenge: Although lowering smoking rates is tied to reduced costs, the government has chosen the ‘feasible’ CQM of smoking cessation counseling. A healthcare organization (HCO) wishing to address this measure has added a line to every discharge summary recommending the patient quit smoking. To avoid missing patients, this is added to all discharge summaries of smokers and non-smokers, buried within pages of text. The result is excellent calculated quality scores, but little or no proven influence on outcomes or cost. The risk, well described by Goodhart’s law, is that the measures become a target in themselves rather than a road toward national goals.55 When the targets (measures) are so distant from the goals (reduced costs and improved outcomes), is it really a surprise that goals are not met? In fact, the problem is even worse. Quality measures were initially piloted within motivated institutions, where clinicians worked hard to assign accurately the correct patients to the correct measures. Now, CQMs are being reported by less motivated hospitals and staff, struggling under constrained resources to meet minimal national reporting requirements. If there were a requirement that measurement needed to be accurate, this Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
Perspective Table 1
Relationships among usability, interoperability and quality with current problems, measures, policy changes and rationales
Issue
Current problem
Policy change
Reason
Usability
Rushed HIT adoption with poor usability; vendors claim limited time to work on usability because of meaningful use requirements Only clinical summaries exchanged in a common format, providing inadequate substrate for analytics CQMs based on feasibility rather than on clinical outcomes and evidence; lack of accurate reporting of CQMs
Slow and reduce EHR and analytics functionality requirements and increase usability requirements in MU3 Shift to require sharing of full clinical data, not just patient summaries, in MU3
With extensive adoption of basic HIT, usability and usefulness have become more important than additional functions National goals require analytics; analytics require full clinical data. If the goal was to only use claims data for analytics, EHR implementation was unnecessary CQMs represent the national target; ignoring accuracy and alignment with improved outcomes and reduced costs puts national goals at risk
Inter-operability
Quality
Restore alignment of CQMs with care guidelines rather than feasibility Institute meaningful accuracy requirements in MU3
CQMs, clinical quality measures; EHR, electronic health record; HIT, health information technology; MU3, Meaningful Use Stage 3.
would be part of the healthcare organization focus. But there is currently no requirement for accuracy; payment is for the report, whether right or not. Are we surprised current quality measurement is shown repeatedly to be wrong?56–58 The foundation of national efforts to improve quality is both inaccurate and incomplete.
Recommendations for quality Current quality measures focus on convenience and on process rather than on clinical outcomes; they lack incentive for accurate capture. Instead, we urge that: (1) measures should be aligned with national care guidelines rather than feasibility, targeting what is meaningful rather than what is easy to hit; (2) MU Stage III should include meaningful requirements on accurate measure collection. It will not matter how successful we are at reporting and improving quality measures if the underlying information infrastructure is misdirected (table 1).
CONCLUSION Healthcare’s inefficiency and mixed quality contrast starkly with national goals of increasing value from HIT. If we are to meet national goals of improving outcomes and reducing costs, our intermediate markers must align with those goals. Aiming toward unproven intermediate markers—such as level of EHR adoption, sharing of a small subset of patient data, and unproven and inaccurate quality measures—places our heavy HIT investment at risk of failure. Significant use of analytics is an essential step. Many analytics software vendors have transitioned from exclusive use of claims data to incorporating additional portions of the clinical record. However, with the government decision to not require interoperability of full clinical data, large vendors are now building systems de novo to access EHR databases directly. These custom interfaces, built uniquely for each software system and its EHR, are extremely expensive, stifling innovation. Not requiring safe and consistent movement of data from the EHR to the EDW is like the government paying for and building trains and tracks, but not demanding the trains fit on the tracks. With MU Stage III not yet concrete, and with acknowledgement that current approaches have not yet significantly reduced costs or improved outcomes, this is the time to act. We suggest adjusting current efforts in usability, usefulness, interoperability, and quality measures to reflect national goals better (table 1.) These concepts have appropriately earned broad support, but current approaches have been inadequate or ineffective. Revised policies must not just support these overarching goals, but must rationally understand and address the underlying challenges in achieving them. Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
We also recognize that health care is a complex system and that technology is only part of the solution. We present technology solutions to address weakness in technology policy. Other factors outside the scope of this paper, especially workflow, health IT safety, and implementation processes, are equally critical to achieving clinical goals, improving patient experience, and maintaining public trust. The national health IT infrastructure investment is staggering, but small compared to the prize of addressing our healthcare woes. In the complex healthcare system, results will not be instantaneous or guaranteed, but progress must be closely watched and measured. Attaining the ambitious and noble goals of improving outcomes and reducing costs in health care will meaningfully influence our economy and national well-being. As a nation, we have already spent hundreds of billions of dollars on HIT.59 60 With reasoned and moderate course corrections, our health IT investment will be far more likely to deliver its promised benefits to our population and economy. Contributors All persons listed above are authors on this manuscript, as defined by The International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals —(a) making substantial contributions to the design of the work, (b) drafting and revising the work, (c) giving final approval on the submitted version, and (d) agreeing to be accountable for the accuracy of said version. There were no additional contributors. Competing interests DR is a Founder of Health Fidelity and supported by National Science Foundation grant IIP-1330136 to study quality reporting within clinical documentation. RK is supported by an FDA grant to study CPOE screens (HHSF2232010000081); by a NSF grant to study smart alarms and safe communications within hospitals (CPS CNS 1035715); and by an Army grant to study workarounds to cyber security (W911NF-13-1-0086 Sub Award: 2013-00347-03). Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES 1
2
3
4 5
6
Jamoom E, Beatty P, Bercovitz A, et al. Physician adoption of electronic health record systems: United States, 2011. NCHS data brief, no. 98. Hyattsville, MD: National Center for Health Statistics, 2012. Orszag PR. Opportunities to increase efficiency in health care [Internet]. Statement at the Health Reform Summit of the Committee on Finance, United States Senate. Washington, DC: CBO, 2008. [cited 10 February 2014]. http://www.cbo.gov/ ftpdocs/93xx/doc9384/06-16-HealthSummit.pdf National Research Council. Core measurement needs for better care, better health, and lower costs: counting what counts: workshop summary. Washington, DC: The National Academies Press, 2013. Kellermann AL, Jones SS. What it will take to achieve the as-yet-unfulfilled promises of health information technology. Health Aff 2013;32:63–8. Institute of Medicine. Health IT and patient safety: building safer systems for better care. Committee on patient safety and health information technology. Washington, DC: The National Academies Press, 2012. HealthIT.gov. SAFER Guides. [Internet]. Undated [cited 20 August 2014]. http:// www.healthit.gov/policy-researchers-implementers/safer
3
Perspective 7 8 9
10
11
12 13
14 15
16
17 18
19 20
21 22 23
24
25
26 27
28
29
30 31 32 33 34
4
Schneider EC, Ridgely MS, Meeker D, et al. Promoting patient safety through effective Health Information Technology risk management. Santa Monica, CA: RAND, 2014. Corrigan JM, Kohn LT, Donaldson MS. eds. To err is human: building a safer health system. Washington: National Academies Press, 1999. Office of the National Coordinator of HIT. Get the facts about electronic health records: advancing America’s health care. [Internet]. Washington, DC: ONC, [cited 22 February 2014]. http://www.healthit.gov/sites/default/files/pdf/fact-sheets/ ehrs-advancing-americas-health-care.pdf DesRoches CM, Audet AM, Painter M, et al. Meeting Meaningful Use criteria and managing patient populations: a national survey of practicing physicians. Ann Intern Med 2013;158:791–9. Guerra A. Healthcare providers voice gripes at EHR usability hearing. [Internet]. InformationWeek; 2011 Apr [cited 18 February 2014]. http://www.informationweek. com/healthcare/electronic-medical-records/healthcare-providers-voice-gripes-at-ehr/ 229402180 Bowman S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 2013;10:1c. Makam AN, Lanham HJ, Batchelor K, et al. Use and satisfaction with key functions of a common commercial electronic health record: a survey of primary care providers. BMC Med Inform Decis Mak 2013;13:1–7. Karsh BT, Weinger MB, Abbott PA, et al. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010;17:617–23. Sebelius K. Voluntary 2015 edition electronic health record (EHR) certification criteria; interoperability updates and regulatory improvements. [Internet]. Washington, DC: ONC, 2014. [cited 23 February 2014]. http://www.ofr.gov/ OFRUpload/OFRData/2014-03959_PI.pdf Bush GW. Incentives for the use of health information technology and establishing the position of the national health information technology coordinator. Executive Order 2004;13335. Ettorchi-Tardy A, Levif M, Michel P. Benchmarking: a method for continuous quality improvement in health. Healthc Policy 2012;7:e101. Miyata H, Motomura N, Murakami A, et al. Effect of benchmarking projects on outcomes of coronary artery bypass graft surgery: challenges and prospects regarding the quality improvement initiative. J Thorac Cardiovasc Surg 2012;143:1364–9. National Research Council. Toward quality measures for population health and the leading health indicators. Washington, DC: The National Academies Press, 2013. Baillie CA, VanZandbergen C, Tait G, et al. The readmission risk flag: using the electronic health record to automatically identify patients at risk for 30-day readmission. J Hosp Med 2013;8:689–95. Birk S. Population health: strategies that deliver value and results. Healthc Exec 2012;28:10–12. Sanford KD. Population health management: a “start small” strategy. Healthc Financ Manage 2013;67:44–7. HIMSS. Defining and testing EMR usability: principles and proposed methods of EMR usability evaluation and rating [Internet]. Chicago: Healthcare Information and Management Systems Society (HIMSS), 2009. [cited 8 February 2014]. http://www. himss.org/files/HIMSSorg/content/files/himss_definingandtestingemrusability.pdf Lowry S, Ramaiah M, Patterson E, et al. Integrating electronic health records into clinical workflow: an application of human factors modeling methods to ambulatory care. Natl Inst Stand Technol 2014. http://dx.doi.org/10.6028/NIST.IR.7988 Redish J, Lowry S, eds. Usability in health IT: technical strategy, implementation and research. Summary of workshop; 13 Jul 2010; Gaithersburg, MD. Natl Inst Stand Technol [Internet]: 2010 [cited 12 Oct 2014]. p 11-14. http://www.nist.gov/ healthcare/usability/upload/U-HIT_Workshop_Report_Publication_Version.pdf DesRoches CM, Campbell EG, Rao SR, et al. Electronic health records in ambulatory care—a national survey of physicians. N Engl J Med 2008;359:50–60. Hsiao CJ, Hing E. Use and characteristics of electronic health record systems among office-based physician practices: United States, 2001–2013. NCHS data brief, no. 143. Hyattsville, MD: Natl Center for Health Statistics, 2014. Reider J. Usability of EHRs remains a priority for ONC [Internet]. Washington, DC: HealthITBuzz, 2014 [cited 19 February 2014]. http://www.healthit.gov/buzz-blog/ electronic-health-and-medical-records/usability-ehrs-remains-priority-onc/ Sinsky C, Hess J, Karsh BT. Comparative user experiences of health IT products: how user experiences would be reported and used. Washington, DC: Institute of Medicine of the National Academies, 2012. Koppel R. Demanding utility from health information technology. Ann Intern Med 2013;158:845–6. Leviss J. HIT or miss: lessons learned from health information technology implementations. American Health Information Management Association Press, 2010. Koppel R. Keynote chapter: is HIT evidenced-based? International Medical Informatics Yearbook. Stuttgart, Germany: Schattauer Publishers, 2013. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989;13:319–40. Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010;43:159–72.
35 36 37
38
39
40 41
42
43 44
45
46 47
48
49 50
51
52 53
54 55
56
57
58
59
60
National Research Council. Health IT and patient safety: building safer systems for better care. Washington, DC: The National Academies Press, 2012. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203. Hicks LS, Sequist TD, Ayanian JZ, et al. Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med 2008;23:429–41. Ridgely MS, Greenberg MD. Too many alerts, too much liability: sorting through the malpractice implications of drug-drug interaction clinical decision support. St Louis UJ Health L & Pol’y 2012;5:257–79. Classen DC, Phansalkar S, Bates DW. Critical drug-drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf 2011;7:61–5. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2013(0):1–5. Van Der Sijs H, Aarts J, Van Gelder T, et al. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008;15:439–48. Stolba N, Nguyen TM, Tjoa AM. Data warehouse facilitating evidence-based medicine. Complex data warehousing and knowledge discovery for advanced retrieval development: innovative methods and applications. IGI Global 2010:174–207. Evans RS, Lloyd JF, Pierce LA. Clinical use of an enterprise data warehouse. AMIA Annu Symp Proc 2012;2012:189–98. Health Affairs. Health policy brief: interoperability [Internet]. Robert Woods Johnson Foundation, 2014. [cited 2014 August 13]. http://healthaffairs.org/ healthpolicybriefs/brief_pdfs/healthpolicybrief_122.pdf The Office of the National Coordinator for Health Information Technology. Connecting Health and Care for the Nation: A 10-year vision to achieve an interoperable Health IT infrastructure [Internet]. [cited 20 August 2014]. http://www. healthit.gov/sites/default/files/ONC10yearInteroperabilityConceptPaper.pdf Bitton A, Flier LA, Jha AK. Health Information Technology in the era of care delivery reform: to what end?. JAMA 2012;307:2593–4. Riskin D. Congressional testimony. Hearing on “21st Century Technology for 21st Century Cures,” Subcommittees on Communications and Technology and Health 2014 [cited 20 August 2014]. http://democrats.energycommerce.house.gov/sites/ default/files/documents/Testimony-Riskin-HE-CAT-Telehealth-2014-7-17.pdf Health Information Technology Policy Committee. Meaningful Use workgroup request for comments regarding Meaningful Use Stage 2. Washington, DC: Office of the National Coordinator for Health Information Technology, 2011. Blumenthal D. Implementation of the federal health information technology initiative. N Engl J Med 2011;365:2426–31. Zhou L, Soran CS, Jenter CA, et al. The relationship between electronic health record use and quality of care over time. J Am Med Inform Assoc 2009;16:457–64. Welch WP, Bazarko D, Ritten K, et al. Electronic health records in four community physician practices: impact on quality and cost of care. J Am Med Inform Assoc 2007;14:320–8. Ozhathil DK, Li Y, Smith JK, et al. Colectomy performance improvement within NSQIP 2005–2008. J Surg Res 2011;171:e9–e13. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organizations in an oncologic setting. BMC Health Serv Res 2013;13:211. Panzer RJ, Gitomer RS, Greene WH, et al. Increasing demands for quality measurement. JAMA 2013;310:1971–80. Goodhart C. Problems of monetary management: the U.K. experience. In: Courakis AS. ed. Inflation, depression, and economic policy in the West. Totowa, NJ: Rowman & Littlefield, 1981:111–46. Jollis JG, Ancukiewicz M, DeLong ER, et al. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med 1993;119:844–50. Tang PC, Ralston M, Arrigotti MF, et al. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc 2007;14:10–15. Parsons A, McCullough C, Wang J, et al. Validity of electronic health record-derived quality measurement for performance monitoring. J Am Med Inform Assoc 2012;19:604–9. Shaw G. The cost-benefit calculation of electronic health records systems [Internet]. FierceHealthIT, 2013. [cited 20 August 2014]. http://www.fiercehealthit.com/story/ cost-benefit-calculation-electronic-health-records-systems/2013-08-19 Lewis N. EHR implementation still costs too much [Internet]. InformationWeek: HealthCare; 2012 Jul [cited 20 August 2014]. http://www.informationweek.com/ healthcare/electronic-health-records/ehr-implementation-still-costs-too-much/d/d-id/ 1105211
Riskin L, et al. J Am Med Inform Assoc 2014;0:1–4. doi:10.1136/amiajnl-2014-003065
