bs_bs_banner

Journal of Evaluation in Clinical Practice ISSN 1365-2753

Determining differences in user performance between expert and novice primary care doctors when using an electronic health record (EHR) Martina A. Clarke MS,1 Jeffery L. Belden MD3 and Min Soon Kim PhD2,4 1

PhD Student, Graduate Research Assistant, 2Assistant Professor, Informatics Institute, University of Missouri, Columbia, MO, USA Family Medicine Physician, Professor, Department of Family and Community Medicine, University of Missouri, Columbia, MO, USA 4 Assistant Professor, Department of Health Management and Informatics, University of Missouri, Columbia, MO, USA 3

Keywords evaluation, medical informatics Correspondence Dr Min Soon Kim Department of Health Management and Informatics Informatics Institute University of Missouri CE728 Clinical Support & Education, DC006.00 5 Hospital Drive Columbia, MO 65212 USA E-mail: [email protected] Accepted for publication: 22 September 2014 doi:10.1111/jep.12277

Abstract Rationale, aims and objectives The goal of this study is to determine usability gaps between expert and novice primary care doctors when using an electronic health record (EHR). Methods Usability tests using video analyses with triangular method approach were conducted to analyse usability gaps between 10 novice and seven expert doctors. Doctors completed 19 tasks, using think-aloud strategy, based on an artificial but typical patient visit note. The usability session lasted approximately 20 minutes. The testing room consisted of the participant and the facilitator. Mixed methods approach including four sets of performance measures, system usability scale (SUS), and debriefing session with participants was used. Results While most expert doctors completed tasks more efficiently, and provided a higher SUS score than novice doctors (novice 68, expert 70 out of 100 being perfect score), the result of ‘percent task success rate’ were comparable (74% for expert group, 78% for novice group, P = 0.98) on all 19 tasks. Conclusion This study found a lack of expertise among doctors with more experience using the system demonstrating that although expert doctors have been using the system longer, their proficiency did not increase with EHR experience. These results may potentially improve the EHR training programme, which may increase doctors’ performance when using an EHR. These results may also assist EHR vendors in improving the user interface, which may aid in reducing errors caused from poor usability of the system.

Introduction Centers for Medicare & Medicaid Services (CMS) and the Office of the National Coordinator for Health Information Technology propose the ‘Meaningful Use (MU)’ of interoperable electronic health records (EHRs) in an effort known as The Health Information Technology for Economic and Clinical Health Act. EHRs are ‘patient records of health information created by encounters in any health care delivery setting’ [1]. MU ought to increase efficiency, safety and quality of health care, reduce health inequalities, encourage patient and family members to engage in their health, improve coordination of care and safeguard the privacy and security of personal health information [2]. In clinical practice, the function of health information technology is growing and more doctors are adopting EHRs extensively because of the financial

incentives promised by CMS [3]. A data brief by the National Center for Health Statistics reported that 78% of office-based doctors have adopted EHRs in their practice in 2013 [4]. EHR users report that the benefits of having an EHR include remote access to a patient’s chart, accurate, complete and up-to-date patient information, alerts to critical lab values, decreased paperwork and an overall enhancement to patient care [5,6]. However, there are potential disadvantages to EHRs, which include financial burdens, workflow misalignment, increase in doctors’ time and loss of productivity affiliated with EHR usability issues (i.e. product use is not easy to learn, and its use is both inefficient and unsatisfying) [7–9]. Usability is defined as how well a system can be operated by users to complete a certain task with effectiveness, efficiency and satisfaction [10]. For an EHR to be used effectively and efficiently, the system should allow doctors’ to complete

Journal of Evaluation in Clinical Practice 20 (2014) 1153–1161 © 2014 John Wiley & Sons, Ltd.

1153

EHR usability gaps between doctors

clinical tasks without significant errors caused by confusion and difficulty of cumbersome systems. Adverse drug events (ADEs) caused over $3.5 billion in additional medical costs [11]. As an approach to decrease medication errors and patient harm, the Institute of Medicine advises using a Computerized Provider Order Entry (CPOE) system [12]. CPOEs are computer-aided medication ordering from an electronic device that is often integrated into EHRs. With the ongoing implementation of MU stage 2 as a part of the EHR incentive programme, any licensed health care professional is required to use CPOE for medication, laboratory and radiology orders. A major advantage of CPOE is medication error reductions that are caused by illegible handwriting or incorrect transcription [13]. CPOE systems may include functionalities, such as, alerts about adverse drug reactions, which may promote reduction in errors. However, there is also some evidence that the use of CPOE may cause increase in clinician work, undesirable workflow issues and generation of new kinds of errors [14,15]. Poor usability of CPOE, has been shown to reduce efficiency, decrease quality of patient care and frustrate clinicians [16]. For a CPOE to be used effectively and efficiently, the system should allow doctors to complete medication orders accurately without causing ADEs. Primary care provides medical care for a significant amount of common illnesses and account for a great number of patient visits [17,18]. Between 2009 and 2010, approximately 46% of all ambulatory care visits were by primary care doctors [19] and approximately 75% of doctor office visits involve drug therapy [20]. With the health care reform underway, an increase in patients will induce a shortage of primary care providers, which may reduce the time doctors spend with patients thereby increasing the duties of primary care doctors [21]. Primary care doctors, who frequently are not adequately trained on EHR use in their medical schools and have started learning their new specialty, have to cope with a steep learning curve on EHR use. This may negatively influence their learning experiences, especially in health care settings with poor EHR usability. This, in turn, may lead to high cognitive load, medical errors and decreased quality of patient care. Allowing doctors to efficiently execute clinical tasks within the EHR, particularly CPOE functionalities, may release some time constraints experienced by doctors while caring for patients. Previous studies have shown the importance of usability testing and evaluation in the EHR adoption and implementation process and current best practices promote utilization of cognitive approaches to assess human–computer interactions within the EHR system [22–26]. For instance, a systematic review by Khajouei and Jasper, examining the impact of CPOE medication system’s design aspect on usability, found that proper system design of CPOE is essential to increase doctors’ adoption and to reduce medication errors [16]. A cognitive approach was used when Li et al. conducted usability testing using think-aloud protocol analysis with clinical simulations to evaluate clinical decision support and found 90% of negative comments were associated with navigation and workflow issues [27]. Heuristic evaluation has been also been used to identify usability issues in multiple articles. Chan et al., conducted a heuristic evaluation of a CPOE order set system and uncovered 92 unique heuristic violations across 10 heuristic principles [28]. A study by Harrington et al., analysing an EHR’s usability, identified 14 usability heuristics that were violated 346 times in the intensive care unit’s clinical documentation 1154

M.A. Clarke et al.

[29].These studies were successful in identifying usability issues among users; however, there is a lack of studies comparing novice and expert doctors when using an electronic health record and this paper seeks to identify the differences between the two doctor groups.

Objective The objective of this study is to determine the difference in performance between expert and novice primary care doctors to eventually improve the EHR training programme for new users of the EHR. The study addresses three specific research questions: 1 How do expert and novice doctors differ in performance measures: task percent success, time on task (TOT), mouse clicks (MC) and mouse movements (MM)? 2 Is there a correlation between performance measures of all tasks between expert and novice doctors? 3 Is there a difference in how novice and expert doctors rate the usability of the EHR system? Our hypothesis is that expert doctors will be more competent than novice doctors when using the EHR.

Method Study design To determine usability gaps in use of EHR systems between expert and novice doctors, data was collected through usability testing using video analysis software, Morae® (TechSmith, Okemos, MI, USA). Twelve family medicine and four internal medicine resident doctors, and one attending doctor completed 19 artificial, scenariobased tasks in a laboratory setting. Mixed methods approach was used to identify usability gaps between the novice and expert doctors. This included four sets of performance measures, system usability scale (SUS) measurement and debriefing session with participants. This pilot study was reviewed then approved by the University of Missouri Health Sciences Institutional Review Board.

Organizational setting This study was conducted at the University of Missouri Health System (UMHS), a 536-bed, tertiary care academic medical hospital located in Columbia, Missouri. The Healthcare Information and Management Systems Society, a non-profit organization that rates how hospitals are implementing electronic medical record (EMR) application, has awarded UMHS with stage 7 of the EMR adoption model [30]. This means UMHS uses electronic patient charts, analyses clinical data using data warehousing and electronically shares health information with authorized health care entities [31]. There are more than 70 primary care doctors at UMHS clinics throughout central Missouri and in 2012, had approximately 553 300 clinic visits. The Department of Family and Community Medicine (FCM) oversees six clinics and have over 100 000 patient visits at these clinics, while the Department of Internal Medicine (IM) oversees two clinics [32]. UMHS’ EHR contains a database, which includes all the data from the university’s hospitals and clinics. Within the EHR, CPOE allows providers to electronically and securely access and place lab and

© 2014 John Wiley & Sons, Ltd.

M.A. Clarke et al.

medication orders for patients, and transmit the orders directly to the department that is responsible for completing the request. Determining usability gaps between expert and novice doctors while using the fully implemented EHR system within one of the most wired health care systems makes the aim of this study achievable.

Participants Currently, there is no evidence-based way to measure a user’s EHR experience so novice doctors and expert doctors were differentiated based on clinical training level and number of years using the EHR according to the discussion with an experienced doctor champion (JLB) and two chief residents from both participating departments (FCM, IM). Thus, first year residents were classified as novice users and expert users were classified as second year residents and above. Convenience sampling method was used when selecting participants [33]. UMHS FCM and IM doctors were selected for the sample because, as primary care residents, they have comparable clinical roles and responsibilities. FCM participants were recruited during weekly resident meetings and IM residents were recruited through group emails sent by the IM chief resident. Twelve FCM and four IM residents volunteered to participate in this study. Based on a review of the literature, 10 participants were deemed adequate in explorative usability studies to find relevant usability problems [34,35]. Participation was voluntary and doctors were not obliged to participate in the study. Residents who volunteered were compensated for their participation.

Scenario and tasks The scenario presented to the residents in this study was a ‘scheduled follow-up visit after a hospitalization for pneumonia’. This scenario was used in order to evaluate both inpatient and outpatient patient data to make this scenario realistic. Nineteen tasks commonly performed by both expert and novice primary care doctors were developed for the participants to complete. The tasks that were used in this study were also a part of the EHR training residents received at the beginning of their residency that makes this evaluation practical. The tasks had clear objectives that doctors were able to comprehend without unnecessary clinical cognitive burdens or vagueness, which was not a part of the study’s goals. The tasks completed by the participants were • Task 1: Start a new note. • Task 2: Include visit information. • Task 3: Include chief complaint. • Task 4: Include history of present illness. • Task 5: Review current medications contained in the note. • Task 6: Review problem list contained in the note. • Task 7: Document new medication allergy. • Task 8: Include review of systems. • Task 9: Include family history. • Task 10: Include physical exam. • Task 11: Include last comprehensive metabolic panel (CMP). • Task 12: Save the note. • Task 13: Include diagnosis. • Task 14: Place order for chest X-ray. • Task 15: Place order for basic metabolic panel (BMP).

© 2014 John Wiley & Sons, Ltd.

EHR usability gaps between doctors

• • • •

Task 16: Change a medication. Task 17: Add a medication to your favourites list. Task 18: Renew one of the existing medications. Task 19: Sign the note.

Performance measures Four performance measures were used to evaluate user performance: 1 Percent task success computes the percentage of subtasks that participants successfully complete error free. 2 TOT measures the length of time each participant takes to complete each task. It begins when participant click on ‘start task’ button to when the ‘end task’ button is clicked. 3 MC measures the number of times the participant clicks on the mouse when completing a given task. 4 Mouse movement computes in pixels the distance of the navigation path by the mouse to complete a given task. For percent task success rate, a higher value indicates higher performance, which represents participants’ proficiency with the system. For TOT, MC, and MM, a higher value usually indicates lower performances [35–37]. Higher values may represent that the participant had difficulties using the system.

SUS In addition to the performance measures, each participant completed a SUS, a 10-item Likert scale that is a subjective assessment of a system. SUS is a widely used, validated instrument that provides fairly robust measures of subjective usability [38–40]. SUS yields a single number that exemplify a composite measure of the overall usability of the system under investigation. SUS yields a score from 0 to 100, with 100 being a perfect score [40]. Scores of 0 to 50 is considered not acceptable, 50 to 62 is considered low marginal, 63 to 70 is considered high marginal, 70 to 100 was considered acceptable [41].

Data collection Usability data was collected between 12 November 2013 and 19 December 2013. The usability test was completed in approximately 20 minutes and was conducted on a 15-inch laptop using Windows 7 operating system. To retain consistency and minimize unsolicited distractions, the participant and the facilitator were the only two individuals in the testing room. The participant was reminded from the beginning that their participation in this study was completely voluntary and that they had the right to stop the testing at any time. The participant was then instructed to read the printed instructions, containing a scenario and 19 tasks. A binder was then given to the participant with the task list and then the participant completed each task. Think aloud strategy was used throughout the session and was recorded using Morae Recorder [42]. We prompted participants to talk aloud and describe what they were doing while completing the tasks. Participants completed the tasks without the assistance of the facilitator who only intervened if there were any technical difficulties. After participants completed the tasks they completed the SUS and demographic survey. At the end of the session, participants were asked to comment on tasks they found difficult. 1155

EHR usability gaps between doctors

Data analysis Usability gaps between expert and novice doctors using an EHR were determined by using Morae Recorder to capture audio, video, on-screen activity and inputs from the keyboard and mouse. The recorded sessions were examined using Morae Manager by calculating performance measures using markers to identify difficulties and errors the participants had. Video analysis took approximately 1.5 hours for each 20-minute recorded session. The first step in analysis was to review the recorded sessions and label any tasks that were unmarked during data collection. The second step was to divide each of the 19 tasks into smaller tasks in order to determine the task success rate and identify subtle usability challenges that we may have otherwise failed to notice. The statistical test used to compare performance measures was the t-test. Pearson’s correlation was used to find the relationship among performance measures and SUS.

Results Participants Out of 10 novice doctors, seven were from family medicine and three were from internal medicine at UMHS. Six of the 10 novice doctors (60%) were male, eight (80%) of novice doctors identified their race as white, one (10%) identified their race as Asian and one resident (10%) identified their race as both Asian and white. The age of novice doctors ranged from 27 to 31 and the mean age was 28 years. Four (40%) novice doctors had no other experience with an EHR other than the EHR at UMHC, two (20%) have less than 3 months experience, one (10%) had 7 months to 1 year experience, and three (30%) had over 2 years experience with an EHR other than the current EHR. Six family medicine and one internal medicine expert doctors participated in the study. Five out of the six expert doctors (83%) were female and all (100%) of expert doctors identified their race as white. Two expert doctors did not provide information on their date of birth and EHR experience and was not included in the calculation of age range, mean age and EHR experience. The age of expert doctors ranged from 30 to 33 and the mean age was 31 years. One (17%) expert doctor had no other experience with an EHR other than the EHR at UMHS, one (17%) had 7 months to 1 year experience, and two (33%) had over 2 years’ experience with an EHR other than the current EHR. Because of the small sample size for this type of study, we did not attempt to control for age or gender. We did not attempt to control for EHR experience either because we considered both clinical and EHR experience as a classification scheme.

M.A. Clarke et al.

seven tasks: tasks 1, 3, 4, 5, 6, 9 and 19; and lower success rate in six tasks: task 2, 8, 11, 13, 15, 17 and 16. Three prominent usability issues that may have affected the percent task success rate were discovered from participants attempting task 7: document new medication allergy, task 14: place order for chest X-ray and task 17: add a medication to your favourites list. Three out of seven expert doctors and four out of 10 novice doctors did not successfully complete task 7: Document new medication allergy. To complete this task, doctors would have to use a drop down box to change the status from ‘cancelled’ to ‘active’ but three expert doctors and four novice doctors were not able to do so. The doctors became frustrated and gave up on completing task 7. Four novice doctors and two expert doctors were able to complete this after clicking and searching. Two novice doctors tried to type in the text field labelled ‘Substance’ that was not meant for typing. One novice doctor went into the Histories tab first to find allergies then successfully found allergies below the patients’ name. One novice doctor was not able to order a chest X-ray for task 14: place order for chest X-ray and did not understand the meaning of the alert that was brought up during the task. This resident was the only participant that received the alert, ‘Radiology orders should be placed following downtime procedures during 2200 and 0000’. This message meant that there was going to be an update to the system and so all orders would have to be processed using paper. If a doctor received this error during clinical workflow, the doctor would not be able to order the chest X-ray at that moment and may have to get back to it later. The doctor may then forget to order the X-ray and may miss an important diagnosis that could have been found from the patient receiving the X-ray. Both expert and novice groups had the lowest task success rate in task 17: add a medication to a favourite list, which required participants to add a medication to a list of their frequently used medications. Three out of seven expert doctors and four out of 10 novice doctors did not successfully complete this task. One novice and one expert doctor did not even attempt to add the medication to their favourites. One expert doctor mentioned not knowing how to add a medication to their favourites list although this has already been taught in the EHR training programme at UMHS. Adding a medication to a favourites list can only be done when the order detail view is open, where medications specifics, such as dosage, are included. Medication favourites cannot be added from the main medication list view. This functionality was not intuitive because this feature was not accessible directly from the medication list, which defeats the purpose and reduces the likelihood of doctors using this feature. TOT

Performance measures Percent task success rate Geometric mean values of percent task success rates of 19 tasks (Fig. 1) were compared between the expert and novice doctors [43]. There was no significant difference (P = 0.59) in the success rate between the two doctor groups (70%, expert group vs. 78%, novice group). The expert doctor group achieved higher success rate in six tasks: 7, 10, 12, 14 and 18; the same success rate in 1156

Geometric mean values of TOT were compared between expert and novice doctors. No substantial difference was observed between the two doctor groups (34s, expert group vs. 39s, novice group, P = 0.41). The expert group took less time completing 13 out of 19 tasks (tasks 1, 2, 3, 4, 7, 8, 9, 10, 14, 15, 16 and 18) than the novice doctors and the same time completing task 17. Three conspicuous usability issues that may have affected both doctor groups’ mean time when completing tasks were discovered from participants attempting task 7: document new medication allergy, task 15: place order for BMP and task 16: change a

© 2014 John Wiley & Sons, Ltd.

M.A. Clarke et al.

EHR usability gaps between doctors

Task success (%)

120 100 80 60 40 20 0

Time on task (seconds)

T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 M Expert mean 100 52 100 100 100 100 96 92 96 100 27 84 99 93 93 73 6 82 100 74 Novice mean 100 100 100 100 100 100 36 100 100 99 100 75 100 86 92 88 6 79 100 78 140 120 100 80 60 40 20 0

T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 M Expert mean 36 12 13 37 50 25 90 44 26 86 31 11 70 54 55 59 13 29 34 34 Novice mean 47 14 14 39 37 26 133 62 46 96 28 10 64 68 81 77 13 36 31 39

Figure 1 Geometric mean values of percent task success rates and time on task (TOT) of 19 tasks between two doctor groups. A higher percentage usually indicates higher performance. A lower number in seconds usually indicate higher performance in TOT. No statistical difference was observed for percent task success rate (70%, expert group vs. 78%, novice group, P = 0.58) and TOT (34s, expert group vs. 39s, novice group, P = 0.41). The expert doctor group achieved higher success rate in six tasks: 7, 10, 12, 14 and 18; the same success rate in seven tasks: tasks 1, 3, 4, 5, 6, 9 and 19; and lower success rate in six tasks: tasks 2, 8, 11, 13, 15, 17 and 16. The expert group took less time completing 13 out of 19 tasks (tasks 1, 2, 3, 4, 7, 8, 9, 10, 14, 15, 16 and 18) than the novice doctors and the same time completing task 17.

medication. The task with the most time required for both expert and novice doctors was task 7: document new medication allergy (100s, expert group vs. 133s, novice group). Doctors may have spent more time on task 7 because the status field was not emphasized to indicate to users that the status of the allergy needed to be changed from ‘cancelled’ to ‘active’. Thus doctors struggled to identify the reason that medication allergy would not appear in the patient’s allergy list. When completing task 7, some inactive fields appeared as active text boxes. For example, when adding a medication allergy, a text box was highlighted in yellow, which deceived users into assuming that they should type in the textbox when they could not. Six out of 10 novice and three out of seven expert doctors attempted to type into this field. Novice doctors spent more time on task 15: place order for BMP than expert doctors because eight out of 10 novice doctors did not know how to place two orders at the same time. One novice doctor mentioned that there was a way to order them both at the same time but did not know how. Emphasis on how to place multiple orders concurrently during EHR training may also improve novice doctors’ knowledge of how to efficiently complete this task. On task 16: change a medication, doctors had to choose from the right click menu options ‘Renew’, ‘Cancel/DC’ or ‘Cancel/

© 2014 John Wiley & Sons, Ltd.

Reorder’. One reason novice doctors may have spent extra time on this task was because doctors were unclear of what option to use among the options ‘Renew’, ‘Cancel/DC’ or ‘Cancel/Reorder’. Right clicking a medication brought up a menu list with a menu item labelled ‘Modify without resending’. Six out of 10 novice and two out of seven expert doctors attempted to use this menu option although it was not the correct menu item. To change a medication, the recommend menu item to choose is ‘Cancel/Reorder’. Having a clearly labelled option in the right click menu of the medication list would allow doctors to complete this task more efficiently.

MC and MM Geometric mean values of MC were compared between the two doctor groups. Expert doctors completed the tasks with slightly fewer MC than novice doctors did (five clicks, expert group vs. six clicks, novice group, P = 0.85; Fig. 2). The expert doctor group achieved fewer MC in eight tasks: tasks 1, 5, 7, 9, 12, 14, 15 and 16; higher MC in seven tasks: 4, 6, 8, 11, 13, 17 and 19; and the comparable number of clicks in three tasks: 2, 3 and 18. The task with the most clicks for both groups was task 7: document new 1157

EHR usability gaps between doctors

Mouse clicks

medication allergy (25 clicks, expert group vs. 31 clicks, novice group). Geometric mean values of mouse movement, length of the navigation path to complete a given task, were compared between two doctor groups. Overall, the expert doctors showed slightly shorter MM across the 19 tasks (5328 pixels, expert group vs. 6469 pixels, novice group, P = 0.47). The expert doctors showed shorter MM in 15 of 19 tasks: tasks 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16 and 18; and longer MM in four tasks: tasks 5, 13, 17 and 19. Both groups spent most of the MM on task 7: document new medication allergy, which is consistent with the results of MC being the highest with this task (18 972 pixels, expert vs. 29 364 pixels, novice). Two noticeable usability issues that may have affected both doctor groups’ mean MC and MM when completing tasks were discovered from participants attempting task 13: include diagnosis and task 15: place order for BMP. When completing task 13: include diagnosis, doctors were unclear on how to import a list of diagnosis from the problem list into the visit note. One novice doctor and two expert doctors were unaware that they should

M.A. Clarke et al.

highlight all the diagnoses before clicking ‘Include’ to get the entire list of diagnoses into the visit note. Returning to the diagnosis list because of this design flaw also may have increased the number of MC and MM. When completing task 15: place order for BMP than expert doctors, eight out of 10 novice doctors did not place two orders concurrently, which may have been one reason why they may have more MC and longer MM. Correlation among performance measures Pearson correlation coefficient between performance measures was computed using all tasks between expert and novice doctors to determine if there were any relationships between the performance measures. Correlation coefficients between 0 and 0.3 are considered weak, those between 0.3 and 0.7 are moderate and coefficients between 0.7 and 1 are considered high [44]. For expert doctor group, strong correlations were observed in TOT vs. MC (r = 0.79), TOT vs. MM (r = 0.82) and MC vs. MM (r = 0.85). Weak correlations were observed: TOT vs. task success (r = 0.02),

35 30 25 20 15 10 5 0

Mouse movements (pixels)

T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 M Expert mean 5 2 2 2 0 2 25 16 7 15 8 1 22 11 17 13 3 6 9 5 Novice mean 7 2 2 1 1 1 31 12 9 15 7 2 13 14 23 18 2 6 6 6 35 000 30 000 25 000 20 000 15 000 10 000 5000 0

T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 Expert mean 5149 1194 1696 1450 4513 728 1897 6652 7279 1242 6315 2384 1990 1330 1529 1227 3126 5618 8546 Novice mean 9449 2333 2037 1772 2698 4749 2936 8047 7969 1474 7775 3030 1466 1470 2470 1572 938 6691 7263

Figure 2 Geometric mean values of mouse clicks and mouse movement were compared between the two doctor groups. A lower number of mouse clicks usually indicate higher performance and lower pixel values usually indicate higher performance for mouse movements. Expert doctors completed the tasks with slightly fewer mouse clicks than novice doctors did (five clicks, expert group vs. six clicks, novice group, P = 0.85) and showed slightly shorter mouse movements across the 19 tasks (5328 pixels, expert group vs. 6469 pixels, novice group, P = 0.47). The expert doctor group achieved fewer mouse clicks in eight tasks: tasks 1, 5, 7, 9, 12, 14, 15 and 16; higher mouse clicks in seven tasks: 4, 6, 8, 11, 13, 17 and 19; and the comparable number of clicks in three tasks: 2, 3 and 18. The expert doctors showed shorter mouse movements in 15 of 19 tasks: tasks 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16 and 18; and longer mouse movements in four tasks: tasks 5, 13, 17 and 19.

1158

© 2014 John Wiley & Sons, Ltd.

M.A. Clarke et al.

and MC vs. task success (r = 0.04). Negative correlations were observed between MM vs. task success (r = −0.01). For novice doctor group, strong correlations were observed in TOT vs. MC (r = 0.89), TOT vs. MM (r = 0.89) and MC vs. MM (r = 0.87). Weak correlations were observed: TOT vs. task success (r = 0.03), MC vs. task success (r = 0.05). Negative correlations were observed: MM vs. task success (r = −0.05). Overall, the number of MC and the length of MM increased as expert and novice doctors spent more time completing tasks. When expert and novice doctors’ MM became longer, tasks success rate decreased which show a negative association between the two metrics. This could mean that an increase in time, MC and MM did not guarantee that both doctor groups would successfully complete a task.

SUS After the usability test, six out of seven expert doctors and all 10 novice doctors completed the SUS. The SUS illustrated that novice doctors ranked the system’s usability at a mean of 68 (high marginal) and experts rated the system’s usability at a mean of 74 (acceptable). Two novice doctors and one expert doctor gave a score below 50 (not acceptable). This result may indicate that proficiency or length of time using the system does not affect the acceptance of the EHR by novice and expert doctors. Pearson correlation coefficient between task success rate, the most objective performance measure and participants’ individual SUS score demonstrates that participants task success had close to no relation to how user-friendly participants perceived the EHR system (r = 0.09).

Discussion Relation to prior studies While EHRs have many benefits, such as clinical improvement and greater efficiencies, there are a number of challenges generated from inadequate software design. In our study, there was no statistical difference found between expert and novice doctors’ performance measures, which disproves our hypothesis that expert doctors will be more competent than novice doctors when using the EHR. A study done by Kim et al. [35], identifying usability gaps between expert and novice emergency department nurses, found similar results. There was no statistical difference between expert and novice nurse groups’ task success rate on EHR use. Alternatively, a study by Avansino and Leu evaluating systematically developed clinical decision supports provide usability benefit found that participants collectively preferred using systematically developed order sets and rated the order sets from the study SUS scores of 75 (P < .05) [45]. Another study by Kjeldskov et al., conducting a usability evaluation with novice and expert nurses to identify usability issues when an electronic patient record system was being deployed, found that expert users were more effective than the novice users and solved significantly more tasks as expert subjects than as novice subjects (P = 0.034). Although expert nurses solved more tasks, there was no significant differences in the total task completion times between novice and expert users (P = 0.161) [46]. The results from our study suggest that although © 2014 John Wiley & Sons, Ltd.

EHR usability gaps between doctors

expert doctors have been using the system longer, their proficiency may not increase with EHR experience. How doctors perceived the usability of the system may have had an effect on the SUS scores provided by novice and expert doctors. The SUS scores from our study indicated that novice doctors were slightly less satisfied with the EHR than expert doctors [novice: 68 (high marginal), experts: 74 (acceptable)]. In a study by Haarbrandt et al., assessing perception of a health information exchange system, primary care providers gave the system a similar SUS score of 70.7 (acceptable). Although expert and novice participants found the graphical user interface of the system easy to use, SUS scores were still only acceptable [47]. SUS scores from Kim et al.’s study [35] demonstrated that novice nurses were not satisfied with their system [novice: 55 (marginal) to 43 (unacceptable)] compared with expert nurses who were satisfied [experts: 75 to 81 (excellent)]. Although there were no statistical differences among performance measures in both studies, the SUS scores in Kim et al., indicate that expert users’ perceived level of proficiency had an effect on how user-friendly the experts thought the EHR system was these results were unlike the results from our study where experts’ perceived level of proficiency had close to no relation with percent task success rate. However, novice users in both studies rated the system with a low usability score. Although the SUS survey was able to track major dissimilarities in task performance on an aggregate level, novice doctors might have given the system a lower score because of one complex task.

Limitations to this study This study had several limitations to our methodology. First, this study involves the doctors in one health care institution where only one EHR system is used. As such, the study’s findings may have limited generalizability to other ambulatory clinic settings because of the selection of different types of EHR applications and doctor practice settings. However, the EHR platform to be employed in this study is one of the top commercial products with significant market share. Second, a limited number of clinical tasks was used in the usability test and may not encompass other tasks completed by doctors in other clinical scenarios. Nevertheless, our study’s accomplishment in identifying usability issues constructs a good base for future studies to expand in other health care settings. Third, this study was conducted in a laboratory setting, which does not take into account common distractions doctors may experience during a clinical encounter. Nonetheless, lab-based usability tests allow for flexibility in questioning and give room for more in-depth probing. The direct observation in lab usability testing allows for interaction between participant and facilitator. Forth, the EHR allow doctors the option to tailor his or her interface to fit his or her needs by creating favourite lists, macros, etc., which participants were not able to access during the usability test. However, this may not have affected the results of the performance measures. Fifth, the sample size was small and consisted of residents from one health care institution, and may not be representative of all primary care practice. Although this study contains some methodological limitation, this is a well-controlled study using rigorous triangular evaluation and instructions were clear to the doctors, which allowed participants to complete the required tasks. 1159

EHR usability gaps between doctors

M.A. Clarke et al.

Conclusion Overall, this study was able to identify varying degrees of usability gaps between expert and novice doctors that may be impeding the use of EHRs. Dissatisfaction resulting in usability issues may cause doctors to resist using the EHR. The study suggests that higher experience levels with the EHR may not be equivalent to being an expert, proficient in using EHR. By collecting doctors’ interaction with the EHR, these results may be communicated to EHR vendors to assist in improving the user interface for doctors to use effectively. This study may also assist in the design of EHR education and training programmes by highlighting the areas of difficulty residents are currently facing. Residents in primary care are now offered exhaustive EHR training by their institutions. However, it is a great challenge finding enough time to train busy doctors. It is an arduous chore when attempting to target training specific to the needs of users and provide hands-on, on-site support [48]. There is scarcely any evidence-based guidelines for training residents effectively on how to use EHRs for patient care [49]. Thus, our study may also serve as a guideline to potentially improve the current EHR training programme, which may increase doctors’ performance, by improving competency when using the system. Future studies should include a larger sample of doctors and broaden the scope to specialty doctors.

References 1. Healthcare Information and Management Systems Society (n.d.) Electronic health record [Internet]. Chicago, IL. Available at: http:// www.himss.org/ASP/topics_ehr.asp (last accessed 9 April 2014). 2. Centers for Disease Control and Prevention (2012) Meaningful use introduction [Internet]. Atlanta, GA: Centers for Disease Control and Prevention. Available at: http://www.cdc.gov/ehrmeaningfuluse/ introduction.html (last accessed 9 April 2014). 3. Centers for Medicare and Medicaid Services (2012) Meaningful use. Baltimore, MD. Available at: http://www.cms.gov/Regulations-andGuidance/Legislation/EHRIncentivePrograms/Meaningful_Use.html (last accessed 9 April 2014). 4. Hsiao, C.-J. & Hing, E. (2012) Use and characteristics of electronic health record systems among office-based physician practices: United States, 2001–2012. NCHS Data Brief, (111), 1–8. Available at: http:// www.cdc.gov/nchs/data/databriefs/db143.pdf (last accessed 8 May 2014). 5. Jamoom, E., Beatty, P., Bercovitz, A., Woodwell, D., Palso, K. & Rechtsteiner, E. (2012) Physician adoption of electronic health record systems: United States, 2011. US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. 6. HealthIT.Gov (2014) What are the advantages of electronic health records?: HealthIT.Gov [Internet]. Available at: http:// www.healthit.gov/providers-professionals/faqs/what-are-advantageselectronic-health-records (last accessed 12 April 2014). 7. Menachemi, N. & Collum, T. H. (2011) Benefits and drawbacks of electronic health record systems. Risk Management and Healthcare Policy, 4, 47–55. [Epub 2012/02/09]. 8. Goldzweig, C. L., Towfigh, A., Maglione, M. & Shekelle, P. G. (2009) Costs and benefits of health information technology: new trends from the literature. Health Affairs, 28 (2), w282–w293. 9. Dainty, K. N., Adhikari, N. K. J., Kiss, A., Quan, S. & Zwarenstein, M. (2012) Electronic prescribing in an ambulatory care setting: a

1160

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

cluster randomized trial. Journal of Evaluation in Clinical Practice, 18 (4), 761–767. ISO 9241-11 (1998) Ergonomic requirements for office work with visual display terminals (VDTs): part 11: guidance on usability. 1 ed: International Organization for Standardization; 22 p. Committee on Identifying and Preventing Medication Errors (2006) Preventing Medication Errors. Washington, DC: The National Academies Press. Institute of Medicine (2001) Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press. Went, K., Antoniewicz, P., Corner, D. A., Dailly, S., Gregor, P., Joss, J., McIntyre, F. B., McLeod, S., Ricketts, I. W. & Shearer, A. J. (2010) Reducing prescribing errors: can a well-designed electronic system help? Journal of Evaluation in Clinical Practice, 16 (3), 556–559. Berger, R. G. & Kichak, J. P. (2004) Computerized physician order entry: helpful or harmful? Journal of the American Medical Informatics Association, 11 (2), 100–103. Ash, J. S., Stavri, P. Z. & Kuperman, G. J. (2003) A consensus statement on considerations for a successful CPOE implementation. Journal of the American Medical Informatics Association, 10 (3), 229–234. [Epub 2003/03/11]. Khajouei, R. & Jaspers, M. W. (2010) The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods of Information in Medicine, 49 (1), 3–19. [Epub 2009/07/08]. Schoen, C., Osborn, R., Huynh, P. T., Doty, M., Peugh, J. & Zapert, K. (2006) On the front lines of care: primary care doctors office systems, experiences, and views in seven countries. Health Affairs, 25 (6), w555–w571. Donaldson, M. S., Yordy, K. D., Lohr, K. N. & Vanselow, N. A. (1996) Primary Care: America’s Health in a New Era. Washington, DC: National Academies Press. National Center for Health Statistics (2010) Selected patient and provider characteristics for ambulatory care visits to physician offices and hospital outpatient and emergency departments: United States, 2009– 2010. FastStats – Diabetes. Hyattsville, MD: Centers for Disease Control and Prevention, National Center for Health Statistics. CDC/National Center for Health Statistics (2010) National Ambulatory Medical Care Survey: 2010 summary tables. CDC/National Center for Health Statistics. Health Capital Consultants (2011) Healthcare reform: impact on physicians. Health Capital Topics. HealthCapital.com: Health Capital Consultants; p. 3. Saleem, J. J., Patterson, E. S., Militello, L., Anders, S., Falciglia, M., Wissman, J. A., Roth, E. M. & Asch, S. M. (2007) Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. Journal of the American Medical Informatics Association, 14 (5), 632–640. Patterson, E. S., Doebbeling, B. N., Fung, C. H., Militello, L., Anders, S. & Asch, S. M. (2005) Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. Journal of Biomedical Informatics, 38 (3), 189–199. Mollon, B., Chong, J., Holbrook, A., Sung, M., Thabane, L. & Foster, G. (2009) Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Medical Informatics and Decision Making, 9 (1), 11. Yen, P. Y. & Bakken, S. (2012) Review of health information technology usability study methodologies. Journal of the American Medical Informatics Association, 19 (3), 413–422. [Epub 2011/08/11]. Bürkle, T., Ammenwerth, E., Prokosch, H.-U. & Dudeck, J. (2001) Evaluation of clinical information systems. What can be evaluated and what cannot? Journal of Evaluation in Clinical Practice, 7 (4), 373– 385.

© 2014 John Wiley & Sons, Ltd.

M.A. Clarke et al.

27. Li, A. C., Kannry, J. L., Kushniruk, A., Chrimes, D., McGinn, T. G., Edonyabo, D. & Mann, D. M. (2012) Integrating usability testing and think-aloud protocol analysis with ‘near-live’ clinical simulations in evaluating clinical decision support. International Journal of Medical Informatics, 81 (11), 761–772. [Epub 2012/03/30]. 28. Chan, J., Shojania, K. G., Easty, A. C. & Etchells, E. E. (2011) Usability evaluation of order sets in a computerised provider order entry system. BMJ Quality and Safety, 20 (11), 932–940. [Epub 2011/ 08/03]. 29. Harrington, L., Porch, L., Acosta, K. & Wilkens, K. (2011) Realizing electronic medical record benefits: an easy-to-do usability study. The Journal of Nursing Administration, 41 (7–8), 331–335. [Epub 2011/ 07/30]. 30. University of Missouri Health System (2013) University of Missouri health care achieves highest level of electronic medical record adoption. Columbia, MO. Available at: http://www.muhealth.org/ body.cfm?id=103&action=detail&ref=311 (last accessed 14 April 2014). 31. Health Information Management Systems Society Analytics (2011) US EMR adoption model trends. Chicago, IL. Available at: http:// www.himssanalytics.org/docs/HA_EMRAM_Overview_ENG.pdf (last accessed 21 February 2014). 32. University of Missouri Health System (2011) MU 2011 annual report. Available at: http://www.mydigitalpublication.com/publication/ ?i=106794 (last accessed 15 April 2012). 33. Battaglia, M. (2008) Convenience sampling. In Encyclopedia of Survey Research Methods (ed. P. Lavrakas), pp. 149–150. Thousand Oaks, CA: Sage Publications, Inc. 34. Barnum, C. (2003) The magic number 5: is it enough for web testing? Information Design Journal, 11, 160–170. 35. Kim, M., Shapiro, J., Genes, N., Aguilar, M., Mohrer, D., Baumlin, K. & Belden, J. A. (2012) Pilot study on usability analysis of emergency department information system by nurses. Applied Clinical Informatics, 3, 135–153. 36. Khajouei, R., Peek, N., Wierenga, P. C., Kersten, M. J. & Jaspers, M. W. (2010) Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. International Journal of Medical Informatics, 79 (10), 690–698. [Epub 2010/09/14]. 37. Koopman, R. J., Kochendorfer, K. M., Moore, J. L., Mehr, D. R., Wakefield, D. S., Yadamsuren, B., Coberly, J. S., Kruse, R. L., Wakefield, B. J. & Belden, J. L. (2011) A diabetes dashboard and

© 2014 John Wiley & Sons, Ltd.

EHR usability gaps between doctors

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

48.

49.

physician efficiency and accuracy in accessing data needed for highquality diabetes care. Annals of Family Medicine, 9 (5), 398–405. [Epub 2011/09/14]. Lewis, J. & Sauro, J. (2009) The factor structure of the system usability scale. In Human Centered Design (ed. M. Kurosu), pp. 94–103. Berlin Heidelberg: Springer. Tullis, T. & Albert, W. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: Morgan Kaufmann Publishers Inc. 336 p. Brooke, J. (1996) SUS-A quick and dirty usability scale. In Usability Evaluation in Industry (ed. W. J. Patrick, B. Thomas, I. L. McClelland & B. Weerdmeester), pp. 189–194. Boca Raton, FL: CRC Press. Bangor, A., Kortum, P. & Miller, J. (2009) Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies, 4 (3), 114–123. Van Someren, M. W., Barnard, Y. F. & Sandberg, J. A. (1994) The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. London: Academic Press. Cordes, R. E. (1993) The effects of running fewer subjects on timeon-task measures. International Journal of Human-Computer Interaction, 5 (4), 393–403. Calkins, K. (2005) Correlation coefficient [Internet]. Berrien Springs, MI: Andrew University. Available at: http://www.andrews.edu/ ~calkins/math/edrm611/edrm05.html (last accessed 22 April 2014). Avansino, J. & Leu, M. G. (2012) Effects of CPOE on provider cognitive workload: a randomized crossover trial. Pediatrics, 130 (3), e547–e552. Kjeldskov, J., Skov, M. B. & Stage, J. (2010) A longitudinal study of usability in health care: does time heal? International Journal of Medical Informatics, 79 (6), e135–e143. [Epub 2008/09/02]. Haarbrandt, B., Schwartze, J., Gusew, N., Seidel, C. & Haux, R. (2013) Primary care providers’ acceptance of health information exchange utilizing IHE XDS. Studies in Health Technology and Informatics, 190, 106–108. [Epub 2013/07/05]. Terry, A. L., Thorpe, C. F., Giles, G., Brown, J. B., Harris, S. B., Reid, G. J., Thind, A. & Stewart, M. (2008) Implementing electronic health records: key factors in primary care. Canadian Family Physician, 54 (5), 730–736. [Epub 2008/05/14]. Peled, J. U., Sagher, O., Morrow, J. B. & Dobbie, A. E. (2009) Do electronic health records help or hinder medical education? PLoS Medicine, 6 (5), e1000069. [Epub 2009/05/13].

1161

Copyright of Journal of Evaluation in Clinical Practice is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.