2015, 37: 281–288
Medical school handoff education improves postgraduate trainee performance and confidence* JENNIFER N. STOJAN1, JOCELYN HUANG SCHILLER1, PATRICIA MULLAN1, JAMES T. FITZGERALD2, JENNIFER CHRISTNER3, PAULA T. ROSS1, SARAH MIDDLEMAS1, HILLARY HAFTEL1, R. BRENT STANSFIELD1 & MONICA L. LYPSON1 1
University of Michigan, USA, 2VA Ann Arbor Healthcare System, USA, 3State University of New York at Syracuse, USA
Abstract Objectives: Determine postgraduate first-year (PGY-1) trainees ability to perform patient care handoffs and associated medical school training. Methods: About 173 incoming PGY-1 trainees completed an OSCE handoff station and a survey eliciting their training and confidence in conducting handoffs. Independent t-tests compared OSCE performance of trainees who reported receiving handoff training to those who had not. Analysis of variance examined differences in performance based on prior handoff instruction and across levels of self-assessed abilities, with significance set at p50.05. Results: About 35% of trainees reported receiving instruction and 51% reported receiving feedback about their handoff performance in medical school. Mean handoff performance score was 69.5%. Trainees who received instruction or feedback during medical school had higher total and component handoff performance scores (p50.05); they were also more confident in their handoff abilities (p50.001). Trainees with higher self-assessed skills and preparedness performed better on the OSCE (p50.05). Conclusions: This study provides evidence that incoming trainees are not well prepared to perform handoffs. However, those who received instruction during medical school perform better and are more confident on standardized performance assessments. Given communication failures lead to uncertainty in patient care and increases in medical errors, medical schools should incorporate handoff training as required instruction.
Introduction
Practice points
The American Council for Graduate Medical Education (ACGME) issued a mandate in 2003 requiring that training programs in the United States restrict postgraduate trainee work hours to 80 h a week, with further restrictions in 2011 (ACGME 2014). The European Working Time Directive instituted a similar mandate, limiting the number of hours that junior medical staff could work in hospitals to 48 h a week (UK Parliament 1998). As postgraduate training programs complied and restructured their programs to reduce trainees’ hospital work hours, the number of patient handoffs, defined as the exchange of patient information between health care providers accompanying a transfer of care, subsequently increased (Horwitz et al. 2006; Mir et al. 2011; Antiel et al. 2013). With that, communication failures led to uncertainty in patient care decisions (Arora et al. 2005) and a perceived increase in medical errors (Borman et al. 2012; Shea et al. 2012). In 2007, the Joint Commission’s Report on Quality and
Incoming PGY-1 trainees are expected to be competent in performing handoffs at the start of their postgraduate training, yet only 35% of trainees reported receiving instruction on handoffs. Trainees who received instruction or feedback had significantly higher confidence in their handoff abilities (p50.0001). Trainees with higher self-assessed skills and perceived preparedness performed better on the OSCE. Trainees who received instruction during medical school had higher handoff performance scores. Given that communication failures lead to uncertainty in patient care and a perceived increase in medical errors, medical schools should incorporate handoff training prior to graduation.
Correspondence: Jennifer N. Stojan, MD, Department of Internal Medicine and the Department of Pediatrics, University of Michigan, 3119 Taubman Center, SPC5376, 1500 East Medical Center Drive, Ann Arbor, MI 48109, USA. E-mail: [email protected]
*Previous presentations: Pediatric Academic Society Meeting May 2013 Poster Presentation, Association of Medical Education in Europe Conference in August 2013 Poster Presentation. ISSN 0142-159X print/ISSN 1466-187X online/15/030281–8 ß 2015 Informa UK Ltd. DOI: 10.3109/0142159X.2014.947939
281
J. N. Stojan et al.
Safety identified lapses in communication as the leading root cause of sentinel events (Joint Commission 2007). In recognition of this problem, the Joint Commission and the World Health Organization issued Patient Safety Solutions recommending that health-care organizations implement a standardized approach to handoff communication and incorporate handoff training into their educational and continuing professional development curriculum (WHO 2007). Most of the published research on patient handoffs focuses on the assessment of handoff skills and curriculum development during postgraduate training (Horwitz et al. 2007; Chu et al. 2009; Gakhar & Spencer 2010; Bump et al. 2012), which means that training occurs after these postgraduate firstyear (PGY-1) trainees have already been performing handoffs as part of their patient care responsibilities. Although the need for handoff training and assessment during postgraduate training is critical, consideration of the potential impact of handoff training during medical school is also warranted, given that without adequate preparation, it is difficult to expect incoming PGY-1 trainees to be both competent and confident in performing handoffs at the start of their postgraduate training. Albeit an older study, the last published study in 2006 showed that only 8% of medical schools in the United States formally taught students how to perform handoffs in preparation for their postgraduate training (Solet et al. 2005). PGY-1 trainees, however, need to be able to care for patients on the first day of their postgraduate training and, therefore, waiting to institute a formal handoff curriculum until after this point may compromise patient safety. Postponing instruction may also lead to unnecessary stress, as it has been found that junior doctors feel unprepared to perform handoffs and have described their early handoff experiences negatively (Cleland et al. 2009).
Research questions (1) Do trainees who receive instruction or feedback during medical school perform better on a standardized patient handoff OSCE? (2) Are trainees who receive instruction or feedback during medical school more confident in their abilities to perform a handoff? (3) Do trainees who have higher self-assessed skills and preparedness perform better on a standardized handoff OSCE?
Methods Setting Since 2002, all incoming PGY-1 trainees at the University of Michigan Health System participate in a Postgraduate Orientation Assessment (POA) at their required initial orientation (Lypson et al. 2004). The POA is an Objective Structured Clinical Examination (OSCE) that provides a baseline assessment of clinical skills, communication skills, and general medical knowledge, using four standardized patient stations and five computer stations (Lypson et al. 2010a,b). Each of the nine POA stations’ scores was normalized within year to control for possible differences in difficulty and measurement 282
variance from year to year over a 10-year period. Four sub-scores were computed as un-weighted means of these normalized station scores. The verbal handoff is included in the communication sub-score. The five stations included in the communication sub-score, which include the verbal handoff, had Cronbach’s alpha of 0.42. A stepwise multiple regression of the communication sub-score by USMLE Step 1 and Step 2 (CK) scores provides evidence of validity: communication subscores were not predicted by Step 1 scores (p ¼ 0.952) but were positively predicted by Step 2 scores (p ¼ 0.048), which is what one expects from a valid measure of communication skills. The standardized patient handoff station became part of the POA in 2009. As the PGY-1 trainees are informed, the POA evaluation results are not used to assess individual trainees; rather, they are used to inform program directors of the profile of demonstrated competencies and training needs of entering PGY-1 trainee cohorts. In addition, trainees provide demographic information and respond to questions assessing their knowledge and skills on the topics covered in the POA. This study received educational exemption status from the Institutional Review Board at the University of Michigan.
Participants As part of our POA assessment, all (189) PGY-1 trainees in 2012 entering 18 different specialties from 73 different medical schools participated in a handoff station. Our study is based on the responses of the 173 (173/189 ¼ 91%) interns who completed both the structured performance assessments and the surveys eliciting their handoff training, knowledge, and confidence. The distribution of participants among demographic categories and medical specialties is summarized in Table 1. Trainee’s specialties are shown for ACGME-defined categories, which define medical, hospital-based, and surgicalbased specialties (ACGME 2014). Medical specialties included family medicine, internal medicine, pediatrics, combined internal medicine/pediatrics, psychiatry and pediatric neurology. Surgical specialties included obstetrics and gynecology, general surgery, plastic surgery, neurosurgery, urology, vascular surgery, orthopedic surgery, otolaryngology and oral/ maxillofacial surgery. Hospital-based specialties included anesthesiology, emergency medicine and pathology.
Assessment procedures and measures During the handoff station, the PGY-1 trainee reviewed a simulated patient’s written history and physical and watched a two-minute video demonstrating the patient’s clinical course. Trainees participated in a patient case appropriate for their medical specialty; i.e., internal medicine, surgery, or pediatrics. The trainee was required to synthesize and integrate the information from the video and the history and physical, and give a verbal handoff of the patient to a trained evaluator playing the role of the ‘‘physician assuming care of the patient.’’ All 24 trained evaluators were upper level postgraduate trainees recruited to work with their own specialty’s’ PGY-1 trainees. They received standardized training, including a review of prior POA performance videos of handoffs to standardize their assessments. After receiving the trainees’
Handoff education improves performance
handoff, the evaluator gave immediate verbal feedback to the trainee, and rated the trainee’s performance using a structured assessment tool adopted from existing research (Farnan et al. 2010). Of note, the raters were blinded to the presence or absence of education about handoffs that trainees received during medical school.
Instruments: Performance assessment protocol Table 2 provides a summary of the content and scoring rubric for the handoff performance assessment tool. This assessment tool was adapted from Horwitz and Farnan (Horwitz et al. 2007; Farnan et al. 2010). To ensure that the items on the tool represented the constructs we intended to evaluate, published Table 1. Demographic profiles of incoming PGY-1 trainees participating in the patient care handoff assessment.
Characteristics
N (%)
Gender Male Female
106 (62%) 66 (38%)
Race Caucasian African American Asian/Pacific Islander Hispanic Other
128 (75%) 5 (3%) 30 (18%) 7 (4%) 1 (1%)
Specialty Medical Surgical Hospital based
88 (51%) 41 (24%) 44 (25%)
Table 2. Content of the patient handoff performance assessment tool.
Content – patient history (items 1–8) 1. Identification of sick patients and code status 2. Identifier: Example: Sue Smith is a 56-year-old female, with hypertension, coronary heart disease, and high cholesterol, who presented with syncope 3. Pertinent past medical history 4. General hospital course 5. New events of the day and active issues 6. Overall health status and relevant physical exam findings 7. Relevant lab and study results 8. Medications and allergies Clinical judgment (items 9 and 10) 9. Upcoming possibilities with plan and supporting rationale 10. To do items for overnight with plan Communication and professionalism (items 11–15) 11. Allowing for questions 12. Appropriateness of the setting that the handoff exchange took place in 13. Organization and efficiency 14. Communication skills 15. Professionalism and attitude Total score (sum of ratings for items 1–15) global performance score Items 1–15 were scored on a scale that ranged from 0to 2, with 0 ¼ item was not done, 1 ¼ item was done but needed improvement, and 2 ¼ item was done well. Total Score range of possible scores was from 0 to 30. Global Performance Score was scored on a scale that ranged from 1 to 5, with 1 ¼ novice, 3 ¼ competent and 5 ¼ expert.
literature on handoff curricula was reviewed. The tool identified assessment criteria, combining components of the SIGNOUT mnemonic (Horwitz et al. 2007), which assessed content and clinical judgment, with items assessing communication skills and professionalism (Farnan et al. 2010). Operational definitions of scale points were also developed; ‘‘0’’ indicated an item was not done, ‘‘1’’ indicated an item was done, but needed improvement and a score of ‘‘2’’ indicated an item was done well. In addition, the evaluator assigned a global performance score based on a scale ranging from 1 to 5, with ‘‘1’’ representing novice performance, ‘‘3’’ representing competence, and ‘‘5’’ representing expertise. Feedback was solicited and incorporated from stakeholders including faculty, residents, and students who piloted the tool during actual patient handoffs.
Additional assessment instruments: Previous training, knowledge, and self-assessed confidence related to patient handoffs Following completion of the OSCE, PGY-1 trainees completed an online survey with questions pertaining to the POA stations. The questions reported in this study primarily focused on PGY-1 trainees’ characterization of having received formal training (with Likert-format responses) or feedback (with ‘‘yes/no’’ responses) about patient handoffs during their medical school training and their knowledge, self-assessment, and comfort level regarding handoffs. An expert panel, consisting of faculty from the departments of pediatrics, internal medicine, and medical education, with experience in clinical care, medical education, survey design, and patient safety, reviewed relevant published literature (Horwitz et al. 2007; Chu et al. 2010; Gakhar & Spencer 2010), developed questions, and elicited and incorporated feedback from stakeholders into the final survey (Table 3).
Statistical analysis Descriptive statistics were calculated for demographic variables, survey question responses, and the assessment of directly observed OSCE handoff performance. OSCE scores included a total score (items 1–15) and three sub-scores: content (items 1–8); clinical judgment (items 9 and 10); and global performance score. Independent t-tests were used to compare performance of PGY-1 trainees who had received feedback during medical school about handoffs to PGY-1 trainees who had not received feedback during medical school; effect sizes (Cohen’s d) were calculated to guide interpretation of the magnitude of the differences. Analysis of variance (ANOVA) was used to examine differences in performance of trainees across groups constituted by: students’ ordinal ratings of handoff instruction during medical school; their ordinal ratings of their self-reported confidence; race; and ACGME-categorized medical specialties. Chi-square analysis compared levels of ordinal-level self-reported confidence of trainees who had received prior handoff training and feedback during medical school to those who had not. JMP Pro 10.0.0 (SAS Inc., Cary, NC) was the statistical program used for these analyses.
283
J. N. Stojan et al.
Results Table 4 summarizes PGY-1 trainees’ mean performances on the OSCE handoff station. The mean total score was 69.5%. The mean content and clinical judgment scores were 68% and 69%, respectively. The mean global performance score was 2.9, which corresponded to the trained rater’s characterization of observed performance as slightly less than competent. The results of trainees’ responses to the handoff survey are summarized in Table 3. Approximately one-half (51%) of the PGY-1 trainees indicated that they had received feedback about their handoff performance in medical school, but only 35% responded that they were formally taught how to perform a handoff as a medical student. Eighty-three percent of respondents felt that they could communicate all the necessary information in a handoff and 71% of respondents felt that they were prepared to perform handoffs as a PGY-1 trainee. Ninetysix percent of trainees correctly identified lapses in communication as the leading root cause of sentinel events. PGY-1 trainees who reported that they had received feedback about their handoff performance or who received instruction on how to perform a handoff during their medical school education had higher confidence about their ability and Table 3. PGY-1 trainees’ medical school training, knowledge, and self-reported confidence associated with patient care handoffs.
Survey questions
N ¼ 173
Medical school training Have you ever received feedback about your handoff performance
Yes, 51%
How to perform handoffs is taught in my medical school Self-assessed confidence I can communicate all the information that is needed in a handoff I can communicate all the information that is needed in a handoff in an efficient manner I am prepared to perform handoffs as a PGY-1 trainee Objective Knowledge Test Items Cross covering providers provide better patient care when a standardized format is used What is the leading root cause of sentinel events?
Agree/strongly agree 35%
83% 63%
71%
preparedness with regards to transitions of care. Trainees who had received feedback on handoffs during medical school felt more prepared to perform a complete (p ¼ 0.007) and efficient handoff (p ¼ 0.023). Trainees who reported that they received instruction in medical school on how to perform handoffs were more likely to feel prepared to perform handoffs as a PGY-1 trainee (p50.001). They were also more likely to feel that they could perform a complete (p50.001) and efficient handoff (p50.001). Table 5 summarizes the mean performance scores of PGY-1 trainees in the context of the residents’ self-reported medical school training and current confidence related to patient handoff care responsibilities. Significant differences were found among those who received feedback on their handoff performance in medical school compared with those who did not in content (p ¼ 0.041), total (p ¼ 0.041), and global performance scores (p ¼ 0.035), with associated effect sizes of d ¼ 0.27, 0.27, and 0.25, but not in clinical judgment scores. Those who were taught how to perform handoffs in medical school performed better on content (p ¼ 0.003), clinical judgment (p ¼ 0.005), total (p ¼ 0.001), and global performance scores (p50.001). Similarly, PGY-1 trainees who felt that they could communicate all the information needed in a handoff were accorded higher content (p ¼ 0.018), clinical judgment (p ¼ 0.029), total (p ¼ 0.009), and global performance scores (p ¼ 0.009). Those who characterized themselves as having been prepared to perform a handoff as a PGY-1 trainee also received higher content (p ¼ 0.011), clinical judgment (p ¼ 0.022), total (p ¼ 0.011), and global performance scores (p ¼ 0.023). There were no significant differences in the communication and professionalism scores based on self-reported medical school training and confidence. In terms of differences in performance across PGY-1 trainees, no differences were found between races. As illustrated in Table 4, there were also no significant differences in content, total, or global performance scores among ACGMEdefined specialties. However, some differences were found among the specialties in clinical judgment scores, with medical specialties scoring 61.4%, while hospital-based specialties scored 65.3% and surgical specialties scored 51.8% (p ¼ 0.009).
95%
Communication 96%
Discussion An unintended consequence of restricting postgraduate trainee duty hours has been increasing the number of patient care
Table 4. Overall and medical specialty performance of the PGY-1 trainees’ on the patient-handoff assessment.
Performance assessment domain Content (items 1–8) Clinical judgment (Items 9 and 10)a Total score (items 1–15) Global Performance Score (score range 1–5) a
Combined trainees (n ¼ 173), mean % (SD) 68.0 60.1 69.5 2.9
(16.6) (21.2) (14.4) (0.8)
Medical specialties (n ¼ 88), mean % (SD) 70.9 61.4 71.4 2.9
(16.5) (20.4) (14.0) (0.7)
Surgical specialties (n ¼ 41), mean % (SD) 64.5 51.8 66.7 2.7
(15.3) (16.2) (13.1) (0.8)
Hospital-based specialties (n ¼ 44), mean % (SD) 65.6 65.3 68.5 3.0
(17.3) (24.8) (15.9) (1.0)
p ¼ 0.009. One-way analysis of variance (with Tukey–Kramer HSD, global p ¼ 0.05) was used to compare differences in mean scores among medical, surgical, and hospitalbased specialties for all scores. The only significant difference that emerged was for the clinical judgment score; the surgical specialties scored lower than the medical and hospital-based specialties.
284
Handoff education improves performance
Table 5. Patient handoff performance results across PGY-1 trainees’ level of medical school training and current self-reported confidence in patient handoff competencies.
Survey question Have you ever received feedback about your handoff performance? (n) Yes (89) No (84) How to perform handoffs is taught in my medical school (n) Strongly disagree (18) Disagree (50) Neutral (44) Agree (49) Strongly agree (12) I can communicate all the information that is needed in a handoff (n) Strongly disagree (2) Disagree (2) Neutral (25) Agree (123) Strongly agree (21) I can communicate all the information that is needed in a handoff in an efficient manner (n) Strongly disagree (3) Disagree (14) Neutral (47) Agree (96) Strongly agree (13) I am prepared to perform a handoff as a PGY-1 trainee (n) Strongly disagree (2) Disagree (6) Neutral (42) Agree (113) Strongly agree (10)
Content (items 1–8), mean (SD)
Clinical judgment (items 9 and 10), mean (SD)
Total score (items 1–15) mean (SD)
Global performance score, mean (SD)
70.2 (15.8) 65.8 (17.2) p ¼ 0.041*
60.1 (20.9) 60.1 (21.6) p ¼ 0.500
71.4 (13.2) 67.6 (15.3) p ¼ 0.041*
3.0 (0.8) 2.8 (0.8) p ¼ 0.035*
63.5 (15.0) 66.1 (15.6) 62.9 (18.1) 74.4 (13.8) 75.5 (19.3) p ¼ 0.003*
52.8 (22.5) 58.0 (18.5) 55.7 (20.8) 64.8 (21.0) 77.1 (22.5) p ¼ 0.005*
64.8 (12.6) 67.7 (13.4) 65.3 (15.5) 74.8 (12.0) 78.1 (17.0) p ¼ 0.001*
2.6 (0.7) 2.9 (0.8) 2.5 (0.7) 3.2 (0.8) 3.5 (1.0) p50.001*
43.8 (17.7) 53.1 (22.1) 64.0 (14.1) 68.2 (16.8) 75.6 (14.1) p ¼ 0.018*
25.0 (35.4) 62.5 (17.7) 59.0 (23.8) 59.2 (19.4) 70.2 (23.2) p ¼ 0.029*
45.0 (11.8) 60.0 (23.6) 65.6 (13.7) 69.8 (14.1) 76.2 (13.1) p ¼ 0.009*
1.5 (0.7) 2.5 (0.7) 2.6 (0.8) 2.9 (0.8) 3.3 (0.9) p ¼ 0.009*
47.9 (14.4) 63.0 (12.4) 64.0 (16.6) 70.3 (16.9) 76.0 (11.4) p ¼ 0.009*
33.3 (28.9) 57.1 (18.2) 56.4 (18.4) 62.2 (21.8) 67.3 (23.7) p ¼ 0.060
50.0 (12.0) 66.2 (12.7) 65.9 (14.0) 71.5 (14.2) 76.2 (12.8) p ¼ 0.006*
1.7 (0.6) 2.8 (0.7) 2.7 (0.8) 3.0 (0.8) 3.4 (1.0) p ¼ 0.005*
53.1 (4.4) 69.8 (13.4) 61.3 (14.6) 69.9 (16.9) 76.9 (16.2) p ¼ 0.011*
50.0 (0.0) 66.7 (25.8) 51.2 (21.4) 62.6 (20.1) 67.5 (23.7) p ¼ 0.0221*
56.7 (4.7) 70.6 (15.4) 63.8 (13.0) 71.2 (14.2) 77.3 (15.4) p ¼ 0.011*
2.0 (0.0) 2.8 (0.8) 2.6 (0.8) 3.0 (0.8) 3.3 (0.8) p ¼ 0.023*
*p50.05. One-way analysis of variance was used to determine differences by response category for the (five) questions with five Likert-format response options. For the ‘‘yes/ no’’ question about feedback during medical school, an independent t test (one-tailed) was used.
handoffs. This increase in handoffs poses a potential threat to patient safety. Most existing educational interventions represented in current research have focused on training and assessment during postgraduate training (Horwitz et al. 2008; Chu et al. 2009; Gakhar & Spencer 2010; Bump et al. 2012). Of note, even within that scope, an inter-institutional patient handoff education working group recently characterized their need to formulate a curriculum blueprint for patient handoff education as warranted, given that, prior to their working group’s efforts, no one had ‘‘developed a comprehensive approach to teaching and assessing residents’ competence in patient handoffs (Wohlauer et al. 2012).’’ Our study builds on and extends current understanding of the need for and potential impact of patient handoff training. Others have shown that trained students and residents are more confident in their skills (Arora et al. 2013; Didwania et al. 2013). This study further provides empirically based insights derived from incoming residents’ directly observed performance on a standardized handoff assessment, as well as these trainees’ characterization of their ability and preparation they received during medical school for patient handoff responsibilities they reported having received.
We found that only 35% of incoming PGY-1 trainees reported receiving any instruction about handoffs during their medical school education and only 51% reported receiving feedback about their handoff performance. Trainees who had received instruction in medical school performed better than their peers who had not received instruction on objective performance assessments of their handoff content, clinical judgment, total scores, and global performance scores. In addition to their superior performance scores, trainees who had received training about handoffs were more confident. Similarly, residents who had received feedback during medical school about their handoff performance had significantly higher content scores, total scores, and global performance scores. The group who received feedback, however, did not have superior clinical judgment scores. This may reflect that receiving feedback is not equivalent to a formal medical handoff curriculum. If, for example, feedback was specific to the content of the handoff, clinical judgment may not have been addressed. Existing research seeking to evaluate the outcome of medical education training has been criticized for its dependence on trainees’ self-assessed confidence and competence.
285
J. N. Stojan et al.
Reservations often note that self-assessment has been shown to be a predictor of questionable reliability as it pertains to individual performance (Davis et al. 2006). It is difficult to trust that individuals who have not yet practiced extensively in their field can adequately judge the success with which an educational program has prepared them for such practice (Peterson et al. 2012). Our study places reports of trainees’ selfassessments in the context of these residents’ reflections about root causes of adverse outcomes and their own preparedness for patient care handoff practice, informed by their actual performance and feedback by trained raters who directly observe their handoff performance on a structured clinical simulation performance task. Our study found that despite the relative absence of handoff education during their medical school training, entering PGY-1 trainees clearly recognized the importance of communication with regards to patient safety, as indicated by their ability to identify that communication lapses were the leading root cause of sentinel events in the hospital. However, while the majority of entering PGY-1 trainees in our institution’s study of trainees across medical specialties lack formal training, they characterize themselves as confident in their ability to communicate patient information. Despite their confidence, trainees as a whole scored a mean of 69.5% on the OSCE total score and elicited trained observers’ characterization of their global performance as slightly less than competent. Our finding of over-confidence is consistent with existing research showing that 60% of the time the PGY-1 trainee giving a handoff did not communicate the most vital piece of patient information during that handoff, despite that trainee believing that it was communicated (Chang et al. 2010). This discrepancy has major implications for patient safety as it has been shown that inadequate handoffs can result in delays in diagnosis or treatment, intensive care unit transfers, near misses and inefficiencies or redundancies in work (Horwitz et al. 2008). Although the trainees in our study were, as a whole, overly confident in their abilities to perform a handoff, there was a statistically significant difference substantiating greater confidence and greater abilities for trainees who had received feedback instruction. In our study, residents receiving any type of feedback or handoff instruction in medical school demonstrated both better performance and higher confidence. Our findings of more confidence in performing handoffs following medical school instruction is consistent with prior research studies on handoff education (Klamen et al. 2009; Chu et al. 2010; Farnan et al. 2010). Eva and Regeher caution that attention to self-assessment as an end in itself is, at best misguided, and instead needs to be considered in terms of motivating learning and self-directed assessment seeking (Eva et al. 2008). Our study adds evidence of the long-term impact of training, as demonstrated by the trainees’ performance. As a recent review of educational interventions to improve nontechnical skills indicated, further research is needed to explicate the underlying theoretical frameworks, replicable instructional methods, and extent of the impact of these educational interventions (Gordon et al. 2012). Limitations of this study include that it was performed at a single institution, which may limit the extent to which its 286
findings can be generalized to other study sites. The observed handoff performance was on one OSCE station, conducted as part of a required assessment for all entering first-year residents at the study institution (Lypson et al. 2004). The potential differences in the difficulty of the specialty-specific scenarios constitute a study limitation. The source of information characterizing trainees’ instruction and feedback about handoff performance during medical school depended on the residents’ self-report, and may then have been subject to recall bias. Nonetheless, it does represent a full training cohort representing a diverse group of 173 PGY-1 trainees, coming from 73 different medical schools and entering 18 different specialties. The large number of trainees provided sufficient power for us to identify differences between those who had training compared with those who had not, and to explore the generalizability of these findings across disciplines, establishing that there were no differences among medical specialties, surgical specialties, and hospital-based specialties in content, total, and global performance scores. The sole difference found between specialties was in the clinical judgment score, a finding that warrants further study. The paucity of formal handoff instruction and research in an era where handoffs are ubiquitous and in a field where failures in communication can have such a devastating human impact must be addressed. As a profession, physicians cannot expect PGY-1 trainees to have acquired the necessary skills in communication if these skills were not formally taught as part of the medical school curriculum. In addition, it should not be expected that PGY-1 trainees would master these skills through trial and error during their postgraduate training while they are already engaged in patient care. Therefore, with the changes in work hours and the accompanying increase in patient handoffs, physician educators must take notice of the Joint Commission’s and the World Health Organization’s recommendations to implement handoff curriculum into medical education and adopt a standardized approach. Further research will then be necessary to determine how to incorporate handoff education into the medical school curriculum and to evaluate the long-term impact of this curriculum on handoff performance and patient safety.
Glossary Handoff: Transfer of patient care responsibility information
Conclusions Despite the increase in the number of care transitions after the residency work restriction mandates, the majority of medical schools in the United States do not appear to be preparing their medical students for the reality of postgraduate training as evidenced by the relative lack of formalized instruction on handoffs in undergraduate medical education curricula. It is encouraging to confirm that PGY-1 trainees who were taught how to perform a handoff in medical school had higher performance scores, based on the direct observations of
Handoff education improves performance
trained raters blinded to the trainees’ preparation during medical school. Trainees who received instruction were also more confident in their ability to perform handoffs as incoming trainees. Medical schools should consider making this important patient safety instruction a mandatory part of the later phase of the curriculum.
Notes on contributors JENNIFER N. STOJAN, MD, is an Assistant Professor in the Department of Internal Medicine and the Department of Pediatrics, University of Michigan Medical School, Ann Arbor, MI. JOCELYN HUANG SCHILLER, MD, is an Associate Professor in the Department of Pediatrics, University of Michigan Medical School, Ann Arbor, MI. PATRICIA MULLAN, PhD, is a Professor in the Department of Medical Education, University of Michigan Medical School, Ann Arbor, MI. JAMES T. FITZGERALD, PhD, is a Professor in the Department of Medical Education, University of Michigan Medical School, Ann Arbor, Michigan and the Associate Director of Education and Evaluation at the Geriatric Research, Education and Clinical Center, VA Ann Arbor Healthcare System, Ann Arbor, MI. JENNIFER CHRISTNER, MD, is an Associate Professor of Pediatrics in the Department of Pediatrics and Associate Dean for Undergraduate Medical Education at SUNY Upstate College of Medicine, Syracuse, NY. PAULA T. ROSS, PhD, is a Project Manager, University of Michigan Medical School, Ann Arbor, MI. SARAH MIDDLEMAS, MPH, is at the University of Michigan Medical School, Ann Arbor, MI. HILARY M. HAFTEL, MD, MHPE, is a Professor of Pediatrics, Internal Medicine, and Medical Education and Associate Chair for Education in the Department of Pediatrics, University of Michigan Medical School, Ann Arbor, MI. R. BRENT STANSFIELD, PhD, is an Assistant Professor in the Department of Medical Education, University of Michigan Medical School, Ann Arbor, MI. MONICA L. LYPSON, MD, MHPE, is a Professor of Internal Medicine & Medical Education and Assistant Dean of Graduate Medical Education, University of Michigan Medical School, Ann Arbor, MI.
Declaration of interest: The authors report no declarations of interest.
References Accreditation Council for Graduate Medical Education. 2014. ACGME Program and Institutional Guidelines. [Accessed 6 June 2014] Available from https://www.acgme.org/acgmeweb/tabid/158/Programand InstitutionalAccreditation. Antiel RM, Reed DA, Van Arendonk KJ, Wightman SC, Hall DE, Porterfield JR, Horvath KD, Terhune KP, Tarpley JL, Farley DR. 2013. Effects of duty hour restrictions on core competencies, education, quality of life, and burnout among general surgery interns. JAMA Surg 16:1–8. Arora VM, Eastment MC, Bethea ED, Farnan JM, Friedman ES. 2013. Participation and experience of third-year medical students in handoffs: Time to sign out? J Gen Intern Med 28(8):994–998. Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. 2005. Communication failures in patient sign-out and suggestions for improvement: A critical incident analysis. Qual Saf Health Care 14(6):401–407.
Borman KR, Jones AT, Shea JA. 2012. Duty hours, quality of care, and patient safety: General surgery resident perceptions. J Am Coll Surg 215(1):70–77. Bump GM, Bost JE, Buranosky R, Elnicki M. 2012. Faculty member review and feedback using a sign-out checklist: Improving intern written signout. Acad Med 87(8):1125–1131. Chang V, Arora V, Lev-Ari S, D’Arcy M, Keysay B. 2010. Interns overestimate the effectiveness of their handoff communication. Pediatrics 125(3):491–496. Chu ES, Reid M, Burden M, Mancini D, Schulz T, Keniston A, Sarcone E, Albert RK. 2010. Effectiveness of a course designed to teach handoffs to medical students. J Hosp Med 5(6):344–348. Chu ES, Reid M, Schulz T, Burden M, Mancini D, Ambardekar AV, Keniston A, Albert RK. 2009. A structured hand-off programme for interns. Acad Med 84(3):347–-352. Cleland JA, Ross S, Miller SC, Patey R. 2009. ‘‘There is a chain of Chinese whispers . . .’’: Empirical data support the call to formally teach handover to prequalification doctors. Qual Saf Health Care 18:267–271. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. 2006. Accuracy of physician self-assessment compared with observed measures of competence: A systemic review. JAMA 296(9):1094–1102. Didwania A, Kriss M, Cohen ER, McGaghie WC, Wayne DB. 2013. Internal medicine postgraduate training and assessment of patient handoff skills. J Grad Med Educ 5(3):394–398. Eva KW, Regehr G. 2008. ‘‘I’ll never play professional football’’ and other fallacies of self-assessment. J Continuing Educ Health Prof 28(1):14–19. Farnan JM, Paro JA, Rodriguez RM, Reddy ST, Horwitz LI, Johnson JK, Arora VM. 2010. Hand-off education and evaluation: Piloting the observed simulated hand-off experience (OSHE). J Gen Intern Med 25(2):129–134. Gakhar B, Spencer AL. 2010. Using direct observation, formal evaluation, and an interactive curriculum to improve the sign-out practices of internal medicine interns. Acad Med 85(7):1182–1188. Gordon M, Darbyshire D, Baker P. 2012. Non-technical skills training to enhance patient safety: A systematic review. Med Educ 46(11):1042–1054. Horwitz LI, Krumholz HM, Green MI, Huot SJ. 2006. Transfers of patient care between house staff on the internal medicine wards: A national survey. Arch Intern Med 166:1173–1176. Horwitz LI, Moin T, Green ML. 2007. Development and implementation of an oral sign-out skills curriculum. J Gen Intern Med 22(10):1470–1474. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. 2008. Consequences of inadequate sign-out for patient care. Arch Intern Med 168(16):1755–1760. Joint Commission. 2007. Improving America’s hospitals: The Joint Commission’s annual report on quality and safety 2007. [Accessed 6 June 2014] Available from http://www.jointcommission.org/assets/1/6/ 2007_Annual_Report.pdf. Klamen DL, Reynolds KL, Yale B, Aiello M. 2009. Students learning handovers in a simulated in-patient unit. Med Educ 43(11):1097–1098. Lypson ML, Frohna JG, Gruppen L, Woolliscroft JO. 2004. Assessing residents’ competencies at baseline: Identifying the gaps. Acad Med 79(6):564–570. Lypson ML, Hamstra SJ, Ross PT, Gruppen LD, Colletti LM. 2010a. An assessment tool for aseptic technique in resident physicians: A journey towards validation in the real world of limited supervision. J Grad Med Educ 2(1):85–89. Lypson ML, Ross PT, Hamstra SJ, Haftel HM, Gruppen LD, Colletti LM. 2010b. Evidence for increasing diversity in graduate medical education: The competence of underrepresented minority residents measured by an intern objective structured clinical examination. J Grad Med Educ 2(3):354–359. Mir HR, Cannada LK, Murray JN, Black KP, Wolf JM. 2011. Orthopaedic resident and program director opinions of resident duty hours: A national survey. J Bone Joint Surg Am 93:e1421–e1429. Peterson LN, Eva KW, Rusticus SA, Lovato CY. 2012. The readiness for clerkship survey: Can self-assessment data be used to evaluate program effectiveness?. Acad Med 87(10):1355–1360.
287
J. N. Stojan et al.
Shea JA, Weissman A, McKinney S, Silber JH, Volpp KG. 2012. Internal medicine trainees’ views of training adequacy and duty hours restrictions in 2009. Acad Med 87(7):889–894. Solet DJ, Norvell JM, Rutan GH, Frankel RM. 2005. Lost in translation: Challenges and opportunities in physician-to-physician communication during patient handoffs. Acad Med 80:1094–1099. UK Parliament. 1998. The Working Time Regulations. 1998. [Accessed 6 June 2014] Available from http://www.hse.gov.uk/contact/faqs/ workingtimedirective.htm.
288
WHO Collaborating Centre for Patient Safety Solutions. 2007. Communication during patient hand-overs. Patient Safety Solutions. [Accessed 6 June 2014] Available from http://www.who.int/patientsafety/solutions/patientsafety/PS-Solution3.pdf. Wohlauer MV, Arora VM, Horwitz LI, Bass EJ, Mahar SE, Philbert I. 2012. For the Handoff Education and Assessment for Residents (HEAR) Computer Supported Cooperative Workgroup. The patient handoff: A comprehensive curricular blueprint for resident education to improve continuity of care. Acad Med 87:411–418.
Copyright of Medical Teacher is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Medical school handoff education improves postgraduate trainee performance and confidence* JENNIFER N. STOJAN1, JOCELYN HUANG SCHILLER1, PATRICIA MULLAN1, JAMES T. FITZGERALD2, JENNIFER CHRISTNER3, PAULA T. ROSS1, SARAH MIDDLEMAS1, HILLARY HAFTEL1, R. BRENT STANSFIELD1 & MONICA L. LYPSON1 1
University of Michigan, USA, 2VA Ann Arbor Healthcare System, USA, 3State University of New York at Syracuse, USA
Abstract Objectives: Determine postgraduate first-year (PGY-1) trainees ability to perform patient care handoffs and associated medical school training. Methods: About 173 incoming PGY-1 trainees completed an OSCE handoff station and a survey eliciting their training and confidence in conducting handoffs. Independent t-tests compared OSCE performance of trainees who reported receiving handoff training to those who had not. Analysis of variance examined differences in performance based on prior handoff instruction and across levels of self-assessed abilities, with significance set at p50.05. Results: About 35% of trainees reported receiving instruction and 51% reported receiving feedback about their handoff performance in medical school. Mean handoff performance score was 69.5%. Trainees who received instruction or feedback during medical school had higher total and component handoff performance scores (p50.05); they were also more confident in their handoff abilities (p50.001). Trainees with higher self-assessed skills and preparedness performed better on the OSCE (p50.05). Conclusions: This study provides evidence that incoming trainees are not well prepared to perform handoffs. However, those who received instruction during medical school perform better and are more confident on standardized performance assessments. Given communication failures lead to uncertainty in patient care and increases in medical errors, medical schools should incorporate handoff training as required instruction.
Introduction
Practice points
The American Council for Graduate Medical Education (ACGME) issued a mandate in 2003 requiring that training programs in the United States restrict postgraduate trainee work hours to 80 h a week, with further restrictions in 2011 (ACGME 2014). The European Working Time Directive instituted a similar mandate, limiting the number of hours that junior medical staff could work in hospitals to 48 h a week (UK Parliament 1998). As postgraduate training programs complied and restructured their programs to reduce trainees’ hospital work hours, the number of patient handoffs, defined as the exchange of patient information between health care providers accompanying a transfer of care, subsequently increased (Horwitz et al. 2006; Mir et al. 2011; Antiel et al. 2013). With that, communication failures led to uncertainty in patient care decisions (Arora et al. 2005) and a perceived increase in medical errors (Borman et al. 2012; Shea et al. 2012). In 2007, the Joint Commission’s Report on Quality and
Incoming PGY-1 trainees are expected to be competent in performing handoffs at the start of their postgraduate training, yet only 35% of trainees reported receiving instruction on handoffs. Trainees who received instruction or feedback had significantly higher confidence in their handoff abilities (p50.0001). Trainees with higher self-assessed skills and perceived preparedness performed better on the OSCE. Trainees who received instruction during medical school had higher handoff performance scores. Given that communication failures lead to uncertainty in patient care and a perceived increase in medical errors, medical schools should incorporate handoff training prior to graduation.
Correspondence: Jennifer N. Stojan, MD, Department of Internal Medicine and the Department of Pediatrics, University of Michigan, 3119 Taubman Center, SPC5376, 1500 East Medical Center Drive, Ann Arbor, MI 48109, USA. E-mail: [email protected]
*Previous presentations: Pediatric Academic Society Meeting May 2013 Poster Presentation, Association of Medical Education in Europe Conference in August 2013 Poster Presentation. ISSN 0142-159X print/ISSN 1466-187X online/15/030281–8 ß 2015 Informa UK Ltd. DOI: 10.3109/0142159X.2014.947939
281
J. N. Stojan et al.
Safety identified lapses in communication as the leading root cause of sentinel events (Joint Commission 2007). In recognition of this problem, the Joint Commission and the World Health Organization issued Patient Safety Solutions recommending that health-care organizations implement a standardized approach to handoff communication and incorporate handoff training into their educational and continuing professional development curriculum (WHO 2007). Most of the published research on patient handoffs focuses on the assessment of handoff skills and curriculum development during postgraduate training (Horwitz et al. 2007; Chu et al. 2009; Gakhar & Spencer 2010; Bump et al. 2012), which means that training occurs after these postgraduate firstyear (PGY-1) trainees have already been performing handoffs as part of their patient care responsibilities. Although the need for handoff training and assessment during postgraduate training is critical, consideration of the potential impact of handoff training during medical school is also warranted, given that without adequate preparation, it is difficult to expect incoming PGY-1 trainees to be both competent and confident in performing handoffs at the start of their postgraduate training. Albeit an older study, the last published study in 2006 showed that only 8% of medical schools in the United States formally taught students how to perform handoffs in preparation for their postgraduate training (Solet et al. 2005). PGY-1 trainees, however, need to be able to care for patients on the first day of their postgraduate training and, therefore, waiting to institute a formal handoff curriculum until after this point may compromise patient safety. Postponing instruction may also lead to unnecessary stress, as it has been found that junior doctors feel unprepared to perform handoffs and have described their early handoff experiences negatively (Cleland et al. 2009).
Research questions (1) Do trainees who receive instruction or feedback during medical school perform better on a standardized patient handoff OSCE? (2) Are trainees who receive instruction or feedback during medical school more confident in their abilities to perform a handoff? (3) Do trainees who have higher self-assessed skills and preparedness perform better on a standardized handoff OSCE?
Methods Setting Since 2002, all incoming PGY-1 trainees at the University of Michigan Health System participate in a Postgraduate Orientation Assessment (POA) at their required initial orientation (Lypson et al. 2004). The POA is an Objective Structured Clinical Examination (OSCE) that provides a baseline assessment of clinical skills, communication skills, and general medical knowledge, using four standardized patient stations and five computer stations (Lypson et al. 2010a,b). Each of the nine POA stations’ scores was normalized within year to control for possible differences in difficulty and measurement 282
variance from year to year over a 10-year period. Four sub-scores were computed as un-weighted means of these normalized station scores. The verbal handoff is included in the communication sub-score. The five stations included in the communication sub-score, which include the verbal handoff, had Cronbach’s alpha of 0.42. A stepwise multiple regression of the communication sub-score by USMLE Step 1 and Step 2 (CK) scores provides evidence of validity: communication subscores were not predicted by Step 1 scores (p ¼ 0.952) but were positively predicted by Step 2 scores (p ¼ 0.048), which is what one expects from a valid measure of communication skills. The standardized patient handoff station became part of the POA in 2009. As the PGY-1 trainees are informed, the POA evaluation results are not used to assess individual trainees; rather, they are used to inform program directors of the profile of demonstrated competencies and training needs of entering PGY-1 trainee cohorts. In addition, trainees provide demographic information and respond to questions assessing their knowledge and skills on the topics covered in the POA. This study received educational exemption status from the Institutional Review Board at the University of Michigan.
Participants As part of our POA assessment, all (189) PGY-1 trainees in 2012 entering 18 different specialties from 73 different medical schools participated in a handoff station. Our study is based on the responses of the 173 (173/189 ¼ 91%) interns who completed both the structured performance assessments and the surveys eliciting their handoff training, knowledge, and confidence. The distribution of participants among demographic categories and medical specialties is summarized in Table 1. Trainee’s specialties are shown for ACGME-defined categories, which define medical, hospital-based, and surgicalbased specialties (ACGME 2014). Medical specialties included family medicine, internal medicine, pediatrics, combined internal medicine/pediatrics, psychiatry and pediatric neurology. Surgical specialties included obstetrics and gynecology, general surgery, plastic surgery, neurosurgery, urology, vascular surgery, orthopedic surgery, otolaryngology and oral/ maxillofacial surgery. Hospital-based specialties included anesthesiology, emergency medicine and pathology.
Assessment procedures and measures During the handoff station, the PGY-1 trainee reviewed a simulated patient’s written history and physical and watched a two-minute video demonstrating the patient’s clinical course. Trainees participated in a patient case appropriate for their medical specialty; i.e., internal medicine, surgery, or pediatrics. The trainee was required to synthesize and integrate the information from the video and the history and physical, and give a verbal handoff of the patient to a trained evaluator playing the role of the ‘‘physician assuming care of the patient.’’ All 24 trained evaluators were upper level postgraduate trainees recruited to work with their own specialty’s’ PGY-1 trainees. They received standardized training, including a review of prior POA performance videos of handoffs to standardize their assessments. After receiving the trainees’
Handoff education improves performance
handoff, the evaluator gave immediate verbal feedback to the trainee, and rated the trainee’s performance using a structured assessment tool adopted from existing research (Farnan et al. 2010). Of note, the raters were blinded to the presence or absence of education about handoffs that trainees received during medical school.
Instruments: Performance assessment protocol Table 2 provides a summary of the content and scoring rubric for the handoff performance assessment tool. This assessment tool was adapted from Horwitz and Farnan (Horwitz et al. 2007; Farnan et al. 2010). To ensure that the items on the tool represented the constructs we intended to evaluate, published Table 1. Demographic profiles of incoming PGY-1 trainees participating in the patient care handoff assessment.
Characteristics
N (%)
Gender Male Female
106 (62%) 66 (38%)
Race Caucasian African American Asian/Pacific Islander Hispanic Other
128 (75%) 5 (3%) 30 (18%) 7 (4%) 1 (1%)
Specialty Medical Surgical Hospital based
88 (51%) 41 (24%) 44 (25%)
Table 2. Content of the patient handoff performance assessment tool.
Content – patient history (items 1–8) 1. Identification of sick patients and code status 2. Identifier: Example: Sue Smith is a 56-year-old female, with hypertension, coronary heart disease, and high cholesterol, who presented with syncope 3. Pertinent past medical history 4. General hospital course 5. New events of the day and active issues 6. Overall health status and relevant physical exam findings 7. Relevant lab and study results 8. Medications and allergies Clinical judgment (items 9 and 10) 9. Upcoming possibilities with plan and supporting rationale 10. To do items for overnight with plan Communication and professionalism (items 11–15) 11. Allowing for questions 12. Appropriateness of the setting that the handoff exchange took place in 13. Organization and efficiency 14. Communication skills 15. Professionalism and attitude Total score (sum of ratings for items 1–15) global performance score Items 1–15 were scored on a scale that ranged from 0to 2, with 0 ¼ item was not done, 1 ¼ item was done but needed improvement, and 2 ¼ item was done well. Total Score range of possible scores was from 0 to 30. Global Performance Score was scored on a scale that ranged from 1 to 5, with 1 ¼ novice, 3 ¼ competent and 5 ¼ expert.
literature on handoff curricula was reviewed. The tool identified assessment criteria, combining components of the SIGNOUT mnemonic (Horwitz et al. 2007), which assessed content and clinical judgment, with items assessing communication skills and professionalism (Farnan et al. 2010). Operational definitions of scale points were also developed; ‘‘0’’ indicated an item was not done, ‘‘1’’ indicated an item was done, but needed improvement and a score of ‘‘2’’ indicated an item was done well. In addition, the evaluator assigned a global performance score based on a scale ranging from 1 to 5, with ‘‘1’’ representing novice performance, ‘‘3’’ representing competence, and ‘‘5’’ representing expertise. Feedback was solicited and incorporated from stakeholders including faculty, residents, and students who piloted the tool during actual patient handoffs.
Additional assessment instruments: Previous training, knowledge, and self-assessed confidence related to patient handoffs Following completion of the OSCE, PGY-1 trainees completed an online survey with questions pertaining to the POA stations. The questions reported in this study primarily focused on PGY-1 trainees’ characterization of having received formal training (with Likert-format responses) or feedback (with ‘‘yes/no’’ responses) about patient handoffs during their medical school training and their knowledge, self-assessment, and comfort level regarding handoffs. An expert panel, consisting of faculty from the departments of pediatrics, internal medicine, and medical education, with experience in clinical care, medical education, survey design, and patient safety, reviewed relevant published literature (Horwitz et al. 2007; Chu et al. 2010; Gakhar & Spencer 2010), developed questions, and elicited and incorporated feedback from stakeholders into the final survey (Table 3).
Statistical analysis Descriptive statistics were calculated for demographic variables, survey question responses, and the assessment of directly observed OSCE handoff performance. OSCE scores included a total score (items 1–15) and three sub-scores: content (items 1–8); clinical judgment (items 9 and 10); and global performance score. Independent t-tests were used to compare performance of PGY-1 trainees who had received feedback during medical school about handoffs to PGY-1 trainees who had not received feedback during medical school; effect sizes (Cohen’s d) were calculated to guide interpretation of the magnitude of the differences. Analysis of variance (ANOVA) was used to examine differences in performance of trainees across groups constituted by: students’ ordinal ratings of handoff instruction during medical school; their ordinal ratings of their self-reported confidence; race; and ACGME-categorized medical specialties. Chi-square analysis compared levels of ordinal-level self-reported confidence of trainees who had received prior handoff training and feedback during medical school to those who had not. JMP Pro 10.0.0 (SAS Inc., Cary, NC) was the statistical program used for these analyses.
283
J. N. Stojan et al.
Results Table 4 summarizes PGY-1 trainees’ mean performances on the OSCE handoff station. The mean total score was 69.5%. The mean content and clinical judgment scores were 68% and 69%, respectively. The mean global performance score was 2.9, which corresponded to the trained rater’s characterization of observed performance as slightly less than competent. The results of trainees’ responses to the handoff survey are summarized in Table 3. Approximately one-half (51%) of the PGY-1 trainees indicated that they had received feedback about their handoff performance in medical school, but only 35% responded that they were formally taught how to perform a handoff as a medical student. Eighty-three percent of respondents felt that they could communicate all the necessary information in a handoff and 71% of respondents felt that they were prepared to perform handoffs as a PGY-1 trainee. Ninetysix percent of trainees correctly identified lapses in communication as the leading root cause of sentinel events. PGY-1 trainees who reported that they had received feedback about their handoff performance or who received instruction on how to perform a handoff during their medical school education had higher confidence about their ability and Table 3. PGY-1 trainees’ medical school training, knowledge, and self-reported confidence associated with patient care handoffs.
Survey questions
N ¼ 173
Medical school training Have you ever received feedback about your handoff performance
Yes, 51%
How to perform handoffs is taught in my medical school Self-assessed confidence I can communicate all the information that is needed in a handoff I can communicate all the information that is needed in a handoff in an efficient manner I am prepared to perform handoffs as a PGY-1 trainee Objective Knowledge Test Items Cross covering providers provide better patient care when a standardized format is used What is the leading root cause of sentinel events?
Agree/strongly agree 35%
83% 63%
71%
preparedness with regards to transitions of care. Trainees who had received feedback on handoffs during medical school felt more prepared to perform a complete (p ¼ 0.007) and efficient handoff (p ¼ 0.023). Trainees who reported that they received instruction in medical school on how to perform handoffs were more likely to feel prepared to perform handoffs as a PGY-1 trainee (p50.001). They were also more likely to feel that they could perform a complete (p50.001) and efficient handoff (p50.001). Table 5 summarizes the mean performance scores of PGY-1 trainees in the context of the residents’ self-reported medical school training and current confidence related to patient handoff care responsibilities. Significant differences were found among those who received feedback on their handoff performance in medical school compared with those who did not in content (p ¼ 0.041), total (p ¼ 0.041), and global performance scores (p ¼ 0.035), with associated effect sizes of d ¼ 0.27, 0.27, and 0.25, but not in clinical judgment scores. Those who were taught how to perform handoffs in medical school performed better on content (p ¼ 0.003), clinical judgment (p ¼ 0.005), total (p ¼ 0.001), and global performance scores (p50.001). Similarly, PGY-1 trainees who felt that they could communicate all the information needed in a handoff were accorded higher content (p ¼ 0.018), clinical judgment (p ¼ 0.029), total (p ¼ 0.009), and global performance scores (p ¼ 0.009). Those who characterized themselves as having been prepared to perform a handoff as a PGY-1 trainee also received higher content (p ¼ 0.011), clinical judgment (p ¼ 0.022), total (p ¼ 0.011), and global performance scores (p ¼ 0.023). There were no significant differences in the communication and professionalism scores based on self-reported medical school training and confidence. In terms of differences in performance across PGY-1 trainees, no differences were found between races. As illustrated in Table 4, there were also no significant differences in content, total, or global performance scores among ACGMEdefined specialties. However, some differences were found among the specialties in clinical judgment scores, with medical specialties scoring 61.4%, while hospital-based specialties scored 65.3% and surgical specialties scored 51.8% (p ¼ 0.009).
95%
Communication 96%
Discussion An unintended consequence of restricting postgraduate trainee duty hours has been increasing the number of patient care
Table 4. Overall and medical specialty performance of the PGY-1 trainees’ on the patient-handoff assessment.
Performance assessment domain Content (items 1–8) Clinical judgment (Items 9 and 10)a Total score (items 1–15) Global Performance Score (score range 1–5) a
Combined trainees (n ¼ 173), mean % (SD) 68.0 60.1 69.5 2.9
(16.6) (21.2) (14.4) (0.8)
Medical specialties (n ¼ 88), mean % (SD) 70.9 61.4 71.4 2.9
(16.5) (20.4) (14.0) (0.7)
Surgical specialties (n ¼ 41), mean % (SD) 64.5 51.8 66.7 2.7
(15.3) (16.2) (13.1) (0.8)
Hospital-based specialties (n ¼ 44), mean % (SD) 65.6 65.3 68.5 3.0
(17.3) (24.8) (15.9) (1.0)
p ¼ 0.009. One-way analysis of variance (with Tukey–Kramer HSD, global p ¼ 0.05) was used to compare differences in mean scores among medical, surgical, and hospitalbased specialties for all scores. The only significant difference that emerged was for the clinical judgment score; the surgical specialties scored lower than the medical and hospital-based specialties.
284
Handoff education improves performance
Table 5. Patient handoff performance results across PGY-1 trainees’ level of medical school training and current self-reported confidence in patient handoff competencies.
Survey question Have you ever received feedback about your handoff performance? (n) Yes (89) No (84) How to perform handoffs is taught in my medical school (n) Strongly disagree (18) Disagree (50) Neutral (44) Agree (49) Strongly agree (12) I can communicate all the information that is needed in a handoff (n) Strongly disagree (2) Disagree (2) Neutral (25) Agree (123) Strongly agree (21) I can communicate all the information that is needed in a handoff in an efficient manner (n) Strongly disagree (3) Disagree (14) Neutral (47) Agree (96) Strongly agree (13) I am prepared to perform a handoff as a PGY-1 trainee (n) Strongly disagree (2) Disagree (6) Neutral (42) Agree (113) Strongly agree (10)
Content (items 1–8), mean (SD)
Clinical judgment (items 9 and 10), mean (SD)
Total score (items 1–15) mean (SD)
Global performance score, mean (SD)
70.2 (15.8) 65.8 (17.2) p ¼ 0.041*
60.1 (20.9) 60.1 (21.6) p ¼ 0.500
71.4 (13.2) 67.6 (15.3) p ¼ 0.041*
3.0 (0.8) 2.8 (0.8) p ¼ 0.035*
63.5 (15.0) 66.1 (15.6) 62.9 (18.1) 74.4 (13.8) 75.5 (19.3) p ¼ 0.003*
52.8 (22.5) 58.0 (18.5) 55.7 (20.8) 64.8 (21.0) 77.1 (22.5) p ¼ 0.005*
64.8 (12.6) 67.7 (13.4) 65.3 (15.5) 74.8 (12.0) 78.1 (17.0) p ¼ 0.001*
2.6 (0.7) 2.9 (0.8) 2.5 (0.7) 3.2 (0.8) 3.5 (1.0) p50.001*
43.8 (17.7) 53.1 (22.1) 64.0 (14.1) 68.2 (16.8) 75.6 (14.1) p ¼ 0.018*
25.0 (35.4) 62.5 (17.7) 59.0 (23.8) 59.2 (19.4) 70.2 (23.2) p ¼ 0.029*
45.0 (11.8) 60.0 (23.6) 65.6 (13.7) 69.8 (14.1) 76.2 (13.1) p ¼ 0.009*
1.5 (0.7) 2.5 (0.7) 2.6 (0.8) 2.9 (0.8) 3.3 (0.9) p ¼ 0.009*
47.9 (14.4) 63.0 (12.4) 64.0 (16.6) 70.3 (16.9) 76.0 (11.4) p ¼ 0.009*
33.3 (28.9) 57.1 (18.2) 56.4 (18.4) 62.2 (21.8) 67.3 (23.7) p ¼ 0.060
50.0 (12.0) 66.2 (12.7) 65.9 (14.0) 71.5 (14.2) 76.2 (12.8) p ¼ 0.006*
1.7 (0.6) 2.8 (0.7) 2.7 (0.8) 3.0 (0.8) 3.4 (1.0) p ¼ 0.005*
53.1 (4.4) 69.8 (13.4) 61.3 (14.6) 69.9 (16.9) 76.9 (16.2) p ¼ 0.011*
50.0 (0.0) 66.7 (25.8) 51.2 (21.4) 62.6 (20.1) 67.5 (23.7) p ¼ 0.0221*
56.7 (4.7) 70.6 (15.4) 63.8 (13.0) 71.2 (14.2) 77.3 (15.4) p ¼ 0.011*
2.0 (0.0) 2.8 (0.8) 2.6 (0.8) 3.0 (0.8) 3.3 (0.8) p ¼ 0.023*
*p50.05. One-way analysis of variance was used to determine differences by response category for the (five) questions with five Likert-format response options. For the ‘‘yes/ no’’ question about feedback during medical school, an independent t test (one-tailed) was used.
handoffs. This increase in handoffs poses a potential threat to patient safety. Most existing educational interventions represented in current research have focused on training and assessment during postgraduate training (Horwitz et al. 2008; Chu et al. 2009; Gakhar & Spencer 2010; Bump et al. 2012). Of note, even within that scope, an inter-institutional patient handoff education working group recently characterized their need to formulate a curriculum blueprint for patient handoff education as warranted, given that, prior to their working group’s efforts, no one had ‘‘developed a comprehensive approach to teaching and assessing residents’ competence in patient handoffs (Wohlauer et al. 2012).’’ Our study builds on and extends current understanding of the need for and potential impact of patient handoff training. Others have shown that trained students and residents are more confident in their skills (Arora et al. 2013; Didwania et al. 2013). This study further provides empirically based insights derived from incoming residents’ directly observed performance on a standardized handoff assessment, as well as these trainees’ characterization of their ability and preparation they received during medical school for patient handoff responsibilities they reported having received.
We found that only 35% of incoming PGY-1 trainees reported receiving any instruction about handoffs during their medical school education and only 51% reported receiving feedback about their handoff performance. Trainees who had received instruction in medical school performed better than their peers who had not received instruction on objective performance assessments of their handoff content, clinical judgment, total scores, and global performance scores. In addition to their superior performance scores, trainees who had received training about handoffs were more confident. Similarly, residents who had received feedback during medical school about their handoff performance had significantly higher content scores, total scores, and global performance scores. The group who received feedback, however, did not have superior clinical judgment scores. This may reflect that receiving feedback is not equivalent to a formal medical handoff curriculum. If, for example, feedback was specific to the content of the handoff, clinical judgment may not have been addressed. Existing research seeking to evaluate the outcome of medical education training has been criticized for its dependence on trainees’ self-assessed confidence and competence.
285
J. N. Stojan et al.
Reservations often note that self-assessment has been shown to be a predictor of questionable reliability as it pertains to individual performance (Davis et al. 2006). It is difficult to trust that individuals who have not yet practiced extensively in their field can adequately judge the success with which an educational program has prepared them for such practice (Peterson et al. 2012). Our study places reports of trainees’ selfassessments in the context of these residents’ reflections about root causes of adverse outcomes and their own preparedness for patient care handoff practice, informed by their actual performance and feedback by trained raters who directly observe their handoff performance on a structured clinical simulation performance task. Our study found that despite the relative absence of handoff education during their medical school training, entering PGY-1 trainees clearly recognized the importance of communication with regards to patient safety, as indicated by their ability to identify that communication lapses were the leading root cause of sentinel events in the hospital. However, while the majority of entering PGY-1 trainees in our institution’s study of trainees across medical specialties lack formal training, they characterize themselves as confident in their ability to communicate patient information. Despite their confidence, trainees as a whole scored a mean of 69.5% on the OSCE total score and elicited trained observers’ characterization of their global performance as slightly less than competent. Our finding of over-confidence is consistent with existing research showing that 60% of the time the PGY-1 trainee giving a handoff did not communicate the most vital piece of patient information during that handoff, despite that trainee believing that it was communicated (Chang et al. 2010). This discrepancy has major implications for patient safety as it has been shown that inadequate handoffs can result in delays in diagnosis or treatment, intensive care unit transfers, near misses and inefficiencies or redundancies in work (Horwitz et al. 2008). Although the trainees in our study were, as a whole, overly confident in their abilities to perform a handoff, there was a statistically significant difference substantiating greater confidence and greater abilities for trainees who had received feedback instruction. In our study, residents receiving any type of feedback or handoff instruction in medical school demonstrated both better performance and higher confidence. Our findings of more confidence in performing handoffs following medical school instruction is consistent with prior research studies on handoff education (Klamen et al. 2009; Chu et al. 2010; Farnan et al. 2010). Eva and Regeher caution that attention to self-assessment as an end in itself is, at best misguided, and instead needs to be considered in terms of motivating learning and self-directed assessment seeking (Eva et al. 2008). Our study adds evidence of the long-term impact of training, as demonstrated by the trainees’ performance. As a recent review of educational interventions to improve nontechnical skills indicated, further research is needed to explicate the underlying theoretical frameworks, replicable instructional methods, and extent of the impact of these educational interventions (Gordon et al. 2012). Limitations of this study include that it was performed at a single institution, which may limit the extent to which its 286
findings can be generalized to other study sites. The observed handoff performance was on one OSCE station, conducted as part of a required assessment for all entering first-year residents at the study institution (Lypson et al. 2004). The potential differences in the difficulty of the specialty-specific scenarios constitute a study limitation. The source of information characterizing trainees’ instruction and feedback about handoff performance during medical school depended on the residents’ self-report, and may then have been subject to recall bias. Nonetheless, it does represent a full training cohort representing a diverse group of 173 PGY-1 trainees, coming from 73 different medical schools and entering 18 different specialties. The large number of trainees provided sufficient power for us to identify differences between those who had training compared with those who had not, and to explore the generalizability of these findings across disciplines, establishing that there were no differences among medical specialties, surgical specialties, and hospital-based specialties in content, total, and global performance scores. The sole difference found between specialties was in the clinical judgment score, a finding that warrants further study. The paucity of formal handoff instruction and research in an era where handoffs are ubiquitous and in a field where failures in communication can have such a devastating human impact must be addressed. As a profession, physicians cannot expect PGY-1 trainees to have acquired the necessary skills in communication if these skills were not formally taught as part of the medical school curriculum. In addition, it should not be expected that PGY-1 trainees would master these skills through trial and error during their postgraduate training while they are already engaged in patient care. Therefore, with the changes in work hours and the accompanying increase in patient handoffs, physician educators must take notice of the Joint Commission’s and the World Health Organization’s recommendations to implement handoff curriculum into medical education and adopt a standardized approach. Further research will then be necessary to determine how to incorporate handoff education into the medical school curriculum and to evaluate the long-term impact of this curriculum on handoff performance and patient safety.
Glossary Handoff: Transfer of patient care responsibility information
Conclusions Despite the increase in the number of care transitions after the residency work restriction mandates, the majority of medical schools in the United States do not appear to be preparing their medical students for the reality of postgraduate training as evidenced by the relative lack of formalized instruction on handoffs in undergraduate medical education curricula. It is encouraging to confirm that PGY-1 trainees who were taught how to perform a handoff in medical school had higher performance scores, based on the direct observations of
Handoff education improves performance
trained raters blinded to the trainees’ preparation during medical school. Trainees who received instruction were also more confident in their ability to perform handoffs as incoming trainees. Medical schools should consider making this important patient safety instruction a mandatory part of the later phase of the curriculum.
Notes on contributors JENNIFER N. STOJAN, MD, is an Assistant Professor in the Department of Internal Medicine and the Department of Pediatrics, University of Michigan Medical School, Ann Arbor, MI. JOCELYN HUANG SCHILLER, MD, is an Associate Professor in the Department of Pediatrics, University of Michigan Medical School, Ann Arbor, MI. PATRICIA MULLAN, PhD, is a Professor in the Department of Medical Education, University of Michigan Medical School, Ann Arbor, MI. JAMES T. FITZGERALD, PhD, is a Professor in the Department of Medical Education, University of Michigan Medical School, Ann Arbor, Michigan and the Associate Director of Education and Evaluation at the Geriatric Research, Education and Clinical Center, VA Ann Arbor Healthcare System, Ann Arbor, MI. JENNIFER CHRISTNER, MD, is an Associate Professor of Pediatrics in the Department of Pediatrics and Associate Dean for Undergraduate Medical Education at SUNY Upstate College of Medicine, Syracuse, NY. PAULA T. ROSS, PhD, is a Project Manager, University of Michigan Medical School, Ann Arbor, MI. SARAH MIDDLEMAS, MPH, is at the University of Michigan Medical School, Ann Arbor, MI. HILARY M. HAFTEL, MD, MHPE, is a Professor of Pediatrics, Internal Medicine, and Medical Education and Associate Chair for Education in the Department of Pediatrics, University of Michigan Medical School, Ann Arbor, MI. R. BRENT STANSFIELD, PhD, is an Assistant Professor in the Department of Medical Education, University of Michigan Medical School, Ann Arbor, MI. MONICA L. LYPSON, MD, MHPE, is a Professor of Internal Medicine & Medical Education and Assistant Dean of Graduate Medical Education, University of Michigan Medical School, Ann Arbor, MI.
Declaration of interest: The authors report no declarations of interest.
References Accreditation Council for Graduate Medical Education. 2014. ACGME Program and Institutional Guidelines. [Accessed 6 June 2014] Available from https://www.acgme.org/acgmeweb/tabid/158/Programand InstitutionalAccreditation. Antiel RM, Reed DA, Van Arendonk KJ, Wightman SC, Hall DE, Porterfield JR, Horvath KD, Terhune KP, Tarpley JL, Farley DR. 2013. Effects of duty hour restrictions on core competencies, education, quality of life, and burnout among general surgery interns. JAMA Surg 16:1–8. Arora VM, Eastment MC, Bethea ED, Farnan JM, Friedman ES. 2013. Participation and experience of third-year medical students in handoffs: Time to sign out? J Gen Intern Med 28(8):994–998. Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. 2005. Communication failures in patient sign-out and suggestions for improvement: A critical incident analysis. Qual Saf Health Care 14(6):401–407.
Borman KR, Jones AT, Shea JA. 2012. Duty hours, quality of care, and patient safety: General surgery resident perceptions. J Am Coll Surg 215(1):70–77. Bump GM, Bost JE, Buranosky R, Elnicki M. 2012. Faculty member review and feedback using a sign-out checklist: Improving intern written signout. Acad Med 87(8):1125–1131. Chang V, Arora V, Lev-Ari S, D’Arcy M, Keysay B. 2010. Interns overestimate the effectiveness of their handoff communication. Pediatrics 125(3):491–496. Chu ES, Reid M, Burden M, Mancini D, Schulz T, Keniston A, Sarcone E, Albert RK. 2010. Effectiveness of a course designed to teach handoffs to medical students. J Hosp Med 5(6):344–348. Chu ES, Reid M, Schulz T, Burden M, Mancini D, Ambardekar AV, Keniston A, Albert RK. 2009. A structured hand-off programme for interns. Acad Med 84(3):347–-352. Cleland JA, Ross S, Miller SC, Patey R. 2009. ‘‘There is a chain of Chinese whispers . . .’’: Empirical data support the call to formally teach handover to prequalification doctors. Qual Saf Health Care 18:267–271. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. 2006. Accuracy of physician self-assessment compared with observed measures of competence: A systemic review. JAMA 296(9):1094–1102. Didwania A, Kriss M, Cohen ER, McGaghie WC, Wayne DB. 2013. Internal medicine postgraduate training and assessment of patient handoff skills. J Grad Med Educ 5(3):394–398. Eva KW, Regehr G. 2008. ‘‘I’ll never play professional football’’ and other fallacies of self-assessment. J Continuing Educ Health Prof 28(1):14–19. Farnan JM, Paro JA, Rodriguez RM, Reddy ST, Horwitz LI, Johnson JK, Arora VM. 2010. Hand-off education and evaluation: Piloting the observed simulated hand-off experience (OSHE). J Gen Intern Med 25(2):129–134. Gakhar B, Spencer AL. 2010. Using direct observation, formal evaluation, and an interactive curriculum to improve the sign-out practices of internal medicine interns. Acad Med 85(7):1182–1188. Gordon M, Darbyshire D, Baker P. 2012. Non-technical skills training to enhance patient safety: A systematic review. Med Educ 46(11):1042–1054. Horwitz LI, Krumholz HM, Green MI, Huot SJ. 2006. Transfers of patient care between house staff on the internal medicine wards: A national survey. Arch Intern Med 166:1173–1176. Horwitz LI, Moin T, Green ML. 2007. Development and implementation of an oral sign-out skills curriculum. J Gen Intern Med 22(10):1470–1474. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. 2008. Consequences of inadequate sign-out for patient care. Arch Intern Med 168(16):1755–1760. Joint Commission. 2007. Improving America’s hospitals: The Joint Commission’s annual report on quality and safety 2007. [Accessed 6 June 2014] Available from http://www.jointcommission.org/assets/1/6/ 2007_Annual_Report.pdf. Klamen DL, Reynolds KL, Yale B, Aiello M. 2009. Students learning handovers in a simulated in-patient unit. Med Educ 43(11):1097–1098. Lypson ML, Frohna JG, Gruppen L, Woolliscroft JO. 2004. Assessing residents’ competencies at baseline: Identifying the gaps. Acad Med 79(6):564–570. Lypson ML, Hamstra SJ, Ross PT, Gruppen LD, Colletti LM. 2010a. An assessment tool for aseptic technique in resident physicians: A journey towards validation in the real world of limited supervision. J Grad Med Educ 2(1):85–89. Lypson ML, Ross PT, Hamstra SJ, Haftel HM, Gruppen LD, Colletti LM. 2010b. Evidence for increasing diversity in graduate medical education: The competence of underrepresented minority residents measured by an intern objective structured clinical examination. J Grad Med Educ 2(3):354–359. Mir HR, Cannada LK, Murray JN, Black KP, Wolf JM. 2011. Orthopaedic resident and program director opinions of resident duty hours: A national survey. J Bone Joint Surg Am 93:e1421–e1429. Peterson LN, Eva KW, Rusticus SA, Lovato CY. 2012. The readiness for clerkship survey: Can self-assessment data be used to evaluate program effectiveness?. Acad Med 87(10):1355–1360.
287
J. N. Stojan et al.
Shea JA, Weissman A, McKinney S, Silber JH, Volpp KG. 2012. Internal medicine trainees’ views of training adequacy and duty hours restrictions in 2009. Acad Med 87(7):889–894. Solet DJ, Norvell JM, Rutan GH, Frankel RM. 2005. Lost in translation: Challenges and opportunities in physician-to-physician communication during patient handoffs. Acad Med 80:1094–1099. UK Parliament. 1998. The Working Time Regulations. 1998. [Accessed 6 June 2014] Available from http://www.hse.gov.uk/contact/faqs/ workingtimedirective.htm.
288
WHO Collaborating Centre for Patient Safety Solutions. 2007. Communication during patient hand-overs. Patient Safety Solutions. [Accessed 6 June 2014] Available from http://www.who.int/patientsafety/solutions/patientsafety/PS-Solution3.pdf. Wohlauer MV, Arora VM, Horwitz LI, Bass EJ, Mahar SE, Philbert I. 2012. For the Handoff Education and Assessment for Residents (HEAR) Computer Supported Cooperative Workgroup. The patient handoff: A comprehensive curricular blueprint for resident education to improve continuity of care. Acad Med 87:411–418.
Copyright of Medical Teacher is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
