Simulation
Error management training and simulation education Aimee Gardner, Surgery Department, UT Southwestern Medical Center, Dallas, Texas, USA Michelle Rich, Radiology Department, Akron Children’s Hospital, Ohio, USA
SUMMARY Background: The integration of simulation into the training of health care professionals provides context for decision making and procedural skills in a high-fidelity environment, without risk to actual patients. It was hypothesised that a novel approach to simulation-based education – error management training – would produce higher performance ratings compared with traditional step-by-step instruction. Method: Radiology technology students were randomly assigned to participate in traditional
procedural-based instruction (n = 11) or vicarious error management training (n = 11). All watched an instructional video and discussed how well each incident was handled (traditional instruction group) or identified where the errors were made (vicarious error management training). Students then participated in a 30–minute case-based simulation. Simulations were videotaped for performance analysis. Blinded experts evaluated performance using a predefined evaluation tool created specifically for the scenario.
Results: The vicarious error management group scored higher on observer-rated performance (Mean = 9.49) than students in the traditional instruction group (Mean = 9.02; p < 0.01). Conclusions: These findings suggest that incorporating the discussion of errors and how to handle errors during the learning session will better equip students when performing hands-on procedures and skills. This pilot study provides preliminary evidence for integrating error management skills into medical curricula and for the design of learning goals in simulation-based education.
Blinded experts evaluated performance using a predefined evaluation tool created specifically for the scenario
© 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540 537
tct_12217.indd 537
11/11/2014 4:12:24 PM
People are more motivated to learn when things go wrong
INTRODUCTION
A
lthough traditional learning environments often reproach errors,1 theories as far back as 25 years ago suggest that errors can be highly informative when learners discover why such mistakes occur and how they can be corrected.2 Other learning theories suggest that people are more motivated to learn when things go wrong.3 Indeed, recent developments have indicated that errors can be beneficial in training settings. One specific educational approach, error management training (EMT), is a training method that incorporates active exploration and explicit encouragement for trainees to make errors during training, and to learn from them.4 EMT recognises that errors are a natural by-product of active learning, and that errors will inevitably emerge as students explore their environment. These errors, though, can play a fundamental role in trainee development. By working through any potential mistakes that could arise, learners are actively involved and can obtain a better mental model of the task or procedure at hand, what could go wrong, and strategies for managing the error. Research outside of the medical domain has demonstrated that EMT leads to better skill acquisition,5 decision making,6 transfer of training,7 and performance;7 however, its application to medical education has yet to be empirically validated.8 Because of the high-risk of errors within health care settings, however, the integration of EMT within medicine must be conducted in a safe setting. Simulation-based training is the ideal avenue for such education because it allows trainees to freely make and learn from mistakes, without risk to patient safety. As noted by others,9 the central premise of
simulation-based medical education is to provide students with increased practise in learning from mistakes, so as to reduce such mistakes in real life. Thus, EMT can be seamlessly incorporated within simulationbased education. The current study examines the effectiveness of these training techniques among radiology technician students. As radiographers perform critical imaging examinations to demonstrate fractures and other pathological conditions, their training can have an important impact on patient safety. Additionally, properly trained radiographers can help to optimise the efficiency of the health care provider, can increase patient throughput and can reduce the patient’s total wait time.
METHODS The pilot study was conducted at the Austen BioInnovation Institute, Akron, Ohio, a midwestern regional simulation centre. This initial pilot study was integrated into course clinical activity, and was therefore considered to be exempt for ethics approval by the University Institutional Review Board. Twenty-two radiology technician students in their first year of training participated in a 4–hour simulation-based education training programme. Students were randomly assigned to either
a traditional learning group (TLG; n = 11) or an error management group (EMG; n = 11). At the beginning of the training programme students participated in one of two case-based lectures, including a video demonstration. The TLG watched a video in which the radiology technicians performed procedures accurately throughout, whereas the EMG was shown the same patient presentation with radiology technicians who committed various errors. Mistakes performed in the EMG included unintentional errors as well as deliberate violations. Errors occurred in obtaining patient history (e.g. not obtaining initial pregnancy information), technical skills (e.g. using the wrong size of leads), safety (e.g. not shielding the patient), professionalism (e.g. chatting with a coworker and ignoring the patient) and efficiency (e.g. shooting images in illogical order). The video for the TLG, however, presented the same patient-based case scenario with the radiology technicians performing each behaviour correctly, and accurately, according to the protocol and policies in place. After the case study, each group participated in a short facilitator-led discussion similar to those used in other EMT studies.10 In the TLG the instructor discussed how well each incident was handled. Students were asked to discuss what went well and why it was important. In the EMG the instructor asked
538 © 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540
tct_12217.indd 538
11/11/2014 4:12:27 PM
students to systematically identify where the errors were made. Students were asked to provide examples of where errors were made, and what the implications of those errors might be.
training programme. As shown in Table 1, the EMG scored higher on observer-rated performance (Mean = 9.49, on a scale of 1–10) than students in the TLG (Mean = 9.02; p < 0.01). Additionally, and of note, the results reveal that the TLG had almost twice the variability of the EMG (TLG SD = 0.43; EMG SD = 0.24), suggesting that individuals within the TLG had performance scores that were more variable. Given that the EMG was very close to the maximum value, however, these differences in variability should be interpreted with caution.
Students then individually participated in a 30–minute scenario involving a case in which a patient had sustained a fall from a significant distance. The patient had injured her forearm, elbow and shoulder, and suffered from limited mobility. The student had to make any necessary adjustments to the equipment, while minimising the patient’s discomfort, and ultimately obtain quality diagnostic images (the same standardised patient actor was used during each scenario). Each simulation was recorded for performance ratings. Two blinded experts trained on the evaluation tool both rated each of the 22 video recordings.
DISCUSSION The findings from this study provide preliminary support for implementing error management strategies within medical education. Given the plethora of errors and mistakes committed by health care professionals, identifying avenues for educators to instill error reduction and management strategies among trainees is essential to enhance patient safety. Our study found that engaging in a discussion and meditation on one 30–minute error-ridden case scenario is powerful enough to have an impact on subsequent performance in a simulated scenario. These results suggest that actively considering the role of errors and their implications in facilitated discussion produces important cognitions concerning critical thinking, strategies and improvement techniques that have a direct impact on behaviour. Providing examples of solely correctly performed behaviours and procedures may not develop enriched mental models among health care students and
RESULTS Data analysis was conducted with SPSS 18.0 (SPSS Inc, Chicago, IL, USA). A significance level of p < 0.05 was chosen. Inter-rater reliability was assessed for the performance ratings with intraclass correlation coefficients (ICCs). The ICC value representing the reliability of ratings across raters was 0.734 (95% CI 0.541–0.871; p < 0.001), warranting the use of mean ratings of performance. Differences between performance ratings were analysed using an independent samples Student’s t–test. Twenty-two radiology technician students participated in the
Table 1. Summary of results Group
n
Mean (SD)
Difference
Traditional
11
9.01 (.43)
p < 0.01
Errors
11
9.49 (.24)
professionals in the same way that providing error-based stories can, combined with instruction about correct performance. Thus, errors can be the allies of medical educators.10
Errors can be the allies of medical educators
Although the beneficial roles of errors within medical education has not gone unacknowledged (i.e. morbidity and mortality conferences, autopsies, etc.), this study suggests that educators should make a pointed effort to discuss errors, their consequences, and strategies to manage them among learners. While the current study examined the effect of vicarious EMT, similar or even more powerful effects will undoubtedly be seen with more active error training. Having students commit the errors themselves and strategise to manage those errors may produce even more pronounced cognitive, emotional and behavioural outcomes. After all, students are more motivated to learn when things go wrong.3 Simulation-based education can uniquely complement this novel educational strategy, in that it provides a safe setting for students to make mistakes and learn from them. This research also highlights the efficaciousness of observational learning for educational training design. In settings in which learners are presented with practise examples, or observe other more senior residents or faculty staff, they should be encouraged to actively think about errors and to learn from them. For example, before beginning a new rotation, residents can be instructed to compile a list of procedures that they might encounter on that rotation, the errors frequently associated with performing those procedures, and how students plan to manage and adapt to those situations. By considering a range of alternative approaches to the procedure or task and thinking adaptively about the
© 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540 539
tct_12217.indd 539
11/11/2014 4:12:29 PM
Adopting error training in educational initiatives can...help revolutionise the culture of medical education
situation, learners will engage with the material at a deeper level.10 Adopting error training in educational initiatives can also help revolutionise the culture of medical education. Institutions that encourage the continued and open communication of errors, accidents and near misses can create an atmosphere in which faculty staff and learners are encouraged to voice concerns, which may ultimately lead to an increased awareness and to better safety systems. Despite the significant contributions of this pilot study, there are a few limitations that should be noted. The first limitation concerns the sample size of the study. Replication of these results with a larger and more diverse sample is needed to fully understand the effects of EMT. Although students were randomly assigned to engage in discussion of ideal versus error behaviours, we did not obtain data regarding the involvement of each student in the discussion. Thus, we cannot identify the extent to which students identified errors, merely listened or were completely unengaged in
the discussion. Finally, using a broader array of criteria would help extend the understanding of this educational process. The current study examined blinded observer-rated performance, but examining the impact of error training on more robust, novel or team-based performance events would be beneficial.
CONCLUSION These findings suggest that incorporating a discussion of errors and how to handle errors during the learning session will better equip students when performing handson procedures and skills. This pilot study provides preliminary evidence for integrating error management skills into medical curricula and designing learning goals in simulation-based education. REFERENCES 1. Frese M, Brodbeck FC, Heinbokel T, Mooser C, Schleiffenbaum E, Thiemann P. Errors in training computer skills: On the positive function of errors. Human-Computer Interaction 1991;6:77–93. 2. Arnold B, Roe R. User errors in human–computer interaction. In: Human Computer Interaction in the Work Place. Frese M, Ulich E, Dzida W (eds). Amsterdam: Elsevier; 1987: pp. 203–220.
3. Argyris C. On organizational learning. Oxford: Blackwell Publishers; 1992. 4. Keith N, Frese M. Self-regulation in error management training: Emotional control and metacognition as mediators of performance effects. J Appl Psychol 2005;90:677–691. 5. Chillarege K, Nordstrom C, Williams K. Learning from our mistakes: Error exposure training for mature learners. Journal of Business and Psychology 2003;17:369–385. 6. Gully S, Payne S, Koles K, Whiteman J. The impact of error training and individual differences on training outcomes: An attribute–treatment interaction perspective. J Appl Psychol 2002;87:143–155. 7. Keith N, Frese M. Effectiveness of error management training: A meta-analysis. J Appl Psychol 2008;93:59–69. 8. King A, Holder MG, Ahmed RA. Errors as allies: Error management training in health professions education. BMJ Qual Saf 2013;22:516–519. 9. Ziv A, Ben-David S, Ziv M. Simulation based medical education: an opportunity to learn from errors. Med Teach 2005; 27:193–199. 10. Joung W, Hesketh B, Neal A. Using, “war stories” to train for adaptive performance: Is it better to learn from error or success? App Psychol 2006;55:292–302.
Corresponding author’s contact details: Aimee K. Gardner, PhD, Surgery Department, UT Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, Texas, 75390–9092, USA. E-mail: [email protected]
Funding: None. Conflict of interest: None. Ethical approval: This project was deemed exempt from ethical approval by the University of Akron’s Institutional Review Board. doi: 10.1111/tct.12217
540 © 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540
tct_12217.indd 540
11/11/2014 4:12:29 PM
Copyright of Clinical Teacher is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Error management training and simulation education Aimee Gardner, Surgery Department, UT Southwestern Medical Center, Dallas, Texas, USA Michelle Rich, Radiology Department, Akron Children’s Hospital, Ohio, USA
SUMMARY Background: The integration of simulation into the training of health care professionals provides context for decision making and procedural skills in a high-fidelity environment, without risk to actual patients. It was hypothesised that a novel approach to simulation-based education – error management training – would produce higher performance ratings compared with traditional step-by-step instruction. Method: Radiology technology students were randomly assigned to participate in traditional
procedural-based instruction (n = 11) or vicarious error management training (n = 11). All watched an instructional video and discussed how well each incident was handled (traditional instruction group) or identified where the errors were made (vicarious error management training). Students then participated in a 30–minute case-based simulation. Simulations were videotaped for performance analysis. Blinded experts evaluated performance using a predefined evaluation tool created specifically for the scenario.
Results: The vicarious error management group scored higher on observer-rated performance (Mean = 9.49) than students in the traditional instruction group (Mean = 9.02; p < 0.01). Conclusions: These findings suggest that incorporating the discussion of errors and how to handle errors during the learning session will better equip students when performing hands-on procedures and skills. This pilot study provides preliminary evidence for integrating error management skills into medical curricula and for the design of learning goals in simulation-based education.
Blinded experts evaluated performance using a predefined evaluation tool created specifically for the scenario
© 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540 537
tct_12217.indd 537
11/11/2014 4:12:24 PM
People are more motivated to learn when things go wrong
INTRODUCTION
A
lthough traditional learning environments often reproach errors,1 theories as far back as 25 years ago suggest that errors can be highly informative when learners discover why such mistakes occur and how they can be corrected.2 Other learning theories suggest that people are more motivated to learn when things go wrong.3 Indeed, recent developments have indicated that errors can be beneficial in training settings. One specific educational approach, error management training (EMT), is a training method that incorporates active exploration and explicit encouragement for trainees to make errors during training, and to learn from them.4 EMT recognises that errors are a natural by-product of active learning, and that errors will inevitably emerge as students explore their environment. These errors, though, can play a fundamental role in trainee development. By working through any potential mistakes that could arise, learners are actively involved and can obtain a better mental model of the task or procedure at hand, what could go wrong, and strategies for managing the error. Research outside of the medical domain has demonstrated that EMT leads to better skill acquisition,5 decision making,6 transfer of training,7 and performance;7 however, its application to medical education has yet to be empirically validated.8 Because of the high-risk of errors within health care settings, however, the integration of EMT within medicine must be conducted in a safe setting. Simulation-based training is the ideal avenue for such education because it allows trainees to freely make and learn from mistakes, without risk to patient safety. As noted by others,9 the central premise of
simulation-based medical education is to provide students with increased practise in learning from mistakes, so as to reduce such mistakes in real life. Thus, EMT can be seamlessly incorporated within simulationbased education. The current study examines the effectiveness of these training techniques among radiology technician students. As radiographers perform critical imaging examinations to demonstrate fractures and other pathological conditions, their training can have an important impact on patient safety. Additionally, properly trained radiographers can help to optimise the efficiency of the health care provider, can increase patient throughput and can reduce the patient’s total wait time.
METHODS The pilot study was conducted at the Austen BioInnovation Institute, Akron, Ohio, a midwestern regional simulation centre. This initial pilot study was integrated into course clinical activity, and was therefore considered to be exempt for ethics approval by the University Institutional Review Board. Twenty-two radiology technician students in their first year of training participated in a 4–hour simulation-based education training programme. Students were randomly assigned to either
a traditional learning group (TLG; n = 11) or an error management group (EMG; n = 11). At the beginning of the training programme students participated in one of two case-based lectures, including a video demonstration. The TLG watched a video in which the radiology technicians performed procedures accurately throughout, whereas the EMG was shown the same patient presentation with radiology technicians who committed various errors. Mistakes performed in the EMG included unintentional errors as well as deliberate violations. Errors occurred in obtaining patient history (e.g. not obtaining initial pregnancy information), technical skills (e.g. using the wrong size of leads), safety (e.g. not shielding the patient), professionalism (e.g. chatting with a coworker and ignoring the patient) and efficiency (e.g. shooting images in illogical order). The video for the TLG, however, presented the same patient-based case scenario with the radiology technicians performing each behaviour correctly, and accurately, according to the protocol and policies in place. After the case study, each group participated in a short facilitator-led discussion similar to those used in other EMT studies.10 In the TLG the instructor discussed how well each incident was handled. Students were asked to discuss what went well and why it was important. In the EMG the instructor asked
538 © 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540
tct_12217.indd 538
11/11/2014 4:12:27 PM
students to systematically identify where the errors were made. Students were asked to provide examples of where errors were made, and what the implications of those errors might be.
training programme. As shown in Table 1, the EMG scored higher on observer-rated performance (Mean = 9.49, on a scale of 1–10) than students in the TLG (Mean = 9.02; p < 0.01). Additionally, and of note, the results reveal that the TLG had almost twice the variability of the EMG (TLG SD = 0.43; EMG SD = 0.24), suggesting that individuals within the TLG had performance scores that were more variable. Given that the EMG was very close to the maximum value, however, these differences in variability should be interpreted with caution.
Students then individually participated in a 30–minute scenario involving a case in which a patient had sustained a fall from a significant distance. The patient had injured her forearm, elbow and shoulder, and suffered from limited mobility. The student had to make any necessary adjustments to the equipment, while minimising the patient’s discomfort, and ultimately obtain quality diagnostic images (the same standardised patient actor was used during each scenario). Each simulation was recorded for performance ratings. Two blinded experts trained on the evaluation tool both rated each of the 22 video recordings.
DISCUSSION The findings from this study provide preliminary support for implementing error management strategies within medical education. Given the plethora of errors and mistakes committed by health care professionals, identifying avenues for educators to instill error reduction and management strategies among trainees is essential to enhance patient safety. Our study found that engaging in a discussion and meditation on one 30–minute error-ridden case scenario is powerful enough to have an impact on subsequent performance in a simulated scenario. These results suggest that actively considering the role of errors and their implications in facilitated discussion produces important cognitions concerning critical thinking, strategies and improvement techniques that have a direct impact on behaviour. Providing examples of solely correctly performed behaviours and procedures may not develop enriched mental models among health care students and
RESULTS Data analysis was conducted with SPSS 18.0 (SPSS Inc, Chicago, IL, USA). A significance level of p < 0.05 was chosen. Inter-rater reliability was assessed for the performance ratings with intraclass correlation coefficients (ICCs). The ICC value representing the reliability of ratings across raters was 0.734 (95% CI 0.541–0.871; p < 0.001), warranting the use of mean ratings of performance. Differences between performance ratings were analysed using an independent samples Student’s t–test. Twenty-two radiology technician students participated in the
Table 1. Summary of results Group
n
Mean (SD)
Difference
Traditional
11
9.01 (.43)
p < 0.01
Errors
11
9.49 (.24)
professionals in the same way that providing error-based stories can, combined with instruction about correct performance. Thus, errors can be the allies of medical educators.10
Errors can be the allies of medical educators
Although the beneficial roles of errors within medical education has not gone unacknowledged (i.e. morbidity and mortality conferences, autopsies, etc.), this study suggests that educators should make a pointed effort to discuss errors, their consequences, and strategies to manage them among learners. While the current study examined the effect of vicarious EMT, similar or even more powerful effects will undoubtedly be seen with more active error training. Having students commit the errors themselves and strategise to manage those errors may produce even more pronounced cognitive, emotional and behavioural outcomes. After all, students are more motivated to learn when things go wrong.3 Simulation-based education can uniquely complement this novel educational strategy, in that it provides a safe setting for students to make mistakes and learn from them. This research also highlights the efficaciousness of observational learning for educational training design. In settings in which learners are presented with practise examples, or observe other more senior residents or faculty staff, they should be encouraged to actively think about errors and to learn from them. For example, before beginning a new rotation, residents can be instructed to compile a list of procedures that they might encounter on that rotation, the errors frequently associated with performing those procedures, and how students plan to manage and adapt to those situations. By considering a range of alternative approaches to the procedure or task and thinking adaptively about the
© 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540 539
tct_12217.indd 539
11/11/2014 4:12:29 PM
Adopting error training in educational initiatives can...help revolutionise the culture of medical education
situation, learners will engage with the material at a deeper level.10 Adopting error training in educational initiatives can also help revolutionise the culture of medical education. Institutions that encourage the continued and open communication of errors, accidents and near misses can create an atmosphere in which faculty staff and learners are encouraged to voice concerns, which may ultimately lead to an increased awareness and to better safety systems. Despite the significant contributions of this pilot study, there are a few limitations that should be noted. The first limitation concerns the sample size of the study. Replication of these results with a larger and more diverse sample is needed to fully understand the effects of EMT. Although students were randomly assigned to engage in discussion of ideal versus error behaviours, we did not obtain data regarding the involvement of each student in the discussion. Thus, we cannot identify the extent to which students identified errors, merely listened or were completely unengaged in
the discussion. Finally, using a broader array of criteria would help extend the understanding of this educational process. The current study examined blinded observer-rated performance, but examining the impact of error training on more robust, novel or team-based performance events would be beneficial.
CONCLUSION These findings suggest that incorporating a discussion of errors and how to handle errors during the learning session will better equip students when performing handson procedures and skills. This pilot study provides preliminary evidence for integrating error management skills into medical curricula and designing learning goals in simulation-based education. REFERENCES 1. Frese M, Brodbeck FC, Heinbokel T, Mooser C, Schleiffenbaum E, Thiemann P. Errors in training computer skills: On the positive function of errors. Human-Computer Interaction 1991;6:77–93. 2. Arnold B, Roe R. User errors in human–computer interaction. In: Human Computer Interaction in the Work Place. Frese M, Ulich E, Dzida W (eds). Amsterdam: Elsevier; 1987: pp. 203–220.
3. Argyris C. On organizational learning. Oxford: Blackwell Publishers; 1992. 4. Keith N, Frese M. Self-regulation in error management training: Emotional control and metacognition as mediators of performance effects. J Appl Psychol 2005;90:677–691. 5. Chillarege K, Nordstrom C, Williams K. Learning from our mistakes: Error exposure training for mature learners. Journal of Business and Psychology 2003;17:369–385. 6. Gully S, Payne S, Koles K, Whiteman J. The impact of error training and individual differences on training outcomes: An attribute–treatment interaction perspective. J Appl Psychol 2002;87:143–155. 7. Keith N, Frese M. Effectiveness of error management training: A meta-analysis. J Appl Psychol 2008;93:59–69. 8. King A, Holder MG, Ahmed RA. Errors as allies: Error management training in health professions education. BMJ Qual Saf 2013;22:516–519. 9. Ziv A, Ben-David S, Ziv M. Simulation based medical education: an opportunity to learn from errors. Med Teach 2005; 27:193–199. 10. Joung W, Hesketh B, Neal A. Using, “war stories” to train for adaptive performance: Is it better to learn from error or success? App Psychol 2006;55:292–302.
Corresponding author’s contact details: Aimee K. Gardner, PhD, Surgery Department, UT Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, Texas, 75390–9092, USA. E-mail: [email protected]
Funding: None. Conflict of interest: None. Ethical approval: This project was deemed exempt from ethical approval by the University of Akron’s Institutional Review Board. doi: 10.1111/tct.12217
540 © 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 537–540
tct_12217.indd 540
11/11/2014 4:12:29 PM
Copyright of Clinical Teacher is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
