j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
Available online at www.sciencedirect.com
ScienceDirect journal homepage: www.JournalofSurgicalResearch.com
Association for Academic Surgery
Perception does not equal reality for resident vascular trauma skills Mark W. Bowyer, MD,a,* Stacy A. Shackelford, MD,b Evan Garofalo, PhD,c Kristy Pugh, MS,d and Colin F. Mackenzie, MBChBd a
The Norman M. Rich Department of Surgery, Uniformed Services University of the Health Sciences, Bethesda, Maryland b USAF Center for Sustainment of Trauma and Readiness Skills, R Adams Cowley Shock Trauma Center, University of Maryland, Baltimore, Maryland c Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland d Shock Trauma Anesthesiology Research Center, University of Maryland School of Medicine, Baltimore, Maryland
article info
abstract
Article history:
Background: Experience with the management of vascular trauma by senior surgical resi-
Received 2 January 2015
dents is increasingly limited. When queried about their understanding of anatomy and
Received in revised form
ability to perform specific vascular exposures, residents express a moderately high level of
17 March 2015
confidence. We hypothesized that this perception does not equal reality.
Accepted 27 March 2015
Methods: A total of 42 senior surgical residents participating in an ongoing validation study
Available online 1 April 2015
of the Advanced Surgical Skills for Exposures in Trauma course were asked to self-assess their baseline (precourse) confidence of their understanding of the anatomy required to
Keywords:
perform and their ability to perform exposure and control of the axillary, brachial, and
Vascular exposure
femoral arteries, as well as lower extremity fasciotomy using a 5-point Likert scale. Resi-
Trauma
dents then performed the four procedures on a fresh cadaver model and were scored in
Surgical education
real time by experts using a global assessment of anatomic knowledge and readiness to
Surgical skills
perform.” The Student t-test was used with a set at P < 0.05.
Self-assessment
Results: Residents consistently rated their understanding of anatomy and their ability to
Anatomical knowledge
perform the procedures significantly higher than expert evaluator ultimately scored them. Evaluators also deemed that residents would be unable to perform without help 65%e86% of the time. Conclusions: Senior residents are ill-prepared to perform the procedures studied and have an unwarranted confidence in their knowledge and abilities. Perception clearly does not equal reality in preparing these trainees to perform as advertized. The low global scores for anatomy and performance should be a wake-up call for surgical educators prompting curricular reform and evaluation. Published by Elsevier Inc.
* Corresponding author. The Norman M. Rich Department of Surgery, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. Tel.: þ1 301 295 8155; fax: þ1 301 295 7268. E-mail address: [email protected] (M.W. Bowyer). 0022-4804/$ e see front matter Published by Elsevier Inc. http://dx.doi.org/10.1016/j.jss.2015.03.082
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
1.
Introduction
Traumatic injury continues to be a leading cause of worldwide morbidity and mortality [1,2]. Rapid identification and control of bleeding blood vessels are paramount to survival. Additionally, the ability to perform limb-saving skills such as extremity fasciotomy is vital to optimal outcomes. It is therefore vital that surgical trainees be proficient in the management of trauma to include the exposure and operative control of major hemorrhage. Trauma operative management has experienced a significant decline since the introduction of resident workhour restrictions [3e5]. Additionally, the increasing reliance on the nonoperative management of many traumatic injuries has further limited operative experience during training [6e9]. A recent 20-y review of Accreditation Council for Graduate Medical Education caselogs demonstrated that graduating chief residents performed half the number of designated trauma operations (39.4 21) compared with those of two decades ago (72.5 46) [5]. The average number of trauma and vascular cases submitted to the American Board of Surgery for 2014 graduates of U.S. surgical residencies is remarkably low. In fact, the average number of operative cases for vascular trauma (all vessels) over a 5-y residency was reported as only 2.1, with exposure and repair of peripheral arteries reported as 1.0 and fasciotomy for trauma reported as 0.8. Additionally, nontraumatic vascular experience is also limited with average numbers of cases involving any exposure of the axillary artery at 0.6, open brachial artery exposure at 0.1, and femoral artery at approximately 10.0 [10]. The National Resident Report also lists operative experiences in which the brachial artery might be exposed, namely arteriovenous fistulas and grafts as well as with revision of arteriovenous access with averages of 17.7, 6.6, and 6.3 over a 5-y residency [10]. This limited experience is of great concern, as several investigators have identified experience as one factor in gaining competence [11e13]. An additional consequence of declining experience is a decrease in surgeons’ confidence to manage injuries and the potential increase in morbidity and mortality [14]. It is increasingly clear that current surgical trainees receive a very limited exposure to the operative management of vascular trauma and that we are graduating specialists who when called on to care for victims of trauma may or may not have the requisite skillset to ensure optimal outcomes. As such there is a critical need to change the way we train and maintain the skills of the surgeon caring for trauma in the future. To meet this perceived need, the American College of Surgeons Committee on Trauma leadership established a Surgical Skills Committee in 2005. This committee was charged with developing a standardized, skills-based cadaver course designed to teach surgical exposure of vital structures that are most likely to pose an immediate threat to life or limb when injured [14e17]. The result of this effort was the development of the Advanced Surgical Skills for Exposure in Trauma (ASSET) course. We have previously reported the experience with the first 25 ASSET courses in a prospective fashion detailing marked improvement over baseline of self-reported comfort levels and confidence in all the skills taught in the course by virtue of
281
taking the course [17]. The course is continuing to be evaluated in a prospective fashion, and we are conducting a study to validate the course skills and evaluate skills retention and degradation. In this ongoing study, we are testing the baseline skills of novice (have not taken the ASSET course or received the course materials) senior surgical residents on four of the skills taught in the course as follows: exposure and control of the axillary artery; the brachial artery; the femoral artery (to include control of the common femoral, superficial femoral (SFA), and profunda); and performance of a two incision, four compartment fasciotomy of the lower leg. After establishing baseline performance, the subjects participate in an ASSET course as a student and are then retested to establish a postcourse trained baseline and will return at either 12 or 18 mo to test for skills retention and durability. While conducting this study, it was noted anecdotally that residents tended to overestimate their knowledge of anatomy and their ability to perform these procedures. Based on this observation, we hypothesized that the resident participants self-reported perception of skills and knowledge does not equal reality when asked to perform. This article details the resultant study to test this hypothesis.
2.
Methods
The studies described in this article were approved by the University of Maryland School of Medicine Institutional Review Board, the Maryland State Anatomy Board, and the US Army Office of Research Protection for research involving humans, human data, human specimens, or cadavers. Research subjects underwent a consent process and completed informed consent before participation. Senior surgical residents (postgraduate year [PGY] 3e5) participating in an ongoing study to validate the ASSET course comprised the subject population in this report. Between August 2013 and July 2014, 42 surgical residents from 11 different residency programs in the greater Baltimore area and adjacent states were recruited to participate in the aforementioned ASSET validation study. Ten of these residents were used to establish a baseline performance on four skills taught in the ASSET course; exposure and control of the axillary artery, brachial artery, femoral artery (common, SFA, and profunda) at the groin, and fasciotomy of the lower extremity before participation in the course. The other 32 residents are involved in a longitudinal study of the ASSET course in which they established a baseline performance on the four skills listed previously, followed by a reevaluation on these skills after participation in an ASSET course and then again at 12 or 18 mo to validate the course and evaluate retention of the skills learned. The metrics used to evaluate the resident performances were developed by interviewing four expert surgeons regarding the key points of patient management and decision making for each of the procedures to include key anatomic landmarks and structures, common errors, and complications. Ten expert surgeons were then asked to perform the procedures on a fresh cadaver model while talking out loud with audio and visual capture using a cognitive work analysis
282
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
methodology [18,19]. The “experts” who were interviewed and used to establish optimal performance were experienced trauma surgeons and ASSET instructors practicing in level I trauma centers in the greater Baltimore and Washington, DC areas. Using the detailed task deconstruction and the recorded performance of experts, a consensus group composed of two expert trauma surgeons, a nonsurgeon trauma clinician and/ or researcher, a human factors psychologist, and two human anatomists who developed performance metrics for the four procedures. A case-based script was developed for each of the four procedures allowing for a standardized interrogation of both cognitive and technical skills. Anatomic and clinical management, procedure steps, and technical skills were scored with binary yes or no answers. Additionally, global ratings using a 5-point Likert scale were developed for each procedure to rate the overall understanding of the surgical anatomy and an assessment by the evaluator if the participant is currently ready to perform the procedure in question (Table 1). Finally, an overall rating of 1e100 was included with 100 being the idealized “expert” surgeon performance with the following broad categories: 90e100 “Excellent, I hope this individual is on call if I am injured”; 80e89 “This individual will be able to perform the exposure with minimal difficulty in an expeditious fashion”; 70e79 “The participant might need to look at a text to refresh their memory but will be able to perform the procedure”; 60e69 “This participant could do the exposure only with experienced help but will struggle if left alone”; and 0.80) using video clips from expert and novice performances to illustrate components of the surgical performance to be evaluated. Evaluators subsequently applied the metrics to both real-time evaluation of surgical residents and blinded review of novice and expert surgeon video recordings. The resultant inter-rater reliability (>0.70) has established these performance assessment tools as valid
(unpublished data). Additionally, the performance assessment tool has been loaded onto Android-based tablet platforms to allow rapid and secure data entry. For the purposes of the present reported study, the 42 surgical residents were all “novices” in that they had not yet taken the ASSET course or received any of the instructional materials from the course. These residents were asked to complete a demographic questionnaire that detailed their level of training and their self-reported experience with the management of trauma cases. They were asked to state the number of vascular procedures that they had done for the upper and lower extremity, as well as the number of fasciotomies of the lower extremity. These numbers were selfreported and not verified by actual caselogs. Additionally, the subjects were asked to rate (on a 5-point Likert scale) their current confidence in their understanding of the anatomy required to perform as well as their current confidence in their ability to perform vascular exposure and control of the axillary artery; the brachial artery; the femoral artery (common, SFA, and profunda) at the groin; and performance of a two-incision four compartment fasciotomy of the lower extremity. The subjects each performed all four of the procedures as directed by the case-based standardized script without any coaching or correction of errors. They were allowed no >20 min to complete each of the procedures. The performance assessment tools were used by two real-time evaluators to include the script reader and a silent second expert evaluator. All data points for the performance assessment tool were collected, but for the purpose of this study, we specifically extracted the global ratings for understanding of anatomy and the evaluator’s assessment of whether the resident was currently able to perform the procedure for the four procedures in question. Additionally, we collected the overall global (1e100) rating by the evaluators for each of the four procedures as well. The residents’ level of training, self-reported trauma experience and case numbers, self-assessed knowledge of anatomy, and confidence in their ability were compared to the global assessments provided by the evaluators for each of the procedures. Statistical analysis was accomplished using the
Table 1 e Definitions of the 5-point Likert scale used by the evaluators to globally rate the participants on their overall knowledge of anatomy required, and the confidence that they would be able to perform each of the procedures. Definitions of the 5-point Likert scale 1
2
3
Overall understanding of necessary anatomy Inadequate Knowledge of regional Average understanding of the knowledge of the anatomy is below average. anatomy. May not be able to regional anatomy. Can name most of the immediately point out or name Unable to identify major structures but all the structures but can do so with minimal prompting major structures of requires some prompting their relationships Confidence that participant is ready to perform the procedure The participant might need to Take me to another This participant could do hospital please! the exposure fine only with look at a text to refresh their experienced help, but will memory but will be able to perform the exposure struggle if left alone
4
5
Above average understanding of anatomy. Able to point out all of the relevant structures without prompting
Superior grasp of the anatomy. Knows the minutia, should be teaching anatomy class
This individual will be able to perform the exposure with minimal difficulty in an expeditious fashion.
Absolutely, I hope that this individual is on call if I am injured
283
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
Table 2 e Self-reported trauma patient evaluation and relevant operative experience of 42 surgical residents to include the average plus or minus standard deviation, the range, and median. Self-reported experience
Average
Months on trauma service as resident Number of trauma Patients treated Percent with penetrating trauma Number of upper extremity open trauma vascular cases Number of lower extremity open trauma vascular cases Number of lower extremity open nontrauma vascular cases Number of lower extremity trauma fasciotomies Number of lower extremity nontrauma fasciotomies
5.66 158 19.8 1.37 2.7 21.4 3.05 3.19
t-test (modified for unequal variance) and the Pearson product-moment correlation coefficient with a set at P < 0.05.
3.
3.7 129 18.7% 1.6 3.7 19.7 3.7 3.0
Range
Median
1e16 10e650 0e80 0e5 0e20 0e100 0e16 0e10
4.5 100 10% 1.0 2.0 15 2.0 2.0
5-point Likert scale) were compared with the evaluators scoring of each individual (5-point Likert scale) on their global understanding of the anatomy for each procedure and their current ability to perform each procedure independently. As seen in Table 3, residents rated themselves significantly higher when compared with expert evaluators in their understanding of the anatomy required to perform all four tasks. Additionally, residents rated their current abilities to perform three of the four tasks (brachial artery and femoral artery exposures and lower leg fasciotomy) significantly higher than did the expert evaluators with a trend toward the same for the axillary artery exposure. A global overall score (1e100) was also provided by the evaluators for each of the subjects for each of the four procedures. The average global score for axillary artery exposure and control was 61.8 12.9; for brachial artery exposure and control 64.0 10.8; and for femoral artery (common, SFA, and profunda) exposure and control 66.8 9.72. The average global score for lower leg fasciotomy was 61.8 þ 7.76. With all these average global scores between 61 and 67, it should be remembered that a global score of 60e69 was defined in the performance assessment tool as “This participant could do the exposure only with experienced help but will struggle if left alone.” At the end of the day, it is important that the practicing surgeon be able to perform the requisite exposure or skill independently and ideally at the level of an idealized expert. The evaluators’ confidence of each resident to perform the four procedures was compared with each resident’s self-assessed confidence in their ability to perform these procedures as listed in Table 3. To further examine these differences and the current state of resident preparedness to do these procedures,
Results
The 42 surgical resident subjects in this study included 19 PGY 3 (45%), 15 PGY 4 (36%), and 8 PGY 5 (19%). The average age was 31.8 3.1 with a range of 27e41, and 18 (43%) were female and 24 (57%) were male. Self-reported months on the trauma service, number of trauma patients treated, percentage of penetrating trauma, and numbers for specific cases are detailed in Table 2. As seen in Table 2, the median time spent on the trauma service was 4.5 mo with 100 trauma patients evaluated treated of which 10% had penetrating trauma. The median self-reported operative caseload for vascular trauma and fasciotomy of the lower extremity were low as expected, ranging from 1e2 cases, with nontraumatic exposure of the femoral artery with a reported median of 15 cases (Table 2). When compared to the Accreditation Council for Graduate Medical Education National data set for graduating chief residents, it is important to note that the averages selfreported by the residents in this study are well above the national averages and would place them in most instances above the 70th percentile, even though the majority (81%) are still PGY 3s and 4s [10]. However, it is important to note that these numbers are self-reported and were not verified by actual caselog entries. The residents’ self-assessed comfort with their understanding of anatomy and their self-assessed confidence in their ability to perform each exposure or procedure (rated on a
Table 3 e The self-assessed confidence levels (5-point Likert scale) of 42 surgical residents for their understanding of the anatomy required, and their confidence in their ability to perform each of the four listed procedures compared with the global scores of evaluators on the same items. Procedure
Expose axillary artery Expose brachial artery Expose femoral artery Lower leg fasciotomy
Understanding of anatomy
Confidence of performance
Resident
Evaluator
P value
2.50 2.60 3.58 3.35
1.99 2.14 2.78 2.24
Available online at www.sciencedirect.com
ScienceDirect journal homepage: www.JournalofSurgicalResearch.com
Association for Academic Surgery
Perception does not equal reality for resident vascular trauma skills Mark W. Bowyer, MD,a,* Stacy A. Shackelford, MD,b Evan Garofalo, PhD,c Kristy Pugh, MS,d and Colin F. Mackenzie, MBChBd a
The Norman M. Rich Department of Surgery, Uniformed Services University of the Health Sciences, Bethesda, Maryland b USAF Center for Sustainment of Trauma and Readiness Skills, R Adams Cowley Shock Trauma Center, University of Maryland, Baltimore, Maryland c Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland d Shock Trauma Anesthesiology Research Center, University of Maryland School of Medicine, Baltimore, Maryland
article info
abstract
Article history:
Background: Experience with the management of vascular trauma by senior surgical resi-
Received 2 January 2015
dents is increasingly limited. When queried about their understanding of anatomy and
Received in revised form
ability to perform specific vascular exposures, residents express a moderately high level of
17 March 2015
confidence. We hypothesized that this perception does not equal reality.
Accepted 27 March 2015
Methods: A total of 42 senior surgical residents participating in an ongoing validation study
Available online 1 April 2015
of the Advanced Surgical Skills for Exposures in Trauma course were asked to self-assess their baseline (precourse) confidence of their understanding of the anatomy required to
Keywords:
perform and their ability to perform exposure and control of the axillary, brachial, and
Vascular exposure
femoral arteries, as well as lower extremity fasciotomy using a 5-point Likert scale. Resi-
Trauma
dents then performed the four procedures on a fresh cadaver model and were scored in
Surgical education
real time by experts using a global assessment of anatomic knowledge and readiness to
Surgical skills
perform.” The Student t-test was used with a set at P < 0.05.
Self-assessment
Results: Residents consistently rated their understanding of anatomy and their ability to
Anatomical knowledge
perform the procedures significantly higher than expert evaluator ultimately scored them. Evaluators also deemed that residents would be unable to perform without help 65%e86% of the time. Conclusions: Senior residents are ill-prepared to perform the procedures studied and have an unwarranted confidence in their knowledge and abilities. Perception clearly does not equal reality in preparing these trainees to perform as advertized. The low global scores for anatomy and performance should be a wake-up call for surgical educators prompting curricular reform and evaluation. Published by Elsevier Inc.
* Corresponding author. The Norman M. Rich Department of Surgery, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. Tel.: þ1 301 295 8155; fax: þ1 301 295 7268. E-mail address: [email protected] (M.W. Bowyer). 0022-4804/$ e see front matter Published by Elsevier Inc. http://dx.doi.org/10.1016/j.jss.2015.03.082
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
1.
Introduction
Traumatic injury continues to be a leading cause of worldwide morbidity and mortality [1,2]. Rapid identification and control of bleeding blood vessels are paramount to survival. Additionally, the ability to perform limb-saving skills such as extremity fasciotomy is vital to optimal outcomes. It is therefore vital that surgical trainees be proficient in the management of trauma to include the exposure and operative control of major hemorrhage. Trauma operative management has experienced a significant decline since the introduction of resident workhour restrictions [3e5]. Additionally, the increasing reliance on the nonoperative management of many traumatic injuries has further limited operative experience during training [6e9]. A recent 20-y review of Accreditation Council for Graduate Medical Education caselogs demonstrated that graduating chief residents performed half the number of designated trauma operations (39.4 21) compared with those of two decades ago (72.5 46) [5]. The average number of trauma and vascular cases submitted to the American Board of Surgery for 2014 graduates of U.S. surgical residencies is remarkably low. In fact, the average number of operative cases for vascular trauma (all vessels) over a 5-y residency was reported as only 2.1, with exposure and repair of peripheral arteries reported as 1.0 and fasciotomy for trauma reported as 0.8. Additionally, nontraumatic vascular experience is also limited with average numbers of cases involving any exposure of the axillary artery at 0.6, open brachial artery exposure at 0.1, and femoral artery at approximately 10.0 [10]. The National Resident Report also lists operative experiences in which the brachial artery might be exposed, namely arteriovenous fistulas and grafts as well as with revision of arteriovenous access with averages of 17.7, 6.6, and 6.3 over a 5-y residency [10]. This limited experience is of great concern, as several investigators have identified experience as one factor in gaining competence [11e13]. An additional consequence of declining experience is a decrease in surgeons’ confidence to manage injuries and the potential increase in morbidity and mortality [14]. It is increasingly clear that current surgical trainees receive a very limited exposure to the operative management of vascular trauma and that we are graduating specialists who when called on to care for victims of trauma may or may not have the requisite skillset to ensure optimal outcomes. As such there is a critical need to change the way we train and maintain the skills of the surgeon caring for trauma in the future. To meet this perceived need, the American College of Surgeons Committee on Trauma leadership established a Surgical Skills Committee in 2005. This committee was charged with developing a standardized, skills-based cadaver course designed to teach surgical exposure of vital structures that are most likely to pose an immediate threat to life or limb when injured [14e17]. The result of this effort was the development of the Advanced Surgical Skills for Exposure in Trauma (ASSET) course. We have previously reported the experience with the first 25 ASSET courses in a prospective fashion detailing marked improvement over baseline of self-reported comfort levels and confidence in all the skills taught in the course by virtue of
281
taking the course [17]. The course is continuing to be evaluated in a prospective fashion, and we are conducting a study to validate the course skills and evaluate skills retention and degradation. In this ongoing study, we are testing the baseline skills of novice (have not taken the ASSET course or received the course materials) senior surgical residents on four of the skills taught in the course as follows: exposure and control of the axillary artery; the brachial artery; the femoral artery (to include control of the common femoral, superficial femoral (SFA), and profunda); and performance of a two incision, four compartment fasciotomy of the lower leg. After establishing baseline performance, the subjects participate in an ASSET course as a student and are then retested to establish a postcourse trained baseline and will return at either 12 or 18 mo to test for skills retention and durability. While conducting this study, it was noted anecdotally that residents tended to overestimate their knowledge of anatomy and their ability to perform these procedures. Based on this observation, we hypothesized that the resident participants self-reported perception of skills and knowledge does not equal reality when asked to perform. This article details the resultant study to test this hypothesis.
2.
Methods
The studies described in this article were approved by the University of Maryland School of Medicine Institutional Review Board, the Maryland State Anatomy Board, and the US Army Office of Research Protection for research involving humans, human data, human specimens, or cadavers. Research subjects underwent a consent process and completed informed consent before participation. Senior surgical residents (postgraduate year [PGY] 3e5) participating in an ongoing study to validate the ASSET course comprised the subject population in this report. Between August 2013 and July 2014, 42 surgical residents from 11 different residency programs in the greater Baltimore area and adjacent states were recruited to participate in the aforementioned ASSET validation study. Ten of these residents were used to establish a baseline performance on four skills taught in the ASSET course; exposure and control of the axillary artery, brachial artery, femoral artery (common, SFA, and profunda) at the groin, and fasciotomy of the lower extremity before participation in the course. The other 32 residents are involved in a longitudinal study of the ASSET course in which they established a baseline performance on the four skills listed previously, followed by a reevaluation on these skills after participation in an ASSET course and then again at 12 or 18 mo to validate the course and evaluate retention of the skills learned. The metrics used to evaluate the resident performances were developed by interviewing four expert surgeons regarding the key points of patient management and decision making for each of the procedures to include key anatomic landmarks and structures, common errors, and complications. Ten expert surgeons were then asked to perform the procedures on a fresh cadaver model while talking out loud with audio and visual capture using a cognitive work analysis
282
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
methodology [18,19]. The “experts” who were interviewed and used to establish optimal performance were experienced trauma surgeons and ASSET instructors practicing in level I trauma centers in the greater Baltimore and Washington, DC areas. Using the detailed task deconstruction and the recorded performance of experts, a consensus group composed of two expert trauma surgeons, a nonsurgeon trauma clinician and/ or researcher, a human factors psychologist, and two human anatomists who developed performance metrics for the four procedures. A case-based script was developed for each of the four procedures allowing for a standardized interrogation of both cognitive and technical skills. Anatomic and clinical management, procedure steps, and technical skills were scored with binary yes or no answers. Additionally, global ratings using a 5-point Likert scale were developed for each procedure to rate the overall understanding of the surgical anatomy and an assessment by the evaluator if the participant is currently ready to perform the procedure in question (Table 1). Finally, an overall rating of 1e100 was included with 100 being the idealized “expert” surgeon performance with the following broad categories: 90e100 “Excellent, I hope this individual is on call if I am injured”; 80e89 “This individual will be able to perform the exposure with minimal difficulty in an expeditious fashion”; 70e79 “The participant might need to look at a text to refresh their memory but will be able to perform the procedure”; 60e69 “This participant could do the exposure only with experienced help but will struggle if left alone”; and 0.80) using video clips from expert and novice performances to illustrate components of the surgical performance to be evaluated. Evaluators subsequently applied the metrics to both real-time evaluation of surgical residents and blinded review of novice and expert surgeon video recordings. The resultant inter-rater reliability (>0.70) has established these performance assessment tools as valid
(unpublished data). Additionally, the performance assessment tool has been loaded onto Android-based tablet platforms to allow rapid and secure data entry. For the purposes of the present reported study, the 42 surgical residents were all “novices” in that they had not yet taken the ASSET course or received any of the instructional materials from the course. These residents were asked to complete a demographic questionnaire that detailed their level of training and their self-reported experience with the management of trauma cases. They were asked to state the number of vascular procedures that they had done for the upper and lower extremity, as well as the number of fasciotomies of the lower extremity. These numbers were selfreported and not verified by actual caselogs. Additionally, the subjects were asked to rate (on a 5-point Likert scale) their current confidence in their understanding of the anatomy required to perform as well as their current confidence in their ability to perform vascular exposure and control of the axillary artery; the brachial artery; the femoral artery (common, SFA, and profunda) at the groin; and performance of a two-incision four compartment fasciotomy of the lower extremity. The subjects each performed all four of the procedures as directed by the case-based standardized script without any coaching or correction of errors. They were allowed no >20 min to complete each of the procedures. The performance assessment tools were used by two real-time evaluators to include the script reader and a silent second expert evaluator. All data points for the performance assessment tool were collected, but for the purpose of this study, we specifically extracted the global ratings for understanding of anatomy and the evaluator’s assessment of whether the resident was currently able to perform the procedure for the four procedures in question. Additionally, we collected the overall global (1e100) rating by the evaluators for each of the four procedures as well. The residents’ level of training, self-reported trauma experience and case numbers, self-assessed knowledge of anatomy, and confidence in their ability were compared to the global assessments provided by the evaluators for each of the procedures. Statistical analysis was accomplished using the
Table 1 e Definitions of the 5-point Likert scale used by the evaluators to globally rate the participants on their overall knowledge of anatomy required, and the confidence that they would be able to perform each of the procedures. Definitions of the 5-point Likert scale 1
2
3
Overall understanding of necessary anatomy Inadequate Knowledge of regional Average understanding of the knowledge of the anatomy is below average. anatomy. May not be able to regional anatomy. Can name most of the immediately point out or name Unable to identify major structures but all the structures but can do so with minimal prompting major structures of requires some prompting their relationships Confidence that participant is ready to perform the procedure The participant might need to Take me to another This participant could do hospital please! the exposure fine only with look at a text to refresh their experienced help, but will memory but will be able to perform the exposure struggle if left alone
4
5
Above average understanding of anatomy. Able to point out all of the relevant structures without prompting
Superior grasp of the anatomy. Knows the minutia, should be teaching anatomy class
This individual will be able to perform the exposure with minimal difficulty in an expeditious fashion.
Absolutely, I hope that this individual is on call if I am injured
283
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 8 ( 2 0 1 5 ) 2 8 0 e2 8 8
Table 2 e Self-reported trauma patient evaluation and relevant operative experience of 42 surgical residents to include the average plus or minus standard deviation, the range, and median. Self-reported experience
Average
Months on trauma service as resident Number of trauma Patients treated Percent with penetrating trauma Number of upper extremity open trauma vascular cases Number of lower extremity open trauma vascular cases Number of lower extremity open nontrauma vascular cases Number of lower extremity trauma fasciotomies Number of lower extremity nontrauma fasciotomies
5.66 158 19.8 1.37 2.7 21.4 3.05 3.19
t-test (modified for unequal variance) and the Pearson product-moment correlation coefficient with a set at P < 0.05.
3.
3.7 129 18.7% 1.6 3.7 19.7 3.7 3.0
Range
Median
1e16 10e650 0e80 0e5 0e20 0e100 0e16 0e10
4.5 100 10% 1.0 2.0 15 2.0 2.0
5-point Likert scale) were compared with the evaluators scoring of each individual (5-point Likert scale) on their global understanding of the anatomy for each procedure and their current ability to perform each procedure independently. As seen in Table 3, residents rated themselves significantly higher when compared with expert evaluators in their understanding of the anatomy required to perform all four tasks. Additionally, residents rated their current abilities to perform three of the four tasks (brachial artery and femoral artery exposures and lower leg fasciotomy) significantly higher than did the expert evaluators with a trend toward the same for the axillary artery exposure. A global overall score (1e100) was also provided by the evaluators for each of the subjects for each of the four procedures. The average global score for axillary artery exposure and control was 61.8 12.9; for brachial artery exposure and control 64.0 10.8; and for femoral artery (common, SFA, and profunda) exposure and control 66.8 9.72. The average global score for lower leg fasciotomy was 61.8 þ 7.76. With all these average global scores between 61 and 67, it should be remembered that a global score of 60e69 was defined in the performance assessment tool as “This participant could do the exposure only with experienced help but will struggle if left alone.” At the end of the day, it is important that the practicing surgeon be able to perform the requisite exposure or skill independently and ideally at the level of an idealized expert. The evaluators’ confidence of each resident to perform the four procedures was compared with each resident’s self-assessed confidence in their ability to perform these procedures as listed in Table 3. To further examine these differences and the current state of resident preparedness to do these procedures,
Results
The 42 surgical resident subjects in this study included 19 PGY 3 (45%), 15 PGY 4 (36%), and 8 PGY 5 (19%). The average age was 31.8 3.1 with a range of 27e41, and 18 (43%) were female and 24 (57%) were male. Self-reported months on the trauma service, number of trauma patients treated, percentage of penetrating trauma, and numbers for specific cases are detailed in Table 2. As seen in Table 2, the median time spent on the trauma service was 4.5 mo with 100 trauma patients evaluated treated of which 10% had penetrating trauma. The median self-reported operative caseload for vascular trauma and fasciotomy of the lower extremity were low as expected, ranging from 1e2 cases, with nontraumatic exposure of the femoral artery with a reported median of 15 cases (Table 2). When compared to the Accreditation Council for Graduate Medical Education National data set for graduating chief residents, it is important to note that the averages selfreported by the residents in this study are well above the national averages and would place them in most instances above the 70th percentile, even though the majority (81%) are still PGY 3s and 4s [10]. However, it is important to note that these numbers are self-reported and were not verified by actual caselog entries. The residents’ self-assessed comfort with their understanding of anatomy and their self-assessed confidence in their ability to perform each exposure or procedure (rated on a
Table 3 e The self-assessed confidence levels (5-point Likert scale) of 42 surgical residents for their understanding of the anatomy required, and their confidence in their ability to perform each of the four listed procedures compared with the global scores of evaluators on the same items. Procedure
Expose axillary artery Expose brachial artery Expose femoral artery Lower leg fasciotomy
Understanding of anatomy
Confidence of performance
Resident
Evaluator
P value
2.50 2.60 3.58 3.35
1.99 2.14 2.78 2.24
