RANDOMIZED CONTROLLED TRIAL
Video-based Peer Feedback Through Social Networking for Robotic Surgery Simulation A Multicenter Randomized Controlled Trial Stacey C. Carter, MD,∗ Alexander Chiang, MD,† Galaxy Shah, MD,‡ Lorna Kwan, MPH,∗ Jeffrey S. Montgomery, MD,‡ Amer Karam, MD,§ Christopher Tarnay, MD,† Khurshid A. Guru, MD, and Jim C. Hu, MD, MPH∗
Objective: To examine the feasibility and outcomes of video-based peer feedback through social networking to facilitate robotic surgical skill acquisition. Background: The acquisition of surgical skills may be challenging for novel techniques and/or those with prolonged learning curves. Methods: Randomized controlled trial involving 41 resident physicians performing the Tubes (Da Vinci Intuitive Surgical, Sunnyvale, CA) simulator exercise with versus without peer feedback of video-recorded performance through a social networking Web page. Data collected included simulator exercise score, time to completion, and comfort and satisfaction with robotic surgery simulation. Results: There were no baseline differences between the intervention group (n = 20) and controls (n = 21). The intervention group showed improvement in mean scores from session 1 to sessions 2 and 3 (60.7 vs 75.5, P < 0.001, and 60.7 vs 80.1, P < 0.001, respectively). The intervention group scored significantly higher than controls at sessions 2 and 3 (75.5 vs 59.6, P = 0.009, and 80.1 vs 65.9, P = 0.019, respectively). The mean time (seconds) to complete the task was shorter for the intervention group than for controls during sessions 2 and 3 (217.4 vs 279.0, P = 0.004, and 201.4 vs 261.9, P = 0.006, respectively). At the study conclusion, feedback subjects were more comfortable with robotic surgery than controls (90% vs 62%, P = 0.021) and expressed greater satisfaction with the learning experience (100% vs 67%, P = 0.014). Of the intervention subjects, 85% found that peer feedback was useful and 100% found it effective. Conclusions: Video-based peer feedback through social networking appears to be an effective paradigm for surgical education and accelerates the robotic surgery learning curve during simulation. Keywords: learning curve, randomized control trial, robotic surgery, simulation, surgical education (Ann Surg 2015;261:870–875)
S
urgical education has evolved over the past century, adapting to advances in surgical technology and adjusting to the demands of increasingly stringent regulations on residency training. The traditional foundation for surgeon training is based on the Halsted model for apprenticeship; namely, younger surgeons learn from their mentors, who lead and teach based on prior experience and individual
From the ∗ Department of Urology, UCLA, Los Angeles, CA; †Department of Obstetrics and Gynecology, UCLA, Los Angeles, CA; ‡Department of Urology, University of Michigan, Ann Arbor, MI; §Department of Obstetrics and Gynecology, Stanford, Palo Alto, CA; and Roswell Park Cancer Institute, Buffalo, NY. Disclosure: No funding was received in support of this work. All authors have no conflicts of interest, including financial, consultant, institutional, or other. Reprints: Jim C. Hu, MD, MPH, Department of Urology, UCLA, 924 Westwood Blvd, Ste 1000, Los Angeles, CA 90024. E-mail: [email protected]. C 2014 Wolters Kluwer Health, Inc. All rights reserved. Copyright ISSN: 0003-4932/14/26105-0870 DOI: 10.1097/SLA.0000000000000756
870 | www.annalsofsurgery.com
gestalt.1 This relatively unstructured approach remained the standard until the 1990s when Reznick2 applied a framework of systematic teaching using models and learning principles with formalized assessment. Since this time, techniques for efficient and productive learning have been proposed, fueled by increased restrictions on resident work hours and oversight by regulatory committees.3–5 In addition, it is challenging after completion of residency or fellowship to advance efficiently along surgical learning curves, particularly with the advent of new technologies and techniques. For instance, Hu et al6 described the benefit of intraoperative mentorship from more experienced surgeons, beyond completion of residency training. However, this is often difficult to do because of busy surgeon schedules and the hierarchical nature of surgery.6 Birkmeyer et al7 recently demonstrated through video review that better surgeon technique was associated with better outcomes. In addition, postoperative video debriefing has shown to be an effective educational tool, with broad implications for reducing adverse events,8 and expert surgeons also benefit from immediate structured feedback from colleagues.6 Moreover, surgical societies are turning to the Internet as a forum for Web-based education modules and for discussion and interaction among residents and physicians.9–11 Recently, there has been a deluge of negative press surrounding injuries attributed to robotic assisted surgery, and Web pages such as badrobotsurgery.com have been launched to attract potential plaintiffs. Robotic surgery has a prolonged learning curve, and robotic surgery simulators aid the transition for neophytes.12 In addition, robotic skills acquired through simulation are transferrable to the operating room.13 Given the need for novel paradigms of surgical training, the goal of our study was to conduct a randomized controlled trial to evaluate peer feedback through social networking for acquisition of robotic surgery skills through simulation. We hypothesized that trainees receiving online feedback regarding robotic simulator performance and technique from their peers acquire skills more rapidly than those without feedback.
METHODS Our study was approved by the University of California Los Angeles (UCLA) and University of Michigan institutional review boards. We recruited residents from the UCLA urology and obstetrics and gynecology programs and the University of Michigan urology program. Participants were recruited via e-mail, and participation was voluntary. A power calculation estimated that 25 participants were required in each arm to achieve 97% power to detect a 7% improvement from the intervention. Of the 71 eligible subjects, 53 participants were randomized (Fig. 1) into 2 groups using an online program (www.random.org) and completed a prestudy survey to assess prior intraoperative and simulator experience with robotic assisted surgery. A total of 41 subjects completed the Tubes robotic simulator exercise (Da Vinci Intuitive Surgical, Sunnyvale, CA) 3 times over a 6-week period, 20 in the intervention group and 21 from the control group. Annals of Surgery r Volume 261, Number 5, May 2015
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
Peer Feedback Through Social Networking
tion and control group subjects received their own simulator scores and times to completion, which are generated at the completion of the exercise. A poststudy questionnaire was administered to all subjects to assess comfort level and satisfaction with simulator training. In addition, intervention subjects were asked whether they felt peer feedback was useful or effective.
Statistical Analysis We analyzed data for subjects who completed all 3 simulator sessions. We compared the characteristics between the intervention and control groups with the χ 2 and Fisher exact tests. We also compared the comfort level with the simulator before and after the intervention and satisfaction with the simulator experience. Our main outcomes of interest included the simulation score and completion time, measured in seconds. We calculated the adjusted mean scores for each intervention group at each of the 3 sessions with repeated-measures analyses and compared intervention subjects with controls. In addition, we tested within groups to assess for improvement over time. All tests were 2-sided and conducted in SAS 9.3 (SAS Inc, Cary, NC).
RESULTS
FIGURE 1. Enrollment and randomization. OB-GYN indicates obstetrics and gynecology. The Tubes exercise simulates a running anastamotic suture and has been validated as a training tool.12 Subjects randomized to the intervention group video recorded and uploaded their simulation sessions to the Google+ social networking site in a de-identified fashion. Google+ was selected because it allows uploading an unlimited number of large video files, which were viewable from any computer and did not require access to the institution VPN. Thus, participants at multiple institutions may access the videos. Log-in and video access was not public and was restricted only to study participants. All intervention subjects logged onto a common Google+ account to give blinded feedback regarding peer performance and to receive feedback on their own training session videos (Fig. 2). Video and comment from each study participant were labeled with the study participant’s unique identification number. Feedback after the first and second sessions comprised both open commentary and structured feedback through GEARS, a modified validated robotic skill assessment form.14 Although controls also performed the Tubes exercise 3 times over a 6-week period, they were not given access to social networking, nor did they receive peer feedback. Both interven C 2014 Wolters Kluwer Health, Inc. All rights reserved.
Characteristics of the intervention and control groups did not differ (Table 1). Although mean scores were similar for session 1 (60.7 vs 53.5, P = 0.314), intervention subjects scored higher for the second (mean 75.5 vs 59.6, P = 0.009) and third sessions (mean 80.1 vs 65.9, P = 0.019) than controls (Table 2). In addition, the intervention group demonstrated significant improvement between the first and second sessions (60.7 vs 75.5, P < 0.001) whereas there was only a trend for improvement in controls (53.5 vs 59.6, P = 0.080). Moreover, mean scores improved between sessions 1 and 3 for both the intervention group and controls (60.7 vs 80.1, P < 0.001, and 53.5 vs 65.9, P < 0.001, respectively). Both groups had similar mean times in seconds to complete the first session (287.2 for intervention vs 361.1 for control, P = 0.083). Although both groups demonstrated shorter times to complete simulation over the course of the study, the times were significantly shorter for the intervention group than for controls for sessions 2 (mean 217.4 vs 279.0, P = 0.004) and 3 (mean 201.4 vs 261.9, P = 0.006). In a sensitivity analysis, we ran a second repeated-measures analysis to adjust for prior robotic experience by including the number of cases logged as a covariable (P = 0.161). The results were similar to the original analysis and therefore we reported the results from the original regression. In terms of poststudy survey findings, the feedback group was more comfortable with robotic surgery than controls (90% vs 62%, P = 0.02105) (Table 3). In addition, 100% of the intervention group compared with 67% of controls (P = 0.0138) expressed satisfaction with the learning exercise. Finally, within the intervention group, 85% found peer feedback to be useful and 100% found it to be effective.
DISCUSSION With rapid advancement of surgical tools and technologies, surgeons must constantly update their armamentarium. As such, continued professional development and learning are critical to maintaining a contemporary and competitive skill set. Walsh et al15 demonstrated that surgeon video review leads to improved technique and outcomes. In addition, a recent study by Hu et al6 suggests that career-long continuous education to further improve surgical technique and judgment is beneficial even among experienced surgeons. This is increasingly important, as highly publicized complications receive national attention involving the use of new surgical technologies, such as robotic assisted surgery, by undertrained surgeons, and this is of great concern for patient safety.16 www.annalsofsurgery.com | 871
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
Carter et al
FIGURE 2. Sample screenshot and peer feedback. Birkmeyer et al7 demonstrated that the technical skill of practicing surgeons for laparoscopic bariatric surgery was highly variable and surgeons deemed more skillful by their peers also experienced fewer postoperative complications. The same study found that peer review of a recorded operation was effective in accurately assessing a surgeon’s level of proficiency.7 Likewise, many efforts have been made to decrease variability among surgeons to improve quality and eliminate heterogeneity that may contribute to poor outcomes.17,18 In contrast to medical therapies that are standardized by medication type or dosage, subtle differences in surgical technique lead to significant differences in outcomes. For instance, radical prostatectomy has a challenging learning curve with a series of complex steps that contributes to variation in surgeon technique and outcomes. At a high-volume cancer referral center, the adjusted 1-year postprostatectomy continence rate ranged from 65% to 96% whereas that of erectile function/potency ranged from 8% to 50%.18 Therefore, there is opportunity for improving the significant variation in radical prostatectomy functional outcome recovery; however, barriers to improvement of surgeon technique include departmental hierarchy and surgeon ego. Anonymous peer feedback of surgical video may overcome these challenges and facilitate continuous quality improvement. Our study has several important findings. First, peer feedback was associated with better simulation scores for the intervention group than for controls. Traditionally, surgical training is structured so that trainees are taught by attending surgeons who pass along relatively unstructured feedback and instruction during a live operation. Our study shows that feedback provided at a peer level has benefit for acquisition and improvement of technical skill. Moreover, survey findings reinforce these findings because all intervention subjects found feedback to be effective. Second, the time to complete the exercise was significantly shorter in the intervention group than in controls in sessions 2 and 3, respectively. Operative time is often used as a tool to estimate surgical 872 | www.annalsofsurgery.com
learning curves. A recent study by Tasian et al19 illustrates this point by demonstrating that the median operative time for fellows performing robotic assisted pyeloplasty decreases with each subsequent operation. This finding has also been demonstrated among surgeons performing robotic assisted radical prostatectomy.20 Furthermore, this is consistent with population-based findings that high-volume surgeons have shorter operative times for radical prostatectomies.21 The findings from our study support this previously established notion that the time it takes to perform a task decreases as more experience is gained. However, although both of our study groups took less time to perform the simulation over time, those receiving peer feedback had significantly shorter times than the control group, suggesting that the utilization of online video-based peer feedback helps promote faster acquisition of skills and accelerates the learning curve. Third, peer feedback was associated with greater comfort level with the robotic surgery and greater learning satisfaction than controls. By utilizing an online forum, feedback was given and received on a trainee’s own time and convenience. This is particularly important in a time of increasing resident work-hour restrictions and greater need posttraining for medical record documentation. By demonstrating that online feedback allows for improved performance of a surgical skill, novel forums and strategies for surgical training and mentorship at both a trainee level and an expert level should be explored to reduce variation in technique and improve patient outcomes. Finally, our study demonstrates the efficacy of peer feedback through social networking, a novel paradigm for technical mentorship. Although our study involved a simulated surgical exercise, prior studies have illustrated that the acquisition of simulated surgical skills are transferable to a live operative setting.13 A benefit of using social networking as a forum is that it may serve as a vehicle for neophyte surgeons to receive mentorship from experts without the limitation of geographic distance or time constraints. Peer feedback of online surgical video offers anonymity for all involved and therefore removes C 2014 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
Peer Feedback Through Social Networking
TABLE 1. Study Participant Characteristics Total (N = 41) Variable
Intervention (n = 20)
%
Sex Male 59 Female 41 Field Urology 80 OB-GYN 20 Postgraduate year 1–3 54 4–5 20 6 or fellow 27 Robotic surgery experience (no. cases) 0 24 1–5 17 5–10 17 10–15 15 15–20 12 20+ 15 Robotic simulation in curriculum Yes 44 No 56 Prior simulator experience Yes 80 No 20 Prior time on the simulator, h 0 24 1–5 59 5–10 5 10–15 10 15–20 2 Prior sessions on the simulator 0 20 1–5 59 5–10 10 10–15 10 15–20 0 20+ 2
Control (n = 21)
n
%
n
%
n
P
24 17
65 35
13 7
52 48
11 10
0.4123
33 8
85 15
17 3
76 24
16 5
0.6965∗
22 8 11
45 25 30
9 5 6
62 14 24
13 3 5
0.5414∗
10 7 7 6 5 6
20 5 25 25 10 15
4 1 5 5 2 3
29 29 10 5 14 14
6 6 2 1 3 3
0.1610∗
18 23
40 60
8 12
48 52
10 11
0.6232
33 8
85 15
17 3
76 24
16 5
0.6965∗
10 24 2 4 1
15 60 10 15 0
3 12 2 3 0
33 57 0 5 5
7 12 0 1 1
0.2482∗
8 24 4 4 0 1
15 60 10 10 0 5
3 12 2 2 0 1
24 57 10 10 0 0
5 12 2 2 0 0
0.9513∗
All P values from the χ 2 or Fisher exact tests when denoted with an asterisk. OB-GYN indicates obstetrics and gynecology.
TABLE 2. Comparison of Adjusted Means for Simulator Score and Completion Time Variable Tubes exercise score Session 1 Session 2 Session 3 Session 1 vs 2 Session 1 vs 3 Time to completion Session 1 Session 2 Session 3 Session 1 vs 2 Session 1 vs 3
Total (N = 41), Mean (SE)
Intervention (n = 20), Mean (SE)
Control (n = 21), Mean (SE)
57.1 (3.5) 67.5 (2.9) 73.0 (2.9)
60.7 (5.0) 75.5 (4.1) 80.1 (4.1) 0.0001
Video-based Peer Feedback Through Social Networking for Robotic Surgery Simulation A Multicenter Randomized Controlled Trial Stacey C. Carter, MD,∗ Alexander Chiang, MD,† Galaxy Shah, MD,‡ Lorna Kwan, MPH,∗ Jeffrey S. Montgomery, MD,‡ Amer Karam, MD,§ Christopher Tarnay, MD,† Khurshid A. Guru, MD, and Jim C. Hu, MD, MPH∗
Objective: To examine the feasibility and outcomes of video-based peer feedback through social networking to facilitate robotic surgical skill acquisition. Background: The acquisition of surgical skills may be challenging for novel techniques and/or those with prolonged learning curves. Methods: Randomized controlled trial involving 41 resident physicians performing the Tubes (Da Vinci Intuitive Surgical, Sunnyvale, CA) simulator exercise with versus without peer feedback of video-recorded performance through a social networking Web page. Data collected included simulator exercise score, time to completion, and comfort and satisfaction with robotic surgery simulation. Results: There were no baseline differences between the intervention group (n = 20) and controls (n = 21). The intervention group showed improvement in mean scores from session 1 to sessions 2 and 3 (60.7 vs 75.5, P < 0.001, and 60.7 vs 80.1, P < 0.001, respectively). The intervention group scored significantly higher than controls at sessions 2 and 3 (75.5 vs 59.6, P = 0.009, and 80.1 vs 65.9, P = 0.019, respectively). The mean time (seconds) to complete the task was shorter for the intervention group than for controls during sessions 2 and 3 (217.4 vs 279.0, P = 0.004, and 201.4 vs 261.9, P = 0.006, respectively). At the study conclusion, feedback subjects were more comfortable with robotic surgery than controls (90% vs 62%, P = 0.021) and expressed greater satisfaction with the learning experience (100% vs 67%, P = 0.014). Of the intervention subjects, 85% found that peer feedback was useful and 100% found it effective. Conclusions: Video-based peer feedback through social networking appears to be an effective paradigm for surgical education and accelerates the robotic surgery learning curve during simulation. Keywords: learning curve, randomized control trial, robotic surgery, simulation, surgical education (Ann Surg 2015;261:870–875)
S
urgical education has evolved over the past century, adapting to advances in surgical technology and adjusting to the demands of increasingly stringent regulations on residency training. The traditional foundation for surgeon training is based on the Halsted model for apprenticeship; namely, younger surgeons learn from their mentors, who lead and teach based on prior experience and individual
From the ∗ Department of Urology, UCLA, Los Angeles, CA; †Department of Obstetrics and Gynecology, UCLA, Los Angeles, CA; ‡Department of Urology, University of Michigan, Ann Arbor, MI; §Department of Obstetrics and Gynecology, Stanford, Palo Alto, CA; and Roswell Park Cancer Institute, Buffalo, NY. Disclosure: No funding was received in support of this work. All authors have no conflicts of interest, including financial, consultant, institutional, or other. Reprints: Jim C. Hu, MD, MPH, Department of Urology, UCLA, 924 Westwood Blvd, Ste 1000, Los Angeles, CA 90024. E-mail: [email protected]. C 2014 Wolters Kluwer Health, Inc. All rights reserved. Copyright ISSN: 0003-4932/14/26105-0870 DOI: 10.1097/SLA.0000000000000756
870 | www.annalsofsurgery.com
gestalt.1 This relatively unstructured approach remained the standard until the 1990s when Reznick2 applied a framework of systematic teaching using models and learning principles with formalized assessment. Since this time, techniques for efficient and productive learning have been proposed, fueled by increased restrictions on resident work hours and oversight by regulatory committees.3–5 In addition, it is challenging after completion of residency or fellowship to advance efficiently along surgical learning curves, particularly with the advent of new technologies and techniques. For instance, Hu et al6 described the benefit of intraoperative mentorship from more experienced surgeons, beyond completion of residency training. However, this is often difficult to do because of busy surgeon schedules and the hierarchical nature of surgery.6 Birkmeyer et al7 recently demonstrated through video review that better surgeon technique was associated with better outcomes. In addition, postoperative video debriefing has shown to be an effective educational tool, with broad implications for reducing adverse events,8 and expert surgeons also benefit from immediate structured feedback from colleagues.6 Moreover, surgical societies are turning to the Internet as a forum for Web-based education modules and for discussion and interaction among residents and physicians.9–11 Recently, there has been a deluge of negative press surrounding injuries attributed to robotic assisted surgery, and Web pages such as badrobotsurgery.com have been launched to attract potential plaintiffs. Robotic surgery has a prolonged learning curve, and robotic surgery simulators aid the transition for neophytes.12 In addition, robotic skills acquired through simulation are transferrable to the operating room.13 Given the need for novel paradigms of surgical training, the goal of our study was to conduct a randomized controlled trial to evaluate peer feedback through social networking for acquisition of robotic surgery skills through simulation. We hypothesized that trainees receiving online feedback regarding robotic simulator performance and technique from their peers acquire skills more rapidly than those without feedback.
METHODS Our study was approved by the University of California Los Angeles (UCLA) and University of Michigan institutional review boards. We recruited residents from the UCLA urology and obstetrics and gynecology programs and the University of Michigan urology program. Participants were recruited via e-mail, and participation was voluntary. A power calculation estimated that 25 participants were required in each arm to achieve 97% power to detect a 7% improvement from the intervention. Of the 71 eligible subjects, 53 participants were randomized (Fig. 1) into 2 groups using an online program (www.random.org) and completed a prestudy survey to assess prior intraoperative and simulator experience with robotic assisted surgery. A total of 41 subjects completed the Tubes robotic simulator exercise (Da Vinci Intuitive Surgical, Sunnyvale, CA) 3 times over a 6-week period, 20 in the intervention group and 21 from the control group. Annals of Surgery r Volume 261, Number 5, May 2015
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
Peer Feedback Through Social Networking
tion and control group subjects received their own simulator scores and times to completion, which are generated at the completion of the exercise. A poststudy questionnaire was administered to all subjects to assess comfort level and satisfaction with simulator training. In addition, intervention subjects were asked whether they felt peer feedback was useful or effective.
Statistical Analysis We analyzed data for subjects who completed all 3 simulator sessions. We compared the characteristics between the intervention and control groups with the χ 2 and Fisher exact tests. We also compared the comfort level with the simulator before and after the intervention and satisfaction with the simulator experience. Our main outcomes of interest included the simulation score and completion time, measured in seconds. We calculated the adjusted mean scores for each intervention group at each of the 3 sessions with repeated-measures analyses and compared intervention subjects with controls. In addition, we tested within groups to assess for improvement over time. All tests were 2-sided and conducted in SAS 9.3 (SAS Inc, Cary, NC).
RESULTS
FIGURE 1. Enrollment and randomization. OB-GYN indicates obstetrics and gynecology. The Tubes exercise simulates a running anastamotic suture and has been validated as a training tool.12 Subjects randomized to the intervention group video recorded and uploaded their simulation sessions to the Google+ social networking site in a de-identified fashion. Google+ was selected because it allows uploading an unlimited number of large video files, which were viewable from any computer and did not require access to the institution VPN. Thus, participants at multiple institutions may access the videos. Log-in and video access was not public and was restricted only to study participants. All intervention subjects logged onto a common Google+ account to give blinded feedback regarding peer performance and to receive feedback on their own training session videos (Fig. 2). Video and comment from each study participant were labeled with the study participant’s unique identification number. Feedback after the first and second sessions comprised both open commentary and structured feedback through GEARS, a modified validated robotic skill assessment form.14 Although controls also performed the Tubes exercise 3 times over a 6-week period, they were not given access to social networking, nor did they receive peer feedback. Both interven C 2014 Wolters Kluwer Health, Inc. All rights reserved.
Characteristics of the intervention and control groups did not differ (Table 1). Although mean scores were similar for session 1 (60.7 vs 53.5, P = 0.314), intervention subjects scored higher for the second (mean 75.5 vs 59.6, P = 0.009) and third sessions (mean 80.1 vs 65.9, P = 0.019) than controls (Table 2). In addition, the intervention group demonstrated significant improvement between the first and second sessions (60.7 vs 75.5, P < 0.001) whereas there was only a trend for improvement in controls (53.5 vs 59.6, P = 0.080). Moreover, mean scores improved between sessions 1 and 3 for both the intervention group and controls (60.7 vs 80.1, P < 0.001, and 53.5 vs 65.9, P < 0.001, respectively). Both groups had similar mean times in seconds to complete the first session (287.2 for intervention vs 361.1 for control, P = 0.083). Although both groups demonstrated shorter times to complete simulation over the course of the study, the times were significantly shorter for the intervention group than for controls for sessions 2 (mean 217.4 vs 279.0, P = 0.004) and 3 (mean 201.4 vs 261.9, P = 0.006). In a sensitivity analysis, we ran a second repeated-measures analysis to adjust for prior robotic experience by including the number of cases logged as a covariable (P = 0.161). The results were similar to the original analysis and therefore we reported the results from the original regression. In terms of poststudy survey findings, the feedback group was more comfortable with robotic surgery than controls (90% vs 62%, P = 0.02105) (Table 3). In addition, 100% of the intervention group compared with 67% of controls (P = 0.0138) expressed satisfaction with the learning exercise. Finally, within the intervention group, 85% found peer feedback to be useful and 100% found it to be effective.
DISCUSSION With rapid advancement of surgical tools and technologies, surgeons must constantly update their armamentarium. As such, continued professional development and learning are critical to maintaining a contemporary and competitive skill set. Walsh et al15 demonstrated that surgeon video review leads to improved technique and outcomes. In addition, a recent study by Hu et al6 suggests that career-long continuous education to further improve surgical technique and judgment is beneficial even among experienced surgeons. This is increasingly important, as highly publicized complications receive national attention involving the use of new surgical technologies, such as robotic assisted surgery, by undertrained surgeons, and this is of great concern for patient safety.16 www.annalsofsurgery.com | 871
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
Carter et al
FIGURE 2. Sample screenshot and peer feedback. Birkmeyer et al7 demonstrated that the technical skill of practicing surgeons for laparoscopic bariatric surgery was highly variable and surgeons deemed more skillful by their peers also experienced fewer postoperative complications. The same study found that peer review of a recorded operation was effective in accurately assessing a surgeon’s level of proficiency.7 Likewise, many efforts have been made to decrease variability among surgeons to improve quality and eliminate heterogeneity that may contribute to poor outcomes.17,18 In contrast to medical therapies that are standardized by medication type or dosage, subtle differences in surgical technique lead to significant differences in outcomes. For instance, radical prostatectomy has a challenging learning curve with a series of complex steps that contributes to variation in surgeon technique and outcomes. At a high-volume cancer referral center, the adjusted 1-year postprostatectomy continence rate ranged from 65% to 96% whereas that of erectile function/potency ranged from 8% to 50%.18 Therefore, there is opportunity for improving the significant variation in radical prostatectomy functional outcome recovery; however, barriers to improvement of surgeon technique include departmental hierarchy and surgeon ego. Anonymous peer feedback of surgical video may overcome these challenges and facilitate continuous quality improvement. Our study has several important findings. First, peer feedback was associated with better simulation scores for the intervention group than for controls. Traditionally, surgical training is structured so that trainees are taught by attending surgeons who pass along relatively unstructured feedback and instruction during a live operation. Our study shows that feedback provided at a peer level has benefit for acquisition and improvement of technical skill. Moreover, survey findings reinforce these findings because all intervention subjects found feedback to be effective. Second, the time to complete the exercise was significantly shorter in the intervention group than in controls in sessions 2 and 3, respectively. Operative time is often used as a tool to estimate surgical 872 | www.annalsofsurgery.com
learning curves. A recent study by Tasian et al19 illustrates this point by demonstrating that the median operative time for fellows performing robotic assisted pyeloplasty decreases with each subsequent operation. This finding has also been demonstrated among surgeons performing robotic assisted radical prostatectomy.20 Furthermore, this is consistent with population-based findings that high-volume surgeons have shorter operative times for radical prostatectomies.21 The findings from our study support this previously established notion that the time it takes to perform a task decreases as more experience is gained. However, although both of our study groups took less time to perform the simulation over time, those receiving peer feedback had significantly shorter times than the control group, suggesting that the utilization of online video-based peer feedback helps promote faster acquisition of skills and accelerates the learning curve. Third, peer feedback was associated with greater comfort level with the robotic surgery and greater learning satisfaction than controls. By utilizing an online forum, feedback was given and received on a trainee’s own time and convenience. This is particularly important in a time of increasing resident work-hour restrictions and greater need posttraining for medical record documentation. By demonstrating that online feedback allows for improved performance of a surgical skill, novel forums and strategies for surgical training and mentorship at both a trainee level and an expert level should be explored to reduce variation in technique and improve patient outcomes. Finally, our study demonstrates the efficacy of peer feedback through social networking, a novel paradigm for technical mentorship. Although our study involved a simulated surgical exercise, prior studies have illustrated that the acquisition of simulated surgical skills are transferable to a live operative setting.13 A benefit of using social networking as a forum is that it may serve as a vehicle for neophyte surgeons to receive mentorship from experts without the limitation of geographic distance or time constraints. Peer feedback of online surgical video offers anonymity for all involved and therefore removes C 2014 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
Peer Feedback Through Social Networking
TABLE 1. Study Participant Characteristics Total (N = 41) Variable
Intervention (n = 20)
%
Sex Male 59 Female 41 Field Urology 80 OB-GYN 20 Postgraduate year 1–3 54 4–5 20 6 or fellow 27 Robotic surgery experience (no. cases) 0 24 1–5 17 5–10 17 10–15 15 15–20 12 20+ 15 Robotic simulation in curriculum Yes 44 No 56 Prior simulator experience Yes 80 No 20 Prior time on the simulator, h 0 24 1–5 59 5–10 5 10–15 10 15–20 2 Prior sessions on the simulator 0 20 1–5 59 5–10 10 10–15 10 15–20 0 20+ 2
Control (n = 21)
n
%
n
%
n
P
24 17
65 35
13 7
52 48
11 10
0.4123
33 8
85 15
17 3
76 24
16 5
0.6965∗
22 8 11
45 25 30
9 5 6
62 14 24
13 3 5
0.5414∗
10 7 7 6 5 6
20 5 25 25 10 15
4 1 5 5 2 3
29 29 10 5 14 14
6 6 2 1 3 3
0.1610∗
18 23
40 60
8 12
48 52
10 11
0.6232
33 8
85 15
17 3
76 24
16 5
0.6965∗
10 24 2 4 1
15 60 10 15 0
3 12 2 3 0
33 57 0 5 5
7 12 0 1 1
0.2482∗
8 24 4 4 0 1
15 60 10 10 0 5
3 12 2 2 0 1
24 57 10 10 0 0
5 12 2 2 0 0
0.9513∗
All P values from the χ 2 or Fisher exact tests when denoted with an asterisk. OB-GYN indicates obstetrics and gynecology.
TABLE 2. Comparison of Adjusted Means for Simulator Score and Completion Time Variable Tubes exercise score Session 1 Session 2 Session 3 Session 1 vs 2 Session 1 vs 3 Time to completion Session 1 Session 2 Session 3 Session 1 vs 2 Session 1 vs 3
Total (N = 41), Mean (SE)
Intervention (n = 20), Mean (SE)
Control (n = 21), Mean (SE)
57.1 (3.5) 67.5 (2.9) 73.0 (2.9)
60.7 (5.0) 75.5 (4.1) 80.1 (4.1) 0.0001
