Surgical Education
Augmented-reality-based skills training for robot-assisted urethrovesical anastomosis: a multi-institutional randomised controlled trial Ashirwad Chowriappa*†, Syed Johar Raza*, Anees Fazili‡, Erinn Field*, Chelsea Malito*, Dinesh Samarasekera§, Yi Shi*, Kamran Ahmed¶, Gregory Wilding*, Jihad Kaouk§, Daniel D. Eun**, Ahmed Ghazi‡, James O. Peabody††, Thenkurussi Kesavadas†, James L. Mohler* and Khurshid A. Guru* *Roswell Park Cancer Institute, †State University of New York, Buffalo, ‡University of Rochester Medical Center, Rochester, NY, §Cleveland Clinic Foundation, Cleveland, OH, **Temple University Hospital, Philadelphia, PA, ††Henry Ford Health System, Detroit, MI, USA, and ¶King's College, London, UK
Objective To validate robot-assisted surgery skills acquisition using an augmented reality (AR)-based module for urethrovesical anastomosis (UVA).
Methods Participants at three institutions were randomised to a Hands-on Surgical Training (HoST) technology group or a control group. The HoST group was given procedure-based training for UVA within the haptic-enabled AR-based HoST environment. The control group did not receive any training. After completing the task, the control group was offered to cross over to the HoST group (cross-over group). A questionnaire administered after HoST determined the feasibility and acceptability of the technology. Performance of UVA using an inanimate model on the daVinci Surgical System (Intuitive Surgical Inc., Sunnyvale, CA, USA) was assessed using a UVA evaluation score and a Global Evaluative Assessment of Robotic Skills (GEARS) score. Participants completed the National Aeronautics and Space Administration Task Load Index (NASA TLX) questionnaire for cognitive assessment, as outcome measures. A Wilcoxon rank-sum test was used to compare outcomes among the groups (HoST
Introduction Surgical simulation-training platforms such as virtual reality (VR) have emerged as useful adjuncts to training in and outside the operating theatre [1]. Simulation-based training can enhance safety profile by bridging the gap between the safe acquisition of technical skills and effective operative performance [2,3]. Procedure-based training allows the trainee to put into practice fundamental surgical skills to perform more complex
BJU Int 2015; 115: 336–345 wileyonlinelibrary.com
group vs control group and control group vs cross-over group).
Results A total of 52 individuals participated in the study. UVA evaluation scores showed significant differences in needle driving (3.0 vs 2.3; P = 0.042), needle positioning (3.0 vs 2.4; P = 0.033) and suture placement (3.4 vs 2.6; P = 0.014) in the HoST vs the control group. The HoST group obtained significantly higher scores (14.4 vs 11.9; P 0.012) on the GEARS. The NASA TLX indicated lower temporal demand and effort in the HoST group (5.9 vs 9.3; P = 0.001 and 5.8 vs 11.9; P = 0.035, respectively). In all, 70% of participants found that HoST was similar to the real surgical procedure, and 75% believed that HoST could improve confidence for carrying out the real intervention.
Conclusion Training in UVA in an AR environment improves technical skill acquisition with minimal cognitive demand.
Keywords anastomosis, augmented reality, robot-assisted, robotic, skills training, urethrovesical anastomosis
procedural tasks. To master robot-assisted surgery, a surgeon needs to develop an understanding of three-dimensional spatial orientation, depth perception and the role of magnification, with coordinated hand manoeuvres [4]. A recent review on laparoscopic training concluded that simulators offering procedure-based training were more efficient at training [5,6]. Although VR training devices have been shown to be valid and effective for training, their ability remains limited in terms of their representation of a realistic procedure-specific environment [7]. Augmented reality (AR) is © 2014 The Authors BJU International © 2014 BJU International | doi:10.1111/bju.12704 Published by John Wiley & Sons Ltd. www.bjui.org
Augmented-reality-based procedure specific training
Fig. 1 Hands-on Surgical Training is a framework that provides haptic-enabled augmented-reality-based training for robot-assisted surgery. This exercise divides the entire operation into various steps and provides the trainees with simultaneous proctoring throughout the procedure.
an effective alternative that enhances the face validity of training modules [8–10]. AR offers a real view of an operating environment in which the events on the field are enhanced and supplemented [11]. Difficulty in mastering intracorporeal suturing and anastomosis may prevent practising surgeons from performing advanced procedures. Despite the presence of Endo-wrist® technology in robot-assisted surgery, developing this skill has limitations in terms of complexity of steps, time restraints and related complications [12–14]. AR environments could promote trainees’ cognitive skills and conceptual understanding and correct their misconceptions [15,16]. The aim of the present randomised controlled trial was to evaluate the effectiveness of Hand-on Surgical Training (HoST) technology-based (Fig. 1) urethrovesical anastomosis (UVA) by comparing this technology with the existing standards of training.
Materials and Methods Study Design and Participants After obtaining institutional review board approval (I 228012), we enrolled participants from the Roswell Park Cancer Institute, the Cleveland Clinic Foundation and the University of Rochester Medical Center in a multicentre, randomised controlled trial. In phase I, all eligible participants enrolled were randomised to receive HoST-based UVA training (HoST group) or to a control group that did not receive HoST. In phase II (Fig. 2), the control group participants were offered the opportunity to cross over and complete HoST-based training (cross-over group). Participants in the study had minimal (≤25 h on robotic console) or no previous robotic experience; those who had previous simulation and robotic console experience (>25 h) were excluded from the study. Before randomisation all participants completed a basic demographic pre-study questionnaire. All participants were randomised based on a 1:1 stratified permuted block scheme.
Development of Hands-on Surgical Training The HoST-AR-based procedures used in the present study were developed by Roswell Park Cancer Institute and the State University of New York at Buffalo, Virtual Reality Laboratory in 2009. HoST technology provides a novel simulation-based environment that augments a real surgical procedure within a VR framework. It analyses the steps of a procedure performed by expert surgeons and converts them to a sequence of haptic-enabled AR-based procedures. The HoST environment is enhanced with didactic audio and visual explanations along with anatomically relevant annotations/illustrations of the critical steps of the procedure (Fig. 3). Once a trainee is immersed in the HoST environment on the Robot-assisted Surgical Simulator (RoSS), the RoSS manipulator takes over and guides the trainee through haptic-enabled prompts during the procedure. Robot-assisted Surgical Simulator and Fundamental Skills of Robotic Surgery Curriculum We used a RoSS console [17,18] with its Fundamental Skills of Robotic Surgery (FSRS) curriculum [19,20] and HoST-based UVA in the study. The FSRS curriculum is a sequential, modular criterion-based structured curriculum for acquiring the basic skills of robot-assisted surgery. The FSRS curriculum helps develop a safe human–machine interface, allowing the novice surgeon to master the basic functions of the daVinci Surgical System (dVSS™; Intuitive Surgical Inc., Sunnyvale, CA, USA) console. Training and Testing for Urethrovesical Anastomosis Performance on the daVinci Surgical System Participants in the HoST group were introduced to the RoSS and completed four basic simulation tasks once from the FSRS curriculum; ball placement, spatial control, tissue retraction and knot-tying, before moving on to HoST training. Participants in the HoST group were given four HoST sessions, with each session lasting for no more than 20 min. © 2014 The Authors BJU International © 2014 BJU International
337
Chowriappa et al.
Fig. 2 Algorithm for randomisation process
HoST Study Consort Diagram
used in the study. HoST, Hands-on Surgical Training; RoSS, Robot-assisted Surgical Simulator; NASA TLX, National Aeronautics and
Enrolment
Assessed for Eligibility (n=52)
Space Administration Task Load Index; GEARS, Global Evaluative Assessment of Robotic Skills.
Excluded (n=0)
Allocation
Lost to Follow-Up (n=2)
Lost to Follow-Up (n=4)
Control Group (n=24) Performed tasks on the daVinci Surgical SystemTM
Phase I
Randomisation
HoST Group (n=22) Performed tasks on the RoSS Simulator & daVinci Surgical SystemTM
Test: Urethrovesical Anastomosis on an Inanimate Model
Video-recordings of the Test assessed by Expert Reviewers using 1. GEARS global rating scale 2. Objective Urethrovesical Anastamosis Evaluation
Lost to Follow-Up (n=5)
Phase II
Cross-over
Analysis
NASA TLX
Cross-over to HoST (n=17)
Fig. 3 Final test performed on daVinci Surgical
UVA test on an in-animate Model
system™.
Description: Participants performed UVA using V-LocTM (Covidien) suture on the daVinci Surgical SystemTM (Intuitive Surgical Inc.).
HoST and RoSS training were completed within 2 weeks of recruitment to the study. Assessment of UVA performance was carried out within 24 h of the last HoST session. Before the assessment, the participants were given a short didactic session to introduce them to the features of the dVSS. This session included a one-on-one familiarisation by an
338
© 2014 The Authors BJU International © 2014 BJU International
experienced robot operator. After a 5-min period of acclimation, participants completed the ball placement, suture pass, fourth arm manipulation and suturing tasks on the dVSS. These tasks were selected because they encompass the important robotic skills required to successfully perform UVA on an inanimate model using the dVSS.
Augmented-reality-based procedure specific training
Participants in the control group were introduced to the RoSS system in the same way as the HoST group. The control group did not undergo HoST, instead they watched the four videos showing UVA being performed. They did this for the same length of time as those who underwent HoST. Within 24 h of last viewing of the videos the control group were assessed on their performance of UVA. The control group were given the same short didactic session as the HoST group to introduce them to robotic surgery and the features of the dVSS. After a 5-min period of acclimation on the console, participants completed the four tasks on the dVSS console. After completion of the four tasks using the robot, both groups performed UVA on an inanimate model and their performance was video-recorded for expert analysis. On completion of this session, participants in the control group were offered the opportunity to cross over and complete HoST, and form the cross-over group. All participants completed the National Aeronautics and Space Administration task load index (NASA TLX) assessment questionnaire after completing UVA on the dVSS console.
Cognitive assessment The NASA TLX was used to evaluate mental fatigue (Table 2). The participants completed the NASA TLX questionnaire immediately after completing UVA on the dVSS. Tools for Evaluation GEARS assessment score Basic robotic skills deemed essential for performing UVA were assessed using the GEARS assessment score [21]. GEARS is composed of six domains, including depth perception, bimanual dexterity, efficiency, autonomy, force sensitivity and robotic control. Proficiency is measured on a five-point Likert scale, with specific performance anchors at 1, 3 and 5. Urethrovesical anastomosis evaluation score A UVA evaluation score was developed by deconstructing the fundamental elements of UVA (needle positioning, needle driving, suture placement and tissue manipulation) in consultation with an expert robotic surgeon (>1000 procedures with >50% console time). A rating of 1 corresponded to the lowest quality of performance (Fig. 4).
Outcome Measures
NASA Task Load Index assessment
Feasibility and acceptability
The level of cognitive workload while performing UVA was evaluated using the NASA TLX scale [22]. The NASA TLX is a subjective workload assessment technique that relies on a multidimensional construct to derive an overall workload score based on six weighted subscales: mental demand, physical demand, temporal demand, performance, effort and frustration level (Table 2).
Information was collected, using the post-HoST quantitative questionnaire, about the realism of the HoST-based augmented environment, technical issues related to the HoST, and its overall efficacy pertinent to UVA training (Table 1). Concurrent validity Concurrent validity was established by comparing the performances of the different participants in the HoST-based training on the ‘gold standard’ dVSS, using an inanimate UVA model. Performance was recorded from a direct video feed from the dVSS. The performance was assessed by expert robotic surgeons using the Global Evaluative Assessment of Robotic Skills (GEARS) score and a UVA assessment tool (Fig. 4).
In summary, participants were assessed using the following: GEARS score; objective UVA evaluation score; and cognitive task load using the NASA TLX. Videos were blinded and analysed by four expert independent surgeons. Inter-rater reliability for both GEARS and the object UVA evaluation scores was evaluated. Statistical Analyses Subjects were randomised to the HoST group or control group using a permuted block randomisation scheme with a block of
Table 1 Results of the questionnaire administered after Hands-on Surgical Training. Question type Realism Technical issues
Flow and overall efficacy
Description HoST simulation was as realistic as the surgical procedure The rehearsal environment and simulated surgical steps were appropriate and a learning experience The simulation can possibly increase the overall efficiency of procedure The simulation can possibly alter preconceived choice of surgical steps The simulation is useful to practise the real case before performing it on the patient The simulation can possibly aid in patient safety The HoST can possibly enhance confidence for the real intervention
Participants who agreed with description, % 70 76 76 54 70 76 75
HoST, Hands-on Surgical Training.
© 2014 The Authors BJU International © 2014 BJU International
339
Chowriappa et al.
Fig. 4 Objective urethrovesical anastomosis
Needle Position
*Level (1-5)
CE Check
Critical Error (CE)
Held needle at 1/3 to 2/3 from tip
Not able to see tip due to inappropriate position
Angles entering tissue are appropriate
Needle leading to tearing of adjacent tissue
Use wrist for rotation
Tear tissue and not using the rotation of the endo-wrist
Less than two attempts at proper positioning of needle
Trauma to adjacent area
evaluation.
Needle Driving Tissue torn while manipulating tissue
If yes then critical error
Remove needle along curve of the needle
Touch adjacent tissue
Appropriate tension and pull around the suture
Disrupt suture from needle in-vivo
Suture Placement & Tissue Manipulation Appropriate distance between each suture
Wide gaps
Proper tissue approximation
Unable to align opening
Proper alignment of posterior urethral plate
Wide opening/easily will lead to false passage
Appropriate knot placement
Air-knot
*Level: 1=poor; 2=below average; 3=average; 4=above average; 5=excellent
Table 2 National Aeronautics and Space Administration Task Load Index assessment. NASA TLX rating scale description Endpoint
Level
Rating scale definitions
Mental Demand
Low/High
Physical Demand Temporal Demand
Low/High Low/High
Performance
Low/High
Effort
Low/High
Frustration Level
Low/High
How much mental and perceptual activity was required (e.g. thinking, deciding, etc.)? How much physical activity was required? How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? How hard did you have to work (mentally and physically) to accomplish your level of performance? How successful do you think you were inaccomplishing your responsibilities? How discouraged, irritated, stressed and annoyed versus gratified, content, relaxed and complacent did you feel?
NASA TLX, National Aeronautics and Space Administration Task Load Index.
size 2. The measured outcomes were summarised both overall and according to relevant demographic and baseline variables. Descriptive statistics such as frequencies and relative frequencies were computed for all categorical variables. Numeric variables were summarised using simple descriptive
340
© 2014 The Authors BJU International © 2014 BJU International
statistics such as mean, SD and range values. Wilcoxon rank-sum tests were used to compare outcomes statistically between the independent groups (HoST group vs control group) as well as the dependent groups (control group vs cross-over group). A sample size of 15 participants in each
Augmented-reality-based procedure specific training
group was calculated to be sufficient to detect a difference of 1.1 SD points with 80% power. Fisher’s exact test was used to assess the statistical association between questionnaire outcome variables and group variables. All tests were two-sided and tested at a 0.05 nominal significance level.
Results Demographic Characteristics Out of 52 participants, 22 were residents and 30 were fellows. There were no significant differences in the baseline participant characteristics between the HoST and control groups with respect to demographics, education or previous surgical experience (Table 3).
Concurrent Validity GEARS assessment The HoST group scored significantly better than the control group in overall GEARS score (P = 0.012). Differences were predominantly seen in bimanual dexterity (P = 0.016) and force sensitivity (P < 0.001). The cross-over group also had a significantly higher overall GEARS score (P = 0.008) than the control group. Cross-over group performance was significantly higher in the bimanual dexterity (P = 0.048), efficiency (0.046), force sensitivity (P = 0.011) and autonomy (P = 0.004) domains (Table 4).
Objective urethrovesical anastomosis assessment Feasibility and Acceptability In all, 70% of participants believed that the HoST AR-based environment was as realistic as the actual surgical procedure (Fig. 4). A total of 76% of participants felt that the AR environment and surgical steps were appropriate for learning and 70% believed that HoST would prove useful for practice before performing the real case procedure.
The HoST group demonstrated significantly better performance with regard to needle positioning, needle driving, suture placement and tissue manipulation compared with the control group (P = 0.008, 0.042 and 0.014, respectively). The cross-over group similarly outperformed the control group in these three domains (P = 0.018, 0.037 and 0.013, respectively [Table 5]).
Table 3 Demographic characteristics of the participants. Participant characteristic Demographics Age, n 40 years Gender, n Female Male Dominant hand, n Left Right Education Level of training, n Fellow Resident PGY level, n 5 Formal laparoscopic training, n No Yes dvSS experience Previous robotic console experience, n No Yes Console experience, n 0h 25 h Robotic simulator experience Previous simulator training for the dVSS No Yes
HoST group, N = 26
Control group, N = 26
P
0.27 25 1
23 3
10 16
7 19
2 24
3 23
14 12
16 10
12 14
10 16
13 13
11 15
24 2
23 3
24 2 0
23 3 0
26 0
26 0
0.24
0.91
0.87
0.94
0.91
1.0
0.47
1.0
HoST, Hands-on Surgical Training; dVSS, daVinci Surgical System; PGY, Post Graduate Year.
© 2014 The Authors BJU International © 2014 BJU International
341
Chowriappa et al.
Table 4 Global Evaluative Assessment of Robotic Skills assessment scores for phase I (control group vs HoST group) and phase II (control group vs cross-over group). Skill
Control group, n = 24 Mean (SD) GEARS score
HoST group, n = 22 Mean (SD) GEARS score
P
2.4 (1.0) 2.5 (0.9) 2.0 (0.8) 2.0 (1.1) 3.0 (0.7) 11.9 (4.1)
2.9 (0.2) 2.8 (0.3) 2.5 (0.2) 2.7 (0.4) 3.5 (0.3) 14.4 (1.2)
0.016 0.915
Augmented-reality-based skills training for robot-assisted urethrovesical anastomosis: a multi-institutional randomised controlled trial Ashirwad Chowriappa*†, Syed Johar Raza*, Anees Fazili‡, Erinn Field*, Chelsea Malito*, Dinesh Samarasekera§, Yi Shi*, Kamran Ahmed¶, Gregory Wilding*, Jihad Kaouk§, Daniel D. Eun**, Ahmed Ghazi‡, James O. Peabody††, Thenkurussi Kesavadas†, James L. Mohler* and Khurshid A. Guru* *Roswell Park Cancer Institute, †State University of New York, Buffalo, ‡University of Rochester Medical Center, Rochester, NY, §Cleveland Clinic Foundation, Cleveland, OH, **Temple University Hospital, Philadelphia, PA, ††Henry Ford Health System, Detroit, MI, USA, and ¶King's College, London, UK
Objective To validate robot-assisted surgery skills acquisition using an augmented reality (AR)-based module for urethrovesical anastomosis (UVA).
Methods Participants at three institutions were randomised to a Hands-on Surgical Training (HoST) technology group or a control group. The HoST group was given procedure-based training for UVA within the haptic-enabled AR-based HoST environment. The control group did not receive any training. After completing the task, the control group was offered to cross over to the HoST group (cross-over group). A questionnaire administered after HoST determined the feasibility and acceptability of the technology. Performance of UVA using an inanimate model on the daVinci Surgical System (Intuitive Surgical Inc., Sunnyvale, CA, USA) was assessed using a UVA evaluation score and a Global Evaluative Assessment of Robotic Skills (GEARS) score. Participants completed the National Aeronautics and Space Administration Task Load Index (NASA TLX) questionnaire for cognitive assessment, as outcome measures. A Wilcoxon rank-sum test was used to compare outcomes among the groups (HoST
Introduction Surgical simulation-training platforms such as virtual reality (VR) have emerged as useful adjuncts to training in and outside the operating theatre [1]. Simulation-based training can enhance safety profile by bridging the gap between the safe acquisition of technical skills and effective operative performance [2,3]. Procedure-based training allows the trainee to put into practice fundamental surgical skills to perform more complex
BJU Int 2015; 115: 336–345 wileyonlinelibrary.com
group vs control group and control group vs cross-over group).
Results A total of 52 individuals participated in the study. UVA evaluation scores showed significant differences in needle driving (3.0 vs 2.3; P = 0.042), needle positioning (3.0 vs 2.4; P = 0.033) and suture placement (3.4 vs 2.6; P = 0.014) in the HoST vs the control group. The HoST group obtained significantly higher scores (14.4 vs 11.9; P 0.012) on the GEARS. The NASA TLX indicated lower temporal demand and effort in the HoST group (5.9 vs 9.3; P = 0.001 and 5.8 vs 11.9; P = 0.035, respectively). In all, 70% of participants found that HoST was similar to the real surgical procedure, and 75% believed that HoST could improve confidence for carrying out the real intervention.
Conclusion Training in UVA in an AR environment improves technical skill acquisition with minimal cognitive demand.
Keywords anastomosis, augmented reality, robot-assisted, robotic, skills training, urethrovesical anastomosis
procedural tasks. To master robot-assisted surgery, a surgeon needs to develop an understanding of three-dimensional spatial orientation, depth perception and the role of magnification, with coordinated hand manoeuvres [4]. A recent review on laparoscopic training concluded that simulators offering procedure-based training were more efficient at training [5,6]. Although VR training devices have been shown to be valid and effective for training, their ability remains limited in terms of their representation of a realistic procedure-specific environment [7]. Augmented reality (AR) is © 2014 The Authors BJU International © 2014 BJU International | doi:10.1111/bju.12704 Published by John Wiley & Sons Ltd. www.bjui.org
Augmented-reality-based procedure specific training
Fig. 1 Hands-on Surgical Training is a framework that provides haptic-enabled augmented-reality-based training for robot-assisted surgery. This exercise divides the entire operation into various steps and provides the trainees with simultaneous proctoring throughout the procedure.
an effective alternative that enhances the face validity of training modules [8–10]. AR offers a real view of an operating environment in which the events on the field are enhanced and supplemented [11]. Difficulty in mastering intracorporeal suturing and anastomosis may prevent practising surgeons from performing advanced procedures. Despite the presence of Endo-wrist® technology in robot-assisted surgery, developing this skill has limitations in terms of complexity of steps, time restraints and related complications [12–14]. AR environments could promote trainees’ cognitive skills and conceptual understanding and correct their misconceptions [15,16]. The aim of the present randomised controlled trial was to evaluate the effectiveness of Hand-on Surgical Training (HoST) technology-based (Fig. 1) urethrovesical anastomosis (UVA) by comparing this technology with the existing standards of training.
Materials and Methods Study Design and Participants After obtaining institutional review board approval (I 228012), we enrolled participants from the Roswell Park Cancer Institute, the Cleveland Clinic Foundation and the University of Rochester Medical Center in a multicentre, randomised controlled trial. In phase I, all eligible participants enrolled were randomised to receive HoST-based UVA training (HoST group) or to a control group that did not receive HoST. In phase II (Fig. 2), the control group participants were offered the opportunity to cross over and complete HoST-based training (cross-over group). Participants in the study had minimal (≤25 h on robotic console) or no previous robotic experience; those who had previous simulation and robotic console experience (>25 h) were excluded from the study. Before randomisation all participants completed a basic demographic pre-study questionnaire. All participants were randomised based on a 1:1 stratified permuted block scheme.
Development of Hands-on Surgical Training The HoST-AR-based procedures used in the present study were developed by Roswell Park Cancer Institute and the State University of New York at Buffalo, Virtual Reality Laboratory in 2009. HoST technology provides a novel simulation-based environment that augments a real surgical procedure within a VR framework. It analyses the steps of a procedure performed by expert surgeons and converts them to a sequence of haptic-enabled AR-based procedures. The HoST environment is enhanced with didactic audio and visual explanations along with anatomically relevant annotations/illustrations of the critical steps of the procedure (Fig. 3). Once a trainee is immersed in the HoST environment on the Robot-assisted Surgical Simulator (RoSS), the RoSS manipulator takes over and guides the trainee through haptic-enabled prompts during the procedure. Robot-assisted Surgical Simulator and Fundamental Skills of Robotic Surgery Curriculum We used a RoSS console [17,18] with its Fundamental Skills of Robotic Surgery (FSRS) curriculum [19,20] and HoST-based UVA in the study. The FSRS curriculum is a sequential, modular criterion-based structured curriculum for acquiring the basic skills of robot-assisted surgery. The FSRS curriculum helps develop a safe human–machine interface, allowing the novice surgeon to master the basic functions of the daVinci Surgical System (dVSS™; Intuitive Surgical Inc., Sunnyvale, CA, USA) console. Training and Testing for Urethrovesical Anastomosis Performance on the daVinci Surgical System Participants in the HoST group were introduced to the RoSS and completed four basic simulation tasks once from the FSRS curriculum; ball placement, spatial control, tissue retraction and knot-tying, before moving on to HoST training. Participants in the HoST group were given four HoST sessions, with each session lasting for no more than 20 min. © 2014 The Authors BJU International © 2014 BJU International
337
Chowriappa et al.
Fig. 2 Algorithm for randomisation process
HoST Study Consort Diagram
used in the study. HoST, Hands-on Surgical Training; RoSS, Robot-assisted Surgical Simulator; NASA TLX, National Aeronautics and
Enrolment
Assessed for Eligibility (n=52)
Space Administration Task Load Index; GEARS, Global Evaluative Assessment of Robotic Skills.
Excluded (n=0)
Allocation
Lost to Follow-Up (n=2)
Lost to Follow-Up (n=4)
Control Group (n=24) Performed tasks on the daVinci Surgical SystemTM
Phase I
Randomisation
HoST Group (n=22) Performed tasks on the RoSS Simulator & daVinci Surgical SystemTM
Test: Urethrovesical Anastomosis on an Inanimate Model
Video-recordings of the Test assessed by Expert Reviewers using 1. GEARS global rating scale 2. Objective Urethrovesical Anastamosis Evaluation
Lost to Follow-Up (n=5)
Phase II
Cross-over
Analysis
NASA TLX
Cross-over to HoST (n=17)
Fig. 3 Final test performed on daVinci Surgical
UVA test on an in-animate Model
system™.
Description: Participants performed UVA using V-LocTM (Covidien) suture on the daVinci Surgical SystemTM (Intuitive Surgical Inc.).
HoST and RoSS training were completed within 2 weeks of recruitment to the study. Assessment of UVA performance was carried out within 24 h of the last HoST session. Before the assessment, the participants were given a short didactic session to introduce them to the features of the dVSS. This session included a one-on-one familiarisation by an
338
© 2014 The Authors BJU International © 2014 BJU International
experienced robot operator. After a 5-min period of acclimation, participants completed the ball placement, suture pass, fourth arm manipulation and suturing tasks on the dVSS. These tasks were selected because they encompass the important robotic skills required to successfully perform UVA on an inanimate model using the dVSS.
Augmented-reality-based procedure specific training
Participants in the control group were introduced to the RoSS system in the same way as the HoST group. The control group did not undergo HoST, instead they watched the four videos showing UVA being performed. They did this for the same length of time as those who underwent HoST. Within 24 h of last viewing of the videos the control group were assessed on their performance of UVA. The control group were given the same short didactic session as the HoST group to introduce them to robotic surgery and the features of the dVSS. After a 5-min period of acclimation on the console, participants completed the four tasks on the dVSS console. After completion of the four tasks using the robot, both groups performed UVA on an inanimate model and their performance was video-recorded for expert analysis. On completion of this session, participants in the control group were offered the opportunity to cross over and complete HoST, and form the cross-over group. All participants completed the National Aeronautics and Space Administration task load index (NASA TLX) assessment questionnaire after completing UVA on the dVSS console.
Cognitive assessment The NASA TLX was used to evaluate mental fatigue (Table 2). The participants completed the NASA TLX questionnaire immediately after completing UVA on the dVSS. Tools for Evaluation GEARS assessment score Basic robotic skills deemed essential for performing UVA were assessed using the GEARS assessment score [21]. GEARS is composed of six domains, including depth perception, bimanual dexterity, efficiency, autonomy, force sensitivity and robotic control. Proficiency is measured on a five-point Likert scale, with specific performance anchors at 1, 3 and 5. Urethrovesical anastomosis evaluation score A UVA evaluation score was developed by deconstructing the fundamental elements of UVA (needle positioning, needle driving, suture placement and tissue manipulation) in consultation with an expert robotic surgeon (>1000 procedures with >50% console time). A rating of 1 corresponded to the lowest quality of performance (Fig. 4).
Outcome Measures
NASA Task Load Index assessment
Feasibility and acceptability
The level of cognitive workload while performing UVA was evaluated using the NASA TLX scale [22]. The NASA TLX is a subjective workload assessment technique that relies on a multidimensional construct to derive an overall workload score based on six weighted subscales: mental demand, physical demand, temporal demand, performance, effort and frustration level (Table 2).
Information was collected, using the post-HoST quantitative questionnaire, about the realism of the HoST-based augmented environment, technical issues related to the HoST, and its overall efficacy pertinent to UVA training (Table 1). Concurrent validity Concurrent validity was established by comparing the performances of the different participants in the HoST-based training on the ‘gold standard’ dVSS, using an inanimate UVA model. Performance was recorded from a direct video feed from the dVSS. The performance was assessed by expert robotic surgeons using the Global Evaluative Assessment of Robotic Skills (GEARS) score and a UVA assessment tool (Fig. 4).
In summary, participants were assessed using the following: GEARS score; objective UVA evaluation score; and cognitive task load using the NASA TLX. Videos were blinded and analysed by four expert independent surgeons. Inter-rater reliability for both GEARS and the object UVA evaluation scores was evaluated. Statistical Analyses Subjects were randomised to the HoST group or control group using a permuted block randomisation scheme with a block of
Table 1 Results of the questionnaire administered after Hands-on Surgical Training. Question type Realism Technical issues
Flow and overall efficacy
Description HoST simulation was as realistic as the surgical procedure The rehearsal environment and simulated surgical steps were appropriate and a learning experience The simulation can possibly increase the overall efficiency of procedure The simulation can possibly alter preconceived choice of surgical steps The simulation is useful to practise the real case before performing it on the patient The simulation can possibly aid in patient safety The HoST can possibly enhance confidence for the real intervention
Participants who agreed with description, % 70 76 76 54 70 76 75
HoST, Hands-on Surgical Training.
© 2014 The Authors BJU International © 2014 BJU International
339
Chowriappa et al.
Fig. 4 Objective urethrovesical anastomosis
Needle Position
*Level (1-5)
CE Check
Critical Error (CE)
Held needle at 1/3 to 2/3 from tip
Not able to see tip due to inappropriate position
Angles entering tissue are appropriate
Needle leading to tearing of adjacent tissue
Use wrist for rotation
Tear tissue and not using the rotation of the endo-wrist
Less than two attempts at proper positioning of needle
Trauma to adjacent area
evaluation.
Needle Driving Tissue torn while manipulating tissue
If yes then critical error
Remove needle along curve of the needle
Touch adjacent tissue
Appropriate tension and pull around the suture
Disrupt suture from needle in-vivo
Suture Placement & Tissue Manipulation Appropriate distance between each suture
Wide gaps
Proper tissue approximation
Unable to align opening
Proper alignment of posterior urethral plate
Wide opening/easily will lead to false passage
Appropriate knot placement
Air-knot
*Level: 1=poor; 2=below average; 3=average; 4=above average; 5=excellent
Table 2 National Aeronautics and Space Administration Task Load Index assessment. NASA TLX rating scale description Endpoint
Level
Rating scale definitions
Mental Demand
Low/High
Physical Demand Temporal Demand
Low/High Low/High
Performance
Low/High
Effort
Low/High
Frustration Level
Low/High
How much mental and perceptual activity was required (e.g. thinking, deciding, etc.)? How much physical activity was required? How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? How hard did you have to work (mentally and physically) to accomplish your level of performance? How successful do you think you were inaccomplishing your responsibilities? How discouraged, irritated, stressed and annoyed versus gratified, content, relaxed and complacent did you feel?
NASA TLX, National Aeronautics and Space Administration Task Load Index.
size 2. The measured outcomes were summarised both overall and according to relevant demographic and baseline variables. Descriptive statistics such as frequencies and relative frequencies were computed for all categorical variables. Numeric variables were summarised using simple descriptive
340
© 2014 The Authors BJU International © 2014 BJU International
statistics such as mean, SD and range values. Wilcoxon rank-sum tests were used to compare outcomes statistically between the independent groups (HoST group vs control group) as well as the dependent groups (control group vs cross-over group). A sample size of 15 participants in each
Augmented-reality-based procedure specific training
group was calculated to be sufficient to detect a difference of 1.1 SD points with 80% power. Fisher’s exact test was used to assess the statistical association between questionnaire outcome variables and group variables. All tests were two-sided and tested at a 0.05 nominal significance level.
Results Demographic Characteristics Out of 52 participants, 22 were residents and 30 were fellows. There were no significant differences in the baseline participant characteristics between the HoST and control groups with respect to demographics, education or previous surgical experience (Table 3).
Concurrent Validity GEARS assessment The HoST group scored significantly better than the control group in overall GEARS score (P = 0.012). Differences were predominantly seen in bimanual dexterity (P = 0.016) and force sensitivity (P < 0.001). The cross-over group also had a significantly higher overall GEARS score (P = 0.008) than the control group. Cross-over group performance was significantly higher in the bimanual dexterity (P = 0.048), efficiency (0.046), force sensitivity (P = 0.011) and autonomy (P = 0.004) domains (Table 4).
Objective urethrovesical anastomosis assessment Feasibility and Acceptability In all, 70% of participants believed that the HoST AR-based environment was as realistic as the actual surgical procedure (Fig. 4). A total of 76% of participants felt that the AR environment and surgical steps were appropriate for learning and 70% believed that HoST would prove useful for practice before performing the real case procedure.
The HoST group demonstrated significantly better performance with regard to needle positioning, needle driving, suture placement and tissue manipulation compared with the control group (P = 0.008, 0.042 and 0.014, respectively). The cross-over group similarly outperformed the control group in these three domains (P = 0.018, 0.037 and 0.013, respectively [Table 5]).
Table 3 Demographic characteristics of the participants. Participant characteristic Demographics Age, n 40 years Gender, n Female Male Dominant hand, n Left Right Education Level of training, n Fellow Resident PGY level, n 5 Formal laparoscopic training, n No Yes dvSS experience Previous robotic console experience, n No Yes Console experience, n 0h 25 h Robotic simulator experience Previous simulator training for the dVSS No Yes
HoST group, N = 26
Control group, N = 26
P
0.27 25 1
23 3
10 16
7 19
2 24
3 23
14 12
16 10
12 14
10 16
13 13
11 15
24 2
23 3
24 2 0
23 3 0
26 0
26 0
0.24
0.91
0.87
0.94
0.91
1.0
0.47
1.0
HoST, Hands-on Surgical Training; dVSS, daVinci Surgical System; PGY, Post Graduate Year.
© 2014 The Authors BJU International © 2014 BJU International
341
Chowriappa et al.
Table 4 Global Evaluative Assessment of Robotic Skills assessment scores for phase I (control group vs HoST group) and phase II (control group vs cross-over group). Skill
Control group, n = 24 Mean (SD) GEARS score
HoST group, n = 22 Mean (SD) GEARS score
P
2.4 (1.0) 2.5 (0.9) 2.0 (0.8) 2.0 (1.1) 3.0 (0.7) 11.9 (4.1)
2.9 (0.2) 2.8 (0.3) 2.5 (0.2) 2.7 (0.4) 3.5 (0.3) 14.4 (1.2)
0.016 0.915
