Surg Endosc DOI 10.1007/s00464-014-3996-6
and Other Interventional Techniques
Simulation-based ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial Oliver Brunckhorst • Shahab Shahid • Abdullatif Aydin • Craig McIlhenny • Shahid Khan • Syed Johar Raza • Arun Sahai • James Brewin • Fernando Bello Roger Kneebone • Muhammad Shamim Khan • Prokar Dasgupta • Kamran Ahmed
•
Received: 20 September 2014 / Accepted: 7 November 2014 Ó Springer Science+Business Media New York 2014
Abstract Background Current training modalities within ureteroscopy have been extensively validated and must now be integrated within a comprehensive curriculum. Additionally, non-technical skills often cause surgical error and little research has been conducted to combine this with technical skills teaching. This study therefore aimed to develop and validate a curriculum for semi-rigid ureteroscopy, integrating both technical and non-technical skills teaching within the programme. Methods Delphi methodology was utilised for curriculum development and content validation, with a randomised
O. Brunckhorst S. Shahid A. Aydin A. Sahai J. Brewin M. S. Khan P. Dasgupta (&) K. Ahmed MRC Centre for Transplantation, King’s College London, Department of Urology, Guy’s and St. Thomas’ NHS Foundation Trust, King’s Health Partners, London, UK e-mail: [email protected] K. Ahmed e-mail: [email protected] C. McIlhenny Forth Valley Royal Hospital, NHS Forth Valley, Larbet, UK S. Khan Department of Urology, East Surrey Hospital, Surrey and Sussex Healthcare NHS Trust, Redhill, UK S. J. Raza Department of Urology, Roswell Park Cancer Institute, Buffalo, NY, USA F. Bello R. Kneebone Department of Surgery and Cancer, Imperial College London, St. Mary’s Hospital Campus, London, UK
trial then conducted (n = 32) for curriculum evaluation. The developed curriculum consisted of four modules; initially developing basic technical skills and subsequently integrating non-technical skills teaching. Sixteen participants underwent the simulation-based curriculum and were subsequently assessed, together with the control cohort (n = 16) within a full immersion environment. Both technical (Time to completion, OSATS and a task specific checklist) and non-technical (NOTSS) outcome measures were recorded with parametric and non-parametric analyses used depending on the distribution of our data as evaluated by a Shapiro–Wilk test. Results Improvements within the intervention cohort demonstrated educational value across all technical and non-technical parameters recorded, including time to completion (p \ 0.01), OSATS scores (p \ 0.001), task specific checklist scores (p = 0.011) and NOTSS scores (p \ 0.001). Content validity, feasibility and acceptability were all demonstrated through curriculum development and post-study questionnaire results. Conclusions The current developed curriculum demonstrates that integrating both technical and non-technical skills teaching is both educationally valuable and feasible. Additionally, the curriculum offers a validated simulationbased training modality within ureteroscopy and a framework for the development of other simulation-based programmes. Keywords Ureteroscopy Education Curriculum Nontechnical skills Simulation
It has become increasingly difficult to rely solely on traditional observership and mentorship to train surgeons within the modern era. Variable patient flow and a lack of
123
Surg Endosc
a structured training pathway mean that these methods produce differing educational outcomes [1]. Additionally, minimally invasive procedures such as ureteroscopy have become increasingly popular. Ureteroscopy is a technically difficult procedure with a relatively long learning curve between 50 and 60 cases, and therefore poses a challenge to the training surgeon [2]. This is further compounded by decreasing training hours across the globe and increasing ethical concerns of the training surgeon practicing on patients to acquire proficiency [3]. With these concerns it is clear that adjuncts to training are required with simulation-based training offering one option. Utilisation of simulation as a training tool is believed to improve the progression along the initial phase of the learning curve [4] and has been demonstrated to improve a surgeons’ technical ability in the Operating Room [5]. This highlights its role as a valuable adjunct to traditional training pathways. Various individual ureteroscopy training modalities are available and have all been validated to varying degrees [6]. With current evidence levels these can reliably be utilised into a training program. However, whilst these offer a validated modality of training, simulation should not be used as the only method for education, but should be integrated into a comprehensive proficiency-based curriculum [7]. Furthermore, whilst training for technical skills is vital, surgery additionally requires mastery of non-technical skills. These have been highlighted as a common cause of surgical error [8] and like technical skills, which are acquired over many years of practice and training, can also be acquired through training. There are three distinct categories of non-technical skills including social (communication, teamwork and leadership) and cognitive skills (decision making and situational awareness) alongside the personal resource factors (ability to cope with stress and fatigue) one possesses [9]. There is a current lack in the literature as to a formalised and validated curriculum, which utilises the available training modalities for ureteroscopy. Furthermore, whilst many studies have analysed technical and non-technical skills as separate entities there is paucity in the literature analysing these two skills, within one curriculum, in any surgical procedure. Therefore, this prospective study was conducted with the following aims: 1.
2.
To develop a simulation-based ureteroscopy curriculum, integrating both technical and non-technical skills within the training pathway. Evaluate the developed curriculum in terms of feasibility, acceptability, content validity and educational impact as defined by McDougall [10] and Van der Vleuten [11].
123
Materials and methods Curriculum development The curriculum was drafted via literature review and identification of key steps within the ureteroscopy procedure. This was then content validated utilising the Delphi methodology. Quantitative surveys addressing the proposed modules and tasks, training duration, non-technical skills training and assessment protocol were distributed to six experts from two countries (UK and USA) and across three institutions. These were circulated until a saturation of new information was achieved to produce a finalised curriculum. The final developed curriculum was divided into four modules: 1. basic skills development, 2. modular training, 3. integrated skills development and 4. full immersion simulation (Fig. 1). Each module consisted of tasks addressing important basic knowledge, technical skills and non-technical skills to safely and effectively conduct a semi-rigid ureteroscopy. Module 1 (basic skills development) was designed to consolidate the knowledge delivered in a previous didactic session and allows for basic manoeuvring skills to be developed. Modular training tasks within module 2 (modular training) then develops individual procedural step skills developing basic technical skills. These were then brought together within module 3 (integrated skills development) via completion of full case simulation. Finally, with all the necessary basic technical skills developed the candidate is introduced to non-technical skills in module 4 (full immersion simulation) via didactic teaching and performance of a full case simulation within the full immersion environment utilising the ‘‘Igloo’’ simulator. Individuals were mentored along the curriculum modules with tasks set out to be proficiencybased according to predetermined time criteria. Study materials Two models were included within the curriculum; the UroScopic TrainerTM (Limbs & Things Ltd., Bristol, UK), a physical bench top model, and the URO MentorTM (Simbionix, Cleveland, USA), a virtual reality simulator. Both have undergone extensive validation and posses a proven face, construct and concurrent validity with a proven educational impact [6]. Additionally, for non-technical skills teaching the validated full immersion environment of the Distributed Simulator (Fig. 2) was used [12]. This is a portable environment that utilises an inflatable ‘‘Igloo’’ to provide a 360-degree simulation of an operating room. Within the Distributed environment only the Uro-Scopic Trainer was utilised so as to utilise true operative equipment.
Surg Endosc
Module 1 Basic Skills Development
Task 1: Snapshots of urethral and vesical landmarks
Knowledge Task 2: Snapshots of ureteral landmarks
Task 3: Bladder Visualisation
Module 2 Modular Training
Task 4: Orifice Catheterization
Technical Skills Task 5: Ureteral Stone Extraction Task 6: Stent Insertion
Module 3 Integrated Skills Development
Case 1: Mid Ureteric Stone Fragmentation and Extraction
Case 2: Mid Ureteric Stone Extraction
Module 4 Integration of Case 2 in Distributed Environment
Full Immersion Simulation
Integration of Knowledge and Technical Skills
Technical and Non-Technical Skills
Fig. 1 Developed curriculum
Participants Thirty-two medical student participants, with no prior ureteroscopy training were recruited from six university institutions within the United Kingdom (Table 1). Medical students were selected as opposed to trainees as this ensures subjects do not possess previous endourological exposure allowing for more comparable groups and ensuring any differences recorded are due to the educational intervention as opposed to previous experience. Post hoc power analysis utilising time to completion and rating scales demonstrated a power of greater than 80 %, at the significance level of 0.05, to reveal a significant difference between cohorts with the obtained sample size. Study design Validation was evaluated via a randomised controlled trial (Fig. 3). Participants filled out a demographic questionnaire
containing age, gender and previous endourological surgery experience information and were then randomised via blocked randomisation so as to ensure equal sized cohorts. Groups of four were allocated to alternating arms of the study according to order of recruitment until 16 participants were randomised to each cohort. An introductory 30-min didactic session was delivered to all participants covering relevant basic anatomy, equipment familiarisation and steps of the procedure. The intervention cohort then underwent full ureteroscopy training via the developed curriculum; the control group had no training. Both cohorts were subsequently assessed. Assessment protocol and outcome measures A standardised assessment protocol was set up, with all participants assessed on the Uro-Scopic TrainerTM utilising real operative equipment, within the ‘‘Igloo’’ environment. A standardised case of a left-sided mid-ureteric stone was
123
Surg Endosc
Fig. 2 Full immersion simulation utilised for ureteroscopy training
performed using the same scope instrumentation, stack system, scrub nurse and anaesthetist for every participant. Participants were asked to perform the WHO surgical safety checklist [13] both pre and post-operatively. The performance of each participant within the assessment protocol was recorded and sent to two independent and blinded expert assessors. Videos of all participants were assessed for both technical and non-technical skills parameters along the primary outcome measures proposed. On completion of the curriculum all participants were asked to fill out a questionnaire addressing various aspects of the curriculum undertaken. Subjective aspects covered included enjoyment and productivity, perceived difficulty, feasibility of delivery and perceived skill development through curriculum. Primary outcome measures utilised for this study included time to task completion, time to ureteral orifice catheterisation, stone withdrawal and stent insertion. Furthermore, the extensively validated objective structured assessment of technical skills (OSATS) [14] rating scale was utilised, assessing seven global rating aspects of technical skills. The final technical skills parameters included an objective task specific rigid ureteroscopy evaluation score (RUES) checklist developed for this study, which was content validated via expert consultation. Non-technical skills were assessed utilising the extensively validated non-technical skills for surgeons rating system (NOTSS) which analyses four key nontechnical parameters; situational awareness, decision making, communication and teamwork and leadership [15].
123
Secondary outcome measures included quantitative data collected from the post-study questionnaires. Statistical analysis Parametric and non-parametric analyses were used depending on the distribution of our data as evaluated by a Shapiro–Wilk test. Statistical analysis was undergone utilising SigmaPlot version 12.0 software. A two-tailed p value of less than 0.05 was considered to be statistically significant.
Results Content validation Quantitative surveys distributed within the curriculum development phase revealed that 86 % of experts agreed that the developed curriculum would be effective in teaching novices to perform ureteroscopies. Furthermore, 100 % of experts agreed that the integration of full immersion simulation within the curriculum was a useful addition to the curriculum for teaching non-technical skills. Educational impact All parameters recorded demonstrated a significant improvement in those that underwent the simulation-based
Surg Endosc Table 1 Demographics of study participants Variable
Intervention group
Control group
p value
Number
16
16
–
Age
21.4
20.2
0.08
Gender (Male/Female)
12/4
11/5
0.69
Previous endourological observation
1
3
0.29
curriculum (Table 2) when compared to the control group. The curriculum-trained group were significantly faster in performing the overall task (p \ 0.001), and all sub-tasks (all p \ 0.05) than the control group (Fig. 4). Additional technical skills measures showed significantly better performance in the intervention group in both OSATS (p \ 0.001) and RUES (p = 0.011) scores. When evaluating non-technical skills training, the curriculum-trained group had significantly higher NOTSS scores (p \ 0.001) than the non-trained group. Furthermore, when subdividing the NOTSS scores via individual scoring components, it was found that previous training within the surgical ‘‘igloo’’, significantly improved all NOTSS components (all p \ 0.05) (Table 2). Feasibility and acceptability Running the training sessions were feasible, with delivery of the curriculum carried out by all of those who underwent
it in less than 4 h in total, spread over the course of two sessions. Staffing requirements included a primary instructor who was present throughout the curriculum for the teaching and supervision of modules 1–3 with a secondary assistant additionally required for module 4 within the full immersive environment. Therefore, on average for every hour undergone by participants a total of one and a quarter hours of labour were required. Whilst large variance in completion time was present no individual undertook a task more than three times for proficiency to be achieved. Additionally, post-curriculum questionnaires completed by participants demonstrated that the feasibility of delivery of the curriculum was rated on average rating 9.27 on a 10-point likert scale. Furthermore, when analysing acceptability, 100 % of participants agreed that simulationbased training needs to be incorporated into training programmes with 93 % either agreeing or strongly agreeing that doing so is essential for patient safety. On a 10-point likert scale enjoyment and productivity of the curriculum was scored at 9.07 with difficulty of curriculum rated at 4.93 on average.
Discussion Within ureteroscopy there is a variety of literature assessing different simulation-based training methods and different assessment methods. Various studies have presented examples of training pathways utilised for the purpose of
Fig. 3 Study protocol
Randomised (n= 32)
Allocated to Intervention [Ureteroscopy Curriculum] (n= 16 )
Allocated to no Intervention [Control Group] (n= 16 )
Introduction to Ureteroscopy didactic session
Introduction to Ureteroscopy didactic session
Simulation Based Ureteroscopy Curriculum Training [2 Sessions]
Lost to Follow Up (n=1)
Assessment and Video Analysis (n=15)
Assessment and Video Analysis (n=16)
123
Surg Endosc Table 2 Mean educational impact outcome results Outcome measure
Intervention (n = 15), mean (SD)
Control (n = 16), mean (SD)
Total time to completion (sec)
p value
933 (376)
1617 (481)
p \ 0.001
Catheterisation time (sec)
112 (54)
194 (37)
p \ 0.001
Stone extraction time (sec)
209 (124)
394 (234)
p = 0.011 p = 0.010
464 (363)
793 (388)
OSATS score (7–35)
Stent insertion time (sec)
23.5 (6.4)
14.8 (6.5)
p \ 0.001
RUES scores (0–100)
73.4 (15.8)
60.4 (20.7)
p = 0.011
NOTSS score (4–16)
13.1 (2.49)
9.1 (3.42)
p \ 0.001
Situational awareness (1–4)
3.33 (0.66)
2.40 (0.93)
p \ 0.001
Decision making (1–4)
3.17 (0.70)
2.47 (0.86)
p = 0.002
Communication and teamwork (1–4)
3.33 (0.66)
2.17 (0.94)
p \ 0.001
Leadership (1–4)
3.30 (0.75)
2.13 (0.94)
p \ 0.001
Key: Mean Median
B
2500
OSAT Score (Out of 35)
A
Time (Seconds)
2000
1500
1000
35
30
25
20
15
500 10 0 Control
D
100
14
80
12
70 60
Control
Intervention
Control
Intervention
16
90
NOTSS Score
Score (Out of 100)
C
5
Intervention
10 8
50
6
40
4
30
2 0
20 Control
Intervention
Fig. 4 Outcome measure results. A Total time to completion, B OSATS scores, C RUES scores, D NOTSS scores
individual model validation [16–18]. However, these studies focus primarily on the utility of the models as opposed to developing a method of delivering the teaching.
123
Additionally, it is evident various assessment tools are available within surgery including global assessment scales or task specific methods [19], with some tools already having
Surg Endosc
been demonstrated to be effective within ureteroscopy [20]. Furthermore, research within ureteroscopy is currently focused within technical skills training and assessment. Therefore, the intention of this curriculum was to go beyond the current literature, by formally developing and evaluating a simulation-based curriculum, which combined both training and assessment for both technical and non-technical skills within one structured pathway. The results of this study have demonstrated several aspects of validation for the developed ureteroscopy curriculum. Content validation was demonstrated via positive results from expert consultation via a Delphi methodology. Furthermore, every outcome measure utilised, including all time to completion components, OSATS, RUES and NOTSS scores significantly improved in those undergoing the simulation-based curriculum demonstrating the curriculums educational impact. Finally, acceptability and feasibility were shown through positive subjective questionnaire results completed by participants. Whilst many see the need for technical skills training, non-technical skills within surgery are vital with poor skillsets increasing post-operative mortality [21]. This demonstrates that these skills should not be overlooked when training surgeons. A large body of evidence exists to demonstrate that simulation-based training is an effective modality to teach both technical and non-technical skills separately [5, 22]. However, little research has been previously conducted to investigate the impact of teaching these two critical skills together. One of the most important outcomes of the current study is that this curriculum demonstrates that integrating the teaching and assessment of the two skillsets within one curriculum is both feasible and educationally valuable. This shows that rather than both skills being separately trained, they can be incorporated within one structured simulation-based curriculum to develop the two skillsets in parallel. Previous research within ureteroscopy simulation has focused primarily on individual model validation. However, utilising these models within a validated proficiencybased curricula allows for a simulation-based modality that can be used within the initial phase of training pathways for urologists to learn a core urological procedure. Additionally, this curriculum may offer a much wider use as a framework, which can be adapted for training of any surgical procedure and level of training. The content delivered within the four proposed core modules can easily be adapted to any procedure with the inbuilt proficiency benchmarks adapted to the specific procedure, and level of training being delivered. However, despite offering the potential to act as a framework, further research is required to assess if the current curriculum can be altered and adapted to meet the needs for a simulation programme for any surgical procedure.
The principal limitation of this study is that training assessment was conducted via change in simulator performance and in a simulated operative environment. Therefore, whilst it is demonstrated that those who practice on a simulator improve both technical and non-technical performance on a simulator it does not demonstrate that this is transferable to the operating room. As participants were medical students, the predictive element of the curriculum and the transferability of skills to the operating room was not ethical to investigate. Finally, whilst skill acquisition was demonstrated within the current study, it demonstrates no evidence on the maintenance of this skill acquisition. Future work within this developed curriculum includes the potential for incorporation of additional simulation modalities such as wet lab training and e-modules to analyse the impact these additional training modalities can have. Furthermore assessing the feasibility of running such a curriculum alongside the standard training pathways at a multi-institutional level would assess the full applicability of this curriculum. Finally, further research investigating the current curriculum, as a framework for any surgical procedure may facilitate future simulation-based programmes by offering a simple framework that can be built upon for teaching both technical and non-technical skills.
Conclusions The developed curriculum offers a valid, acceptable and educationally useful modality for the training and development of skills within ureteroscopy. This structured simulation-based curriculum demonstrates that training and assessing both technical and non-technical skills simultaneously within a structured training programme is both feasible and educationally valuable. Finally, the developed curriculum may offer a framework that may be utilised for the development of an integrated simulation-based training programme within any surgical procedure, which may facilitate future curriculum developments. Acknowledgments The authors would like to Thank Dr. Saied Froghi for his invaluable contribution with the statistical analysis of the collected data. The research was supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. Additionally, Prokar Dasgupta and Kamran Ahmed acknowledge educational research support from The Urology Foundation, Olympus and the Royal College of Surgeons of England. Prokar Dasgupta further acknowledges support from the Vattikuti Foundation and Medical Research Council (MRC) Centre for Transplantation at Kings College London, UK - MRC Grant No. MR/ J006742/1. Disclosures Oliver Brunckhorst, Shahab Shahid, Abdullatif Aydin, Craig McIlhenny, Shahid Khan, Johar Raza Syed, Arun Sahai, James
123
Surg Endosc Brewin, Fernando Bello, Roger Kneebone and Muhammad Shamim Khan have no conflicts of interest or financial ties to disclose.
References 1. Reznick RK, MacRae H (2006) Teaching surgical skills—changes in the wind. N Engl J Med 355(25):2664–2669 2. Skolarikos A, Gravas S, Laguna MP, Traxer O, Preminger GM, de la Rosette J (2011) Training in ureteroscopy: a critical appraisal of the literature. BJU Int 108(6):798–805 discussion 805 3. Gallagher AG, Traynor O (2008) Simulation in surgery: opportunity or threat? Ir J Med Sci 177(4):283–287 4. Abboudi H, Khan MS, Guru KA, Froghi S, de Win G, Van Poppel H, Dasgupta P, Ahmed K (2013) Learning curves for urological procedures: a systematic review. BJU Int 114(4):617–629 5. Dawe SR, Pena GN, Windsor JA, Broeders JA, Cregan PC, Hewett PJ, Maddern GJ (2014) Systematic review of skills transfer after surgical simulation-based training. Br J Surg 101(9):1063–1076 6. Brunckhorst O, Aydin A, Abboudi H, Sahai A, Khan MS, Dasgupta P, Ahmed K (2014) Simulation-based ureteroscopy training: a systematic review. J Surg Educ 7. Ahmed K, Jawad M, Abboudi M, Gavazzi A, Darzi A, Athanasiou T, Vale J, Khan MS, Dasgupta P (2011) Effectiveness of procedural simulation in urology: a systematic review. J Urol 186(1):26–34 8. Gawande AA, Zinner MJ, Studdert DM, Brennan TA (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133(6):614–621 9. Flin RH, O’Connor P, Crichton M (2008) Safety at the sharp end : a guide to non-technical skills. Ashgate, Aldershot 10. McDougall EM (2007) Validation of surgical simulators. J Endourol 21(3):244–247 11. Van Der Vleuten CP (1996) The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract 1(1):41–67
123
12. Kassab E, Tun JK, Arora S, King D, Ahmed K, Miskovic D, Cope A, Vadhwana B, Bello F, Sevdalis N, Kneebone R (2011) ‘‘Blowing up the barriers’’ in surgical training: exploring and validating the concept of distributed simulation. Ann Surg 254(6):1059–1065 13. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AA, Safe Surgery Saves Lives Study G (2009) A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 360(5):491–499 14. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278 15. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D (2006) Development of a rating system for surgeons’ non-technical skills. Med Educ 40(11):1098–1104 16. Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD (2001) A novel approach to endourological training: training at the surgical skills center. J Urol 166(4):1261–1266 17. Brehmer M, Swartz R (2005) Training on bench models improves dexterity in ureteroscopy. Eur Urol 48(3):458–463 discussion 463 18. Chou DS, Abdelshehid C, Clayman RV, McDougall EM (2006) Comparison of results of virtual-reality simulator and training model for basic ureteroscopy training. J Endourol 20(4):266–271 19. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202(4):469–480 e466 20. Kishore TA, Pedro RN, Monga M, Sweet RM (2008) Assessment of validity of an OSATS for cystoscopic and ureteroscopic cognitive and psychomotor skills. J Endourol 22(12):2707–2711 21. Mazzocco K, Petitti DB, Fong KT, Bonacum D, Brookey J, Graham S, Lasky RE, Sexton JB, Thomas EJ (2009) Surgical team behaviors and patient outcomes. Am J Surg 197(5):678–685 22. Gordon M, Darbyshire D, Baker P (2012) Non-technical skills training to enhance patient safety: a systematic review. Med Educ 46(11):1042–1054
and Other Interventional Techniques
Simulation-based ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial Oliver Brunckhorst • Shahab Shahid • Abdullatif Aydin • Craig McIlhenny • Shahid Khan • Syed Johar Raza • Arun Sahai • James Brewin • Fernando Bello Roger Kneebone • Muhammad Shamim Khan • Prokar Dasgupta • Kamran Ahmed
•
Received: 20 September 2014 / Accepted: 7 November 2014 Ó Springer Science+Business Media New York 2014
Abstract Background Current training modalities within ureteroscopy have been extensively validated and must now be integrated within a comprehensive curriculum. Additionally, non-technical skills often cause surgical error and little research has been conducted to combine this with technical skills teaching. This study therefore aimed to develop and validate a curriculum for semi-rigid ureteroscopy, integrating both technical and non-technical skills teaching within the programme. Methods Delphi methodology was utilised for curriculum development and content validation, with a randomised
O. Brunckhorst S. Shahid A. Aydin A. Sahai J. Brewin M. S. Khan P. Dasgupta (&) K. Ahmed MRC Centre for Transplantation, King’s College London, Department of Urology, Guy’s and St. Thomas’ NHS Foundation Trust, King’s Health Partners, London, UK e-mail: [email protected] K. Ahmed e-mail: [email protected] C. McIlhenny Forth Valley Royal Hospital, NHS Forth Valley, Larbet, UK S. Khan Department of Urology, East Surrey Hospital, Surrey and Sussex Healthcare NHS Trust, Redhill, UK S. J. Raza Department of Urology, Roswell Park Cancer Institute, Buffalo, NY, USA F. Bello R. Kneebone Department of Surgery and Cancer, Imperial College London, St. Mary’s Hospital Campus, London, UK
trial then conducted (n = 32) for curriculum evaluation. The developed curriculum consisted of four modules; initially developing basic technical skills and subsequently integrating non-technical skills teaching. Sixteen participants underwent the simulation-based curriculum and were subsequently assessed, together with the control cohort (n = 16) within a full immersion environment. Both technical (Time to completion, OSATS and a task specific checklist) and non-technical (NOTSS) outcome measures were recorded with parametric and non-parametric analyses used depending on the distribution of our data as evaluated by a Shapiro–Wilk test. Results Improvements within the intervention cohort demonstrated educational value across all technical and non-technical parameters recorded, including time to completion (p \ 0.01), OSATS scores (p \ 0.001), task specific checklist scores (p = 0.011) and NOTSS scores (p \ 0.001). Content validity, feasibility and acceptability were all demonstrated through curriculum development and post-study questionnaire results. Conclusions The current developed curriculum demonstrates that integrating both technical and non-technical skills teaching is both educationally valuable and feasible. Additionally, the curriculum offers a validated simulationbased training modality within ureteroscopy and a framework for the development of other simulation-based programmes. Keywords Ureteroscopy Education Curriculum Nontechnical skills Simulation
It has become increasingly difficult to rely solely on traditional observership and mentorship to train surgeons within the modern era. Variable patient flow and a lack of
123
Surg Endosc
a structured training pathway mean that these methods produce differing educational outcomes [1]. Additionally, minimally invasive procedures such as ureteroscopy have become increasingly popular. Ureteroscopy is a technically difficult procedure with a relatively long learning curve between 50 and 60 cases, and therefore poses a challenge to the training surgeon [2]. This is further compounded by decreasing training hours across the globe and increasing ethical concerns of the training surgeon practicing on patients to acquire proficiency [3]. With these concerns it is clear that adjuncts to training are required with simulation-based training offering one option. Utilisation of simulation as a training tool is believed to improve the progression along the initial phase of the learning curve [4] and has been demonstrated to improve a surgeons’ technical ability in the Operating Room [5]. This highlights its role as a valuable adjunct to traditional training pathways. Various individual ureteroscopy training modalities are available and have all been validated to varying degrees [6]. With current evidence levels these can reliably be utilised into a training program. However, whilst these offer a validated modality of training, simulation should not be used as the only method for education, but should be integrated into a comprehensive proficiency-based curriculum [7]. Furthermore, whilst training for technical skills is vital, surgery additionally requires mastery of non-technical skills. These have been highlighted as a common cause of surgical error [8] and like technical skills, which are acquired over many years of practice and training, can also be acquired through training. There are three distinct categories of non-technical skills including social (communication, teamwork and leadership) and cognitive skills (decision making and situational awareness) alongside the personal resource factors (ability to cope with stress and fatigue) one possesses [9]. There is a current lack in the literature as to a formalised and validated curriculum, which utilises the available training modalities for ureteroscopy. Furthermore, whilst many studies have analysed technical and non-technical skills as separate entities there is paucity in the literature analysing these two skills, within one curriculum, in any surgical procedure. Therefore, this prospective study was conducted with the following aims: 1.
2.
To develop a simulation-based ureteroscopy curriculum, integrating both technical and non-technical skills within the training pathway. Evaluate the developed curriculum in terms of feasibility, acceptability, content validity and educational impact as defined by McDougall [10] and Van der Vleuten [11].
123
Materials and methods Curriculum development The curriculum was drafted via literature review and identification of key steps within the ureteroscopy procedure. This was then content validated utilising the Delphi methodology. Quantitative surveys addressing the proposed modules and tasks, training duration, non-technical skills training and assessment protocol were distributed to six experts from two countries (UK and USA) and across three institutions. These were circulated until a saturation of new information was achieved to produce a finalised curriculum. The final developed curriculum was divided into four modules: 1. basic skills development, 2. modular training, 3. integrated skills development and 4. full immersion simulation (Fig. 1). Each module consisted of tasks addressing important basic knowledge, technical skills and non-technical skills to safely and effectively conduct a semi-rigid ureteroscopy. Module 1 (basic skills development) was designed to consolidate the knowledge delivered in a previous didactic session and allows for basic manoeuvring skills to be developed. Modular training tasks within module 2 (modular training) then develops individual procedural step skills developing basic technical skills. These were then brought together within module 3 (integrated skills development) via completion of full case simulation. Finally, with all the necessary basic technical skills developed the candidate is introduced to non-technical skills in module 4 (full immersion simulation) via didactic teaching and performance of a full case simulation within the full immersion environment utilising the ‘‘Igloo’’ simulator. Individuals were mentored along the curriculum modules with tasks set out to be proficiencybased according to predetermined time criteria. Study materials Two models were included within the curriculum; the UroScopic TrainerTM (Limbs & Things Ltd., Bristol, UK), a physical bench top model, and the URO MentorTM (Simbionix, Cleveland, USA), a virtual reality simulator. Both have undergone extensive validation and posses a proven face, construct and concurrent validity with a proven educational impact [6]. Additionally, for non-technical skills teaching the validated full immersion environment of the Distributed Simulator (Fig. 2) was used [12]. This is a portable environment that utilises an inflatable ‘‘Igloo’’ to provide a 360-degree simulation of an operating room. Within the Distributed environment only the Uro-Scopic Trainer was utilised so as to utilise true operative equipment.
Surg Endosc
Module 1 Basic Skills Development
Task 1: Snapshots of urethral and vesical landmarks
Knowledge Task 2: Snapshots of ureteral landmarks
Task 3: Bladder Visualisation
Module 2 Modular Training
Task 4: Orifice Catheterization
Technical Skills Task 5: Ureteral Stone Extraction Task 6: Stent Insertion
Module 3 Integrated Skills Development
Case 1: Mid Ureteric Stone Fragmentation and Extraction
Case 2: Mid Ureteric Stone Extraction
Module 4 Integration of Case 2 in Distributed Environment
Full Immersion Simulation
Integration of Knowledge and Technical Skills
Technical and Non-Technical Skills
Fig. 1 Developed curriculum
Participants Thirty-two medical student participants, with no prior ureteroscopy training were recruited from six university institutions within the United Kingdom (Table 1). Medical students were selected as opposed to trainees as this ensures subjects do not possess previous endourological exposure allowing for more comparable groups and ensuring any differences recorded are due to the educational intervention as opposed to previous experience. Post hoc power analysis utilising time to completion and rating scales demonstrated a power of greater than 80 %, at the significance level of 0.05, to reveal a significant difference between cohorts with the obtained sample size. Study design Validation was evaluated via a randomised controlled trial (Fig. 3). Participants filled out a demographic questionnaire
containing age, gender and previous endourological surgery experience information and were then randomised via blocked randomisation so as to ensure equal sized cohorts. Groups of four were allocated to alternating arms of the study according to order of recruitment until 16 participants were randomised to each cohort. An introductory 30-min didactic session was delivered to all participants covering relevant basic anatomy, equipment familiarisation and steps of the procedure. The intervention cohort then underwent full ureteroscopy training via the developed curriculum; the control group had no training. Both cohorts were subsequently assessed. Assessment protocol and outcome measures A standardised assessment protocol was set up, with all participants assessed on the Uro-Scopic TrainerTM utilising real operative equipment, within the ‘‘Igloo’’ environment. A standardised case of a left-sided mid-ureteric stone was
123
Surg Endosc
Fig. 2 Full immersion simulation utilised for ureteroscopy training
performed using the same scope instrumentation, stack system, scrub nurse and anaesthetist for every participant. Participants were asked to perform the WHO surgical safety checklist [13] both pre and post-operatively. The performance of each participant within the assessment protocol was recorded and sent to two independent and blinded expert assessors. Videos of all participants were assessed for both technical and non-technical skills parameters along the primary outcome measures proposed. On completion of the curriculum all participants were asked to fill out a questionnaire addressing various aspects of the curriculum undertaken. Subjective aspects covered included enjoyment and productivity, perceived difficulty, feasibility of delivery and perceived skill development through curriculum. Primary outcome measures utilised for this study included time to task completion, time to ureteral orifice catheterisation, stone withdrawal and stent insertion. Furthermore, the extensively validated objective structured assessment of technical skills (OSATS) [14] rating scale was utilised, assessing seven global rating aspects of technical skills. The final technical skills parameters included an objective task specific rigid ureteroscopy evaluation score (RUES) checklist developed for this study, which was content validated via expert consultation. Non-technical skills were assessed utilising the extensively validated non-technical skills for surgeons rating system (NOTSS) which analyses four key nontechnical parameters; situational awareness, decision making, communication and teamwork and leadership [15].
123
Secondary outcome measures included quantitative data collected from the post-study questionnaires. Statistical analysis Parametric and non-parametric analyses were used depending on the distribution of our data as evaluated by a Shapiro–Wilk test. Statistical analysis was undergone utilising SigmaPlot version 12.0 software. A two-tailed p value of less than 0.05 was considered to be statistically significant.
Results Content validation Quantitative surveys distributed within the curriculum development phase revealed that 86 % of experts agreed that the developed curriculum would be effective in teaching novices to perform ureteroscopies. Furthermore, 100 % of experts agreed that the integration of full immersion simulation within the curriculum was a useful addition to the curriculum for teaching non-technical skills. Educational impact All parameters recorded demonstrated a significant improvement in those that underwent the simulation-based
Surg Endosc Table 1 Demographics of study participants Variable
Intervention group
Control group
p value
Number
16
16
–
Age
21.4
20.2
0.08
Gender (Male/Female)
12/4
11/5
0.69
Previous endourological observation
1
3
0.29
curriculum (Table 2) when compared to the control group. The curriculum-trained group were significantly faster in performing the overall task (p \ 0.001), and all sub-tasks (all p \ 0.05) than the control group (Fig. 4). Additional technical skills measures showed significantly better performance in the intervention group in both OSATS (p \ 0.001) and RUES (p = 0.011) scores. When evaluating non-technical skills training, the curriculum-trained group had significantly higher NOTSS scores (p \ 0.001) than the non-trained group. Furthermore, when subdividing the NOTSS scores via individual scoring components, it was found that previous training within the surgical ‘‘igloo’’, significantly improved all NOTSS components (all p \ 0.05) (Table 2). Feasibility and acceptability Running the training sessions were feasible, with delivery of the curriculum carried out by all of those who underwent
it in less than 4 h in total, spread over the course of two sessions. Staffing requirements included a primary instructor who was present throughout the curriculum for the teaching and supervision of modules 1–3 with a secondary assistant additionally required for module 4 within the full immersive environment. Therefore, on average for every hour undergone by participants a total of one and a quarter hours of labour were required. Whilst large variance in completion time was present no individual undertook a task more than three times for proficiency to be achieved. Additionally, post-curriculum questionnaires completed by participants demonstrated that the feasibility of delivery of the curriculum was rated on average rating 9.27 on a 10-point likert scale. Furthermore, when analysing acceptability, 100 % of participants agreed that simulationbased training needs to be incorporated into training programmes with 93 % either agreeing or strongly agreeing that doing so is essential for patient safety. On a 10-point likert scale enjoyment and productivity of the curriculum was scored at 9.07 with difficulty of curriculum rated at 4.93 on average.
Discussion Within ureteroscopy there is a variety of literature assessing different simulation-based training methods and different assessment methods. Various studies have presented examples of training pathways utilised for the purpose of
Fig. 3 Study protocol
Randomised (n= 32)
Allocated to Intervention [Ureteroscopy Curriculum] (n= 16 )
Allocated to no Intervention [Control Group] (n= 16 )
Introduction to Ureteroscopy didactic session
Introduction to Ureteroscopy didactic session
Simulation Based Ureteroscopy Curriculum Training [2 Sessions]
Lost to Follow Up (n=1)
Assessment and Video Analysis (n=15)
Assessment and Video Analysis (n=16)
123
Surg Endosc Table 2 Mean educational impact outcome results Outcome measure
Intervention (n = 15), mean (SD)
Control (n = 16), mean (SD)
Total time to completion (sec)
p value
933 (376)
1617 (481)
p \ 0.001
Catheterisation time (sec)
112 (54)
194 (37)
p \ 0.001
Stone extraction time (sec)
209 (124)
394 (234)
p = 0.011 p = 0.010
464 (363)
793 (388)
OSATS score (7–35)
Stent insertion time (sec)
23.5 (6.4)
14.8 (6.5)
p \ 0.001
RUES scores (0–100)
73.4 (15.8)
60.4 (20.7)
p = 0.011
NOTSS score (4–16)
13.1 (2.49)
9.1 (3.42)
p \ 0.001
Situational awareness (1–4)
3.33 (0.66)
2.40 (0.93)
p \ 0.001
Decision making (1–4)
3.17 (0.70)
2.47 (0.86)
p = 0.002
Communication and teamwork (1–4)
3.33 (0.66)
2.17 (0.94)
p \ 0.001
Leadership (1–4)
3.30 (0.75)
2.13 (0.94)
p \ 0.001
Key: Mean Median
B
2500
OSAT Score (Out of 35)
A
Time (Seconds)
2000
1500
1000
35
30
25
20
15
500 10 0 Control
D
100
14
80
12
70 60
Control
Intervention
Control
Intervention
16
90
NOTSS Score
Score (Out of 100)
C
5
Intervention
10 8
50
6
40
4
30
2 0
20 Control
Intervention
Fig. 4 Outcome measure results. A Total time to completion, B OSATS scores, C RUES scores, D NOTSS scores
individual model validation [16–18]. However, these studies focus primarily on the utility of the models as opposed to developing a method of delivering the teaching.
123
Additionally, it is evident various assessment tools are available within surgery including global assessment scales or task specific methods [19], with some tools already having
Surg Endosc
been demonstrated to be effective within ureteroscopy [20]. Furthermore, research within ureteroscopy is currently focused within technical skills training and assessment. Therefore, the intention of this curriculum was to go beyond the current literature, by formally developing and evaluating a simulation-based curriculum, which combined both training and assessment for both technical and non-technical skills within one structured pathway. The results of this study have demonstrated several aspects of validation for the developed ureteroscopy curriculum. Content validation was demonstrated via positive results from expert consultation via a Delphi methodology. Furthermore, every outcome measure utilised, including all time to completion components, OSATS, RUES and NOTSS scores significantly improved in those undergoing the simulation-based curriculum demonstrating the curriculums educational impact. Finally, acceptability and feasibility were shown through positive subjective questionnaire results completed by participants. Whilst many see the need for technical skills training, non-technical skills within surgery are vital with poor skillsets increasing post-operative mortality [21]. This demonstrates that these skills should not be overlooked when training surgeons. A large body of evidence exists to demonstrate that simulation-based training is an effective modality to teach both technical and non-technical skills separately [5, 22]. However, little research has been previously conducted to investigate the impact of teaching these two critical skills together. One of the most important outcomes of the current study is that this curriculum demonstrates that integrating the teaching and assessment of the two skillsets within one curriculum is both feasible and educationally valuable. This shows that rather than both skills being separately trained, they can be incorporated within one structured simulation-based curriculum to develop the two skillsets in parallel. Previous research within ureteroscopy simulation has focused primarily on individual model validation. However, utilising these models within a validated proficiencybased curricula allows for a simulation-based modality that can be used within the initial phase of training pathways for urologists to learn a core urological procedure. Additionally, this curriculum may offer a much wider use as a framework, which can be adapted for training of any surgical procedure and level of training. The content delivered within the four proposed core modules can easily be adapted to any procedure with the inbuilt proficiency benchmarks adapted to the specific procedure, and level of training being delivered. However, despite offering the potential to act as a framework, further research is required to assess if the current curriculum can be altered and adapted to meet the needs for a simulation programme for any surgical procedure.
The principal limitation of this study is that training assessment was conducted via change in simulator performance and in a simulated operative environment. Therefore, whilst it is demonstrated that those who practice on a simulator improve both technical and non-technical performance on a simulator it does not demonstrate that this is transferable to the operating room. As participants were medical students, the predictive element of the curriculum and the transferability of skills to the operating room was not ethical to investigate. Finally, whilst skill acquisition was demonstrated within the current study, it demonstrates no evidence on the maintenance of this skill acquisition. Future work within this developed curriculum includes the potential for incorporation of additional simulation modalities such as wet lab training and e-modules to analyse the impact these additional training modalities can have. Furthermore assessing the feasibility of running such a curriculum alongside the standard training pathways at a multi-institutional level would assess the full applicability of this curriculum. Finally, further research investigating the current curriculum, as a framework for any surgical procedure may facilitate future simulation-based programmes by offering a simple framework that can be built upon for teaching both technical and non-technical skills.
Conclusions The developed curriculum offers a valid, acceptable and educationally useful modality for the training and development of skills within ureteroscopy. This structured simulation-based curriculum demonstrates that training and assessing both technical and non-technical skills simultaneously within a structured training programme is both feasible and educationally valuable. Finally, the developed curriculum may offer a framework that may be utilised for the development of an integrated simulation-based training programme within any surgical procedure, which may facilitate future curriculum developments. Acknowledgments The authors would like to Thank Dr. Saied Froghi for his invaluable contribution with the statistical analysis of the collected data. The research was supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. Additionally, Prokar Dasgupta and Kamran Ahmed acknowledge educational research support from The Urology Foundation, Olympus and the Royal College of Surgeons of England. Prokar Dasgupta further acknowledges support from the Vattikuti Foundation and Medical Research Council (MRC) Centre for Transplantation at Kings College London, UK - MRC Grant No. MR/ J006742/1. Disclosures Oliver Brunckhorst, Shahab Shahid, Abdullatif Aydin, Craig McIlhenny, Shahid Khan, Johar Raza Syed, Arun Sahai, James
123
Surg Endosc Brewin, Fernando Bello, Roger Kneebone and Muhammad Shamim Khan have no conflicts of interest or financial ties to disclose.
References 1. Reznick RK, MacRae H (2006) Teaching surgical skills—changes in the wind. N Engl J Med 355(25):2664–2669 2. Skolarikos A, Gravas S, Laguna MP, Traxer O, Preminger GM, de la Rosette J (2011) Training in ureteroscopy: a critical appraisal of the literature. BJU Int 108(6):798–805 discussion 805 3. Gallagher AG, Traynor O (2008) Simulation in surgery: opportunity or threat? Ir J Med Sci 177(4):283–287 4. Abboudi H, Khan MS, Guru KA, Froghi S, de Win G, Van Poppel H, Dasgupta P, Ahmed K (2013) Learning curves for urological procedures: a systematic review. BJU Int 114(4):617–629 5. Dawe SR, Pena GN, Windsor JA, Broeders JA, Cregan PC, Hewett PJ, Maddern GJ (2014) Systematic review of skills transfer after surgical simulation-based training. Br J Surg 101(9):1063–1076 6. Brunckhorst O, Aydin A, Abboudi H, Sahai A, Khan MS, Dasgupta P, Ahmed K (2014) Simulation-based ureteroscopy training: a systematic review. J Surg Educ 7. Ahmed K, Jawad M, Abboudi M, Gavazzi A, Darzi A, Athanasiou T, Vale J, Khan MS, Dasgupta P (2011) Effectiveness of procedural simulation in urology: a systematic review. J Urol 186(1):26–34 8. Gawande AA, Zinner MJ, Studdert DM, Brennan TA (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133(6):614–621 9. Flin RH, O’Connor P, Crichton M (2008) Safety at the sharp end : a guide to non-technical skills. Ashgate, Aldershot 10. McDougall EM (2007) Validation of surgical simulators. J Endourol 21(3):244–247 11. Van Der Vleuten CP (1996) The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract 1(1):41–67
123
12. Kassab E, Tun JK, Arora S, King D, Ahmed K, Miskovic D, Cope A, Vadhwana B, Bello F, Sevdalis N, Kneebone R (2011) ‘‘Blowing up the barriers’’ in surgical training: exploring and validating the concept of distributed simulation. Ann Surg 254(6):1059–1065 13. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AA, Safe Surgery Saves Lives Study G (2009) A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 360(5):491–499 14. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278 15. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D (2006) Development of a rating system for surgeons’ non-technical skills. Med Educ 40(11):1098–1104 16. Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD (2001) A novel approach to endourological training: training at the surgical skills center. J Urol 166(4):1261–1266 17. Brehmer M, Swartz R (2005) Training on bench models improves dexterity in ureteroscopy. Eur Urol 48(3):458–463 discussion 463 18. Chou DS, Abdelshehid C, Clayman RV, McDougall EM (2006) Comparison of results of virtual-reality simulator and training model for basic ureteroscopy training. J Endourol 20(4):266–271 19. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202(4):469–480 e466 20. Kishore TA, Pedro RN, Monga M, Sweet RM (2008) Assessment of validity of an OSATS for cystoscopic and ureteroscopic cognitive and psychomotor skills. J Endourol 22(12):2707–2711 21. Mazzocco K, Petitti DB, Fong KT, Bonacum D, Brookey J, Graham S, Lasky RE, Sexton JB, Thomas EJ (2009) Surgical team behaviors and patient outcomes. Am J Surg 197(5):678–685 22. Gordon M, Darbyshire D, Baker P (2012) Non-technical skills training to enhance patient safety: a systematic review. Med Educ 46(11):1042–1054
