Dharmawardana, N.,* Ruthenbeck, G.,† Woods, C.,*† Elmiyeh, B.,* Diment, L.,† Ooi, E.H.,*† Reynolds, K.† & Carney, A.S.*† *Flinders Medical Centre, †Medical Device Research Institute, Flinders University, Adelaide, Australia Accepted for publication 17 March 2015 Clin. Otolaryngol. 2015, 40, 569–579
Background: Virtual reality (VR) simulators provide an alternative to real patients for practicing surgical skills but require validation to ensure accuracy. Here, we validate the use of a virtual reality sinus surgery simulator with haptic feedback for training in Otorhinolaryngology - Head & Neck Surgery (OHNS). Methods: Participants were recruited from final-year medical students, interns, resident medical officers (RMOs), OHNS registrars and consultants. All participants completed an online questionnaire after performing four separate simulation tasks. These were then used to assess face, content and construct validity. ANOVA with post hoc correlation was used for statistical analysis. Results: The following groups were compared: (i) medical students/interns, (ii) RMOs, (iii) registrars and (iv) consultants. Face validity results had a statistically significant
Introduction
Simulation training provides an attractive method for teaching surgical skills outside the operating theatres (OT).1 Traditionally, surgical training has been an apprenticeship-based training following graduation from medical school. While this method has successfully trained our current surgical workforce to practise safely, this method inherently requires many hours of hands on experience in OT. With restrictions in junior doctor working hours and decreased exposure to OT time, it is possible that future surgeons will struggle to reach the same standards as their predecessors.2 Therefore, it is increasingly important to teach advanced surgical skills outside the OT to ensure safe surgical practice. Simulation training allows the safe teaching of surgical skills and can also be utilised for assessment Correspondence: N. Dharmawardana, Department of Otorhinolaryngology, Flinders ENT, Level 5, Flinders Medical Centre, Flinders Dr, Bedford Park SA 5042, Australia. Tel.: +61 8 8222 4000; Fax:+61 8 8204 7524; e-mail: [email protected] © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
(P < 0.05) difference between the consultant group and others, while there was no significant difference between medical student/intern and RMOs. Variability within groups was not significant. Content validity results based on consultant scoring and comments indicated that the simulations need further development in several areas to be effective for registrar-level teaching. However, students, interns and RMOs indicated that the simulations provide a useful tool for learning OHNS-related anatomy and as an introduction to ENT-specific procedures. Conclusions: The VR simulations have been validated for teaching sinus anatomy and nasendoscopy to medical students, interns and RMOs. However, they require further development before they can be regarded as a valid tool for more advanced surgical training.
purposes if appropriate validation has been performed. The Fitts–Posner learning theory describes cognition, integration and automation as the three stages of motor skill acquisition.3 The individual needs to understand the task at hand before combining this understanding with the required mechanics in order to perform the task efficiently. VR simulators provide access to all stages of this theory without patient compromise by allowing individuals to attempt the procedure with or without errors. There are many different types of surgical simulators including animal models, plastic mannequins, artificial theatre environments and computer-generated virtual reality (VR) simulators.4,5 Multiple Cochrane reviews have been published showing the benefits of using VR simulators for surgical training.6,7 VR simulation uses computer software to generate a virtual model of a specific training environment. With the evolution of simulation technology, it has become easier to construct realistic virtual models with realistic haptic feedback. The most well-validated haptic sinus simulator was constructed by Lockheed Martin Corporation based on flight simulation technology.8 How569
ORIGINAL ARTICLE
Validation of virtual-reality-based simulations for endoscopic sinus surgery
570 N. Dharmawardana et al.
ever, the prohibitive cost of this simulator prevented widespread uptake of the device. Our sinus simulator was developed by the Medical Device Research Institute at the Flinders University, Adelaide, Australia, in cooperation with the Department of OHNS (Flinders Medical Centre, Adelaide, Australia). It provides a laptop-based photorealistic VR sinus simulator with haptic tissue feedback.9 Validity is an advanced concept defined as the property of being true and correct, and is in keeping with reality.10 There are multiple facets to validation that includes face validity and content validity among others. Face validity is defined as the assessment by experts of contents to see whether they are appropriate.10 While content validity more rigorously examines the content of the simulator by subject matter experts. Educational validity is a component of content validity that specifically assesses the suitability to test or teach.12 In this validation study, we describe the face, educational and content validity of our sinus simulation. Methods Simulator components
The simulation software for this study was run on a 17-inch laptop and controlled by the user via two different haptic devices (Fig 11). The first haptic device ‘Novint Falcon’ (Novint Technologies, Inc. Delaware, USA) only senses three-dimensional position and provides linear haptic feedback. The second haptic device ‘Phantom Omni’ since renamed ‘Geomagic Touch’ (Three D Systems, SC, USA) senses both three-dimensional position and orientation, again with linear haptic feedback. The laptop was set up on a desk, elevated to eye level, and haptic devices were located at
comfortable distance apart from each other and lower than the laptop screen for the simulation of the real intraoperative position of the monitor and surgical instruments. The Novint Falcon was used to control the endoscope, while the Phantom Omni manipulated the surgical instruments. When a task did not require a second tool, the second haptic device was disabled. The total cost of the components including the laptop computer was approximately $15 000 Australian dollars. Surgical tasks
Four surgical tasks were performed in order (Fig. 1). In Task 1, the participants had to manoeuvre a simulated rigid nasendoscope through the right-sided nasal cavity and identify six important anatomical landmarks considered to be important for accurate nasendoscopy examination of the nasal cavity and lateral nasal wall. These structures were inferior turbinate, middle turbinate, uncinate process, bulla, superior turbinate and the sphenoid ostium (Fig. 2). Once the name of the anatomical landmark appeared on the screen, the corresponding structure was indicated with a yellow marker.13 The participant had to find the landmark, and once within 5 mm of the marker, press a button on the haptic device to confirm that they had found it. As soon as a landmark was identified and the button pressed, the marker was hidden and the name of the next landmark displayed. At the completion of this task, the participants were directed to an online questionnaire to provide feedback and comments specific to the task described above. Task 2 was a simulated tissue removal task. The task required the user to remove a discoloured region of tissue from a larger tissue model. The tissue model (described by Ruthenbeck et al.)14 was hemispherical in shape and
Fig. 1. Four separate simulation tasks and the haptic devices used.
Fig. 2. Images showing each of the simulated anatomical landmarks the participants were instructed to find during Task 1 of the VR simulation. © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 571
deliberately abstract (unlike any real organ or anatomical region). The haptic device was used to control a stylus (depicted as a small white sphere) that could either palpate or volumetrically remove tissue when the haptic device button was pressed. Participants were instructed to remove a section of a hemispherical mass that was different in colour compared to the surrounding (Fig. 3). By pressing a button on the wand of the haptic device, the participant activated the cutting properties of the sphere and was also able to feel the vibration when activated. Detailed recordings of the time taken to complete the task, accuracy of virtual tissue removal, haptic force and position in virtual space were recorded at 1 kHz. The discoloured section had other physical properties such as decreased density and stiffness that were able to be felt through the haptic device. The endoscopic view was static for this task; hence, only one haptic device was required. The participants were then directed to complete an online questionnaire at the end of the simulation specific to this task.
(a)
Task 3 utilised a Phantom Omni controlled backbiter and microdebrider tool (switchable between tools), where participants were instructed to make two cuts using the backbiter tool on a vertical plate of virtual tissue showing where to cut with clear markings (Fig. 4 ). Once the cuts were made, they were instructed to switch the tool to a simulated microdebrider using a key on the laptop keyboard and remove the virtual tissue between the two cuts made earlier. Task 4 utilised the same simulator as Task 1 with the addition of an area of ‘abnormal simulated tissue’ to the middle turbinate. Participants were instructed to remove this ‘abnormal tissue’ using a simulated microdebrider (Fig. 5). The interactive tissue model is described by Ruthenbeck et al.15 Study groups
Twenty-four participants participated in this study: 5 final-year medical students, 2 interns, 7 resident medical
(b)
Fig. 3. Image A shows a screenshot as soon as the VR simulator for Task 2 was activated where the abstract simulated tissue mass has a distinctly differently coloured area to mark abnormal tissue. Image B shows the simulated cutting ball that can be manoeuvred using the haptic device to cut into the simulated tissue with haptic feedback (a)
(b)
(c)
Fig. 4. Image A shows the start-up screen for Task 3 with a sheet of simulated flat tissue with markings on the left and a simulated backbiter tool on the right. Image B shows the simulated backbiter in use. Image C shows the use of the simulated microdebrider to remove the discoloured middle turbinate tissue © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
572 N. Dharmawardana et al.
Each participant was provided with a printed instruction manual to complete each of the tasks described above using the VR simulator. Following each task, they were instructed to complete an online questionnaire (Fig. 6) where each question was rated on a 5-point scale, 1 (poor) and 5 (excellent) (Fig. 6). In the questionnaire, they were also provided with a free text field for any further comments. Questions were focussed to assess face validity (visual and haptic realism, user-friendliness), educational validity (potential as a teaching tool) and content validity. Data analysis Fig. 5. This image shows a screenshot from Task 4 where the participant was instructed to remove a portion of the middle turbinate with abnormal colouring compared to the rest of the simulated tissue using the simulated microdebrider.
officers (RMOs), 6 registrars and 4 consultants. Medical students were in the final year of a 4-year postgraduate medical degree, interns were defined as the postgraduate year 1 (PGY 1), RMOs were defined as a practicing clinician with general registration in Australia without being in a specific vocational training program, registrars were defined as a practicing clinician in OHNS specialist training through the Royal Australasian College of Surgeons (RACS), and consultants were defined as qualified OHNS specialists, awarded the Fellowship of the RACS.
Data analysis was performed using an SPSS statistical package using one-way analysis of variance with post hoc analysis using Dunnett’s T3 test assuming no equal size or variance between groups. A higher questionnaire score indicated a higher satisfactory rating of the utility of performing specific tasks on the simulator. Statistically significant results are reported as P = value, and non-significant results reported as P = ns. Alpha-value of 0.05 was utilised to report significance level. Ethical approval
This project was reviewed and approved by the Southern Adelaide Clinical Human Research Ethics Committee (SAC HREC) on 10 December 2012 in accordance with the National Statement of Ethical Conduct in Human Research (2007).
Fig. 6. Questions asked at the end of each task described above. Participants selected a score from 1 to 5, where 1 = poor and 5 = excellent or 1 = never and 5 = very likely. © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 573
Results Face validity
There were no statistically significant (P = ns) differences between the groups for the visual appearance between four tasks (Fig. 7). There were statistically significant (P = 0.00001) differences between the groups for haptic feedback and user-friendliness for Task 1 (Fig. 1A); however, there were no statistically significant (P = ns) differences between the 4 groups for tasks 2 to 4 for haptic feedback experience and user-friendliness.
Post hoc analysis of Task 1 for haptic feedback experience demonstrated that the intern/student group scored significantly higher compared to the registrars (P = 0.0002) and consultants (P = 0.005). The RMO group also scored significantly higher compared to the registrars (P = 0.0065). However, there was no significant difference (P = ns) between intern/student and RMOs or registrars and consultants. Post hoc analysis for userfriendliness indicated the intern/student group (P = 0.024) scored significantly higher compared to the consultants. There were no significant differences between the other groups for user-friendliness.
(a)
(b)
(c)
(d)
Fig. 7. Face validity: Charts A, B, C and D represent the data for tasks 1 to 4, respectively. Each figure indicates the mean score with associated standard error of mean (SEM) for visual appearance, haptic feedback and user-friendliness of the simulator and the instructions provided. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬). © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
574 N. Dharmawardana et al.
Educational validity
Teaching anatomy. There was a statistically significant difference (P = 0.029) between the intern/student group and the registrars for Task 1 where the registrars rated lower the suitability of the simulator for teaching anatomy for residents and registrars. There were no significant differences between the other groups for the suitability of the simulator for teaching anatomy for students or residents and registrars in tasks 1 and 4 (Fig. 8). Task 1 received higher scores with lower variance than Task 4 for teaching anatomy for both groups described above. Teaching procedural skills. Medical students and interns scored significantly higher than consultants for teaching endoscopy skills to interns/students (P = 0.042) and RMOs/ registrars (P = 0.026). There were no significant differences in scores between other groups (Fig. 9). Content validity
The intern/student group rated highly the suitability of the simulator for their current level of training for all tasks. However, in Task 1, this was significantly different to the RMOs (P = 0.023), registrars (P = 0.0002) and consultants (P = 0.004). For the same question in Task 1, the RMO group also scored significantly higher compared to the registrars (P = 0.0001) and consultants (P = 0.006) (Fig. 10A). For tasks 2 and 4, the rating was significantly different between the intern/student group and the
(a)
consultant group (P = 0.027) for the suitability of the simulator for their current level, but there was no statistically significant difference between other groups for the same question. For the same question in Task 3, the registrar group scored significantly lower when compared to RMO (P = 0.023) and the intern/student groups (P = 0.009). There was no statistically significant difference in scoring for the suitability of the simulator for any level of training. However, the registrar and consultant groups consistently scored lower compared the intern/ student and RMO groups (Fig. 10). The online questionnaire also allowed participants to provide further comments on a free text field that provided qualitative information. The consultant and registrar group consistently commented that the simulated nasendoscope in tasks 1 and 4 was ‘too easy’ to move through the simulated tissue structures and the haptic feedback was not realistic enough compared to a real rigid nasendoscopy. Multiple junior participants identified that a pre-task tutorial on sinus anatomy would have been useful to complete the requested tasks as they did not have specialist knowledge and had not used a rigid nasendoscope before. Task 2 received interesting comments where most participants identified the haptic feedback was much closer to real tissue feedback and that the three-dimensional effect was portrayed better. However, the consultant group commented on the abstract nature of this simulator and the limitation for surgical training. Task 3 received poor feedback from most participants ranging from the lack of haptic
(b)
Fig. 8. Charts A and B represent tasks 1 and 4, respectively. Each bar indicates mean score SEM for the suitability of teaching anatomy to medical students or residents and registrars. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬). © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 575
(a)
(b)
(c)
(d)
Fig. 9. Charts A, B, C and D represent data from tasks 1 to 4, respectively. Where Task 1 specifically addresses suitability to teach nasendoscopy skills, other three tasks questions the suitability of general procedural skills in otorhinolaryngology. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬).
feedback and the unrealistic tissue modelling with poor depth perception without the surrounding simulated tissues. Regarding the instructions, comments reflect minimal confusion for tasks 1, 2 and 4. However, the instructions for Task 3 were found to be confusing as per the comments. There were also comments with regard to having a facilitator to help through the tasks. The final comments were regarding the haptic devices, their placement and body positioning based on the equipment. Consultant, registrar and RMO group commented on difficulty in using the simulated instruments as it was difficult to rest their elbow and the feeling of hovering the instruments in the air made it more tiring and difficult to accurately complete the tasks. © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Discussion
Endoscopic sinus surgery (ESS) is a mainstay treatment for multiple rhinology conditions including chronic rhinosinusitis.16 However, there is an associated long-term revision rate of up to 20% and complication rates that vary from up to 10% reported 20 years ago to more recent reports of complication rates up to 5%.17 Nonetheless, 20 years after the introduction of this method of surgery, minor and major disabling complications continue to occur. The ability to prevent these complications relies mainly on knowledge and surgical experience. Hence, it is necessary to provide further training in ESS for otorhinolaryngology trainees. An advanced VR sinus simulator provides a safe environment
576 N. Dharmawardana et al.
(a)
(b)
(c)
(d)
Fig. 10. Charts A, B, C and D represent data from tasks 1 to 4, respectively. Each chart represents two data series comparing the scores for the suitability of the simulator for training at any level from student to consultant versus the suitability for training at the level of the participant at present. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬).
where mistakes can be made without risks or harm to patients before a trainee operates on live patients. Comparison with other studies
There are many VR simulators described in literature for use in otorhinolaryngology training with at least 12 temporal bone surgery simulators, 6 ESS simulators and 3 myringotomy simulators previously reported by Arora et al. 2014, a review of current otorhinolaryngology simulators.18 Of these, only a few have had full validation studies conducted. As previously defined, the measurement of face validity is a very subjective; nonetheless, it is the only way to assess this component. In this study, we analysed the quantified
opinion of both experts and novice to reduce the subjective influence from an expert only opinion. For this particular VR simulator, the face validity was assessed in three separate domains: visual appearance (photorealism), haptic feedback and user-friendliness. All four groups agreed that the visual appearance was a realistic representation of the real endoscopic view of the sinuses with no statistically significant difference between the scoring (Fig. 7). However, the haptic feedback scoring was different between groups with the consultant and registrar groups scoring significantly lower than the student/intern and RMO groups. The lower scoring by the consultant and registrar group corresponded well with the comments made in the questionnaire, where the haptic feedback was frequently © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 577
Fig. 11. Photograph showing a researcher using the haptic devices for Task 1 of the simulator.
described as ‘clunky’. Despite the intern/student and RMO groups scoring high for this aspect of the simulator, it is evident that more improvements are required to improve face validation in this simulator. The consultant group scored significantly lower for user-friendliness of this VR simulator compared to the other groups. While the scoring for the user-friendliness was only one scale, this score may have been influenced by multiple domains that may have contributed to the final score including the clarity of the instructions provided, the ability to complete the task and familiarity of using VR simulators or previous experience of video gaming. Evidence already exists to suggest that individuals who are familiar with VR simulators and video gamers acquire virtual endoscopic skills quicker than others.19,20 The VR sinus simulator used here was modelled based on a multislice computed tomography imaging of a human with added tissue modelling with the help of experienced otorhinolaryngologists.9 Current results suggest that the technology used to create this photorealistic simulator has reached a standard of visual appearance that is accepted by both novice and expert users. Educational applicability of the study
Teaching and assessment of detailed anatomy are a core component of surgical education programmes. Traditionally, this has been via textbooks, anatomical models, cadaveric samples and lastly exposure and observation of real patients in OT or clinic environments. Here, we evaluated the educational validity of the VR sinus simulator specific to anatomy and procedural skills. The intern/student and RMO groups scored highly for the ability of this simulator to teach anatomy to their current level, whereas the registrar group scored significantly lower. However, all participants agreed the
© 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
simulator is effective for teaching anatomy for medical students but not necessarily for RMO or registrars. In Task 1 of our study described earlier, we only requested basic sinus structures to be identified to complete the task. This may have been useful for the novice participants as this was their first exposure to specific sinus anatomy in a photorealistic VR simulator, whereas more experienced participants have a level of detailed anatomy already known. This simulator is not limited to identifying basic anatomy, and with simple manipulation, any aspect of sinus anatomy can be labelled. However, it is clear that medical students and junior medical staff will benefit from having exposure to a VR sinus simulator prior to attending otorhinolaryngology clinics and OT sessions to acquire the basic anatomy for nasal endoscopy or basic ESS. The studies using the Lockheed Martin ESS simulator claimed that sinus simulators could be effectively utilised for teaching sino-nasal anatomy to many different levels of clinicians.21 They found that students who used the sinus simulator to study sino-nasal anatomy not only scored higher in testing, but they also completed the tests significantly faster that the comparison group who studied using only textbooks.21 Another study evaluating medical student attitudes towards using ESS-VR simulators for learning anatomy indicated that the simulator provided significant training benefits.22 While this is a subjective interpretation of student attitudes, coupled with the objective evidence in our study and others described above, we believe VR sinus surgery simulators have a definite role in teaching anatomy, especially at medical student level. Clinical applicability of the study
VR simulator training exercises have shown to improve operative performance in laparoscopic6 and gastrointestinal7 VR simulators. We also evaluated the use of this VR simulator for teaching procedural skills. Both the intern/ student and RMO groups agreed that this simulator is appropriate to teach nasendoscope skills at their level. However, the registrar and consultant groups disagreed by scoring significantly lower for teaching skills at their level, although the senior clinicians agreed the simulator would be useful for teaching nasendoscopy skills to students. The use of the abstract tumour-removal component of our simulator was scored lower by all groups, when compared to the nasendoscopy tasks. The participant comments described that the task was non-specific and hence difficult to interpret its usefulness for teaching procedural skills. Interestingly, Task 2 received the highest scoring for the haptic feedback and comments indicated that the simulated tissue feedback was more realistic than the nasendoscopy task.
578 N. Dharmawardana et al.
Content validity demonstrated that, for the nasendoscopy tasks (tasks 1 and 4), the intern/student and RMO groups scored significantly higher than the registrar or the consultant groups for training at their current level. However, there was no significant difference in their scoring for training at any level. We believe our findings from this study demonstrate that this simulator’s content is validated for using at medical student, intern and RMO level but requires further development before it can be recommended for teaching at registrar level. A recent Cochrane review indicates that VR simulation training improves operative performance especially of those with limited exposure to laparoscopic or endoscopic procedures.6 Our findings would support the Cochrane review as our study was found to be useful by junior medical staff with limited exposure to ESS to practice at a very basic level prior to operating on live patients. Study limitations
A limitation to this study is the relatively small sample size. While the total number of participants was adequate, once divided into corresponding groups, these numbers resulted in a low statistical power. Ongoing work involving this simulator includes a full evaluation of construct validity including assessment validity to be able to distinguish expert from novice practitioners. We continue to enhance the simulator environment with added computed tomography imaging now incorporated to the simulator. Ongoing validation studies are planned using the improved simulations and a larger sample size involving multiple centres. In time, we hope this tool will become an established training tool for medical students and OHNS trainees. Synopsis of key findings
This validation study has demonstrated that low-cost VR sinus simulation can be used for teaching anatomy and procedural skills with adequate face validity and content validity. While at present, this only validated for use with medical students and interns, with further development and establishment of construct validity, this may become a useful training tool for OHNS surgeons at all levels. Keypoints
• • •
VR simulators provide an alternative to real patients for practicing surgical skills. Validation of VR simulators is necessary to ensure accuracy. This VR simulations have been validated for teaching sinus anatomy and nasendoscopy for medical students.
Acknowledgments
We would like to acknowledge the students, medical, surgical and research staff members of the Department of Otorhinolaryngology at Flinders Medical Centre, Adelaide, Australia, for their help and participation in this study. Conflict of interest
None to declare. References 1 Wais M., Ooi E., Leung R.M. et al. (2012) The effect of low-fidelity endoscopic sinus surgery simulators on surgical skill. Int. Forum Allergy Rhinol. 2, 20–26 2 Ooi E.H. & Witterick I.J. (2010) Rhinologic surgical training. Otolaryngol. Clin. North Am. 43, 673–689, xi. 3 Reznick R.K. & MacRae H. (2006) Teaching surgical skills–changes in the wind. N. Engl. J. Med. 355, 2664–2669 4 Sturm L.P., Windsor J.A., Cosman P.H. et al. (2008) A systematic review of skills transfer after surgical simulation training. Ann. Surg. 248, 166–179 5 Scott D.J., Pugh C.M., Ritter E.M. et al. (2011) New directions in simulation-based surgical education and training: validation and transfer of surgical skills, use of nonsurgeons as faculty, use of simulation to screen and select surgery residents, and long-term follow-up of learners. Surgery 149, 735–744 6 Nagendran M., Gurusamy K.S., Aggarwal R. et al. (2013) Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst. Rev. 8, CD006575 7 Walsh C.M., Sherlock M.E., Ling S.C. et al. (2012) Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst. Rev. 6, CD008237 8 Weghorst S., Airola C., Oppenheimer P. et al. (1998) Validation of the Madigan ESS simulator. Stud. Health Technol. Inform. 50, 399–405 9 Ruthenbeck G.S., Hobson J., Carney A.S. et al. (2013) Toward photorealism in endoscopic sinus surgery simulation. Am. J. Rhinol. Allergy 27, 138–143 10 Gallagher A.G., Ritter E.M. & Satava R.M. (2003) Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg. Endosc. 17, 1525–1529 11 Sireci S. & Faulkner-Bond M. (2014) Validity evidence based on test content. Psicothema 26, 100–107 12 Hoover M.J., Jung R., Jacobs D.M. et al. (2013) Educational testing validity and reliability in pharmacy and medical education literature. Am. J. Pharm. Educ. 77, 213 13 Ruthenbeck G.S., Carney A.S. & Reynolds K.J. (2013) Detail preserving anatomical markers in a virtual reality nasendoscopy simulation in Innovative Computing Technology (INTECH), 2013 Third International 14 Ruthenbeck G.S., Lim F.S. & Reynolds K.J. (2013) Real-time interactive isosurfacing: a new method for improving marching
© 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 579
15
16
17
18
isosurfacing algorithm output and efficiency. Comput. Methods Biomech. Biomed Engin. 18(2), 213–220 Ruthenbeck G.S. & Reynolds K.J. (2014) Edge concealment in a combined surface mesh and scalar-field tissue model for surgical simulations. Simulation 90, 216–223 Fokkens W.J., Lund V.J., Mullol J. et al. (2012) EPOS 2012: European position paper on rhinosinusitis and nasal polyps 2012. A summary for otorhinolaryngologists. Rhinology 50, 1–12 Georgalas C., Cornet M., Adriaensen G. et al. (2014) Evidencebased Surgery for Chronic Rhinosinusitis with and without Nasal Polyps. Curr. Allergy Asthma Rep. 14, 427 Arora A., Lau L.Y., Awad Z. et al. (2014) Virtual reality simulation training in Otolaryngology. Int. J. Surg. 12, 87–94
© 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
19 Lynch J., Aughwane P. & Hammond T.M. (2010) Video games and surgical ability: a literature review. J. Surg. Educ. 67, 184–189 20 Adams B.J., Margaron F. & Kaplan B.J. (2012) Comparing video games and laparoscopic simulators in the development of laparoscopic skills in surgical residents. J. Surg. Educ. 69, 714–717 21 Solyar A., Cuellar H., Sadoughi B. et al. (2008) Endoscopic Sinus Surgery Simulator as a teaching tool for anatomy education. Am. J. Surg. 196, 120–124 22 Glaser A.Y., Hall C.B., Uribe S.J. et al. (2006) Medical students’ attitudes toward the use of an endoscopic sinus surgery simulator as a training tool. Am. J. Rhinol. Allergy 20, 177–179
Background: Virtual reality (VR) simulators provide an alternative to real patients for practicing surgical skills but require validation to ensure accuracy. Here, we validate the use of a virtual reality sinus surgery simulator with haptic feedback for training in Otorhinolaryngology - Head & Neck Surgery (OHNS). Methods: Participants were recruited from final-year medical students, interns, resident medical officers (RMOs), OHNS registrars and consultants. All participants completed an online questionnaire after performing four separate simulation tasks. These were then used to assess face, content and construct validity. ANOVA with post hoc correlation was used for statistical analysis. Results: The following groups were compared: (i) medical students/interns, (ii) RMOs, (iii) registrars and (iv) consultants. Face validity results had a statistically significant
Introduction
Simulation training provides an attractive method for teaching surgical skills outside the operating theatres (OT).1 Traditionally, surgical training has been an apprenticeship-based training following graduation from medical school. While this method has successfully trained our current surgical workforce to practise safely, this method inherently requires many hours of hands on experience in OT. With restrictions in junior doctor working hours and decreased exposure to OT time, it is possible that future surgeons will struggle to reach the same standards as their predecessors.2 Therefore, it is increasingly important to teach advanced surgical skills outside the OT to ensure safe surgical practice. Simulation training allows the safe teaching of surgical skills and can also be utilised for assessment Correspondence: N. Dharmawardana, Department of Otorhinolaryngology, Flinders ENT, Level 5, Flinders Medical Centre, Flinders Dr, Bedford Park SA 5042, Australia. Tel.: +61 8 8222 4000; Fax:+61 8 8204 7524; e-mail: [email protected] © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
(P < 0.05) difference between the consultant group and others, while there was no significant difference between medical student/intern and RMOs. Variability within groups was not significant. Content validity results based on consultant scoring and comments indicated that the simulations need further development in several areas to be effective for registrar-level teaching. However, students, interns and RMOs indicated that the simulations provide a useful tool for learning OHNS-related anatomy and as an introduction to ENT-specific procedures. Conclusions: The VR simulations have been validated for teaching sinus anatomy and nasendoscopy to medical students, interns and RMOs. However, they require further development before they can be regarded as a valid tool for more advanced surgical training.
purposes if appropriate validation has been performed. The Fitts–Posner learning theory describes cognition, integration and automation as the three stages of motor skill acquisition.3 The individual needs to understand the task at hand before combining this understanding with the required mechanics in order to perform the task efficiently. VR simulators provide access to all stages of this theory without patient compromise by allowing individuals to attempt the procedure with or without errors. There are many different types of surgical simulators including animal models, plastic mannequins, artificial theatre environments and computer-generated virtual reality (VR) simulators.4,5 Multiple Cochrane reviews have been published showing the benefits of using VR simulators for surgical training.6,7 VR simulation uses computer software to generate a virtual model of a specific training environment. With the evolution of simulation technology, it has become easier to construct realistic virtual models with realistic haptic feedback. The most well-validated haptic sinus simulator was constructed by Lockheed Martin Corporation based on flight simulation technology.8 How569
ORIGINAL ARTICLE
Validation of virtual-reality-based simulations for endoscopic sinus surgery
570 N. Dharmawardana et al.
ever, the prohibitive cost of this simulator prevented widespread uptake of the device. Our sinus simulator was developed by the Medical Device Research Institute at the Flinders University, Adelaide, Australia, in cooperation with the Department of OHNS (Flinders Medical Centre, Adelaide, Australia). It provides a laptop-based photorealistic VR sinus simulator with haptic tissue feedback.9 Validity is an advanced concept defined as the property of being true and correct, and is in keeping with reality.10 There are multiple facets to validation that includes face validity and content validity among others. Face validity is defined as the assessment by experts of contents to see whether they are appropriate.10 While content validity more rigorously examines the content of the simulator by subject matter experts. Educational validity is a component of content validity that specifically assesses the suitability to test or teach.12 In this validation study, we describe the face, educational and content validity of our sinus simulation. Methods Simulator components
The simulation software for this study was run on a 17-inch laptop and controlled by the user via two different haptic devices (Fig 11). The first haptic device ‘Novint Falcon’ (Novint Technologies, Inc. Delaware, USA) only senses three-dimensional position and provides linear haptic feedback. The second haptic device ‘Phantom Omni’ since renamed ‘Geomagic Touch’ (Three D Systems, SC, USA) senses both three-dimensional position and orientation, again with linear haptic feedback. The laptop was set up on a desk, elevated to eye level, and haptic devices were located at
comfortable distance apart from each other and lower than the laptop screen for the simulation of the real intraoperative position of the monitor and surgical instruments. The Novint Falcon was used to control the endoscope, while the Phantom Omni manipulated the surgical instruments. When a task did not require a second tool, the second haptic device was disabled. The total cost of the components including the laptop computer was approximately $15 000 Australian dollars. Surgical tasks
Four surgical tasks were performed in order (Fig. 1). In Task 1, the participants had to manoeuvre a simulated rigid nasendoscope through the right-sided nasal cavity and identify six important anatomical landmarks considered to be important for accurate nasendoscopy examination of the nasal cavity and lateral nasal wall. These structures were inferior turbinate, middle turbinate, uncinate process, bulla, superior turbinate and the sphenoid ostium (Fig. 2). Once the name of the anatomical landmark appeared on the screen, the corresponding structure was indicated with a yellow marker.13 The participant had to find the landmark, and once within 5 mm of the marker, press a button on the haptic device to confirm that they had found it. As soon as a landmark was identified and the button pressed, the marker was hidden and the name of the next landmark displayed. At the completion of this task, the participants were directed to an online questionnaire to provide feedback and comments specific to the task described above. Task 2 was a simulated tissue removal task. The task required the user to remove a discoloured region of tissue from a larger tissue model. The tissue model (described by Ruthenbeck et al.)14 was hemispherical in shape and
Fig. 1. Four separate simulation tasks and the haptic devices used.
Fig. 2. Images showing each of the simulated anatomical landmarks the participants were instructed to find during Task 1 of the VR simulation. © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 571
deliberately abstract (unlike any real organ or anatomical region). The haptic device was used to control a stylus (depicted as a small white sphere) that could either palpate or volumetrically remove tissue when the haptic device button was pressed. Participants were instructed to remove a section of a hemispherical mass that was different in colour compared to the surrounding (Fig. 3). By pressing a button on the wand of the haptic device, the participant activated the cutting properties of the sphere and was also able to feel the vibration when activated. Detailed recordings of the time taken to complete the task, accuracy of virtual tissue removal, haptic force and position in virtual space were recorded at 1 kHz. The discoloured section had other physical properties such as decreased density and stiffness that were able to be felt through the haptic device. The endoscopic view was static for this task; hence, only one haptic device was required. The participants were then directed to complete an online questionnaire at the end of the simulation specific to this task.
(a)
Task 3 utilised a Phantom Omni controlled backbiter and microdebrider tool (switchable between tools), where participants were instructed to make two cuts using the backbiter tool on a vertical plate of virtual tissue showing where to cut with clear markings (Fig. 4 ). Once the cuts were made, they were instructed to switch the tool to a simulated microdebrider using a key on the laptop keyboard and remove the virtual tissue between the two cuts made earlier. Task 4 utilised the same simulator as Task 1 with the addition of an area of ‘abnormal simulated tissue’ to the middle turbinate. Participants were instructed to remove this ‘abnormal tissue’ using a simulated microdebrider (Fig. 5). The interactive tissue model is described by Ruthenbeck et al.15 Study groups
Twenty-four participants participated in this study: 5 final-year medical students, 2 interns, 7 resident medical
(b)
Fig. 3. Image A shows a screenshot as soon as the VR simulator for Task 2 was activated where the abstract simulated tissue mass has a distinctly differently coloured area to mark abnormal tissue. Image B shows the simulated cutting ball that can be manoeuvred using the haptic device to cut into the simulated tissue with haptic feedback (a)
(b)
(c)
Fig. 4. Image A shows the start-up screen for Task 3 with a sheet of simulated flat tissue with markings on the left and a simulated backbiter tool on the right. Image B shows the simulated backbiter in use. Image C shows the use of the simulated microdebrider to remove the discoloured middle turbinate tissue © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
572 N. Dharmawardana et al.
Each participant was provided with a printed instruction manual to complete each of the tasks described above using the VR simulator. Following each task, they were instructed to complete an online questionnaire (Fig. 6) where each question was rated on a 5-point scale, 1 (poor) and 5 (excellent) (Fig. 6). In the questionnaire, they were also provided with a free text field for any further comments. Questions were focussed to assess face validity (visual and haptic realism, user-friendliness), educational validity (potential as a teaching tool) and content validity. Data analysis Fig. 5. This image shows a screenshot from Task 4 where the participant was instructed to remove a portion of the middle turbinate with abnormal colouring compared to the rest of the simulated tissue using the simulated microdebrider.
officers (RMOs), 6 registrars and 4 consultants. Medical students were in the final year of a 4-year postgraduate medical degree, interns were defined as the postgraduate year 1 (PGY 1), RMOs were defined as a practicing clinician with general registration in Australia without being in a specific vocational training program, registrars were defined as a practicing clinician in OHNS specialist training through the Royal Australasian College of Surgeons (RACS), and consultants were defined as qualified OHNS specialists, awarded the Fellowship of the RACS.
Data analysis was performed using an SPSS statistical package using one-way analysis of variance with post hoc analysis using Dunnett’s T3 test assuming no equal size or variance between groups. A higher questionnaire score indicated a higher satisfactory rating of the utility of performing specific tasks on the simulator. Statistically significant results are reported as P = value, and non-significant results reported as P = ns. Alpha-value of 0.05 was utilised to report significance level. Ethical approval
This project was reviewed and approved by the Southern Adelaide Clinical Human Research Ethics Committee (SAC HREC) on 10 December 2012 in accordance with the National Statement of Ethical Conduct in Human Research (2007).
Fig. 6. Questions asked at the end of each task described above. Participants selected a score from 1 to 5, where 1 = poor and 5 = excellent or 1 = never and 5 = very likely. © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 573
Results Face validity
There were no statistically significant (P = ns) differences between the groups for the visual appearance between four tasks (Fig. 7). There were statistically significant (P = 0.00001) differences between the groups for haptic feedback and user-friendliness for Task 1 (Fig. 1A); however, there were no statistically significant (P = ns) differences between the 4 groups for tasks 2 to 4 for haptic feedback experience and user-friendliness.
Post hoc analysis of Task 1 for haptic feedback experience demonstrated that the intern/student group scored significantly higher compared to the registrars (P = 0.0002) and consultants (P = 0.005). The RMO group also scored significantly higher compared to the registrars (P = 0.0065). However, there was no significant difference (P = ns) between intern/student and RMOs or registrars and consultants. Post hoc analysis for userfriendliness indicated the intern/student group (P = 0.024) scored significantly higher compared to the consultants. There were no significant differences between the other groups for user-friendliness.
(a)
(b)
(c)
(d)
Fig. 7. Face validity: Charts A, B, C and D represent the data for tasks 1 to 4, respectively. Each figure indicates the mean score with associated standard error of mean (SEM) for visual appearance, haptic feedback and user-friendliness of the simulator and the instructions provided. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬). © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
574 N. Dharmawardana et al.
Educational validity
Teaching anatomy. There was a statistically significant difference (P = 0.029) between the intern/student group and the registrars for Task 1 where the registrars rated lower the suitability of the simulator for teaching anatomy for residents and registrars. There were no significant differences between the other groups for the suitability of the simulator for teaching anatomy for students or residents and registrars in tasks 1 and 4 (Fig. 8). Task 1 received higher scores with lower variance than Task 4 for teaching anatomy for both groups described above. Teaching procedural skills. Medical students and interns scored significantly higher than consultants for teaching endoscopy skills to interns/students (P = 0.042) and RMOs/ registrars (P = 0.026). There were no significant differences in scores between other groups (Fig. 9). Content validity
The intern/student group rated highly the suitability of the simulator for their current level of training for all tasks. However, in Task 1, this was significantly different to the RMOs (P = 0.023), registrars (P = 0.0002) and consultants (P = 0.004). For the same question in Task 1, the RMO group also scored significantly higher compared to the registrars (P = 0.0001) and consultants (P = 0.006) (Fig. 10A). For tasks 2 and 4, the rating was significantly different between the intern/student group and the
(a)
consultant group (P = 0.027) for the suitability of the simulator for their current level, but there was no statistically significant difference between other groups for the same question. For the same question in Task 3, the registrar group scored significantly lower when compared to RMO (P = 0.023) and the intern/student groups (P = 0.009). There was no statistically significant difference in scoring for the suitability of the simulator for any level of training. However, the registrar and consultant groups consistently scored lower compared the intern/ student and RMO groups (Fig. 10). The online questionnaire also allowed participants to provide further comments on a free text field that provided qualitative information. The consultant and registrar group consistently commented that the simulated nasendoscope in tasks 1 and 4 was ‘too easy’ to move through the simulated tissue structures and the haptic feedback was not realistic enough compared to a real rigid nasendoscopy. Multiple junior participants identified that a pre-task tutorial on sinus anatomy would have been useful to complete the requested tasks as they did not have specialist knowledge and had not used a rigid nasendoscope before. Task 2 received interesting comments where most participants identified the haptic feedback was much closer to real tissue feedback and that the three-dimensional effect was portrayed better. However, the consultant group commented on the abstract nature of this simulator and the limitation for surgical training. Task 3 received poor feedback from most participants ranging from the lack of haptic
(b)
Fig. 8. Charts A and B represent tasks 1 and 4, respectively. Each bar indicates mean score SEM for the suitability of teaching anatomy to medical students or residents and registrars. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬). © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 575
(a)
(b)
(c)
(d)
Fig. 9. Charts A, B, C and D represent data from tasks 1 to 4, respectively. Where Task 1 specifically addresses suitability to teach nasendoscopy skills, other three tasks questions the suitability of general procedural skills in otorhinolaryngology. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬).
feedback and the unrealistic tissue modelling with poor depth perception without the surrounding simulated tissues. Regarding the instructions, comments reflect minimal confusion for tasks 1, 2 and 4. However, the instructions for Task 3 were found to be confusing as per the comments. There were also comments with regard to having a facilitator to help through the tasks. The final comments were regarding the haptic devices, their placement and body positioning based on the equipment. Consultant, registrar and RMO group commented on difficulty in using the simulated instruments as it was difficult to rest their elbow and the feeling of hovering the instruments in the air made it more tiring and difficult to accurately complete the tasks. © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Discussion
Endoscopic sinus surgery (ESS) is a mainstay treatment for multiple rhinology conditions including chronic rhinosinusitis.16 However, there is an associated long-term revision rate of up to 20% and complication rates that vary from up to 10% reported 20 years ago to more recent reports of complication rates up to 5%.17 Nonetheless, 20 years after the introduction of this method of surgery, minor and major disabling complications continue to occur. The ability to prevent these complications relies mainly on knowledge and surgical experience. Hence, it is necessary to provide further training in ESS for otorhinolaryngology trainees. An advanced VR sinus simulator provides a safe environment
576 N. Dharmawardana et al.
(a)
(b)
(c)
(d)
Fig. 10. Charts A, B, C and D represent data from tasks 1 to 4, respectively. Each chart represents two data series comparing the scores for the suitability of the simulator for training at any level from student to consultant versus the suitability for training at the level of the participant at present. Statistically significant (P < 0.05) pairwise comparison results are indicated with horizontal bars (¬).
where mistakes can be made without risks or harm to patients before a trainee operates on live patients. Comparison with other studies
There are many VR simulators described in literature for use in otorhinolaryngology training with at least 12 temporal bone surgery simulators, 6 ESS simulators and 3 myringotomy simulators previously reported by Arora et al. 2014, a review of current otorhinolaryngology simulators.18 Of these, only a few have had full validation studies conducted. As previously defined, the measurement of face validity is a very subjective; nonetheless, it is the only way to assess this component. In this study, we analysed the quantified
opinion of both experts and novice to reduce the subjective influence from an expert only opinion. For this particular VR simulator, the face validity was assessed in three separate domains: visual appearance (photorealism), haptic feedback and user-friendliness. All four groups agreed that the visual appearance was a realistic representation of the real endoscopic view of the sinuses with no statistically significant difference between the scoring (Fig. 7). However, the haptic feedback scoring was different between groups with the consultant and registrar groups scoring significantly lower than the student/intern and RMO groups. The lower scoring by the consultant and registrar group corresponded well with the comments made in the questionnaire, where the haptic feedback was frequently © 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 577
Fig. 11. Photograph showing a researcher using the haptic devices for Task 1 of the simulator.
described as ‘clunky’. Despite the intern/student and RMO groups scoring high for this aspect of the simulator, it is evident that more improvements are required to improve face validation in this simulator. The consultant group scored significantly lower for user-friendliness of this VR simulator compared to the other groups. While the scoring for the user-friendliness was only one scale, this score may have been influenced by multiple domains that may have contributed to the final score including the clarity of the instructions provided, the ability to complete the task and familiarity of using VR simulators or previous experience of video gaming. Evidence already exists to suggest that individuals who are familiar with VR simulators and video gamers acquire virtual endoscopic skills quicker than others.19,20 The VR sinus simulator used here was modelled based on a multislice computed tomography imaging of a human with added tissue modelling with the help of experienced otorhinolaryngologists.9 Current results suggest that the technology used to create this photorealistic simulator has reached a standard of visual appearance that is accepted by both novice and expert users. Educational applicability of the study
Teaching and assessment of detailed anatomy are a core component of surgical education programmes. Traditionally, this has been via textbooks, anatomical models, cadaveric samples and lastly exposure and observation of real patients in OT or clinic environments. Here, we evaluated the educational validity of the VR sinus simulator specific to anatomy and procedural skills. The intern/student and RMO groups scored highly for the ability of this simulator to teach anatomy to their current level, whereas the registrar group scored significantly lower. However, all participants agreed the
© 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
simulator is effective for teaching anatomy for medical students but not necessarily for RMO or registrars. In Task 1 of our study described earlier, we only requested basic sinus structures to be identified to complete the task. This may have been useful for the novice participants as this was their first exposure to specific sinus anatomy in a photorealistic VR simulator, whereas more experienced participants have a level of detailed anatomy already known. This simulator is not limited to identifying basic anatomy, and with simple manipulation, any aspect of sinus anatomy can be labelled. However, it is clear that medical students and junior medical staff will benefit from having exposure to a VR sinus simulator prior to attending otorhinolaryngology clinics and OT sessions to acquire the basic anatomy for nasal endoscopy or basic ESS. The studies using the Lockheed Martin ESS simulator claimed that sinus simulators could be effectively utilised for teaching sino-nasal anatomy to many different levels of clinicians.21 They found that students who used the sinus simulator to study sino-nasal anatomy not only scored higher in testing, but they also completed the tests significantly faster that the comparison group who studied using only textbooks.21 Another study evaluating medical student attitudes towards using ESS-VR simulators for learning anatomy indicated that the simulator provided significant training benefits.22 While this is a subjective interpretation of student attitudes, coupled with the objective evidence in our study and others described above, we believe VR sinus surgery simulators have a definite role in teaching anatomy, especially at medical student level. Clinical applicability of the study
VR simulator training exercises have shown to improve operative performance in laparoscopic6 and gastrointestinal7 VR simulators. We also evaluated the use of this VR simulator for teaching procedural skills. Both the intern/ student and RMO groups agreed that this simulator is appropriate to teach nasendoscope skills at their level. However, the registrar and consultant groups disagreed by scoring significantly lower for teaching skills at their level, although the senior clinicians agreed the simulator would be useful for teaching nasendoscopy skills to students. The use of the abstract tumour-removal component of our simulator was scored lower by all groups, when compared to the nasendoscopy tasks. The participant comments described that the task was non-specific and hence difficult to interpret its usefulness for teaching procedural skills. Interestingly, Task 2 received the highest scoring for the haptic feedback and comments indicated that the simulated tissue feedback was more realistic than the nasendoscopy task.
578 N. Dharmawardana et al.
Content validity demonstrated that, for the nasendoscopy tasks (tasks 1 and 4), the intern/student and RMO groups scored significantly higher than the registrar or the consultant groups for training at their current level. However, there was no significant difference in their scoring for training at any level. We believe our findings from this study demonstrate that this simulator’s content is validated for using at medical student, intern and RMO level but requires further development before it can be recommended for teaching at registrar level. A recent Cochrane review indicates that VR simulation training improves operative performance especially of those with limited exposure to laparoscopic or endoscopic procedures.6 Our findings would support the Cochrane review as our study was found to be useful by junior medical staff with limited exposure to ESS to practice at a very basic level prior to operating on live patients. Study limitations
A limitation to this study is the relatively small sample size. While the total number of participants was adequate, once divided into corresponding groups, these numbers resulted in a low statistical power. Ongoing work involving this simulator includes a full evaluation of construct validity including assessment validity to be able to distinguish expert from novice practitioners. We continue to enhance the simulator environment with added computed tomography imaging now incorporated to the simulator. Ongoing validation studies are planned using the improved simulations and a larger sample size involving multiple centres. In time, we hope this tool will become an established training tool for medical students and OHNS trainees. Synopsis of key findings
This validation study has demonstrated that low-cost VR sinus simulation can be used for teaching anatomy and procedural skills with adequate face validity and content validity. While at present, this only validated for use with medical students and interns, with further development and establishment of construct validity, this may become a useful training tool for OHNS surgeons at all levels. Keypoints
• • •
VR simulators provide an alternative to real patients for practicing surgical skills. Validation of VR simulators is necessary to ensure accuracy. This VR simulations have been validated for teaching sinus anatomy and nasendoscopy for medical students.
Acknowledgments
We would like to acknowledge the students, medical, surgical and research staff members of the Department of Otorhinolaryngology at Flinders Medical Centre, Adelaide, Australia, for their help and participation in this study. Conflict of interest
None to declare. References 1 Wais M., Ooi E., Leung R.M. et al. (2012) The effect of low-fidelity endoscopic sinus surgery simulators on surgical skill. Int. Forum Allergy Rhinol. 2, 20–26 2 Ooi E.H. & Witterick I.J. (2010) Rhinologic surgical training. Otolaryngol. Clin. North Am. 43, 673–689, xi. 3 Reznick R.K. & MacRae H. (2006) Teaching surgical skills–changes in the wind. N. Engl. J. Med. 355, 2664–2669 4 Sturm L.P., Windsor J.A., Cosman P.H. et al. (2008) A systematic review of skills transfer after surgical simulation training. Ann. Surg. 248, 166–179 5 Scott D.J., Pugh C.M., Ritter E.M. et al. (2011) New directions in simulation-based surgical education and training: validation and transfer of surgical skills, use of nonsurgeons as faculty, use of simulation to screen and select surgery residents, and long-term follow-up of learners. Surgery 149, 735–744 6 Nagendran M., Gurusamy K.S., Aggarwal R. et al. (2013) Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst. Rev. 8, CD006575 7 Walsh C.M., Sherlock M.E., Ling S.C. et al. (2012) Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst. Rev. 6, CD008237 8 Weghorst S., Airola C., Oppenheimer P. et al. (1998) Validation of the Madigan ESS simulator. Stud. Health Technol. Inform. 50, 399–405 9 Ruthenbeck G.S., Hobson J., Carney A.S. et al. (2013) Toward photorealism in endoscopic sinus surgery simulation. Am. J. Rhinol. Allergy 27, 138–143 10 Gallagher A.G., Ritter E.M. & Satava R.M. (2003) Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg. Endosc. 17, 1525–1529 11 Sireci S. & Faulkner-Bond M. (2014) Validity evidence based on test content. Psicothema 26, 100–107 12 Hoover M.J., Jung R., Jacobs D.M. et al. (2013) Educational testing validity and reliability in pharmacy and medical education literature. Am. J. Pharm. Educ. 77, 213 13 Ruthenbeck G.S., Carney A.S. & Reynolds K.J. (2013) Detail preserving anatomical markers in a virtual reality nasendoscopy simulation in Innovative Computing Technology (INTECH), 2013 Third International 14 Ruthenbeck G.S., Lim F.S. & Reynolds K.J. (2013) Real-time interactive isosurfacing: a new method for improving marching
© 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
Validation of VR simulations for ESS 579
15
16
17
18
isosurfacing algorithm output and efficiency. Comput. Methods Biomech. Biomed Engin. 18(2), 213–220 Ruthenbeck G.S. & Reynolds K.J. (2014) Edge concealment in a combined surface mesh and scalar-field tissue model for surgical simulations. Simulation 90, 216–223 Fokkens W.J., Lund V.J., Mullol J. et al. (2012) EPOS 2012: European position paper on rhinosinusitis and nasal polyps 2012. A summary for otorhinolaryngologists. Rhinology 50, 1–12 Georgalas C., Cornet M., Adriaensen G. et al. (2014) Evidencebased Surgery for Chronic Rhinosinusitis with and without Nasal Polyps. Curr. Allergy Asthma Rep. 14, 427 Arora A., Lau L.Y., Awad Z. et al. (2014) Virtual reality simulation training in Otolaryngology. Int. J. Surg. 12, 87–94
© 2015 John Wiley & Sons Ltd Clinical Otolaryngology 40, 569–579
19 Lynch J., Aughwane P. & Hammond T.M. (2010) Video games and surgical ability: a literature review. J. Surg. Educ. 67, 184–189 20 Adams B.J., Margaron F. & Kaplan B.J. (2012) Comparing video games and laparoscopic simulators in the development of laparoscopic skills in surgical residents. J. Surg. Educ. 69, 714–717 21 Solyar A., Cuellar H., Sadoughi B. et al. (2008) Endoscopic Sinus Surgery Simulator as a teaching tool for anatomy education. Am. J. Surg. 196, 120–124 22 Glaser A.Y., Hall C.B., Uribe S.J. et al. (2006) Medical students’ attitudes toward the use of an endoscopic sinus surgery simulator as a training tool. Am. J. Rhinol. Allergy 20, 177–179
