e57(1) C OPYRIGHT Ó 2014
BY
T HE J OURNAL
OF
B ONE
AND J OINT
S URGERY, I NCORPORATED
Topics in Training Evaluation of Skill Level Between Trainees and Community Orthopaedic Surgeons Using a Virtual Reality Arthroscopic Knee Simulator W. Dilworth Cannon, MD, Gregg T. Nicandri, MD, Karl Reinig, PhD, Howard Mevis, MA, and Jocelyn Wittstein, MD Investigation performed at the University of California San Francisco, San Francisco, California, and Duke University, Durham, North Carolina
Background: Several virtual reality simulators have been developed to assist orthopaedic surgeons in acquiring the skills necessary to perform arthroscopic surgery. The purpose of this study was to assess the construct validity of the ArthroSim virtual reality arthroscopy simulator by evaluating whether skills acquired through increased experience in the operating room lead to improved performance on the simulator. Methods: Using the simulator, six postgraduate year-1 orthopaedic residents were compared with six postgraduate year-5 residents and with six community-based orthopaedic surgeons when performing diagnostic arthroscopy. The time to perform the procedure was recorded. To ensure that subjects did not sacrifice the quality of the procedure to complete the task in a shorter time, the simulator was programmed to provide a completeness score that indicated whether the surgeon accurately performed all of the steps of diagnostic arthroscopy in the correct sequence. Results: The mean time to perform the procedure by each group was 610 seconds for community-based orthopaedic surgeons, 745 seconds for postgraduate year-5 residents, and 1028 seconds for postgraduate year-1 residents. Both the postgraduate year-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time (p = 0.006) than the postgraduate year-1 residents. There was a trend toward significance (p = 0.055) in time to complete the procedure when the postgraduate year-5 residents were compared with the community-based orthopaedic surgeons. The mean level of completeness as assigned by the simulator for each group was 85% for the community-based orthopaedic surgeons, 79% for the postgraduate year-5 residents, and 71% for the postgraduate year-1 residents. As expected, these differences were not significant, indicating that the three groups had achieved an acceptable level of consistency in their performance of the procedure. Conclusions: Higher levels of surgeon experience resulted in improved efficiency when performing diagnostic knee arthroscopy on the simulator. Further validation studies utilizing the simulator are currently under way and the additional simulated tasks of arthroscopic meniscectomy, meniscal repair, microfracture, and loose body removal are being developed.
Peer Review: This article was reviewed by the Editor-in-Chief and one Deputy Editor, and it underwent blinded review by two or more outside experts. The Deputy Editor reviewed each revision of the article, and it underwent a final review by the Editor-in-Chief prior to publication. Final corrections and clarifications occurred during one or more exchanges between the author(s) and copyeditors.
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. No author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
J Bone Joint Surg Am. 2014;96:e57(1-7)
d
http://dx.doi.org/10.2106/JBJS.M.00779
e57(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
Surgical skills education has traditionally followed the Halstedian model in which residents are given increasing autonomy while performing live surgical cases at the discretion of a supervising surgeon educator1,2. Due to a number of factors, including an increased public awareness of medical errors, duty-hour limitations for residents, and an increasing emphasis on the efficient use of operating room time, there has been a paradigm shift in orthopaedic education. There is now an increased utilization of skills training modalities outside of the operating room that enable trainees to practice while making and learning from mistakes in an environment where patient harm cannot occur3-10. As part of the next accreditation system, the Accreditation Council for Graduate Medical Education (ACGME) has added a requirement that residency programs include a surgical skills training curriculum. As a result, it is likely that simulators will play an increasingly important role in the future of surgical education11. There are several types of simulators, categorized as lowfidelity and high-fidelity. Low-fidelity simulators (box trainers and dry models such as the Large Arthroscopy Knee Model or Alex Shoulder [Sawbones, Vashon, Washington]) teach basic surgical skills (knot tying, bimanual dexterity, triangulation skills, and grasping) but do not necessarily recreate a realistic environment. It is currently unknown how effectively the skills acquired utilizing this type of simulation training transfer to the operating room. High-fidelity simulators are those that are meant to recreate a more realistic environment for simulated surgery and would include cadaveric simulation, live animal model simulation, and virtual reality simulation. Cadavers are currently the gold standard for simulation training12,13. How-
ever, they have several drawbacks, including the risk of disease transmission and the lack of uniformity among specimens, and their use requires considerable preparation time as well as appropriate storage and disposal protocols. In addition, the cost and supply of specimens may limit their availability to trainees. Virtual reality simulators have been developed to allow surgeons the opportunity for deliberate practice in a high-fidelity simulated environment12-18. Skills acquired using a truly highfidelity simulator should translate to improved performance in the operating room and vice versa. It is currently unknown whether skills acquired through live surgical experience result in improved performance on the ArthroSim virtual reality arthroscopy simulator (Touch of Life Technologies, Aurora, Colorado) used in this study. In this study, we evaluated the construct validity of the ArthroSim virtual reality arthroscopy simulator as a high-fidelity simulation of diagnostic knee arthroscopy, and we hypothesized that surgeons with more arthroscopic experience would perform a complete diagnostic knee arthroscopy (as measured by the completeness score) on the simulator in a shorter amount of time than surgeons with less experience.
d
d
d
Materials and Methods The ArthroSim virtual reality arthroscopy simulator (Fig. 1) was used for this study. The virtual reality knee joint was based on segmentation of 0.1-mm 14 cryosections of a right knee from a twenty-eight-year-old cadaver . Both the simulated arthroscope and the probe were provided with haptic feedback devices allowing six degrees of motion and three degrees of force (3-D Systems, Rock Hill, South Carolina). The surrogate right leg had knee motion from 0° to 90° of flexion and measured varus and valgus forces, which were transformed into medial and lateral compartment opening.
Fig. 1
A photograph showing the ArthroSim virtual reality arthroscopy simulator.
e57(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014 d
d
d
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
TABLE I Objects Visualized and Probed During the Knee Diagnostic Arthroscopy Objects Visualized
Objects Probed
Suprapatellar pouch
Medial meniscus
Patellofemoral joint
Medial femoral condyle articular cartilage
Medial recess
Medial tibial plateau articular cartilage
Medial femoral condyle articular cartilage
Posterior cruciate ligament or anterior cruciate ligament
Medial meniscus
Lateral meniscus
Posterior cruciate ligament or anterior cruciate ligament
Lateral femoral condyle articular cartilage
Lateral femoral condyle articular cartilage
Posteromedial compartment (posterior horn medial meniscus)
Lateral meniscus
Posterior lateral compartment (posterior horn lateral meniscus)
Lateral gutter
Prior to conducting the study, institutional review board approval was acquired from the review boards at both the University of California San Francisco Medical Center and the Duke University Medical Center. A letter of invitation was sent to the six postgraduate year (PGY)-1 residents and six PGY-5 residents in the Department of Orthopedic Surgery at the University of California San Francisco. The first four residents who responded were selected to participate in the study. In addition, a letter of invitation was sent to each of the eight PGY-1 residents and eight PGY-5 residents in the Department of Orthopedic Surgery at Duke University. The first two residents who responded were selected to participate. E-mailed invitations were sent to communitybased orthopaedic surgeons from the local San Francisco area. The first four surgeons who responded positively were selected to participate. E-mailed invitations were also sent to community-based orthopaedic surgeons from Durham, North Carolina, and the first two surgeons who responded were selected to participate. In total, there were six subjects per group. To qualify for participation, the community-based orthopaedic surgeons had to be performing between fifty and 125 knee arthroscopy cases per year. Each subject signed a consent form, filled out a demographic sheet, provided answers to a questionnaire, and then watched a fifteen-minute video of a diagnostic arthroscopy of the right knee instructing them in the technique and sequence that they were expected to follow in the arthroscopic procedure. None of the subjects enrolled in this study had had any prior experience using an arthroscopic virtual reality simulator. The subjects were allowed approximately two minutes to handle the 30° arthroscope and to probe inside the virtual right knee joint to familiarize themselves with the setup and the feel of the instruments. Each subject then performed three trials visualizing and then probing the structures listed in Table I. Each subject was given hints throughout the procedure by a supervising orthopaedic sports medicine subspecialist. The hints were scripted and were the same for all groups regardless of experience. We believed that this was necessary with time being our primary outcome variable. Without hints, it would be possible for subjects to omit certain steps of the procedure. We wanted to ensure that those completing the procedure in a shorter time were doing so as a result of being more efficient and not simply because they were skipping steps. For example, hints for the step of probing the posterior horn of the medial meniscus of the right knee for trial 1 were as follows: probe the superior and inferior surfaces of the posterior horn with the knee flexed 30°, with a medial compartment opening (produced by a valgus force) of ‡3.5°, and aim the arthroscope at 9 o’clock. For trial 2, the recommended knee flexion angle and amount of valgus force were omitted from the verbal cues. For trial 3, cues were given only if the subject failed to progress and was stuck at a step in the procedure. A stopwatch was used to record the time for each step of the procedure for all three trials. The completeness score incrementally increased as the subject progressed through the steps of the diagnostic arthroscopy. A score of 100% would
indicate that the subject had the proper arthroscope position, probing technique, knee flexion angle, and correct valgus or varus force on the knee joint during the procedure as determined by a range of acceptable values programmed into the simulator. The completeness scores from all three trials for both the visualization and probing parts of the procedure were recorded.
Statistical Analysis All statistical tests were conducted with use of SPSS version 15.0 (SPSS, Chicago, Illinois). A one-way analysis of variance (ANOVA) was used to determine the equality of means and pairwise comparisons were made using two a priori orthogonal contrasts, one that compared the PGY-1 residents with the combined group of PGY-5 residents and community-based orthopaedic surgeons and the other that compared the PGY-5 residents with the community-based orthopaedic surgeons. Significance was set at p < 0.05 for both the primary outcome of time and the secondary outcome of completeness. Although each subject completed three trials, only the time and completeness scores from trial 3 were used in the statistical analysis. The first two trials were considered practice. Descriptive statistics were used for the outcomes of time, completeness, and demographic variables.
Source of Funding No external funding source was employed in this investigation.
Results For trial 3, the mean time to perform the procedure by each group was 610 seconds for the community-based orthopaedic surgeons, 745 seconds for the PGY-5 residents, and 1028 seconds for the PGY-1 residents (Table II). The overall ANOVA showed a significant mean difference for all three groups (p = 0.009). The Levene test for homogeneity of variance was not significant, but considering the small sample size and large differences in variance across the groups, tests that did not assume equal variances were performed. The combined PGY-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-1 residents (p = 0.006), and there was a trend (p = 0.055) toward decreased time to completion when the PGY-5 residents were compared with the community-based orthopaedic surgeons. A post hoc power analysis was performed using the data for trial 3. With n = 6, a = 0.05, and a difference in means of 2.4 standard deviations (the Cohen D effect size),
e57(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014 d
d
d
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
TABLE II Total Procedural Time in Trial 3 Group
No. of Participants
Total Procedural Time* 1028.0 ± 228.7
PGY-1 residents
6
PGY-5 residents
6
745.3 ± 155.7
Community-based orthopaedic surgeons
6
609.8 ± 103.7
*The values are given as the mean and the standard deviation in seconds.
the study had 0.95 power to detect a difference in time scores among the three groups. Probing structures in the knee joint took almost twice as much time as visualization of these structures (Fig. 2). Therefore, the times to complete the visualization and probing tasks were separately analyzed. For the visualization task, both the PGY-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-1 residents (p = 0.012). There was no significant difference in time (p = 0.3) for visualization when the PGY-5 residents and communitybased orthopaedic surgeons were compared. For the probing task, both the PGY-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-1 residents (p = 0.015) and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-5 residents (p = 0.046). In Figure 3, the mean times for the three groups performing trials 1 through 3 are shown. The time to complete the procedure decreased corresponding to the level of previous surgical experi-
Fig. 2
ence of the three groups. A training effect was also demonstrated as all three groups’ times decreased as they progressed from trial 1 to trial 3. This effect was the largest for the PGY-1 group. The answers to the questionnaire regarding previous arthroscopic experience were averaged for each group and are presented in Table III. The six community-based orthopaedic surgeons averaged 7.7 years (one, one, two, two, ten, and thirty years) since the completion of their residency. They averaged ninety-five knee arthroscopy cases per year. The PGY-5 residents had manipulated an arthroscope an average of eighty-eight times during their residency in contrast to an average of one time for the PGY-1 residents. Four of the six community-based orthopaedic surgeons had completed a sports medicine fellowship. The mean level of completeness as assigned by the simulator for each group was 85% for the community-based orthopaedic surgeons, 79% for the PGY-5 residents, and 71% for the PGY-1 residents. These differences were not significant, indicating that the three groups had achieved an acceptable level of consistency in their performance of the procedure.
Fig. 3
Fig. 2 A line graph showing the breakdown of trial 3 into the time for visualization and then probing of the knee for the three groups of subjects. CBO = community-based orthopaedic surgeon. Fig. 3 A line graph showing the mean times for the three groups for trials 1 through 3, demonstrating the differences between the three groups as well as a learning effect. CBO = community-based orthopaedic surgeon.
e57(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014 d
d
d
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
TABLE III Past Experience Among the Three Groups in This Study PGY-1 Residents
PGY-5 Residents
Average no. of arthroscopy cases seen
6
101
95*
Average no. of arthroscopy cases in which the subject held the scope
2
88
95*
Average no. of arthroscopy cases in which the subject manipulated the scope
1
88
95*
Average no. of hours per week in which the subject plays video games
0
0
1
Average no. of hours per week in which the subject played video games
1
1
2
Average no. of arthroscopy lectures received
1
10
4
Average no. of arthroscopy videos viewed
1
2
2
Average no. of arthroscopy courses attended
0
0
1
Past Experience
Community-Based Orthopaedic Surgeons
*This experience occurred within the last year before the start of the study.
Discussion Surgical simulation is being increasingly utilized in orthopaedic technical skills training, and the ArthroSim virtual reality arthroscopy simulator is one such model that has been developed for this purpose4,11,14,15. If a simulator is a valid high-fidelity recreation of the live surgical environment, skills acquired through operative experience should transfer to a better performance on the simulator. One component of surgical technical skill is efficiency, and in this investigation, we sought to determine whether individuals with increased arthroscopic experience could perform the task of diagnostic arthroscopy on the simulator in a shorter time when compared with those with less experience. Our study was limited by several factors. First, efficiency (as measured by the time to complete the task) was used as the primary outcome measure of technical skill. Time has also been utilized as an outcome by other similar studies12,18-21. With time as the primary outcome variable, it was imperative to ensure that one of the groups simply did not sacrifice quality for a faster time. We attempted to control for this in two ways: (1) each procedure was observed by one of two orthopaedic sports medicine specialists who provided scripted hints in an effort to standardize the tasks and to ensure that a similar procedure was performed by all subjects, and (2) the simulator was programmed to measure the completeness of the task being performed by the surgeon and to output this value as the completeness score. We did not find a significant difference between any of the groups for the completeness score. The group that took the shortest time (community-based orthopaedic surgeons) also had the highest completeness scores and the group that took the longest time (PGY-1 residents) had the lowest completeness scores. Thus, it is unlikely that the surgeons with the fastest time were simply achieving this by performing a lower-quality procedure. All subjects were given verbal cues during the course of the procedure, which could have introduced some bias. We believed that these cues were necessary to decrease the effect of a knowledge deficit, which was most likely present in the less experienced group and could have resulted in lower time and completeness scores as a result of skipped steps in the procedure. The cues were given by the mod-
erators and were scripted so that only the same information was available to all groups. The amount of help decreased with each trial in a previously agreed upon fashion as described above. The community-based orthopaedic surgeons had an average of only 7.7 years of post-training and a median of two years of post-training. Although we contacted individuals with a breadth of experience, those who were the first to respond tended to have only one to two years of post-training. This was the result of randomly selecting community-based orthopaedic surgeons. The community-based orthopaedic surgeons enrolled said that they averaged performing ninety-five knee arthroscopy cases per year, and the fifth-year residents indicated that they participated in eighty-eight cases during their entire residency. In effect, then, those community-based orthopaedic surgeons who had completed residency two years earlier had at least three times the experience of the PGY-5 group in performing diagnostic arthroscopy. In contrast, PGY-1 residents indicated that they had manipulated an arthroscope an average of one time during their residency (Table III). For a task like diagnostic knee arthroscopy, eighty-eight cases may be enough to attain some proficiency22 and it may have been more appropriate to have selected residents with less experience for the intermediate group. Lastly, only a small number of subjects were evaluated. Although the study demonstrated a high level of statistical power for the outcome of time, we were unable to draw robust conclusions regarding whether other interesting factors such as handedness or video game experience lead to improved efficiency when performing diagnostic arthroscopy on the simulator. With regard to the time required to complete the task of diagnostic arthroscopy, we were able to demonstrate, as other similar studies have, that the time to complete the task decreased as the experience of the surgeon increased12,15,23,24. This provides evidence to the construct validity of the simulator for the task of diagnostic knee arthroscopy. As expected, skills acquired through operative experience resulted in improved time of performance on the simulator. Whether skills learned on the ArthroSim virtual reality arthroscopy simulator lead to improved performance in the operating room remains unknown and is currently being investigated.
e57(6) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
Despite the differences in experience between the PGY-5 residents and the community-based orthopaedic surgeons, we were only able to detect a trend toward a significant difference in the overall time to complete the procedure. This could indicate that the simulator may be unable to discriminate between higher levels of experience. It is possible that if we had chosen community-based orthopaedic surgeons who had completed their residency earlier than the community-based orthopaedic surgeons chosen for this study, we may have been able to demonstrate a significant difference in time of procedure between the community-based orthopaedic surgeons and PGY-5 residents as a larger effect size may be assumed. However, the effect size (the Cohen D = 0.95) found between the two groups in this study was already very large (an effect size of 0.8 is viewed as large by Cohen) suggesting that it is more likely that this study was underpowered to detect even this large effect at the p = 0.05 level25. When the visualization and probing sections of the procedure were analyzed separately, we did find a significant difference between the PGY-5 residents and the community-based orthopaedic surgeons for probing, but not for visualization. Not only did we demonstrate that probing took almost twice the time as visualization, but also the data demonstrated a steeper learning curve for probing than for visualization, which could indicate that probing is a more difficult procedure to learn (Fig. 2). There was no significant difference among the average completeness scores assigned by the simulator among PGY-1 residents, PGY-5 residents, and community-based orthopaedic surgeons, indicating that the individuals performing the procedure did so by completing all steps in order, utilizing similar arthroscope position, probing technique, knee flexion angle, and valgus or varus force on the knee joint during the procedure. In this investigation, two orthopaedic instructors were utilized to provide hints to all of the subjects to minimize these variables and this result was expected and necessary as a requisite for using time as a valid measure of arthroscopic experience, as discussed previously. The ArthroSim virtual reality arthroscopy simulator was designed with a mentor program that teaches many of the skills necessary for safely performing a diagnostic knee arthroscopy outside of the presence of supervising faculty. Given the demands on instructor time at many institutions, this may be a very desirable feature of the simulator. It should be stated that the completeness score output by the simulator was used only as a measure of completeness of the procedure and not as a measure of the surgeon’s arthroscopic proficiency. It is unknown whether this completeness score would correlate with true surgical proficiency. Virtual reality simulators have been utilized for this purpose previously
by recording measures such as instrument collisions, instrument path length, probe velocity, and force data12,18,20,24,26,27. In conclusion, this study provides evidence that surgeons with higher skill levels acquired through surgical experience outperform lesser skilled surgeons by achieving faster completion times when performing diagnostic arthroscopy on the ArthroSim virtual reality arthroscopy simulator. Training on the simulator may likewise lead to improved efficiency for surgeons in the operating room. Further work is ongoing to determine whether simulation-trained residents will outperform non-simulation-trained residents in live surgical cases, and to establish the simulator’s validity as an instructor, as an instrument for the deliberate practice of arthroscopic skills, and as a tool for evaluation and feedback. n
d
d
d
NOTE: We thank Kaaren I. Hoffman, PhD, and the American Academy of Orthopaedic Surgeons’ Department of Research and Scientific Affairs for performing the statistical analyses in this paper. We also thank Howard Sweeney, MD, and Greg Palutsis, MD, for their production of the arthroscopic instructional video used in this study.
W. Dilworth Cannon, MD Department of Orthopaedic Surgery, University of California San Francisco, 1500 Owens Street, San Francisco, CA 94158. E-mail address: [email protected] Gregg T. Nicandri, MD Department of Orthopaedic Surgery and Rehabilitation, University of Rochester Medical Center, 601 Elmwood Avenue, Box 665, Rochester, NY 14642 Karl Reinig, PhD Department of Cell and Developmental Biology, Colorado School of Medicine, 12801 East 17th Avenue, MS 8108, Aurora, CO 80045 Howard Mevis, MA CME Course Operations and Practice Management, American Academy of Orthopaedic Surgeons, 6300 North River Road, Rosemont, IL 60018 Jocelyn Wittstein, MD Bassett Shoulder and Sports Medicine Research Institute, Bassett Healthcare Network, 1 Atwell Road, Cooperstown, NY 13326
References 1. Barnes RW, Lang NP, Whiteside MF. Halstedian technique revisited. Innovations in teaching surgical skills. Ann Surg. 1989 Jul;210(1): 118-21. 2. Michelson JD. Simulation in orthopaedic education: an overview of theory and practice. J Bone Joint Surg Am. 2006 Jun;88(6):1405-11. 3. Aggarwal R, Darzi A. Technical-skills training in the 21st century. N Engl J Med. 2006 Dec 21;355(25):2695-6.
4. Atesok K, Mabrey JD, Jazrawi LM, Egol KA. Surgical simulation in orthopaedic skills training. J Am Acad Orthop Surg. 2012 Jul;20(7):410-22. 5. Callery MP. Expansion beyond compression. Surg Endosc. 2003 May;17(5):677-8. Epub 2003 Apr 3. 6. Darosa DA, Bell RH Jr, Dunnington GL. Residency program models, implications, and evaluation: results of a think tank consortium on resident work hours. Surgery. 2003 Jan;133(1):13-23.
e57(7) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
7. Farnworth LR, Lemay DE, Wooldridge T, Mabrey JD, Blaschak MJ, DeCoster TA, Wascher DC, Schenck RC Jr. A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: the financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J. 2001;21:31-5. 8. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005 Jan;27(1):10-28. 9. Michelson JD, Manning L. Competency assessment in simulation-based procedural education. Am J Surg. 2008 Oct;196(4):609-15. Epub 2008 Jul 9. 10. Scott DJ. Patient safety, competency, and the future of surgical simulation. Simul Healthc. 2006 Fall;1(3):164-70. 11. Pedowitz RA, Marsh JL. Motor skills training in orthopaedic surgery: a paradigm shift toward a simulation-based educational curriculum. J Am Acad Orthop Surg. 2012 Jul;20(7):407-9. 12. Gomoll AH, O’Toole RV, Czarnecki J, Warner JJ. Surgical experience correlates with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports Med. 2007 Jun;35(6):883-8. Epub 2007 Jan 29. 13. Heng PA, Cheng CY, Wong TT, Wu W, Xu Y, Xie Y, Chui YP, Chan KM, Leung KS. Virtual reality techniques. Application to anatomic visualization and orthopaedics training. Clin Orthop Relat Res. 2006 Jan;442:5-12. 14. Cannon WD, Eckhoff DG, Garrett WE Jr, Hunter RE, Sweeney HJ. Report of a group developing a virtual reality simulator for arthroscopic surgery of the knee joint. Clin Orthop Relat Res. 2006 Jan;442:21-9. 15. Gomoll AH, Pappas G, Forsythe B, Warner JJ. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: a 3-year follow-up study. Am J Sports Med. 2008 Jun;36(6):1139-42. Epub 2008 Mar 6. 16. Mabrey JD, Gillogly SD, Kasser JR, Sweeney HJ, Zarins B, Mevis H, Garrett WE Jr, Poss R, Cannon WD. Virtual reality simulation of arthroscopy of the knee. Arthroscopy. 2002 Jul-Aug;18(6):E28. 17. Mabrey JD, Reinig KD, Cannon WD. Virtual reality in orthopaedics: is it a reality? Clin Orthop Relat Res. 2010 Oct;468(10):2586-91.
18. Pedowitz RA, Esch J, Snyder S. Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy. 2002 Jul-Aug;18(6):E29. 19. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002 Oct;236(4):458-63; discussion 463-4. 20. Tanoue K, Uemura M, Kenmotsu H, Ieiri S, Konishi K, Ohuchida K, Onimaru M, Nagao Y, Kumashiro R, Tomikawa M, Hashizume M. Skills assessment using a virtual reality simulator, LapSim, after training to develop fundamental skills for endoscopic surgery. Minim Invasive Ther Allied Technol. 2010;19(1):24-9. 21. Wohaibi EM, Bush RW, Earle DB, Seymour NE. Surgical resident performance on a virtual reality simulator correlates with operating room performance. J Surg Res. 2010 May 1;160(1):67-72. Epub 2008 Dec 10. 22. O’Neill PJ, Cosgarea AJ, Freedman JA, Queale WS, McFarland EG. Arthroscopic proficiency: a survey of orthopaedic sports medicine fellowship directors and orthopaedic surgery department chairs. Arthroscopy. 2002 Sep;18(7):795800. 23. Reznick RK, MacRae H. Teaching surgical skills—changes in the wind. N Engl J Med. 2006 Dec 21;355(25):2664-9. 24. Tashiro Y, Miura H, Nakanishi Y, Okazaki K, Iwamoto Y. Evaluation of skills in arthroscopic training based on trajectory and force data. Clin Orthop Relat Res. 2009 Feb;467(2):546-52. Epub 2008 Sep 13. 25. Cohen J. editor. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: L. Erlbaum Associates; 1988. 26. D˘ anil˘ a R, Gerdes B, Ulrike H, Domı´nguez Ferna´ ndez E, Hassan I. Objective evaluation of minimally invasive surgical skills for transplantation. Surgeons using a virtual reality simulator. Chirurgia (Bucur). 2009 Mar-Apr;104(2): 181-5. 27. Escoto A, Trejos AL, Naish MD, Patel RV, Lebel ME. Force sensing-based simulator for arthroscopic skills assessment in orthopaedic knee surgery. Stud Health Technol Inform. 2012;173:129-35.
d
d
d
BY
T HE J OURNAL
OF
B ONE
AND J OINT
S URGERY, I NCORPORATED
Topics in Training Evaluation of Skill Level Between Trainees and Community Orthopaedic Surgeons Using a Virtual Reality Arthroscopic Knee Simulator W. Dilworth Cannon, MD, Gregg T. Nicandri, MD, Karl Reinig, PhD, Howard Mevis, MA, and Jocelyn Wittstein, MD Investigation performed at the University of California San Francisco, San Francisco, California, and Duke University, Durham, North Carolina
Background: Several virtual reality simulators have been developed to assist orthopaedic surgeons in acquiring the skills necessary to perform arthroscopic surgery. The purpose of this study was to assess the construct validity of the ArthroSim virtual reality arthroscopy simulator by evaluating whether skills acquired through increased experience in the operating room lead to improved performance on the simulator. Methods: Using the simulator, six postgraduate year-1 orthopaedic residents were compared with six postgraduate year-5 residents and with six community-based orthopaedic surgeons when performing diagnostic arthroscopy. The time to perform the procedure was recorded. To ensure that subjects did not sacrifice the quality of the procedure to complete the task in a shorter time, the simulator was programmed to provide a completeness score that indicated whether the surgeon accurately performed all of the steps of diagnostic arthroscopy in the correct sequence. Results: The mean time to perform the procedure by each group was 610 seconds for community-based orthopaedic surgeons, 745 seconds for postgraduate year-5 residents, and 1028 seconds for postgraduate year-1 residents. Both the postgraduate year-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time (p = 0.006) than the postgraduate year-1 residents. There was a trend toward significance (p = 0.055) in time to complete the procedure when the postgraduate year-5 residents were compared with the community-based orthopaedic surgeons. The mean level of completeness as assigned by the simulator for each group was 85% for the community-based orthopaedic surgeons, 79% for the postgraduate year-5 residents, and 71% for the postgraduate year-1 residents. As expected, these differences were not significant, indicating that the three groups had achieved an acceptable level of consistency in their performance of the procedure. Conclusions: Higher levels of surgeon experience resulted in improved efficiency when performing diagnostic knee arthroscopy on the simulator. Further validation studies utilizing the simulator are currently under way and the additional simulated tasks of arthroscopic meniscectomy, meniscal repair, microfracture, and loose body removal are being developed.
Peer Review: This article was reviewed by the Editor-in-Chief and one Deputy Editor, and it underwent blinded review by two or more outside experts. The Deputy Editor reviewed each revision of the article, and it underwent a final review by the Editor-in-Chief prior to publication. Final corrections and clarifications occurred during one or more exchanges between the author(s) and copyeditors.
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. No author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
J Bone Joint Surg Am. 2014;96:e57(1-7)
d
http://dx.doi.org/10.2106/JBJS.M.00779
e57(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
Surgical skills education has traditionally followed the Halstedian model in which residents are given increasing autonomy while performing live surgical cases at the discretion of a supervising surgeon educator1,2. Due to a number of factors, including an increased public awareness of medical errors, duty-hour limitations for residents, and an increasing emphasis on the efficient use of operating room time, there has been a paradigm shift in orthopaedic education. There is now an increased utilization of skills training modalities outside of the operating room that enable trainees to practice while making and learning from mistakes in an environment where patient harm cannot occur3-10. As part of the next accreditation system, the Accreditation Council for Graduate Medical Education (ACGME) has added a requirement that residency programs include a surgical skills training curriculum. As a result, it is likely that simulators will play an increasingly important role in the future of surgical education11. There are several types of simulators, categorized as lowfidelity and high-fidelity. Low-fidelity simulators (box trainers and dry models such as the Large Arthroscopy Knee Model or Alex Shoulder [Sawbones, Vashon, Washington]) teach basic surgical skills (knot tying, bimanual dexterity, triangulation skills, and grasping) but do not necessarily recreate a realistic environment. It is currently unknown how effectively the skills acquired utilizing this type of simulation training transfer to the operating room. High-fidelity simulators are those that are meant to recreate a more realistic environment for simulated surgery and would include cadaveric simulation, live animal model simulation, and virtual reality simulation. Cadavers are currently the gold standard for simulation training12,13. How-
ever, they have several drawbacks, including the risk of disease transmission and the lack of uniformity among specimens, and their use requires considerable preparation time as well as appropriate storage and disposal protocols. In addition, the cost and supply of specimens may limit their availability to trainees. Virtual reality simulators have been developed to allow surgeons the opportunity for deliberate practice in a high-fidelity simulated environment12-18. Skills acquired using a truly highfidelity simulator should translate to improved performance in the operating room and vice versa. It is currently unknown whether skills acquired through live surgical experience result in improved performance on the ArthroSim virtual reality arthroscopy simulator (Touch of Life Technologies, Aurora, Colorado) used in this study. In this study, we evaluated the construct validity of the ArthroSim virtual reality arthroscopy simulator as a high-fidelity simulation of diagnostic knee arthroscopy, and we hypothesized that surgeons with more arthroscopic experience would perform a complete diagnostic knee arthroscopy (as measured by the completeness score) on the simulator in a shorter amount of time than surgeons with less experience.
d
d
d
Materials and Methods The ArthroSim virtual reality arthroscopy simulator (Fig. 1) was used for this study. The virtual reality knee joint was based on segmentation of 0.1-mm 14 cryosections of a right knee from a twenty-eight-year-old cadaver . Both the simulated arthroscope and the probe were provided with haptic feedback devices allowing six degrees of motion and three degrees of force (3-D Systems, Rock Hill, South Carolina). The surrogate right leg had knee motion from 0° to 90° of flexion and measured varus and valgus forces, which were transformed into medial and lateral compartment opening.
Fig. 1
A photograph showing the ArthroSim virtual reality arthroscopy simulator.
e57(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014 d
d
d
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
TABLE I Objects Visualized and Probed During the Knee Diagnostic Arthroscopy Objects Visualized
Objects Probed
Suprapatellar pouch
Medial meniscus
Patellofemoral joint
Medial femoral condyle articular cartilage
Medial recess
Medial tibial plateau articular cartilage
Medial femoral condyle articular cartilage
Posterior cruciate ligament or anterior cruciate ligament
Medial meniscus
Lateral meniscus
Posterior cruciate ligament or anterior cruciate ligament
Lateral femoral condyle articular cartilage
Lateral femoral condyle articular cartilage
Posteromedial compartment (posterior horn medial meniscus)
Lateral meniscus
Posterior lateral compartment (posterior horn lateral meniscus)
Lateral gutter
Prior to conducting the study, institutional review board approval was acquired from the review boards at both the University of California San Francisco Medical Center and the Duke University Medical Center. A letter of invitation was sent to the six postgraduate year (PGY)-1 residents and six PGY-5 residents in the Department of Orthopedic Surgery at the University of California San Francisco. The first four residents who responded were selected to participate in the study. In addition, a letter of invitation was sent to each of the eight PGY-1 residents and eight PGY-5 residents in the Department of Orthopedic Surgery at Duke University. The first two residents who responded were selected to participate. E-mailed invitations were sent to communitybased orthopaedic surgeons from the local San Francisco area. The first four surgeons who responded positively were selected to participate. E-mailed invitations were also sent to community-based orthopaedic surgeons from Durham, North Carolina, and the first two surgeons who responded were selected to participate. In total, there were six subjects per group. To qualify for participation, the community-based orthopaedic surgeons had to be performing between fifty and 125 knee arthroscopy cases per year. Each subject signed a consent form, filled out a demographic sheet, provided answers to a questionnaire, and then watched a fifteen-minute video of a diagnostic arthroscopy of the right knee instructing them in the technique and sequence that they were expected to follow in the arthroscopic procedure. None of the subjects enrolled in this study had had any prior experience using an arthroscopic virtual reality simulator. The subjects were allowed approximately two minutes to handle the 30° arthroscope and to probe inside the virtual right knee joint to familiarize themselves with the setup and the feel of the instruments. Each subject then performed three trials visualizing and then probing the structures listed in Table I. Each subject was given hints throughout the procedure by a supervising orthopaedic sports medicine subspecialist. The hints were scripted and were the same for all groups regardless of experience. We believed that this was necessary with time being our primary outcome variable. Without hints, it would be possible for subjects to omit certain steps of the procedure. We wanted to ensure that those completing the procedure in a shorter time were doing so as a result of being more efficient and not simply because they were skipping steps. For example, hints for the step of probing the posterior horn of the medial meniscus of the right knee for trial 1 were as follows: probe the superior and inferior surfaces of the posterior horn with the knee flexed 30°, with a medial compartment opening (produced by a valgus force) of ‡3.5°, and aim the arthroscope at 9 o’clock. For trial 2, the recommended knee flexion angle and amount of valgus force were omitted from the verbal cues. For trial 3, cues were given only if the subject failed to progress and was stuck at a step in the procedure. A stopwatch was used to record the time for each step of the procedure for all three trials. The completeness score incrementally increased as the subject progressed through the steps of the diagnostic arthroscopy. A score of 100% would
indicate that the subject had the proper arthroscope position, probing technique, knee flexion angle, and correct valgus or varus force on the knee joint during the procedure as determined by a range of acceptable values programmed into the simulator. The completeness scores from all three trials for both the visualization and probing parts of the procedure were recorded.
Statistical Analysis All statistical tests were conducted with use of SPSS version 15.0 (SPSS, Chicago, Illinois). A one-way analysis of variance (ANOVA) was used to determine the equality of means and pairwise comparisons were made using two a priori orthogonal contrasts, one that compared the PGY-1 residents with the combined group of PGY-5 residents and community-based orthopaedic surgeons and the other that compared the PGY-5 residents with the community-based orthopaedic surgeons. Significance was set at p < 0.05 for both the primary outcome of time and the secondary outcome of completeness. Although each subject completed three trials, only the time and completeness scores from trial 3 were used in the statistical analysis. The first two trials were considered practice. Descriptive statistics were used for the outcomes of time, completeness, and demographic variables.
Source of Funding No external funding source was employed in this investigation.
Results For trial 3, the mean time to perform the procedure by each group was 610 seconds for the community-based orthopaedic surgeons, 745 seconds for the PGY-5 residents, and 1028 seconds for the PGY-1 residents (Table II). The overall ANOVA showed a significant mean difference for all three groups (p = 0.009). The Levene test for homogeneity of variance was not significant, but considering the small sample size and large differences in variance across the groups, tests that did not assume equal variances were performed. The combined PGY-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-1 residents (p = 0.006), and there was a trend (p = 0.055) toward decreased time to completion when the PGY-5 residents were compared with the community-based orthopaedic surgeons. A post hoc power analysis was performed using the data for trial 3. With n = 6, a = 0.05, and a difference in means of 2.4 standard deviations (the Cohen D effect size),
e57(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014 d
d
d
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
TABLE II Total Procedural Time in Trial 3 Group
No. of Participants
Total Procedural Time* 1028.0 ± 228.7
PGY-1 residents
6
PGY-5 residents
6
745.3 ± 155.7
Community-based orthopaedic surgeons
6
609.8 ± 103.7
*The values are given as the mean and the standard deviation in seconds.
the study had 0.95 power to detect a difference in time scores among the three groups. Probing structures in the knee joint took almost twice as much time as visualization of these structures (Fig. 2). Therefore, the times to complete the visualization and probing tasks were separately analyzed. For the visualization task, both the PGY-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-1 residents (p = 0.012). There was no significant difference in time (p = 0.3) for visualization when the PGY-5 residents and communitybased orthopaedic surgeons were compared. For the probing task, both the PGY-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-1 residents (p = 0.015) and the community-based orthopaedic surgeons performed the procedure in significantly less time than the PGY-5 residents (p = 0.046). In Figure 3, the mean times for the three groups performing trials 1 through 3 are shown. The time to complete the procedure decreased corresponding to the level of previous surgical experi-
Fig. 2
ence of the three groups. A training effect was also demonstrated as all three groups’ times decreased as they progressed from trial 1 to trial 3. This effect was the largest for the PGY-1 group. The answers to the questionnaire regarding previous arthroscopic experience were averaged for each group and are presented in Table III. The six community-based orthopaedic surgeons averaged 7.7 years (one, one, two, two, ten, and thirty years) since the completion of their residency. They averaged ninety-five knee arthroscopy cases per year. The PGY-5 residents had manipulated an arthroscope an average of eighty-eight times during their residency in contrast to an average of one time for the PGY-1 residents. Four of the six community-based orthopaedic surgeons had completed a sports medicine fellowship. The mean level of completeness as assigned by the simulator for each group was 85% for the community-based orthopaedic surgeons, 79% for the PGY-5 residents, and 71% for the PGY-1 residents. These differences were not significant, indicating that the three groups had achieved an acceptable level of consistency in their performance of the procedure.
Fig. 3
Fig. 2 A line graph showing the breakdown of trial 3 into the time for visualization and then probing of the knee for the three groups of subjects. CBO = community-based orthopaedic surgeon. Fig. 3 A line graph showing the mean times for the three groups for trials 1 through 3, demonstrating the differences between the three groups as well as a learning effect. CBO = community-based orthopaedic surgeon.
e57(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014 d
d
d
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
TABLE III Past Experience Among the Three Groups in This Study PGY-1 Residents
PGY-5 Residents
Average no. of arthroscopy cases seen
6
101
95*
Average no. of arthroscopy cases in which the subject held the scope
2
88
95*
Average no. of arthroscopy cases in which the subject manipulated the scope
1
88
95*
Average no. of hours per week in which the subject plays video games
0
0
1
Average no. of hours per week in which the subject played video games
1
1
2
Average no. of arthroscopy lectures received
1
10
4
Average no. of arthroscopy videos viewed
1
2
2
Average no. of arthroscopy courses attended
0
0
1
Past Experience
Community-Based Orthopaedic Surgeons
*This experience occurred within the last year before the start of the study.
Discussion Surgical simulation is being increasingly utilized in orthopaedic technical skills training, and the ArthroSim virtual reality arthroscopy simulator is one such model that has been developed for this purpose4,11,14,15. If a simulator is a valid high-fidelity recreation of the live surgical environment, skills acquired through operative experience should transfer to a better performance on the simulator. One component of surgical technical skill is efficiency, and in this investigation, we sought to determine whether individuals with increased arthroscopic experience could perform the task of diagnostic arthroscopy on the simulator in a shorter time when compared with those with less experience. Our study was limited by several factors. First, efficiency (as measured by the time to complete the task) was used as the primary outcome measure of technical skill. Time has also been utilized as an outcome by other similar studies12,18-21. With time as the primary outcome variable, it was imperative to ensure that one of the groups simply did not sacrifice quality for a faster time. We attempted to control for this in two ways: (1) each procedure was observed by one of two orthopaedic sports medicine specialists who provided scripted hints in an effort to standardize the tasks and to ensure that a similar procedure was performed by all subjects, and (2) the simulator was programmed to measure the completeness of the task being performed by the surgeon and to output this value as the completeness score. We did not find a significant difference between any of the groups for the completeness score. The group that took the shortest time (community-based orthopaedic surgeons) also had the highest completeness scores and the group that took the longest time (PGY-1 residents) had the lowest completeness scores. Thus, it is unlikely that the surgeons with the fastest time were simply achieving this by performing a lower-quality procedure. All subjects were given verbal cues during the course of the procedure, which could have introduced some bias. We believed that these cues were necessary to decrease the effect of a knowledge deficit, which was most likely present in the less experienced group and could have resulted in lower time and completeness scores as a result of skipped steps in the procedure. The cues were given by the mod-
erators and were scripted so that only the same information was available to all groups. The amount of help decreased with each trial in a previously agreed upon fashion as described above. The community-based orthopaedic surgeons had an average of only 7.7 years of post-training and a median of two years of post-training. Although we contacted individuals with a breadth of experience, those who were the first to respond tended to have only one to two years of post-training. This was the result of randomly selecting community-based orthopaedic surgeons. The community-based orthopaedic surgeons enrolled said that they averaged performing ninety-five knee arthroscopy cases per year, and the fifth-year residents indicated that they participated in eighty-eight cases during their entire residency. In effect, then, those community-based orthopaedic surgeons who had completed residency two years earlier had at least three times the experience of the PGY-5 group in performing diagnostic arthroscopy. In contrast, PGY-1 residents indicated that they had manipulated an arthroscope an average of one time during their residency (Table III). For a task like diagnostic knee arthroscopy, eighty-eight cases may be enough to attain some proficiency22 and it may have been more appropriate to have selected residents with less experience for the intermediate group. Lastly, only a small number of subjects were evaluated. Although the study demonstrated a high level of statistical power for the outcome of time, we were unable to draw robust conclusions regarding whether other interesting factors such as handedness or video game experience lead to improved efficiency when performing diagnostic arthroscopy on the simulator. With regard to the time required to complete the task of diagnostic arthroscopy, we were able to demonstrate, as other similar studies have, that the time to complete the task decreased as the experience of the surgeon increased12,15,23,24. This provides evidence to the construct validity of the simulator for the task of diagnostic knee arthroscopy. As expected, skills acquired through operative experience resulted in improved time of performance on the simulator. Whether skills learned on the ArthroSim virtual reality arthroscopy simulator lead to improved performance in the operating room remains unknown and is currently being investigated.
e57(6) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
Despite the differences in experience between the PGY-5 residents and the community-based orthopaedic surgeons, we were only able to detect a trend toward a significant difference in the overall time to complete the procedure. This could indicate that the simulator may be unable to discriminate between higher levels of experience. It is possible that if we had chosen community-based orthopaedic surgeons who had completed their residency earlier than the community-based orthopaedic surgeons chosen for this study, we may have been able to demonstrate a significant difference in time of procedure between the community-based orthopaedic surgeons and PGY-5 residents as a larger effect size may be assumed. However, the effect size (the Cohen D = 0.95) found between the two groups in this study was already very large (an effect size of 0.8 is viewed as large by Cohen) suggesting that it is more likely that this study was underpowered to detect even this large effect at the p = 0.05 level25. When the visualization and probing sections of the procedure were analyzed separately, we did find a significant difference between the PGY-5 residents and the community-based orthopaedic surgeons for probing, but not for visualization. Not only did we demonstrate that probing took almost twice the time as visualization, but also the data demonstrated a steeper learning curve for probing than for visualization, which could indicate that probing is a more difficult procedure to learn (Fig. 2). There was no significant difference among the average completeness scores assigned by the simulator among PGY-1 residents, PGY-5 residents, and community-based orthopaedic surgeons, indicating that the individuals performing the procedure did so by completing all steps in order, utilizing similar arthroscope position, probing technique, knee flexion angle, and valgus or varus force on the knee joint during the procedure. In this investigation, two orthopaedic instructors were utilized to provide hints to all of the subjects to minimize these variables and this result was expected and necessary as a requisite for using time as a valid measure of arthroscopic experience, as discussed previously. The ArthroSim virtual reality arthroscopy simulator was designed with a mentor program that teaches many of the skills necessary for safely performing a diagnostic knee arthroscopy outside of the presence of supervising faculty. Given the demands on instructor time at many institutions, this may be a very desirable feature of the simulator. It should be stated that the completeness score output by the simulator was used only as a measure of completeness of the procedure and not as a measure of the surgeon’s arthroscopic proficiency. It is unknown whether this completeness score would correlate with true surgical proficiency. Virtual reality simulators have been utilized for this purpose previously
by recording measures such as instrument collisions, instrument path length, probe velocity, and force data12,18,20,24,26,27. In conclusion, this study provides evidence that surgeons with higher skill levels acquired through surgical experience outperform lesser skilled surgeons by achieving faster completion times when performing diagnostic arthroscopy on the ArthroSim virtual reality arthroscopy simulator. Training on the simulator may likewise lead to improved efficiency for surgeons in the operating room. Further work is ongoing to determine whether simulation-trained residents will outperform non-simulation-trained residents in live surgical cases, and to establish the simulator’s validity as an instructor, as an instrument for the deliberate practice of arthroscopic skills, and as a tool for evaluation and feedback. n
d
d
d
NOTE: We thank Kaaren I. Hoffman, PhD, and the American Academy of Orthopaedic Surgeons’ Department of Research and Scientific Affairs for performing the statistical analyses in this paper. We also thank Howard Sweeney, MD, and Greg Palutsis, MD, for their production of the arthroscopic instructional video used in this study.
W. Dilworth Cannon, MD Department of Orthopaedic Surgery, University of California San Francisco, 1500 Owens Street, San Francisco, CA 94158. E-mail address: [email protected] Gregg T. Nicandri, MD Department of Orthopaedic Surgery and Rehabilitation, University of Rochester Medical Center, 601 Elmwood Avenue, Box 665, Rochester, NY 14642 Karl Reinig, PhD Department of Cell and Developmental Biology, Colorado School of Medicine, 12801 East 17th Avenue, MS 8108, Aurora, CO 80045 Howard Mevis, MA CME Course Operations and Practice Management, American Academy of Orthopaedic Surgeons, 6300 North River Road, Rosemont, IL 60018 Jocelyn Wittstein, MD Bassett Shoulder and Sports Medicine Research Institute, Bassett Healthcare Network, 1 Atwell Road, Cooperstown, NY 13326
References 1. Barnes RW, Lang NP, Whiteside MF. Halstedian technique revisited. Innovations in teaching surgical skills. Ann Surg. 1989 Jul;210(1): 118-21. 2. Michelson JD. Simulation in orthopaedic education: an overview of theory and practice. J Bone Joint Surg Am. 2006 Jun;88(6):1405-11. 3. Aggarwal R, Darzi A. Technical-skills training in the 21st century. N Engl J Med. 2006 Dec 21;355(25):2695-6.
4. Atesok K, Mabrey JD, Jazrawi LM, Egol KA. Surgical simulation in orthopaedic skills training. J Am Acad Orthop Surg. 2012 Jul;20(7):410-22. 5. Callery MP. Expansion beyond compression. Surg Endosc. 2003 May;17(5):677-8. Epub 2003 Apr 3. 6. Darosa DA, Bell RH Jr, Dunnington GL. Residency program models, implications, and evaluation: results of a think tank consortium on resident work hours. Surgery. 2003 Jan;133(1):13-23.
e57(7) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 96 -A N U M B E R 7 A P R I L 2, 2 014
S K I L L L E V E L E VA LUAT I O N O F T R A I N E E S A N D O RT H O PA E D I C S U R G E O N S U S I N G V I RT UA L R E A L I T Y A RT H R O S C O P I C K N E E S I M U L AT O R
7. Farnworth LR, Lemay DE, Wooldridge T, Mabrey JD, Blaschak MJ, DeCoster TA, Wascher DC, Schenck RC Jr. A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: the financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J. 2001;21:31-5. 8. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005 Jan;27(1):10-28. 9. Michelson JD, Manning L. Competency assessment in simulation-based procedural education. Am J Surg. 2008 Oct;196(4):609-15. Epub 2008 Jul 9. 10. Scott DJ. Patient safety, competency, and the future of surgical simulation. Simul Healthc. 2006 Fall;1(3):164-70. 11. Pedowitz RA, Marsh JL. Motor skills training in orthopaedic surgery: a paradigm shift toward a simulation-based educational curriculum. J Am Acad Orthop Surg. 2012 Jul;20(7):407-9. 12. Gomoll AH, O’Toole RV, Czarnecki J, Warner JJ. Surgical experience correlates with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports Med. 2007 Jun;35(6):883-8. Epub 2007 Jan 29. 13. Heng PA, Cheng CY, Wong TT, Wu W, Xu Y, Xie Y, Chui YP, Chan KM, Leung KS. Virtual reality techniques. Application to anatomic visualization and orthopaedics training. Clin Orthop Relat Res. 2006 Jan;442:5-12. 14. Cannon WD, Eckhoff DG, Garrett WE Jr, Hunter RE, Sweeney HJ. Report of a group developing a virtual reality simulator for arthroscopic surgery of the knee joint. Clin Orthop Relat Res. 2006 Jan;442:21-9. 15. Gomoll AH, Pappas G, Forsythe B, Warner JJ. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: a 3-year follow-up study. Am J Sports Med. 2008 Jun;36(6):1139-42. Epub 2008 Mar 6. 16. Mabrey JD, Gillogly SD, Kasser JR, Sweeney HJ, Zarins B, Mevis H, Garrett WE Jr, Poss R, Cannon WD. Virtual reality simulation of arthroscopy of the knee. Arthroscopy. 2002 Jul-Aug;18(6):E28. 17. Mabrey JD, Reinig KD, Cannon WD. Virtual reality in orthopaedics: is it a reality? Clin Orthop Relat Res. 2010 Oct;468(10):2586-91.
18. Pedowitz RA, Esch J, Snyder S. Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy. 2002 Jul-Aug;18(6):E29. 19. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002 Oct;236(4):458-63; discussion 463-4. 20. Tanoue K, Uemura M, Kenmotsu H, Ieiri S, Konishi K, Ohuchida K, Onimaru M, Nagao Y, Kumashiro R, Tomikawa M, Hashizume M. Skills assessment using a virtual reality simulator, LapSim, after training to develop fundamental skills for endoscopic surgery. Minim Invasive Ther Allied Technol. 2010;19(1):24-9. 21. Wohaibi EM, Bush RW, Earle DB, Seymour NE. Surgical resident performance on a virtual reality simulator correlates with operating room performance. J Surg Res. 2010 May 1;160(1):67-72. Epub 2008 Dec 10. 22. O’Neill PJ, Cosgarea AJ, Freedman JA, Queale WS, McFarland EG. Arthroscopic proficiency: a survey of orthopaedic sports medicine fellowship directors and orthopaedic surgery department chairs. Arthroscopy. 2002 Sep;18(7):795800. 23. Reznick RK, MacRae H. Teaching surgical skills—changes in the wind. N Engl J Med. 2006 Dec 21;355(25):2664-9. 24. Tashiro Y, Miura H, Nakanishi Y, Okazaki K, Iwamoto Y. Evaluation of skills in arthroscopic training based on trajectory and force data. Clin Orthop Relat Res. 2009 Feb;467(2):546-52. Epub 2008 Sep 13. 25. Cohen J. editor. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: L. Erlbaum Associates; 1988. 26. D˘ anil˘ a R, Gerdes B, Ulrike H, Domı´nguez Ferna´ ndez E, Hassan I. Objective evaluation of minimally invasive surgical skills for transplantation. Surgeons using a virtual reality simulator. Chirurgia (Bucur). 2009 Mar-Apr;104(2): 181-5. 27. Escoto A, Trejos AL, Naish MD, Patel RV, Lebel ME. Force sensing-based simulator for arthroscopic skills assessment in orthopaedic knee surgery. Stud Health Technol Inform. 2012;173:129-35.
d
d
d
