ORIGINAL REPORTS

Communication Skills Training in Surgical Residency: A Needs Assessment and Metacognition Analysis of a Difficult Conversation Objective Structured Clinical Examination John L. Falcone, MD,* René N. Claxton, MD,† and Gary T. Marshall, MD* *

University of Pittsburgh School of Medicine, Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania; and †University of Pittsburgh School of Medicine, Department of Medicine, Division of General Internal Medicine, Section of Palliative Care and Ethics, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania BACKGROUND: The objective structured clinical exami-

nation (OSCE) can be used to evaluate the Accreditation Council for Graduate Medical Education Core Competencies of Professionalism and Interpersonal and Communication Skills. The aim of this study was to describe general surgery resident performance on a “difficult conversation” OSCE. METHODS: In this prospective study, junior and senior

residents participated in a 2-station OSCE. Junior stations involved discussing operative risks and benefits and breaking bad news. Senior stations involved discussing goals of care and discussing transition to comfort measures only status. Residents completed post-OSCE checklist and Likert-based self-evaluations of experience, comfort, and confidence. Trained standardized patients (SPs) evaluated residents using communication skill–based checklists and Likertbased assessments. Pearson correlation coefficients were determined between self-assessment and SP assessment. Mann-Whitney U tests were conducted between junior and senior resident variables, using α ¼ 0.05. RESULTS: There were 27 junior residents (age 28.1 ⫾

1.9 years [29.6% female]) and 27 senior residents (age 32.1 ⫾ 2.5 years [26.9% female]). The correlation of selfassessment and SP assessment of overall communication skills by junior residents was 0.32 on the risks and benefits case and 0.07 on the breaking bad news case. Source of financial support: This study was supported by the Geriatrics for Specialty Residents education grant from the American Geriatrics Society. Correspondence: Inquiries to John L. Falcone, MD, MS, University of Pittsburgh Medical Center, Presbyterian University Hospital, F-675 200 Lothrop Street, Pittsburgh, PA 15213; fax: þ412-647-4889; e-mail: [email protected]

The correlation of self-assessment and SP assessment of overall communication skills by senior residents was 0.30 on the goals of care case and 0.26 on the comfort measures only case. SP assessments showed that junior residents had higher overall communication skills than senior residents (p ¼ 0.03). Senior residents perceived that having difficult conversations was more level appropriate (p o 0.001), and they were less nervous having difficult conversations (p o 0.01) than junior residents. CONCLUSIONS: We found that residents perform difficult conversations well, that subjective and objective skills are correlated, and that skills-based training is needed across all residency levels. This well-received method may be used to observe, document, and provide resident feedback for C 2014 Associthese important skills. ( J Surg 71:309-315. J ation of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: assessment, general surgery, communication, professionalism, objective structured clinical examination COMPETENCIES: Patient Care, Professionalism, Interper-

sonal and Communication Skills

INTRODUCTION Medical education is shifting toward outcomes-based assessment of residents and fellows as evidenced by the Next Accreditation System Milestone project developed by the Accreditation Council for Graduate Medical Education (ACGME). Residencies would be asked to evaluate trainees

Journal of Surgical Education  & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2013.09.020

309

on developmentally based educational achievements in the ACGME core competencies through time.1 General surgery education, similar to general medical education, is adding emphasis on outcomes assessment.2 Obstacles such as resident duty hours and limited protected faculty time for observation and feedback create challenges for residencies in the evaluation of these outcomes. Two of the more challenging competencies to evaluate from an objective, outcomes perspective are Professionalism and Interpersonal and Communication Skills.3 Going forward, general surgery residency programs would need to develop assessment tools to demonstrate resident achievement of educational milestones. In the past, several studies have evaluated the effect of dedicated curricula using standardized patients (SPs) in surgical training.4-7 Some studies evaluate broad curricular topics like professionalism and communication.4-6 Other studies focus on the practice of specific skills such as disclosing operative complications and iatrogenic injury.7 Evidence shows that surgical residents can improve performance skills with SP-based curricula targeting the core competencies of Professionalism And Interpersonal And Communication Skills.4-8 The objective structured clinical examination (OSCE) is an assessment technique for professional and communication behaviors shown to be valid and reliable in the domains of Professionalism And Interpersonal And Communication Skills.9-14 The OSCE allows trainees and SPs to evaluate trainee skills based on objective measures such as skill checklists. This is educationally efficient as the checklist serves as both the curricula and the evaluation method. Moreover, the use of SPs frees faculty time for other educational or clinical endeavors. The OSCE is also learner centered as each trainee receives individual feedback on his or her performance. As resident evaluation moves from time based to outcomes based, the OSCE may serve as a helpful evaluation tool for the General Surgery Milestone Project.15 At the University of Pittsburgh Medical Center, general surgery residents participate in a “difficult conversation” OSCE annually. Junior residents (postgraduate year 1-2) and senior residents (Zpostgraduate year 3) participate in a 2-station OSCE. The first junior resident stations involve discussing operative risks and benefits (RAB) with a patient who has asymptomatic cholelithiasis and significant medical comorbidities. The second junior station involves breaking bad news (BBN) to a patient with recurrent pancreatic cancer. The first senior resident stations involve discussing goals of care (GOC) with a patient who has a perforated peptic ulcer in the setting of locally advanced pancreatic cancer. The second senior station involves discussing the transition to comfort measures only (CMO) status with two family members of a 91-year-old patient with a large intracranial hemorrhage. 310

The purpose of this study was to describe performance on the 2-station OSCE by junior residents and senior residents. Results would be used as a needs assessment for a dedicated communication-based skills curriculum to be used in the evaluation of general surgery residents with regard to Professionalism and Interpersonal and Communication Skills. We hypothesized that senior residents would outperform junior residents in the domain of communication.

METHODS This was a prospective study including categorical general surgery residents at the University of Pittsburgh Medical Center. Residents were included if they attended the OSCE and if they completed both level-appropriate cases. This OSCE occurred during scheduled time for weekly teaching conference. Each station of this OSCE was 15 minutes in length. Each resident was given 12 minutes per patient encounter. This was followed by 2 minutes of selfevaluation and 1 minute of tailored feedback by the SPs. All patient encounters were videotaped. Deidentified surgery resident OSCE performance data were obtained. This study was approved by the Institutional Review Board at the University of Pittsburgh. Two sources of performance data were obtained for each resident: evaluation by SPs and self-evaluation of performance. The first source of performance data was an evaluation by the SP who interacted with the resident. SPs for this OSCE were selected from the University of Pittsburgh School of Medicine Standardized Patient Program. SPs with a history of excellent emotional portrayal and skills in verbal feedback were selected by the Director of the Standardized Patient Program. The selected SPs had a minimum of 20 hours of training in which continuity of case portrayal is an integral part. In training, SPs demonstrate knowledge of essential concepts regarding continuity of case portrayal between learners and continuity of progression of disclosure and emotional response. This, in addition to a minimum of 16 hours of prior SP experience, comprised the training for this activity. These SPs were also experienced in the sophisticated use of behaviorally based, observable and balanced feedback. SPs evaluated the residents with skills checklists and Likert (1-5 scale) assessments in multiple domains of resident performance such as nonverbal and verbal communication skills and empathy. Such evaluative measures have been internally validated, with standardized evaluation outcomes shown in a study using similarly trained SPs.16 The externally validated measure used for the assessment of verbal empathy in our study is the NURSE mnemonic for verbal empathy: Name, Understand, Respect, Support, and Explore.17 The second source was self-evaluation of OSCE performance. General surgery residents completed skills-based checklists and Likert (1-7 scale) self-evaluations of prior

Journal of Surgical Education  Volume 71/Number 3  May/June 2014

experience, comfort, satisfaction, and perceived confidence in OSCE performance. Nonparametric statistics were used to describe junior and senior resident performances on the level-appropriate OSCE. For the metacognition analyses, Pearson correlation coefficients were found comparing self-assessment measures and SP assessment measures. To compare junior and senior resident outcomes in both self-assessment and SP assessment, a Mann-Whitney U test was performed. All statistical tests were performed with Stata 11.1 statistical software (StataCorp, College Station, TX), using α ¼ 0.05.

RESULTS There were 27 junior residents (age 28.1 ⫾ 1.9 years [30% female]) and 27 senior residents (age 32.1 ⫾ 2.5 years [26% female]) who satisfied the inclusion criteria for this study. The male/female ratio between junior residents and senior residents was the same (p ¼ 0.76). SP assessment of junior resident performance on the 2station OSCE is given in Table 1 for the RAB case (Table 1 [A]) and the BBN case (Table 1[B]). SP assessment of senior resident performance on the 2-station OSCE is given in Table 2 for the GOC case and the CMO case. Table 3 shows the results of several junior and senior resident selfassessment measures. Comparisons between junior resident

and senior resident self-assessments and SP assessments in shared outcome variables are given in Table 4. The Pearson correlation coefficient of self-assessment and SP assessment of overall skills by junior residents was 0.40 on the RAB case and 0.07 on the BBN case. (Data not shown.) The Pearson correlation coefficient of selfassessment and SP assessment of overall skills by senior residents was 0.30 on the GOC case and 0.26 on the CMO case. (Data not shown.) In the evaluation of the OSCE exercise, there were no differences between junior residents and senior residents in assessment of overall case realism (6.0 [5.0-7.0) vs 6.0 [5.06.0]) or overall OSCE quality (6.0 [5.0-6.0] vs 6.0 [5.07.0]) (both p 4 0.50).

DISCUSSION This study describes general surgery resident performance on a 2-station “difficult conversation” OSCE in the absence of a dedicated skills-based communication and professionalism curriculum. We found that junior residents and senior residents show a high degree of favorable body language and nonverbal communication skills. This is demonstrated by high percentages of encounters with effective use of eye contact, posture, facial expression, the use of silence, and sitting down when speaking to patients. Senior residents sat

TABLE 1. Standardized Patient-Based Assessment of Junior Residents on a 2-Station OSCE Involving Discussing the Risks and Benefits of an Operation in a Patient With Prohibitive Risk (A) and Breaking Bad News of Recurrent Cancer (B) (A) Risks and Benefits Performance Assessment (%) Determined cardiac and other risk factors present in the patient Determine appropriateness of surgery Explains the importance of assessing risk of nonmandatory surgery Communicates the presence of no risk factors that would indicate the need for immediate surgery States clearly the risks and benefits of the surgery Ascertains the reasoning behind the patient’s request without criticizing the PCP or the patient Explain what the surgery involves Discussed high risk of stroke, anesthesia, and valve and bleeding complications Frankly discussed the pros and cons of surgery Made a clear and respectful recommendation not to have the surgery (B) Breaking Bad News Performance Assessment (%) Greets and shows interest in patient as a person Uses words that show care and concern throughout the encounter Uses tone, pace, and eye contact that show care and concern Used appropriate body language/posture toward patient (e.g., nodding and facial expression) Allowed some silence for patient to absorb the information Explained terms and information clearly to patient Organized the encounter in a comprehensive manner (e.g., good flow of conversation in interview) Responded to questions appropriately Offered to talk to family about diagnosis Offered to have social work/chaplain/other support systems come talk to patient Discussed cost of treatment with patient

Junior Residents (n ¼ 27) 96% 85% 89% 78% 78% 100% 89% 78% 81% 59% Junior Residents (n ¼ 27) 100% 100% 96% 89% 89% 100% 100% 100% 59% 7% 4%

PCP, Primary Care Physician. Journal of Surgical Education  Volume 71/Number 3  May/June 2014

311

TABLE 2. Standardized Patient-Based Assessment of Senior Residents on a 2-Station OSCE Involving Discussing Goals of Care and the Transition of a Patient to Comfort Measures Only Status Senior Residents (n ¼ 27) Resident Performance Assessment (%)

Either Case

Both Cases

100% 100%

100% 100%

52% 81% 81% 70% 100% 100% 100% 100% 100% 100% 96% 100% 81%

4% 22% 44% 41% 78% 70% 93% 85% 78% 85% 78% 70% 41%

0% 30%

0% 4%

Skills Demonstrated nonverbal empathy Sat down Made eye contact Demonstrated verbal empathy Named emotion Understand an emotion Stated respect for patient Offered support Used open-ended questions Stated prognosis Attempted to elicit patient’s treatment goals and expectations Discussed treatment options Used appropriate level of directiveness/made a recommendation Was easily understood Invited questions Suggested a plan Concluded with a review and a plan for follow-up Negative Behaviors Interrupted Made recommendation before eliciting patient preferences

down and made eye contact with the patients 100% of the time. Junior and senior residents, in general, also demonstrate favorable verbal communication skills as shown in Tables 1 and 2. Junior residents reliably demonstrated effective communication skills in both the RAB and the BBN cases. In the RAB case, most junior residents demonstrated all of the communication skills evaluated by the SP, demonstrating that junior residents are successful in having a balanced discussion with patients about perioperative RAB regarding a possible cholecystectomy. In the BBN case, junior residents also exhibit a number of professional and communication skills regarding BBN with patients. A minority of junior residents discussed additional resources available to patients; this could be a target for future skills training.

Most residents are consistent with skills between “difficult conversation” OSCE cases. However, although most senior residents used some skills in the NURSE mnemonic, the use of these skills was inconsistent between cases. Given that emotion handling is crucial in effectively managing difficult conversations, we plan to teach the routine use of the NURSE mnemonic for verbal empathy in our dedicated skill-based curriculum. We found that despite frequently performing difficult communication tasks in the clinical setting, residents are not routinely observed by an attending physician and structured feedback is not routinely delivered. This is concerning, and this serves as a needs assessment in this important domain of communication. Because of this finding, we encourage attending physicians to both observe and provide feedback

TABLE 3. Self-Assessment of Junior and Senior Residents Regarding Skills in Involving “Difficult Conversation” OSCE Cases Using a 7-Point Likert Scale Junior Residents (n ¼ 27) Senior Residents (n ¼ 27) Resident Evaluation/Self-Evaluation

RAB

BBN

GOC

CMO

I have discussed this issue with a patient (%) I have discussed this issue in front of a staff member (%) I received feedback from the staff member (%) This issue is appropriate for my level of training (median [IQR]) I am trained to discuss this issue with patients (median [IQR]) I feel nervous discussing this issue with patients (median [IQR]) I am able to answer questions about this issue (median [IQR]) I am skilled at this difficult conversation (median [IQR])

96 37 30 (4.5-6.5) (4.0-5.0) (2.0-4.0) (4.0-6.0) (4.0-5.0)

78 19 60 (3.5-5.5) (3.5-5.0) (4.0-6.0) (4.0-5.5) (4.0-5.0)

100 52 29 (6.0-7.0) (4.0-6.0) (2.0-4.0) (5.0-7.0) (4.0-6.0)

96 37 30 (6.0-7.0) (4.0-6.0) (2.0-4.0) (5.0-6.0) (4.0-6.0)

5.0 5.0 3.0 5.0 5.0

4.0 4.0 5.0 5.0 4.0

7.0 5.0 2.0 6.0 5.0

7.0 5.0 3.0 6.0 5.0

IQR, inter-quartile range. 312

Journal of Surgical Education  Volume 71/Number 3  May/June 2014

TABLE 4. Comparisons Between Junior and Senior Residents in Standardized Patient-Based Assessments and Self-Assessments of Skills in a 2-Station OSCE Skill Assessment Performance Assessment by Standardized Patients Assessed patient understanding (%) Verbalized a “warning shot” (%) Refrained from using medical jargon (%) Listened attentively (%) Average degree of communication skills (median [IQR]) Average skill of having a “Difficult Conversation” (median [IQR]) Self-Assessment of Skills I am nervous having a “Difficult Conversation” with a patient (median [IQR]) My level of skill in having a “Difficult Conversation” (median [IQR]) Having a “Difficult Conversation” with a patient is appropriate for my level of residency training (median [IQR]) I have been trained to have a “Difficult Conversation” with a patient (median [IQR])

Junior Residents (n ¼ 27)

Senior Residents (n ¼ 27)

37 96 96 96 4.0 (3.5-4.5) 4.0 (3.8-4.5)

89 65 91 89 3.5 (3.0-4.0) 3.5 (2.8-4.0)

5.0 (3.5-5.0) 4.5 (4.0-5.0) 4.5 (4.0-6.0)

3.0 (2.0-4.0) 5.0 (4.0-6.0) 7.0 (6.0-7.0)

4.5 (4.0-5.0)

5.0 (4.0-6.0)

IQR, interquartile range.

for residents in this important core competency domain of Interpersonal and Communication Skills. A checklist of skills from the OSCE would also help faculty give structured feedback and target specific behaviors. A worthwhile and interesting future study would involve defining and exploring some of the barriers surrounding the delivery of communication skill–based feedback to residents by attending physicians. The findings in metacognition and the relationship between self-assessment and SP assessment are important. We found a very small positive correlation between subjective and objective case–specific skills by junior residents on the BBN case. Surprisingly, we found a moderate negative correlation between junior resident selfassessment and SP assessment on the RAB case, indicating that junior residents judged their skills more harshly than SPs. The reason behind this finding is unclear and warrants further evaluation; this likely represents a relative lack of experience with communication skills and structured feedback by junior residents. There were moderate positive correlations between subjective and objective communication skills by senior residents on the GOC and CMO cases. This finding suggests that senior residents are more able to correctly identify their skill level compared with junior residents. This suggests that residents further in residency and with more experience are better able to self-evaluate. Table 4 highlights interesting findings about differences in communication skills between junior and senior residents. We found that senior residents assess patient understanding more than junior residents. This is consistent with our study hypothesis. However, junior residents more commonly verbalized a “warning shot.” Junior residents were also deemed to have a higher degree of communication skills and difficult conversation skills than senior residents; importantly, we cannot conclude that junior residents are more skilled than senior residents because the content of the

cases were different between groups. This, interestingly, directly contradicts our study hypothesis. To determine if this phenomenon is true, junior and senior residents would have to be compared using the same cases and outcome metrics. A possible explanation would be the deterioration of skills after medical school is more pronounced in senior residents compared with the more recently graduated junior residents. This finding supports the conclusion that a progressive communication skill–based curriculum throughout residency would be critically beneficial for general surgery residents and that skills-based training is required even for senior residents. Such comparisons and study of skill retention was a planned future endeavor at our institution. Regarding self-assessment, senior residents thought that having a difficult conversation was more level appropriate, and they were less nervous about having these sorts of conversations than junior residents. Experiential effects are the likely cause of this finding. However, although confidence in communication skills increased with training, performance decreased over time. Both junior and senior residents thought the OSCE cases were realistic and of high quality. Limitations of the study include concerns about generalizability and validity of results. This is a single-institution study, and as a result, these findings may not be generalizable to other general surgery residency programs. The main concern of the outcome measures is that not all of the variables and measures employed have been externally validated. So, it is unclear which conclusions are truly validated by the study. Although 3 of the cases were written with the help of a board-certified general surgeon and a palliative care expert in communication skills, 1 of the cases (BBN) was not written at the University of Pittsburgh. As a result, there is much less overlap in outcome variables for junior residents, whereas the outcome variables were

Journal of Surgical Education  Volume 71/Number 3  May/June 2014

313

identical for senior residents. Additionally, all patient encounters were videotaped. Unfortunately, no validated metrics were analyzed; although some observations would be used for feedback purposes as part of the needs assessment, this data could potentially be used for SP training, for tracking resident performance longitudinally, as well as for evaluating tool validation. Finally, we must be cautious about conclusions drawn using comparisons between junior resident and senior resident performances. As these study participants did not perform the exact same “difficult conversation” OSCE cases using the same SPs, we are unable to conclude with certainty that junior residents outperformed senior residents, despite the same Likertbased outcome variable. Future study of this curriculum should involve these direct comparisons across consistent outcome measures. Strengths of the study include the evaluation of medical education from a skills perspective.18 Residents were also evaluated prospectively, and subsequent effects from educational interventions and communication skills training can be evaluated. Additionally, this study addresses 2 of the ACGME core competencies directly using the validated OSCE modality. There is a relative paucity of metacognition studies looking at subjective and objective communication skills in general surgery residents, and this is the first known study to do so. Moreover, as there is a shift in the methodological evaluation of general surgery residents, this curriculum provides an important method of assessment that may be a useful adjunct for the General Surgery Milestone Project.15 Overall, we conclude that both junior and senior general surgery residents exhibit many favorable nonverbal and verbal communication skills. There are, however, some communication skills for which residents would benefit from a progressive communication skills curriculum throughout residency which includes both didactic and direct observations with feedback. Moreover, subjective and objective OSCE communication skills are correlated. Finally, there is a clear need for skills-based communication training across all residency levels. This study shows some interesting insight into the metacognitive skills of general surgery residents as they relate to the ACGME core competencies of Professionalism and Interpersonal and Communication Skills. Finally, this well-received OSCE technique may be used to observe, document, and provide feedback for these important communication skills.

2. Bell RH. National curricula, certification and creden-

tialing. Surgeon. 2011;9(suppl 1):S10-S11. 3. Stewart MG. Core Competencies. The Accreditation

Council for Graduate Medical Education, 2001. Available at: 〈http://www.acgme.org/acwebsite/RRC_ 280/280_corecomp.asp〉; Accessed December 5, 2012. 4. Hochberg MS, Berman RS, Kalet AL, Zabar SR,

Gillespie C, Pachter HL. The professionalism curriculum as a cultural change agent in surgical residency education. Am J Surg. 2012;203(1):14-20 [Epub 2011 Oct 8]. 5. Larkin AC, Cahan MA, Whalen G, et al. Human

Emotion and Response in Surgery (HEARS): a simulation-based curriculum for communication skills, systems-based practice, and professionalism in surgical residency training. J Am Coll Surg. 2010;211 (2):285-292. 6. Hochberg MS, Kalet A, Zabar S, Kachur E, Gillespie

C, Berman RS. Can professionalism be taught? Encouraging evidence. Am J Surg. 2010;199(1): 86-93. 7. Posner G, Nakajima A. Assessing residents’ commu-

nication skills: disclosure of an adverse event to a standardized patient. J Obstet Gynaecol Can. 2011;33 (3):262-268. 8. Chandawarkar RY, Ruscher KA, Krajewski A, et al.

Pretraining and posttraining assessment of residents’ performance in the fourth accreditation council for graduate medical education competency: patient communication skills. Arch Surg. 2011;146(8):916-921. 9. Yudkowsky R, Alseidi A, Cintron J. Beyond fulfilling

the core competencies: an objective structured clinical examination to assess communication and interpersonal skills in a surgical residency. Curr Surg. 2004;61 (5):499-503. 10. Chipman JG, Beilman GJ, Schmitz CC, Seatter SC.

Development and pilot testing of an OSCE for difficult conversations in surgical intensive care. Surg Educ. 2007;64(2):79-87. 11. Wehbe-Janek H, Song J, Shabahang M. An evaluation

of the usefulness of the standardized patient methodology in the assessment of surgery residents’ communication skills. J Surg Educ. 2011;68(3):172-177. 12. Cohen R, Reznick RK, Taylor BR, Provan J, Rothman

REFERENCES 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next

GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. 314

A. Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg. 1990;160(3):302-305. 13. LeBlanc VR, Tabak D, Kneebone R, Nestel D,

MacRae H, Moulton CA. Psychometric properties of

Journal of Surgical Education  Volume 71/Number 3  May/June 2014

an integrated assessment of technical and communication skills. Am J Surg. 2009;197(1):96-101. 14. Ponton-Carss A, Hutchison C, Violato C. Assessment

of communication, professionalism, and surgical skills in an objective structured performance-related examination (OSPRE): a psychometric study. Am J Surg. 2011;202(4):433-440. 15. General Surgery Milestone Project. The Accreditation

Council for Graduate Medical Education, 2013. Available at: 〈http://www.acgme-nas.org/assets/pdf/ Milestones/GeneralSurgeryMilestones.pdf〉; Accessed April 20, 2013.

16. Falcone JL, Schenarts KD, Ferson PF, Day HD. Using

elements from an acute abdominal pain Objective Structured Clinical Examination (OSCE) leads to more standardized grading in the surgical clerkship for third-year medical students. J Surg Educ. 2011;68 (5):408-413. 17. Back AL, Arnold RM, Baile WF, Tulsky JA, Fryer-

Edwards K. Approaching difficult communication tasks in oncology. CA Cancer J Clin. 2005;55(3):164-177. 18. Kirkpatrick DL. Another Look at Evaluating Training

Programs. Alexandria, VA: American Society for Training & Development; 1998.

Journal of Surgical Education  Volume 71/Number 3  May/June 2014

315