ORIGINAL REPORTS
Portfolios Enhance Clinical Activity in Surgical Clerks Sabine Zundel, PhD,* Gunnar Blumenstock, PhD,† Stephan Zipfel,‡ Anne Herrmann-Werner, PhD,§ and Friederike Holderried, PhD║ Department of Pediatric Surgery, University Hospital of Tuebingen, Germany; †Department of Medical Biometry, University of Tuebingen, Germany; ‡Department of Psychosomatic Medicine, University Hospital of Tuebingen, Germany; §Department of Psychosomatic Medicine and Medical Faculty Tuebingen, Multidisciplinary Skills Lab “DocLab”, University Hospital of Tuebingen, Germany; and ║Department of Internal Medicine, University Hospital of Tuebingen, Germany *
OBJECTIVES: A change in German licensing legislation
imposed a portfolio for surgical clerks. We aimed to analyze whether the implementation of the portfolio changed the amount of clinical exposure and activities during surgical clerkships. DESIGN: The study was conducted with a modified prepost design at the University Hospital of Tuebingen, Germany. Before and after the implementation of the portfolio on April 1, 2013, final-year students (n ¼ 557) who had just finished their surgical clerkship were interviewed with an online questionnaire. A total of 21 basic surgical skills were evaluated. RESULTS: Overall, 230 questionnaires were returned and
analyzed; 51% were preintervention. Overall clinical activity for the whole study cohort varied for different activities between 98% and 32%. For 16 of 21 parameters, there was more clinical activity in the postintervention (portfolio) group. This difference was statistically significant for the following 7 activities: discharge, analgesia, local infiltration, patient positioning, drain in, blood transfusion, and emergency diagnostics. CONCLUSION: The implementation of the portfolio did
enhance clinical activity for surgical clerks in the study cohort. Nevertheless, overall exposure is still unsatisfactory low for some activities. Additional changes and studies are necessary to further improve surgical education. ( J Surg C 2015 Association of Program Directors in ]:]]]-]]]. J Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: portfolio, surgical clerkship, workplace-
based education, surgical teaching COMPETENCIES: Professional, Scholar
Correspondence: Inquiries to Sabine Zundel, PhD, Hoppe-Seyler-Street. 3, 72076 Tuebingen, Germany; fax: (707) 129 4046; e-mail: [email protected]
INTRODUCTION A decline in medical students’ interest in surgical careers has become evident in the past 10 years.1 Departments try to counteract this decline by improving their teaching, as positive experience is regarded to increase students’ interest to pursue a surgical career.2-4 This renewed interest in surgical education has led to an increasing amount of educational research in this field. Resulting studies have revealed significant gaps in clinical exposure during surgical clerkships. An exemplary study stated that only 58% of 116 surveyed students sutured during their surgical clerkship.5 Another study on program variations concluded that curricula from different schools differed greatly and that national standards are necessary to attain a minimal acceptable skill level.6 Owing to an awareness of these issues, a change in German licensing legislation for physicians was executed. Medical schools were asked to create and implement clerkship portfolios. A compulsory minimal standard was agreed on by all 37 medical faculties. This was hoped to establish a homogenous norm and increase clinical exposure by listing learning objectives. Unfortunately, there is little sound evidence regarding the effectiveness of portfolios in students’ education. Existing studies report that portfolios can be used in surgical training in general,7 they present a valuable list of learning objectives,8 and they guide the learning process.9 For postgraduate education, there is good evidence that portfolios increase personal responsibility for learning and support professional development.10 To further analyze the effectiveness of portfolios, we evaluated whether the implementation of our portfolio led to more clinical exposure and more hands-on activities in surgical clerks. We focused on clinical activities because of the recognized gap mentioned previously and owing to the
Journal of Surgical Education & 2015 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2015.03.014
1
fact that an increase in volume of clinical experience is known to enhance students’ clinical skills.11,12
METHODS
miniclerkships in surgical disciplines. No curricular or organizational changes occurred during the time of the survey. There were isolated fluctuations in teaching personnel, but no major changes occurred.
Study Design
Portfolio
The study was planned after the announcement of the upcoming change in licensing regulations. Once the surgical portfolio for the university was drawn up, the questionnaire was created, containing questions about 21 surgical skills, which were components of the portfolio. From November 2012 until March 2013, students who had just finished their surgery rotation were invited to complete a questionnaire about their clinical activity during their clerkship. At the time, the portfolio was not in use, had not been distributed publically, and was not available to the students in any other way. On April 1, 2013, the portfolio was implemented (compulsory for all clerkship students). The same postrotation questionnaire was distributed to all students who finished their clerkships with required portfolios, from August 2013 through April 2014. To rule out possible differences between cohorts, demographic data were gathered and analyzed.
The portfolio was created based on the minimal standard agreed on by the national board. It was supplemented with contents regarded essential or locally important (core competencies of surgical subspecialties located in Tuebingen). The drafting was done by the educational representatives of the 6 surgical specialties and verified by the faculty. The portfolio was distributed in an A6-sized booklet to allow carrying in coat pockets and scrubs. It consists of a list of activities (Table 1) that are to be completed during the clerkship. The clerkship activities are typical performed on entrustment level 3 (trusted to perform activity with indirect supervision). There are boxes to be ticked for each individual activity; free space for additional note-taking is supplemented to facilitate students’ reflection. Students receive regular feedback based on their documentation of activities. Because continuous highquality observation and feedback is not always easily achievable in the workplace setting,13 3 compulsory standardized feedback sessions were implemented and need to be documented in the portfolio booklet. Copies of the portfolio may be obtained from the author. All students received their portfolio on the first day of their clerkship. Its use was explained, and it was clarified that the aim was to complete all listed activities. Once a student had performed an activity successfully, the achievement was documented in the portfolio booklet by a member of the surgical department. Before the implementation of the portfolio, all surgical staff members were informed about the portfolio as a new teaching tool, and its use and handling were explained in detail. A short seminar on how to give feedback was included in the information session. Additional voluntary seminars are continuously offered by continuous professional development staff. The application of the portfolio was supervised by the educational coordinators of each surgical department. All portfolios were checked and signed by the chief teaching coordinator at the end of the surgical rotation.
Population German medical students were target population of our study. The sample population was obtained from the University of Tuebingen. Educational Setting Surgical education differs in the 37 German medical schools, but all schools offer a 6-year curriculum. A compulsory surgical clerkship of 4 months is required at all schools in the sixth year to meet licensing regulations. This clerkship can be performed at the university hospital of the medical school or at associated teaching hospitals. Some students choose to spend their clerkship abroad. To ensure a quality standard, university authorities check and validate individual destinations. The same portfolio is used at all locations. At the University of Tuebingen, surgical education begins in the second year with a seminar on working under sterile conditions, scrubbing, and operating room (OR) procedures. There are skills laboratory seminars on history taking and physical examination in the third year, which are evaluated with an Objective Structured Clinical Examination. Surgical education is continued with lectures from all surgical specialties in the fourth year. Regular workplacebased education is begun in the fifth year, with a training session of 2.5 weeks. Surgical education is completed after the surgical clerkship in the sixth year. Students may additionally choose 4 weeks of electives, seminars, and 2
Data Collection The data collection took place between November 2012 and April 2014. An online, web-based questionnaire was chosen for feasibility reasons and comfort of participants. The online questionnaire was conducted and distributed using the web-based tool “oFb—der onlineFragebogen” from SoSci Survey. All students of the cohorts 2012, 2012/2013, and 2013 received an e-mail with an invitation to participate in the Journal of Surgical Education Volume ]/Number ] ] 2015
TABLE 1. Questionnaire Theme Perioperative management
In the operating room
Activities on the surgical ward
Physical examination
Question
No.
Did you take patients’ histories? Did you do a symptom-orientated physical examination at admission? Did you witness an informed consent for a procedure? Did you witness a discharge? Where you able to prescribe analgesics preoperatively (under supervision) Did you perform a local skin infiltration? Did you do the preoperative skin disinfection? Did you assist in positioning a patient on the table? Did you present an operating field using retractors? Did you do surgical knots (patients or simulation) Did you suture? Did you assist in laying a drain? Did you do a crossmatch for a blood transfusion including the necessary paperwork? Did you assist in doing a blood transfusion? Did you remove a drainage, stent, or splint? Thoracic (16) and abdominal (17), supervision, feedback, and assessment Range of motion Pulse status Glasgow Coma Scale Emergency diagnostics
1 2 3 4 5 6 7 8 9 10 11 12 13
questionnaire (n ¼ 557). A second e-mail with a reminder was sent 1 week after the initial e-mail if the questionnaire had not been opened. Internet protocol addresses were not recorded; the software does not allow tracing of questionnaire responses to the originating e-mail addresses. Personal data of the participants were neither requested nor saved if acquired by chance (direct inquiries by participants). All participants agreed to anonymous publication of data. Serial numbers of e-mail links were recorded to exclude possible numerous participations from individual students. Demographic questions were asked concerning sex, intended career, age, location of clerkship, and use of portfolio. This was followed by 21 multiple-choice questions grouped into the following topics: perioperative management, OR, working on a surgical ward, and physical examination. It was emphasized that only skills performed during the surgical clerkship were relevant. A copy of the questionnaire, with exact phrasing of questions and multiple-choice answers, can be obtained from the author. The questions can be extracted from Table 1. Ethical Approval Ethical approval was obtained by the ethical committee of the Department of Medicine at the University of Tuebingen. Statistical Analysis
14 15 16, 17 18 19 20 21
RESULTS Of 557 questionnaires, 238 were returned; 8 contained no answers and were not included in the analysis. This left 230 sets of data to be analyzed. There was no duplication in serial numbers; therefore, no student answered the questionnaire twice. Overall, 118 students did their clerkship before the implementation of the portfolio, and 112 after intervention. Demographic Data of the Overall Cohort In total, 149 female and 81 male students returned the questionnaire. The percentage of female students in the study represented a sample of the overall number of female students at German medical schools. Among the participants, 36 (15.7%) were planning a career in a surgical discipline, 111 (48.5%) were opting for a nonsurgical discipline, 49 (21.4%) were considering a mixed surgical/ nonsurgical discipline like dermatology or gynecology, and 33 (14.4%) were not decided yet. One participant did not state his preference. The participants’ age in the median was 27 years. Overall, 81 of the students spent their clerkship at the Tuebingen university hospital, 127 at the teaching hospital, and 17 went abroad. Experience of the Overall Cohort (Both Groups, n ¼ 230)
The data were saved in Microsoft Excel and analyzed in SPS Jmp. Chi-square analysis was used to analyze contingency; group sizes were always sufficient. All tests were 2 tailed, and p values less than 0.05 were considered significant.
The overall clinical exposure of clerkship students is demonstrated in Table 2 and Figure 1. The following paragraph describes the activities sorted according to
Journal of Surgical Education Volume ]/Number ] ] 2015
3
Table 2. Overall Clerkship Activity Group
Activity
Perioperative management
Intraoperative
Activities on a surgical ward Specific diagnostics
Performed
%
202 195
89 85
193 154 74 180 201 89 224 168 174 184 180 152 212 199 210 69 206 45 119
84 67 32 79 89 39 98 73 77 81 80 67 94 90 92 31 91 76 52
History taking Physical examination on admission Informed consent Discharge Analgesia Lokal infiltration Patient positioning Desinfection Intraoperative assistance Knoting Suture Drain in Blood match Transfusion Drain ex Thoracic examination Abdominal examination Neutral zero Pulse status GCS Emergency diagnostics
Don't Know
Total N of Answers
Missing Answers
24 33
0 1
226 229
4 1
36 72 152 46 24 138 4 61 49 43 43 74 14 29 18 156 20 171 107
0 2 2 2 1 2 0 0 2 1 2 1 0 0 0 1 0 9 1
229 228 228 228 226 229 228 229 225 228 225 227 226 228 228 226 226 225 227
1 2 2 2 4 1 2 1 5 2 5 3 4 2 2 4 4 5 3
Failed
GCS, Glasgow Coma Scale.
decreasing frequency: There were a number of activities most clerks participated in. These included history taking and physical examination (PE) at admission. Likewise, most students performed a symptom-orientated thoracic and abdominal PE and did a pulse status examination.
However, 7% of students stated they did not have the chance to perform either a thoracic or an abdominal PE. Most students assisted during procedures (98%), removed drains (94%), witnessed the informed consent
250
200
Number of Students
150
100 performed failed 50
don't know
GCS
Emergency Diagnoscs
Puls Status
Neutral Zero
Abdominal examinaon
Drain ex
Thoracic examinaon
Transfusion
Blood Match
Suture
Drain in
Knong
Desinfecon
Intraop Assistance
Paent Posioning
Analgesia
Lokal Infiltraon
Discharge
Informed Consent
History Taking
Physical Examinaon
0
Type of clinical acvity
FIGURE 1. Overall clerkship activity. 4
Journal of Surgical Education Volume ]/Number ] ] 2015
(84%), applied local skin infiltration anesthetics (79%), and observed the discharge of inpatients (67%). Specific involvement in the OR, including surgical knotting (73%), suturing (77%), or assisting in inserting drains (81%), was reported to be achieved by most students. Assisting in positioning the patient on the operating table was a regular activity for 89% of students. Overall, 80% of students did a crossmatch under supervision, but only 67% witnessed a blood transfusion. The Glasgow coma scale as a common diagnostic measure was applied by 76% of students. Moreover, 52% took part in emergency diagnostics. The preoperative disinfection of patients in the OR was reported to be done by only 39% of students. Similarly low was the number of students who participated in the prescription of postoperative analgesia (32%). The neutral zero methods to document the range of movement was performed by 31% of participants. Comparison of Clerkship Activity With and Without Portfolio There were 44 vs 37 male students and 74 vs 75 female students in the nonportfolio group vs portfolio group, respectively; sex differences are not statistically significant. Median age was similar in both cohorts at 27 years. In the nonportfolio group, there were more students
intending to become surgeons (n ¼ 23) than in the portfolio group (n ¼ 13); comparable numbers of students intended on a nonsurgical career (60 in the nonportfolio group vs 51 in the portfolio group) and a mixed specialty (26 in the nonportfolio group vs 23 in the portfolio group). There were more students “undecided” in the portfolio group (n ¼ 25) than in the in nonportfolio group (n ¼ 8). There was a statistically significant difference in the place where students did their clerkship. Within the portfolio group, fewer students were at the university hospital (38 vs 48 in the nonportfolio group) and more at academic teaching hospitals (72 vs 55 in the nonportfolio group). More nonportfolio students spent their clerkship abroad (n ¼ 15). Contingency analysis (distribution of the variables) was done for all tested clinical activities. All findings are demonstrated in Table 3 and Figure 2. There were 4 activities where the nonportfolio students participated in greater numbers: “history taking,” “thoracic PE,” and “pulse status”; the difference was in trend, without being statistically significant. For abdominal PE (96.58 in the nonportfolio group vs 87.39 in the portfolio group), the difference was statistically significant (p ¼ 0.04). Notably, the overall activity for the aforementioned 4 skills was approximately 90%; they represent the activities that were almost always completed.
Table 3. Activity Distributed According to Use of Portfolio Performed (Absolute Numbers) Nonportfolio Enhanced activity in the portfolio group: statistically significant Discharge 70 Analgesia 31 Local infiltration 80 Patient positioning 99 Emergency diagnostics 53 Drain in 89 Transfusion 71 Enhanced activity in the portfolio group: trend Informed consent 98 Desinfection 45 Intraoperative assistance 114 Crossmatch 88 Drain ex 106 Range in motion 30 GCS 20 Knoting 84 Suture 87 Equivocal Physical examination on admission 101 Enhanced activity in the nonportfolio group History taking 107 Thoracic examination 104 Abdominal examination 113 Pulse status 111
Failed
Portfolio
Nonportfolio
Portfolio
p Value
45 43 100 102 66 95 81
84 86 35 17 63 29 46
27 66 11 7 44 14 28
0.02 0.038 0.0001 0.04 0.03 0.02 0.029
95 44 110 92 106 38 25 84 87
20 73 3 26 10 86 94 34 28
16 65 1 17 4 72 77 27 21
0.59 0.73 0.33 0.17 0.12 0.15 0.20 0.44 0.37
94
17
16
0.97
95 95 97 95
10 13 4 6
14 16 14 14
0.29 0.45 0.01 0.04
GCS, Glasgow Coma Scale.
Journal of Surgical Education Volume ]/Number ] ] 2015
5
120.00 100.00 80.00 40.00 20.00
porolio
Significant increase for porolio group
Increase in trend for porolio group
Equivocal
Abdominal examinaon Puls Status
History Taking Thoracic examinaon
Physical Examinaon
Neutral Zero GCS
Cross Match Drain ex
Intraop Assistance Knong Suture
Informed Consent Desinfecon
Transfusion Emergency Diagnoscs
Local Infiltraon Paent Posioning Drain in
0.00 Discharge Analgesia
Number of Students
60.00
no-porolio
Adverse effect
FIGURE 2. Activity distributed according to use of portfolio.
For “physical examination at admission,” the percentages of students in both groups were nearly identical (85.59% in the nonportfolio group vs 85.45% in the portfolio group, p ¼ 0.97). For all remaining parameters (n ¼ 16), there was consistently more clinical activity in the portfolio group. This difference was statistically significant for the following 7 activities: discharge (p ¼ 0.02), analgesia (p ¼ 0.038), local infiltration (achievement 69.5% vs 90%; p ¼ 0.0001), patient positioning (p ¼ 0.04), emergency diagnostics (p ¼ 0.03), drain in (p ¼ 0.02), and blood transfusion (p ¼ 0.029). There was no statistically significant difference when analyzing the activities distributed by location of clerkship.
The use of an online questionnaire proved easily feasible. The technique has a number of strengths. Distribution was easy to handle and enabled partakers to participate with minimal effort. According to the literature, e-mail survey response rates differ from 19% to 61%.14 The response rate of our cohort of 42% is satisfactory and is similar to the evaluation response rates of the university’s permanently ongoing evaluation. High affiliation of students with the topic might be responsible for the response rates in midfield even though no incentive for participation was given. The literature suggests that the recall accuracy of surveys is related to a number of characteristics—appropriate
setting, time provided to answer questions, introduction, and motivation to participate—which is again affected by events happening before the survey. Additionally, supplying possible answers is found to positively influence the recall accuracy.15 These suggestions were incorporated into the questionnaire’s design. With online questionnaires, the setting and timeframe to answer is chosen by participants themselves, which can be assumed as suitable circumstances. The invitations to participate were issued within a week of the end of the surgical rotation to allow affiliation; additionally, the purpose of the study was explained to enhance motivation. Multiple-choice answers were provided. Alternative assessment tools such as personal interviews based on the portfolios or focus groups might have resulted in more multilayered data. These options were discarded in favor of increasing the sample size and for feasibility reasons, as well as the loss of anonymity of the respondent. The option of analyzing the portfolios themselves rather than having students complete follow-up questionnaires regarding their portfolio was discussed during the planning phase of the study. The idea was discarded because it was expected that anonymous questionnaires without any relation to assessment were more likely to reveal genuine data. There is evidence that portfolios are signed off without the task having been completed or that documentation is missed owing to a lack of motivation or time from either students or mentors.16 Nevertheless, all portfolios were checked and signed by one of the authors (S.Z.) in her capacity as the head of surgical teaching representatives. Possible confounding effects of using a modified pre-post design (with a different cohort being asked before and after the intervention) needs to be acknowledged. Possible differences within the 2 study groups were analyzed by comparing demographic data of both subgroups. The data were comparable for all aspects except the fact that more students in the nonportfolio group intended on becoming surgeons. Generally, students intending on becoming surgeons are
6
Journal of Surgical Education Volume ]/Number ] ] 2015
DISCUSSION The results of our study show an enhancement of students’ activity after the implementation of the clerkship portfolio for most activities in the analyzed cohort, with more students participating in the clinical activities expected of clerkship students. Discussion of Method
found to have a higher motivation to participate actively during their surgery rotation. In our results, there were fewer students intending on a surgical career in the portfolio group; nevertheless, we found more clinical activity. Higher motivation is unlikely be responsible for the increase, but may rather be attributed to the implementation of the portfolio. Furthermore, the portfolio group was theoretically able to consult written documentation (their portfolio) of performed activities. It is possible that their data are therefore more accurate. The option of withholding the portfolios at the end-of-clerkship feedback session was initially discussed but discarded. The authors wanted to have their portfolios so that they could be used as a reference during examination preparations. From the experience of the authors, this issue is regarded as a minor confounder. Students value the listed activities as important clinical accomplishments and are most likely to remember. The questionnaire did not analyze the number of times students performed clinical activities. Recollection of these details was presumed to be too vague. Additionally, there is evidence that even a single session of formalized teaching in procedural skills can have long-term effectiveness in basic skills competence and may increase students’ confidence to practice their skills.17
On the contrary, there is 1 activity with equivocal results in both groups, and there are 4 activities with reduced activity in the portfolio group. Uniformly these activities have a high rate of overall achievement (89%-92%). This high rate of achievement might be owing to the facts that they are very basic activities, not specifically surgical, and that these activities are extensively taught before the clerkship. May be there is a ceiling effect, with failure to fulfill these skills in only 10% being incidental. Individual circumstances like a pregnancy and prohibition of patient contact might also be accountable. Nevertheless, the decrease of these activities in the portfolio group is not clearly understood. Larger cohorts and a qualitative research approach (focus groups and interviews) might further evaluate this issue. Reviewing our data and the literature, we believe there are a number of arguments that underline the plausibility of our findings and offer explanations for the success of the portfolios.
Of 21 skills, 16 were achieved more often in the portfolio group. For 7 activities, the increases were statistically significant. The use of portfolios positively influenced students’ clinical activity in the study cohort. Owing to the nature of our data, further explorations on why certain activities showed more increase is not possible. It is but speculation that for some issues students’ were not aware of the issues being learning objectives and that finding them in the portfolio made students aware of them.
1. The portfolio provides students with an explicit list of learning objectives. Many students report on their difficulty to determine learning objectives while learning in workplace-based settings.19,20 Having a list of learning objectives at hand at all times helps the student to focus on relevant tasks and learning objectives. 2. Surgical teachers are also guided by the list of learning objectives. Past studies have reported on inadequate attention to teaching fundamental clinical skills.13 Commonly clerks learn from residents21; these residents usually lack the time to plan their teaching episodes and rarely know details of the catalog of learning objectives.22 The portfolio assists these young teachers to meet their students’ needs. 3. Ideal preparation of feedback sessions entails a more detailed discussion,23 and portfolios enable detailed preparation. Additionally, feedback sessions can be structured according to the makeup of the portfolio and guide the dialog. 4. Usually there are a number of teachers involved in instructing 1 student,13 and the portfolio may function as a source of information for each teacher. By reviewing the portfolio, the teacher may quickly assess the student’s previous achievements. Teaching might become better focused, with an emphasis on the student’s needs. 5. The portfolios are assembled over a period; therefore, they can also be used to support planning and monitoring of students’ development.24 Failure to perform specific activities is revealed more easily, and steps to counteract are possible. 6. The documentation of clinical activities can be regarded as a form of formative assessment and feedback sessions as an oral progress test. It is
Journal of Surgical Education Volume ]/Number ] ] 2015
7
Generalizability The issue of inadequate clinical exposure and failure to develop clinical skills in medical school is prevalent. Our data reveal that only few activities were performed by a high percentage of students. This matches findings from previous studies.5,18 Clerkship settings differ internationally, but the challenges of the workplace-based setting are identical, therefore we argue that the benefit of the use of a portfolio is transferable to other institutions and countries. Expanding our sample population to other German medical schools would have been favorable but failed owing to German privacy protection legislation. Students’ details cannot be shared between universities even for research purposes. Reaching students via postings on individual universities’ homepages was attempted but failed; no questionnaires were returned through these links. Discussion of Results
generally accepted that assessment steers learning, and regular progress tests have shown to improve performance.25 7. Lastly, Goldstein et al. reflected on the decreasing personal relationship between teachers and students.26 The portfolio gives students reason and opportunity to approach their teachers.
CONCLUSION Our data suggest that the change in German licensing regulations fulfilled the set goal of enhancing clinical activity in surgical clerks. Although the data show a significant increase for a number of activities, the overall number of students failing to perform these activities remains alarmingly high. Further measures need to be undertaken to further enhance clinical activity in students. Additionally, the German portfolio needs to be evaluated continuously to ensure the persistence of the reported improvements.
ACKNOWLEDGMENT
7. Amsellem-Ouazana D, Van Pee D, Godin V. Use of
portfolios as a learning and assessment tool in a surgical practical session of urology during undergraduate medical training. Med Teach. 2006;28(4):356-359. 8. Froehlich S, Kasch R, Schwanitz P, et al. Logbook of
learning targets for special educational skills in orthopaedic and trauma surgery for undergraduate medical training. Z Orthop Unfall. 2013;151(6):610-631. 9. Sanchez Gomez S, Ostos EM, Solano JM, Salado TF.
An electronic portfolio for quantitative assessment of surgical skills in undergraduate medical education. BMC Med Educ. 2013;13:65. 10. Tochel C, Haig A, Hesketh A, et al. The effectiveness of
portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 2009;31(4):299-318. 11. Chatenay M, Maguire T, Skakun E, Chang G, Cook
D, Warnock GL. Does volume of clinical experience affect performance of clinical clerks on surgery exit examinations? Am J Surg. 1996;172(4):366-372.
We would like to thank Beth Ryder, a surgeon, for proofreading the manuscript.
12. Lind DS, Marum T, Ledbetter D, Flynn TC, Romrell
REFERENCES
13. Holmboe ES. Faculty and the observation of trainees’
1. Cochran A, Paukert JL, Neumayer LA. Does a general
surgery clerkship influence student perceptions of surgeons and surgical careers? Surgery. 2003;134(2):153-157. 2. Al-Heeti KN, Nassar AK, Decorby K, Winch J,
Reid S. The effect of general surgery clerkship rotation on the attitude of medical students towards general surgery as a future career. J Surg Educ. 2012;69(4): 544-549. 3. Berman L, Rosenthal MS, Curry LA, Evans LV,
Gusberg RJ. Attracting surgical clerks to surgical careers: role models, mentoring, and engagement in the operating room. J Am Coll Surg. 2008;207(6): 793-800, 800 e791-792. 4. Ekenze SO, Ugwumba FO, Obi UM, Ekenze OS.
clinical skills: problems and opportunities. Acad Med. 2004;79(1):16-22. 14. Sheehan KB. E-mail survey response rates: a review. J
Comput Mediated Commun. 2001;6(2):0. 15. Coughlin SS. Recall bias in epidemiologic studies. J
Clin Epidemiol. 1990;43(1):87-91. 16. Heidelberg
MP-U, Heidelberg DSS-UH. Das interaktive chirurgische logbuch im praktischen jahr: eine mehrjährige retrospektive interactive logbook in the surgical training of final year medical students: a fouryear retrospective. GMS Z Med Ausbildung. 2009;26:2.
17. Liddell MJ, Davidson SK, Taub H, Whitecross LE.
Evaluation of procedural skills training in an undergraduate curriculum. Med Educ. 2002;36(11):1035-1041.
Undergraduate surgery clerkship and the choice of surgery as a career: perspective from a developing country. World J Surg. 2013;37(9):2094-2100.
18. Engum SA. Do you know your students’ basic clinical
5. Ladak A, Hanson J, de Gara CJ. What procedures are
Cleland J, Yule S. Undergraduate medical students’ perceptions and expectations of theatre-based learning: how can we improve the student learning experience? Surgeon. 2007;5(5):271-274.
students doing during undergraduate surgical clerkship? Can J Surg. 2006;49(5):329-334. 6. Forbes SS, Fitzgerald PG, Birch DW. Undergraduate
surgical training: variations in program objectives and curriculum implementation across Canada. Can J Surg. 2006;49(1):46-50. 8
LJ, Copeland EM 3rd. The effect of the duration and structure of a surgery clerkship on student performance. J Surg Res. 1999;84(1):106-111.
skills exposure? Am J Surg. 2003;186(2):175-181. 19. Fernando N, McAdam T, Youngson G, McKenzie H,
20. Lee MS, Montague ML, Hussain SS. Student-
perceived benefit from otolaryngology theatre attendance. J Laryngol Otol. 2005;119(4):293-298. Journal of Surgical Education Volume ]/Number ] ] 2015
21. Karani R, Fromme HB, Cayea D, Muller D, Schwartz
24. Van Tartwijk J, Driessen EW. Portfolios for assess-
A, Harris IB. How medical students learn from residents in the workplace: a qualitative study. Acad Med. 2014;89(3):490-496.
ment and learning: AMEE Guide no. 45. Med Teach. 2009;31(9):790-801.
22. Rotenberg
BW, Woodhouse RA, Gilbart M, Hutchison CR. A needs assessment of surgical residents as teachers. Can J Surg. 2000;43(4): 295.
23. Champagne BJ. Effective teaching and feedback strat-
25. Norman G, Neville A, Blake JM, Mueller B. Assess-
ment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach. 2010;32(6):496-499. 26. Goldstein EA, MacLaren CF, Smith S, et al. Promot-
egies in the OR and beyond. Clin Colon Rectal Surg. 2013;26(4):244-249.
ing fundamental clinical skills: a competency-based college approach at the University of Washington. Acad Med. 2005;80(5):423-433.
Journal of Surgical Education Volume ]/Number ] ] 2015
9
Portfolios Enhance Clinical Activity in Surgical Clerks Sabine Zundel, PhD,* Gunnar Blumenstock, PhD,† Stephan Zipfel,‡ Anne Herrmann-Werner, PhD,§ and Friederike Holderried, PhD║ Department of Pediatric Surgery, University Hospital of Tuebingen, Germany; †Department of Medical Biometry, University of Tuebingen, Germany; ‡Department of Psychosomatic Medicine, University Hospital of Tuebingen, Germany; §Department of Psychosomatic Medicine and Medical Faculty Tuebingen, Multidisciplinary Skills Lab “DocLab”, University Hospital of Tuebingen, Germany; and ║Department of Internal Medicine, University Hospital of Tuebingen, Germany *
OBJECTIVES: A change in German licensing legislation
imposed a portfolio for surgical clerks. We aimed to analyze whether the implementation of the portfolio changed the amount of clinical exposure and activities during surgical clerkships. DESIGN: The study was conducted with a modified prepost design at the University Hospital of Tuebingen, Germany. Before and after the implementation of the portfolio on April 1, 2013, final-year students (n ¼ 557) who had just finished their surgical clerkship were interviewed with an online questionnaire. A total of 21 basic surgical skills were evaluated. RESULTS: Overall, 230 questionnaires were returned and
analyzed; 51% were preintervention. Overall clinical activity for the whole study cohort varied for different activities between 98% and 32%. For 16 of 21 parameters, there was more clinical activity in the postintervention (portfolio) group. This difference was statistically significant for the following 7 activities: discharge, analgesia, local infiltration, patient positioning, drain in, blood transfusion, and emergency diagnostics. CONCLUSION: The implementation of the portfolio did
enhance clinical activity for surgical clerks in the study cohort. Nevertheless, overall exposure is still unsatisfactory low for some activities. Additional changes and studies are necessary to further improve surgical education. ( J Surg C 2015 Association of Program Directors in ]:]]]-]]]. J Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: portfolio, surgical clerkship, workplace-
based education, surgical teaching COMPETENCIES: Professional, Scholar
Correspondence: Inquiries to Sabine Zundel, PhD, Hoppe-Seyler-Street. 3, 72076 Tuebingen, Germany; fax: (707) 129 4046; e-mail: [email protected]
INTRODUCTION A decline in medical students’ interest in surgical careers has become evident in the past 10 years.1 Departments try to counteract this decline by improving their teaching, as positive experience is regarded to increase students’ interest to pursue a surgical career.2-4 This renewed interest in surgical education has led to an increasing amount of educational research in this field. Resulting studies have revealed significant gaps in clinical exposure during surgical clerkships. An exemplary study stated that only 58% of 116 surveyed students sutured during their surgical clerkship.5 Another study on program variations concluded that curricula from different schools differed greatly and that national standards are necessary to attain a minimal acceptable skill level.6 Owing to an awareness of these issues, a change in German licensing legislation for physicians was executed. Medical schools were asked to create and implement clerkship portfolios. A compulsory minimal standard was agreed on by all 37 medical faculties. This was hoped to establish a homogenous norm and increase clinical exposure by listing learning objectives. Unfortunately, there is little sound evidence regarding the effectiveness of portfolios in students’ education. Existing studies report that portfolios can be used in surgical training in general,7 they present a valuable list of learning objectives,8 and they guide the learning process.9 For postgraduate education, there is good evidence that portfolios increase personal responsibility for learning and support professional development.10 To further analyze the effectiveness of portfolios, we evaluated whether the implementation of our portfolio led to more clinical exposure and more hands-on activities in surgical clerks. We focused on clinical activities because of the recognized gap mentioned previously and owing to the
Journal of Surgical Education & 2015 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2015.03.014
1
fact that an increase in volume of clinical experience is known to enhance students’ clinical skills.11,12
METHODS
miniclerkships in surgical disciplines. No curricular or organizational changes occurred during the time of the survey. There were isolated fluctuations in teaching personnel, but no major changes occurred.
Study Design
Portfolio
The study was planned after the announcement of the upcoming change in licensing regulations. Once the surgical portfolio for the university was drawn up, the questionnaire was created, containing questions about 21 surgical skills, which were components of the portfolio. From November 2012 until March 2013, students who had just finished their surgery rotation were invited to complete a questionnaire about their clinical activity during their clerkship. At the time, the portfolio was not in use, had not been distributed publically, and was not available to the students in any other way. On April 1, 2013, the portfolio was implemented (compulsory for all clerkship students). The same postrotation questionnaire was distributed to all students who finished their clerkships with required portfolios, from August 2013 through April 2014. To rule out possible differences between cohorts, demographic data were gathered and analyzed.
The portfolio was created based on the minimal standard agreed on by the national board. It was supplemented with contents regarded essential or locally important (core competencies of surgical subspecialties located in Tuebingen). The drafting was done by the educational representatives of the 6 surgical specialties and verified by the faculty. The portfolio was distributed in an A6-sized booklet to allow carrying in coat pockets and scrubs. It consists of a list of activities (Table 1) that are to be completed during the clerkship. The clerkship activities are typical performed on entrustment level 3 (trusted to perform activity with indirect supervision). There are boxes to be ticked for each individual activity; free space for additional note-taking is supplemented to facilitate students’ reflection. Students receive regular feedback based on their documentation of activities. Because continuous highquality observation and feedback is not always easily achievable in the workplace setting,13 3 compulsory standardized feedback sessions were implemented and need to be documented in the portfolio booklet. Copies of the portfolio may be obtained from the author. All students received their portfolio on the first day of their clerkship. Its use was explained, and it was clarified that the aim was to complete all listed activities. Once a student had performed an activity successfully, the achievement was documented in the portfolio booklet by a member of the surgical department. Before the implementation of the portfolio, all surgical staff members were informed about the portfolio as a new teaching tool, and its use and handling were explained in detail. A short seminar on how to give feedback was included in the information session. Additional voluntary seminars are continuously offered by continuous professional development staff. The application of the portfolio was supervised by the educational coordinators of each surgical department. All portfolios were checked and signed by the chief teaching coordinator at the end of the surgical rotation.
Population German medical students were target population of our study. The sample population was obtained from the University of Tuebingen. Educational Setting Surgical education differs in the 37 German medical schools, but all schools offer a 6-year curriculum. A compulsory surgical clerkship of 4 months is required at all schools in the sixth year to meet licensing regulations. This clerkship can be performed at the university hospital of the medical school or at associated teaching hospitals. Some students choose to spend their clerkship abroad. To ensure a quality standard, university authorities check and validate individual destinations. The same portfolio is used at all locations. At the University of Tuebingen, surgical education begins in the second year with a seminar on working under sterile conditions, scrubbing, and operating room (OR) procedures. There are skills laboratory seminars on history taking and physical examination in the third year, which are evaluated with an Objective Structured Clinical Examination. Surgical education is continued with lectures from all surgical specialties in the fourth year. Regular workplacebased education is begun in the fifth year, with a training session of 2.5 weeks. Surgical education is completed after the surgical clerkship in the sixth year. Students may additionally choose 4 weeks of electives, seminars, and 2
Data Collection The data collection took place between November 2012 and April 2014. An online, web-based questionnaire was chosen for feasibility reasons and comfort of participants. The online questionnaire was conducted and distributed using the web-based tool “oFb—der onlineFragebogen” from SoSci Survey. All students of the cohorts 2012, 2012/2013, and 2013 received an e-mail with an invitation to participate in the Journal of Surgical Education Volume ]/Number ] ] 2015
TABLE 1. Questionnaire Theme Perioperative management
In the operating room
Activities on the surgical ward
Physical examination
Question
No.
Did you take patients’ histories? Did you do a symptom-orientated physical examination at admission? Did you witness an informed consent for a procedure? Did you witness a discharge? Where you able to prescribe analgesics preoperatively (under supervision) Did you perform a local skin infiltration? Did you do the preoperative skin disinfection? Did you assist in positioning a patient on the table? Did you present an operating field using retractors? Did you do surgical knots (patients or simulation) Did you suture? Did you assist in laying a drain? Did you do a crossmatch for a blood transfusion including the necessary paperwork? Did you assist in doing a blood transfusion? Did you remove a drainage, stent, or splint? Thoracic (16) and abdominal (17), supervision, feedback, and assessment Range of motion Pulse status Glasgow Coma Scale Emergency diagnostics
1 2 3 4 5 6 7 8 9 10 11 12 13
questionnaire (n ¼ 557). A second e-mail with a reminder was sent 1 week after the initial e-mail if the questionnaire had not been opened. Internet protocol addresses were not recorded; the software does not allow tracing of questionnaire responses to the originating e-mail addresses. Personal data of the participants were neither requested nor saved if acquired by chance (direct inquiries by participants). All participants agreed to anonymous publication of data. Serial numbers of e-mail links were recorded to exclude possible numerous participations from individual students. Demographic questions were asked concerning sex, intended career, age, location of clerkship, and use of portfolio. This was followed by 21 multiple-choice questions grouped into the following topics: perioperative management, OR, working on a surgical ward, and physical examination. It was emphasized that only skills performed during the surgical clerkship were relevant. A copy of the questionnaire, with exact phrasing of questions and multiple-choice answers, can be obtained from the author. The questions can be extracted from Table 1. Ethical Approval Ethical approval was obtained by the ethical committee of the Department of Medicine at the University of Tuebingen. Statistical Analysis
14 15 16, 17 18 19 20 21
RESULTS Of 557 questionnaires, 238 were returned; 8 contained no answers and were not included in the analysis. This left 230 sets of data to be analyzed. There was no duplication in serial numbers; therefore, no student answered the questionnaire twice. Overall, 118 students did their clerkship before the implementation of the portfolio, and 112 after intervention. Demographic Data of the Overall Cohort In total, 149 female and 81 male students returned the questionnaire. The percentage of female students in the study represented a sample of the overall number of female students at German medical schools. Among the participants, 36 (15.7%) were planning a career in a surgical discipline, 111 (48.5%) were opting for a nonsurgical discipline, 49 (21.4%) were considering a mixed surgical/ nonsurgical discipline like dermatology or gynecology, and 33 (14.4%) were not decided yet. One participant did not state his preference. The participants’ age in the median was 27 years. Overall, 81 of the students spent their clerkship at the Tuebingen university hospital, 127 at the teaching hospital, and 17 went abroad. Experience of the Overall Cohort (Both Groups, n ¼ 230)
The data were saved in Microsoft Excel and analyzed in SPS Jmp. Chi-square analysis was used to analyze contingency; group sizes were always sufficient. All tests were 2 tailed, and p values less than 0.05 were considered significant.
The overall clinical exposure of clerkship students is demonstrated in Table 2 and Figure 1. The following paragraph describes the activities sorted according to
Journal of Surgical Education Volume ]/Number ] ] 2015
3
Table 2. Overall Clerkship Activity Group
Activity
Perioperative management
Intraoperative
Activities on a surgical ward Specific diagnostics
Performed
%
202 195
89 85
193 154 74 180 201 89 224 168 174 184 180 152 212 199 210 69 206 45 119
84 67 32 79 89 39 98 73 77 81 80 67 94 90 92 31 91 76 52
History taking Physical examination on admission Informed consent Discharge Analgesia Lokal infiltration Patient positioning Desinfection Intraoperative assistance Knoting Suture Drain in Blood match Transfusion Drain ex Thoracic examination Abdominal examination Neutral zero Pulse status GCS Emergency diagnostics
Don't Know
Total N of Answers
Missing Answers
24 33
0 1
226 229
4 1
36 72 152 46 24 138 4 61 49 43 43 74 14 29 18 156 20 171 107
0 2 2 2 1 2 0 0 2 1 2 1 0 0 0 1 0 9 1
229 228 228 228 226 229 228 229 225 228 225 227 226 228 228 226 226 225 227
1 2 2 2 4 1 2 1 5 2 5 3 4 2 2 4 4 5 3
Failed
GCS, Glasgow Coma Scale.
decreasing frequency: There were a number of activities most clerks participated in. These included history taking and physical examination (PE) at admission. Likewise, most students performed a symptom-orientated thoracic and abdominal PE and did a pulse status examination.
However, 7% of students stated they did not have the chance to perform either a thoracic or an abdominal PE. Most students assisted during procedures (98%), removed drains (94%), witnessed the informed consent
250
200
Number of Students
150
100 performed failed 50
don't know
GCS
Emergency Diagnoscs
Puls Status
Neutral Zero
Abdominal examinaon
Drain ex
Thoracic examinaon
Transfusion
Blood Match
Suture
Drain in
Knong
Desinfecon
Intraop Assistance
Paent Posioning
Analgesia
Lokal Infiltraon
Discharge
Informed Consent
History Taking
Physical Examinaon
0
Type of clinical acvity
FIGURE 1. Overall clerkship activity. 4
Journal of Surgical Education Volume ]/Number ] ] 2015
(84%), applied local skin infiltration anesthetics (79%), and observed the discharge of inpatients (67%). Specific involvement in the OR, including surgical knotting (73%), suturing (77%), or assisting in inserting drains (81%), was reported to be achieved by most students. Assisting in positioning the patient on the operating table was a regular activity for 89% of students. Overall, 80% of students did a crossmatch under supervision, but only 67% witnessed a blood transfusion. The Glasgow coma scale as a common diagnostic measure was applied by 76% of students. Moreover, 52% took part in emergency diagnostics. The preoperative disinfection of patients in the OR was reported to be done by only 39% of students. Similarly low was the number of students who participated in the prescription of postoperative analgesia (32%). The neutral zero methods to document the range of movement was performed by 31% of participants. Comparison of Clerkship Activity With and Without Portfolio There were 44 vs 37 male students and 74 vs 75 female students in the nonportfolio group vs portfolio group, respectively; sex differences are not statistically significant. Median age was similar in both cohorts at 27 years. In the nonportfolio group, there were more students
intending to become surgeons (n ¼ 23) than in the portfolio group (n ¼ 13); comparable numbers of students intended on a nonsurgical career (60 in the nonportfolio group vs 51 in the portfolio group) and a mixed specialty (26 in the nonportfolio group vs 23 in the portfolio group). There were more students “undecided” in the portfolio group (n ¼ 25) than in the in nonportfolio group (n ¼ 8). There was a statistically significant difference in the place where students did their clerkship. Within the portfolio group, fewer students were at the university hospital (38 vs 48 in the nonportfolio group) and more at academic teaching hospitals (72 vs 55 in the nonportfolio group). More nonportfolio students spent their clerkship abroad (n ¼ 15). Contingency analysis (distribution of the variables) was done for all tested clinical activities. All findings are demonstrated in Table 3 and Figure 2. There were 4 activities where the nonportfolio students participated in greater numbers: “history taking,” “thoracic PE,” and “pulse status”; the difference was in trend, without being statistically significant. For abdominal PE (96.58 in the nonportfolio group vs 87.39 in the portfolio group), the difference was statistically significant (p ¼ 0.04). Notably, the overall activity for the aforementioned 4 skills was approximately 90%; they represent the activities that were almost always completed.
Table 3. Activity Distributed According to Use of Portfolio Performed (Absolute Numbers) Nonportfolio Enhanced activity in the portfolio group: statistically significant Discharge 70 Analgesia 31 Local infiltration 80 Patient positioning 99 Emergency diagnostics 53 Drain in 89 Transfusion 71 Enhanced activity in the portfolio group: trend Informed consent 98 Desinfection 45 Intraoperative assistance 114 Crossmatch 88 Drain ex 106 Range in motion 30 GCS 20 Knoting 84 Suture 87 Equivocal Physical examination on admission 101 Enhanced activity in the nonportfolio group History taking 107 Thoracic examination 104 Abdominal examination 113 Pulse status 111
Failed
Portfolio
Nonportfolio
Portfolio
p Value
45 43 100 102 66 95 81
84 86 35 17 63 29 46
27 66 11 7 44 14 28
0.02 0.038 0.0001 0.04 0.03 0.02 0.029
95 44 110 92 106 38 25 84 87
20 73 3 26 10 86 94 34 28
16 65 1 17 4 72 77 27 21
0.59 0.73 0.33 0.17 0.12 0.15 0.20 0.44 0.37
94
17
16
0.97
95 95 97 95
10 13 4 6
14 16 14 14
0.29 0.45 0.01 0.04
GCS, Glasgow Coma Scale.
Journal of Surgical Education Volume ]/Number ] ] 2015
5
120.00 100.00 80.00 40.00 20.00
porolio
Significant increase for porolio group
Increase in trend for porolio group
Equivocal
Abdominal examinaon Puls Status
History Taking Thoracic examinaon
Physical Examinaon
Neutral Zero GCS
Cross Match Drain ex
Intraop Assistance Knong Suture
Informed Consent Desinfecon
Transfusion Emergency Diagnoscs
Local Infiltraon Paent Posioning Drain in
0.00 Discharge Analgesia
Number of Students
60.00
no-porolio
Adverse effect
FIGURE 2. Activity distributed according to use of portfolio.
For “physical examination at admission,” the percentages of students in both groups were nearly identical (85.59% in the nonportfolio group vs 85.45% in the portfolio group, p ¼ 0.97). For all remaining parameters (n ¼ 16), there was consistently more clinical activity in the portfolio group. This difference was statistically significant for the following 7 activities: discharge (p ¼ 0.02), analgesia (p ¼ 0.038), local infiltration (achievement 69.5% vs 90%; p ¼ 0.0001), patient positioning (p ¼ 0.04), emergency diagnostics (p ¼ 0.03), drain in (p ¼ 0.02), and blood transfusion (p ¼ 0.029). There was no statistically significant difference when analyzing the activities distributed by location of clerkship.
The use of an online questionnaire proved easily feasible. The technique has a number of strengths. Distribution was easy to handle and enabled partakers to participate with minimal effort. According to the literature, e-mail survey response rates differ from 19% to 61%.14 The response rate of our cohort of 42% is satisfactory and is similar to the evaluation response rates of the university’s permanently ongoing evaluation. High affiliation of students with the topic might be responsible for the response rates in midfield even though no incentive for participation was given. The literature suggests that the recall accuracy of surveys is related to a number of characteristics—appropriate
setting, time provided to answer questions, introduction, and motivation to participate—which is again affected by events happening before the survey. Additionally, supplying possible answers is found to positively influence the recall accuracy.15 These suggestions were incorporated into the questionnaire’s design. With online questionnaires, the setting and timeframe to answer is chosen by participants themselves, which can be assumed as suitable circumstances. The invitations to participate were issued within a week of the end of the surgical rotation to allow affiliation; additionally, the purpose of the study was explained to enhance motivation. Multiple-choice answers were provided. Alternative assessment tools such as personal interviews based on the portfolios or focus groups might have resulted in more multilayered data. These options were discarded in favor of increasing the sample size and for feasibility reasons, as well as the loss of anonymity of the respondent. The option of analyzing the portfolios themselves rather than having students complete follow-up questionnaires regarding their portfolio was discussed during the planning phase of the study. The idea was discarded because it was expected that anonymous questionnaires without any relation to assessment were more likely to reveal genuine data. There is evidence that portfolios are signed off without the task having been completed or that documentation is missed owing to a lack of motivation or time from either students or mentors.16 Nevertheless, all portfolios were checked and signed by one of the authors (S.Z.) in her capacity as the head of surgical teaching representatives. Possible confounding effects of using a modified pre-post design (with a different cohort being asked before and after the intervention) needs to be acknowledged. Possible differences within the 2 study groups were analyzed by comparing demographic data of both subgroups. The data were comparable for all aspects except the fact that more students in the nonportfolio group intended on becoming surgeons. Generally, students intending on becoming surgeons are
6
Journal of Surgical Education Volume ]/Number ] ] 2015
DISCUSSION The results of our study show an enhancement of students’ activity after the implementation of the clerkship portfolio for most activities in the analyzed cohort, with more students participating in the clinical activities expected of clerkship students. Discussion of Method
found to have a higher motivation to participate actively during their surgery rotation. In our results, there were fewer students intending on a surgical career in the portfolio group; nevertheless, we found more clinical activity. Higher motivation is unlikely be responsible for the increase, but may rather be attributed to the implementation of the portfolio. Furthermore, the portfolio group was theoretically able to consult written documentation (their portfolio) of performed activities. It is possible that their data are therefore more accurate. The option of withholding the portfolios at the end-of-clerkship feedback session was initially discussed but discarded. The authors wanted to have their portfolios so that they could be used as a reference during examination preparations. From the experience of the authors, this issue is regarded as a minor confounder. Students value the listed activities as important clinical accomplishments and are most likely to remember. The questionnaire did not analyze the number of times students performed clinical activities. Recollection of these details was presumed to be too vague. Additionally, there is evidence that even a single session of formalized teaching in procedural skills can have long-term effectiveness in basic skills competence and may increase students’ confidence to practice their skills.17
On the contrary, there is 1 activity with equivocal results in both groups, and there are 4 activities with reduced activity in the portfolio group. Uniformly these activities have a high rate of overall achievement (89%-92%). This high rate of achievement might be owing to the facts that they are very basic activities, not specifically surgical, and that these activities are extensively taught before the clerkship. May be there is a ceiling effect, with failure to fulfill these skills in only 10% being incidental. Individual circumstances like a pregnancy and prohibition of patient contact might also be accountable. Nevertheless, the decrease of these activities in the portfolio group is not clearly understood. Larger cohorts and a qualitative research approach (focus groups and interviews) might further evaluate this issue. Reviewing our data and the literature, we believe there are a number of arguments that underline the plausibility of our findings and offer explanations for the success of the portfolios.
Of 21 skills, 16 were achieved more often in the portfolio group. For 7 activities, the increases were statistically significant. The use of portfolios positively influenced students’ clinical activity in the study cohort. Owing to the nature of our data, further explorations on why certain activities showed more increase is not possible. It is but speculation that for some issues students’ were not aware of the issues being learning objectives and that finding them in the portfolio made students aware of them.
1. The portfolio provides students with an explicit list of learning objectives. Many students report on their difficulty to determine learning objectives while learning in workplace-based settings.19,20 Having a list of learning objectives at hand at all times helps the student to focus on relevant tasks and learning objectives. 2. Surgical teachers are also guided by the list of learning objectives. Past studies have reported on inadequate attention to teaching fundamental clinical skills.13 Commonly clerks learn from residents21; these residents usually lack the time to plan their teaching episodes and rarely know details of the catalog of learning objectives.22 The portfolio assists these young teachers to meet their students’ needs. 3. Ideal preparation of feedback sessions entails a more detailed discussion,23 and portfolios enable detailed preparation. Additionally, feedback sessions can be structured according to the makeup of the portfolio and guide the dialog. 4. Usually there are a number of teachers involved in instructing 1 student,13 and the portfolio may function as a source of information for each teacher. By reviewing the portfolio, the teacher may quickly assess the student’s previous achievements. Teaching might become better focused, with an emphasis on the student’s needs. 5. The portfolios are assembled over a period; therefore, they can also be used to support planning and monitoring of students’ development.24 Failure to perform specific activities is revealed more easily, and steps to counteract are possible. 6. The documentation of clinical activities can be regarded as a form of formative assessment and feedback sessions as an oral progress test. It is
Journal of Surgical Education Volume ]/Number ] ] 2015
7
Generalizability The issue of inadequate clinical exposure and failure to develop clinical skills in medical school is prevalent. Our data reveal that only few activities were performed by a high percentage of students. This matches findings from previous studies.5,18 Clerkship settings differ internationally, but the challenges of the workplace-based setting are identical, therefore we argue that the benefit of the use of a portfolio is transferable to other institutions and countries. Expanding our sample population to other German medical schools would have been favorable but failed owing to German privacy protection legislation. Students’ details cannot be shared between universities even for research purposes. Reaching students via postings on individual universities’ homepages was attempted but failed; no questionnaires were returned through these links. Discussion of Results
generally accepted that assessment steers learning, and regular progress tests have shown to improve performance.25 7. Lastly, Goldstein et al. reflected on the decreasing personal relationship between teachers and students.26 The portfolio gives students reason and opportunity to approach their teachers.
CONCLUSION Our data suggest that the change in German licensing regulations fulfilled the set goal of enhancing clinical activity in surgical clerks. Although the data show a significant increase for a number of activities, the overall number of students failing to perform these activities remains alarmingly high. Further measures need to be undertaken to further enhance clinical activity in students. Additionally, the German portfolio needs to be evaluated continuously to ensure the persistence of the reported improvements.
ACKNOWLEDGMENT
7. Amsellem-Ouazana D, Van Pee D, Godin V. Use of
portfolios as a learning and assessment tool in a surgical practical session of urology during undergraduate medical training. Med Teach. 2006;28(4):356-359. 8. Froehlich S, Kasch R, Schwanitz P, et al. Logbook of
learning targets for special educational skills in orthopaedic and trauma surgery for undergraduate medical training. Z Orthop Unfall. 2013;151(6):610-631. 9. Sanchez Gomez S, Ostos EM, Solano JM, Salado TF.
An electronic portfolio for quantitative assessment of surgical skills in undergraduate medical education. BMC Med Educ. 2013;13:65. 10. Tochel C, Haig A, Hesketh A, et al. The effectiveness of
portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 2009;31(4):299-318. 11. Chatenay M, Maguire T, Skakun E, Chang G, Cook
D, Warnock GL. Does volume of clinical experience affect performance of clinical clerks on surgery exit examinations? Am J Surg. 1996;172(4):366-372.
We would like to thank Beth Ryder, a surgeon, for proofreading the manuscript.
12. Lind DS, Marum T, Ledbetter D, Flynn TC, Romrell
REFERENCES
13. Holmboe ES. Faculty and the observation of trainees’
1. Cochran A, Paukert JL, Neumayer LA. Does a general
surgery clerkship influence student perceptions of surgeons and surgical careers? Surgery. 2003;134(2):153-157. 2. Al-Heeti KN, Nassar AK, Decorby K, Winch J,
Reid S. The effect of general surgery clerkship rotation on the attitude of medical students towards general surgery as a future career. J Surg Educ. 2012;69(4): 544-549. 3. Berman L, Rosenthal MS, Curry LA, Evans LV,
Gusberg RJ. Attracting surgical clerks to surgical careers: role models, mentoring, and engagement in the operating room. J Am Coll Surg. 2008;207(6): 793-800, 800 e791-792. 4. Ekenze SO, Ugwumba FO, Obi UM, Ekenze OS.
clinical skills: problems and opportunities. Acad Med. 2004;79(1):16-22. 14. Sheehan KB. E-mail survey response rates: a review. J
Comput Mediated Commun. 2001;6(2):0. 15. Coughlin SS. Recall bias in epidemiologic studies. J
Clin Epidemiol. 1990;43(1):87-91. 16. Heidelberg
MP-U, Heidelberg DSS-UH. Das interaktive chirurgische logbuch im praktischen jahr: eine mehrjährige retrospektive interactive logbook in the surgical training of final year medical students: a fouryear retrospective. GMS Z Med Ausbildung. 2009;26:2.
17. Liddell MJ, Davidson SK, Taub H, Whitecross LE.
Evaluation of procedural skills training in an undergraduate curriculum. Med Educ. 2002;36(11):1035-1041.
Undergraduate surgery clerkship and the choice of surgery as a career: perspective from a developing country. World J Surg. 2013;37(9):2094-2100.
18. Engum SA. Do you know your students’ basic clinical
5. Ladak A, Hanson J, de Gara CJ. What procedures are
Cleland J, Yule S. Undergraduate medical students’ perceptions and expectations of theatre-based learning: how can we improve the student learning experience? Surgeon. 2007;5(5):271-274.
students doing during undergraduate surgical clerkship? Can J Surg. 2006;49(5):329-334. 6. Forbes SS, Fitzgerald PG, Birch DW. Undergraduate
surgical training: variations in program objectives and curriculum implementation across Canada. Can J Surg. 2006;49(1):46-50. 8
LJ, Copeland EM 3rd. The effect of the duration and structure of a surgery clerkship on student performance. J Surg Res. 1999;84(1):106-111.
skills exposure? Am J Surg. 2003;186(2):175-181. 19. Fernando N, McAdam T, Youngson G, McKenzie H,
20. Lee MS, Montague ML, Hussain SS. Student-
perceived benefit from otolaryngology theatre attendance. J Laryngol Otol. 2005;119(4):293-298. Journal of Surgical Education Volume ]/Number ] ] 2015
21. Karani R, Fromme HB, Cayea D, Muller D, Schwartz
24. Van Tartwijk J, Driessen EW. Portfolios for assess-
A, Harris IB. How medical students learn from residents in the workplace: a qualitative study. Acad Med. 2014;89(3):490-496.
ment and learning: AMEE Guide no. 45. Med Teach. 2009;31(9):790-801.
22. Rotenberg
BW, Woodhouse RA, Gilbart M, Hutchison CR. A needs assessment of surgical residents as teachers. Can J Surg. 2000;43(4): 295.
23. Champagne BJ. Effective teaching and feedback strat-
25. Norman G, Neville A, Blake JM, Mueller B. Assess-
ment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach. 2010;32(6):496-499. 26. Goldstein EA, MacLaren CF, Smith S, et al. Promot-
egies in the OR and beyond. Clin Colon Rectal Surg. 2013;26(4):244-249.
ing fundamental clinical skills: a competency-based college approach at the University of Washington. Acad Med. 2005;80(5):423-433.
Journal of Surgical Education Volume ]/Number ] ] 2015
9
