The Laryngoscope C 2014 The American Laryngological, V
Rhinological and Otological Society, Inc.
Attainment of Surgical Competence in Otolaryngology Training Keith A. Chadwick, MD; Kelley M. Dodson, MD; Wen Wan, PhD; Evan R. Reiter, MD Objectives/Hypothesis: Our aim was to determine the postgraduate year (PGY) of residency at which residents achieve competence in key otolaryngologic procedures as perceived by residents and program directors (PDs), determine resident or programmatic factors affecting PGY at which residents perceive attainment of competence, and evaluate the relationship between resident and PD perceptions for attainment of competence in these procedures. Study Design: Cross-sectional survey. Methods: We surveyed residents and PDs in US otolaryngology residencies in 2011 using SurveyMonkey.com and assessed subjective attainment of competence by residents for 32 otolaryngologic procedures. PGY at which subjective competence achieved per resident perception was determined using a logistic regression model; PD perception was determined by mean calculation. Results: Two hundred seventy-seven residents (19.45%) and 39 PDs (37.86%) completed surveys. Residents achieved subjective competence later than expected by PDs for 25 of 32 procedures, although differences were generally small. The largest disparities were observed for nonsurgical office-based procedures, for which 5
>5
>5
4.513
4.304
4.721
Partial glossectomy with primary closure
3.656
3.372
4.066
3.436
3.150
3.722
Thyroid lobectomy
4.157
3.840
4.613
3.769
3.529
4.010
Tracheostomy
2.550
2.273
2.991
2.410
2.204
2.617
Tympanoplasty without mastoidectomy and without ossicular chain reconstruction
4.485
4.198
4.900
3.974
3.691
4.257
Simple mastoidectomy
Total laryngectomy†
Otology
4.060
3.777
4.464
3.564
3.298
3.830
Stapedectomy
>5
>5
>5
5.000
4.834
5.166
Perform and interpret audiogram†
>5
>5
>5
2.385
2.142
2.627
Open or closed rhinoplasty with removal of dorsal hump and tip revision†
>5
>5
>5
4.667
4.452
4.881
ORIF and MMF for simultaneous subcondylar and body of mandible fractures†
4.932
4.509
>5
3.692
3.423
3.962
Local flap repair of soft tissue defect check (e.g., bilobe/rhomboid)†
4.481
4.066
>5
3.487
3.220
3.754
>5
>5
>5
4.513
4.318
4.708
Facial plastic and reconstructive surgery
Regional flap repair defect oral cavity/pharynx (e.g., pectoralis/trapezius/platysma)† Rhinology Septoplasty
3.154
2.903
3.526
2.949
2.739
3.158
Endoscopic anterior and posterior ethmoidectomy
3.487
3.233
3.855
3.538
3.283
3.794
Endoscopic sphenoidotomy
4.005
3.729
4.399
3.923
3.683
4.163
Administer and interpret allergy skin test†
>5
>5
>5
2.889
2.649
3.129
Allergy emergency protocol (treat anaphylactic reaction)†
>5
>5
>5
2.378
2.070
2.686
Direct laryngoscopy and biopsy tumorof pharynx or larynx
2.022
1.814
2.362
2.236
2.059
2.415
Microsuspension direct laryngoscopy with removal of benign lesion vocal fold
2.949
2.682
3.353
3.027
2.773
3.282
Vocal fold medialization (any technique)†
4.477
4.126
4.986
3.763
3.505
4.022
>5
>5
>5
2.861
2.526
3.196
Laryngotracheal repair for stenosis (resection with reanastamosis or cartilage grafting)
>5
>5
>5
5.000
4.747
5.253
Myringotomy and tube placement
1.540
1.353
2.019
1.736
1.590
1.884
Tonsillectomy and adenoidectomy
1.676
1.483
2.070
1.895
1.767
2.022
Excision of thyroglossal duct cyst†
3.816
3.540
4.211
3.105
2.866
3.344
Bronchoscopy with removal of foreign body from trachea or mainstem bronchus
3.434
3.127
3.887
3.289
3.004
3.574
Esophagoscopy with removal foreign body (coin)
3.113
2.833
3.534
2.974
2.663
3.284
Pediatric tracheotomy (age >2 years old, 5
>5
>5
4.250
3.913
4.587
Laryngology
Videostroboscopy† Pediatric otolaryngology
Sleep Uvulopalatopharyngoplasty Genioglossus advancement†
Values given as >5 indicate that even at the PGY-5 level, the target of 90% of residents perceiving attainment of competence was not reached for the procedure. As such, range boundaries could not be determined. *Range boundaries refer to the intersections between the upper and lower 95% confidence interval curves calculated by logistic regression analysis from the resident response data for each procedure, and the 90% competence threshold line (see Fig. 1 for graphical representation). † Procedures for which 95% confidence interval for program director PGY mean and range boundaries for resident PGY 90% level do not overlap. MMF 5 maxillomandibular fixation; ORIF 5 open reduction and internal fixation; PGY 5postgraduate year; SCM 5 sternocleidomastoid muscle
Laryngoscope 125: February 2015
Chadwick et al.: Surgical Competence in Otolaryngology
335
these procedures by completion of training. We also found that facial plastic and reconstructive surgery revealed the greatest number of procedures for which resident and PD responses differed greatly, among all subspecialty areas. It is unclear if this is an artifact attributable to the specific procedures selected for this and other subspecialty areas, or if other differences in surgical experience or teaching methods exist. There are several limitations of this study that may cloud interpretation of the results. Survey data are inherently subjective in nature, and thus reflect the individual biases of the respondents. Residents were asked to assess their own competence, which was left to their interpretation of “competence” and their assessment of their own skills. Bias may exist if only more self-confident residents chose to participate, leading to earlier perceived PGY attainment of competence than if a better sampling of the resident cohort were achieved. PD responses may reflect not only their current program, but experiences from other programs in which they held prior positions or completed their training. In addition, PDs with charged opinions toward competency assessments (as some demonstrated by written comments made in the survey) may have been more likely to respond, and may be biased toward earlier or later resident attainment of competence. Although our analysis of program characteristics for resident and PD respondents showed no significant differences, the anonymity of the survey precluded matching resident and program director responses. Given the low and different response rates between resident (19.5%) and PD (37.9%) groups, these datasets likely represent different samplings of programs. As residency programs may have 25 or more residents but only one PD, the resident response data may disproportionately weight responses from larger programs compared to the PD data, where each program has only one set of responses. As discussed above, different samplings of programs in the two datasets may be impacted by the differing structure of individual residencies, such as rotation schedules, number and nature of subspecialty rotations, and overall resident case numbers. Also other programmatic differences may exist between resident and PD datasets that were not explored in the survey, such as region, setting (urban vs. rural), or presence of Veterans Administration hospitals. This potential for sampling bias may also impact the applicability of the results to all programs, even within the relatively small number of otolaryngology residencies in the United States. Despite its limitations, we feel the data provided here offer invaluable guidance in establishing milestones for assessment of resident surgical competence in otolaryngology. The systematic trend toward later perceived attainment of competence by residents suggests that relying only on resident determination for readiness for summative competency assessments may actually hinder resident progress. Potentially, residents perceived by
Laryngoscope 125: February 2015
336
their PD as ready to undergo formal assessment of competence should be encouraged to do so, despite their own hesitation. Pushing a resident slightly beyond their comfort zone while still in the supervised setting of residency may speed their skill development. Receiving affirmation and recognizing mastery of a particular procedure would then allow a resident to focus their skill development on areas in which more experience is still required.
CONCLUSION We provided estimates of the PGY at which otolaryngology residents attain competence in 32 common otolaryngologic procedures based on resident self-reports and PD perceptions. Residents may perceive attainment of competence at slightly later PGYs than PDs’ expectations, with the greatest disparities seen with office-based nonsurgical procedures, and the highest concentration of disparities seen in the area of facial plastic and reconstructive surgery. Among resident and program factors, gender had the most pervasive effect on resident selfreported attainment of competence, with male gender suggesting earlier perceived attainment of competence in almost one-third of procedures studied. These data may help guide the establishment of realistic training milestones for procedural skills to more effectively educate future generations of otolaryngologists.
BIBLIOGRAPHY 1. Tsue TT, Dugan JW, Burkey B. Assessment of surgical competency. Otolaryngol Clin North Am 2007;40:1237–1259. 2. Brown DJ, Thompson RE, Bhatti NI. Assessment of operative competency in otolaryngology residency: survey of US program directors. Laryngoscope 2008;118:1761–1764. 3. Carr MM. Program directors’ opinions about surgical competency in otolaryngology residents. Laryngoscope 2005;115:1208–1211. 4. Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope 2005;115:2127–2137. 5. Ahmed A, Ishman SL, Laeeq K, Bhatti NI. Assessment of improvement of trainee surgical skills in the operating room for tonsillectomy. Laryngoscope 2013;123:1639–1644. 6. Laeeq K, Lin SY, Varela DA, Lane AP, Reh D, Bhatti NI. Achievement of competency in endoscopic sinus surgery of otolaryngology residents. Laryngoscope 2013;123:2932–2934. 7. Ishman SL, Benke JR, Johnson KE, et al. Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy. Arch Otolaryngol Head Neck Surg 2012: 1–7. 8. Malik MU, Varela DA, Park E, et al. Determinants of resident competence in mastoidectomy: role of interest and deliberate practice. Laryngoscope 2013;123:3162–3167. 9. Trajkovski T, Veillette C, Backstein D, Wadey VM, Kraemer B. Resident self-assessment of operative experience in primary total knee and total hip arthroplasty: is it accurate? Can J Surg 2012;55:S153–S157. 10. Fronza JS, Prystowsky JP, DaRosa D, Fryer JP. Surgical residents’ perception of competence and relevance of the clinical curriculum to future practice. J Surg Educ 2012;69:792–797. 11. Minter RM, Gruppen LD, Napolitano KS, Gauger PG. Gender differences in the self-assessment of surgical residents. Am J Surg 2005;189: 647–650. 12. Lind DS, Rekkas S, Bui V, Lam T, Beierle E, Copeland EM III. Competency-based student self-assessment on a surgery rotation. J Surg Res 2002;105:31–34. 13. Blanch-Hartigan D. Medical students’ self-assessment of performance: results from three meta-analyses. Patient Educ Couns 2011;84:3–9. 14. Zirkle M, Taplin MA, Anthony R, Dubrowski A. Objective assessment of temporal bone drilling skills. Ann Otol Rhinol Laryngol 2007;116: 793–798.
Chadwick et al.: Surgical Competence in Otolaryngology
Rhinological and Otological Society, Inc.
Attainment of Surgical Competence in Otolaryngology Training Keith A. Chadwick, MD; Kelley M. Dodson, MD; Wen Wan, PhD; Evan R. Reiter, MD Objectives/Hypothesis: Our aim was to determine the postgraduate year (PGY) of residency at which residents achieve competence in key otolaryngologic procedures as perceived by residents and program directors (PDs), determine resident or programmatic factors affecting PGY at which residents perceive attainment of competence, and evaluate the relationship between resident and PD perceptions for attainment of competence in these procedures. Study Design: Cross-sectional survey. Methods: We surveyed residents and PDs in US otolaryngology residencies in 2011 using SurveyMonkey.com and assessed subjective attainment of competence by residents for 32 otolaryngologic procedures. PGY at which subjective competence achieved per resident perception was determined using a logistic regression model; PD perception was determined by mean calculation. Results: Two hundred seventy-seven residents (19.45%) and 39 PDs (37.86%) completed surveys. Residents achieved subjective competence later than expected by PDs for 25 of 32 procedures, although differences were generally small. The largest disparities were observed for nonsurgical office-based procedures, for which 5
>5
>5
4.513
4.304
4.721
Partial glossectomy with primary closure
3.656
3.372
4.066
3.436
3.150
3.722
Thyroid lobectomy
4.157
3.840
4.613
3.769
3.529
4.010
Tracheostomy
2.550
2.273
2.991
2.410
2.204
2.617
Tympanoplasty without mastoidectomy and without ossicular chain reconstruction
4.485
4.198
4.900
3.974
3.691
4.257
Simple mastoidectomy
Total laryngectomy†
Otology
4.060
3.777
4.464
3.564
3.298
3.830
Stapedectomy
>5
>5
>5
5.000
4.834
5.166
Perform and interpret audiogram†
>5
>5
>5
2.385
2.142
2.627
Open or closed rhinoplasty with removal of dorsal hump and tip revision†
>5
>5
>5
4.667
4.452
4.881
ORIF and MMF for simultaneous subcondylar and body of mandible fractures†
4.932
4.509
>5
3.692
3.423
3.962
Local flap repair of soft tissue defect check (e.g., bilobe/rhomboid)†
4.481
4.066
>5
3.487
3.220
3.754
>5
>5
>5
4.513
4.318
4.708
Facial plastic and reconstructive surgery
Regional flap repair defect oral cavity/pharynx (e.g., pectoralis/trapezius/platysma)† Rhinology Septoplasty
3.154
2.903
3.526
2.949
2.739
3.158
Endoscopic anterior and posterior ethmoidectomy
3.487
3.233
3.855
3.538
3.283
3.794
Endoscopic sphenoidotomy
4.005
3.729
4.399
3.923
3.683
4.163
Administer and interpret allergy skin test†
>5
>5
>5
2.889
2.649
3.129
Allergy emergency protocol (treat anaphylactic reaction)†
>5
>5
>5
2.378
2.070
2.686
Direct laryngoscopy and biopsy tumorof pharynx or larynx
2.022
1.814
2.362
2.236
2.059
2.415
Microsuspension direct laryngoscopy with removal of benign lesion vocal fold
2.949
2.682
3.353
3.027
2.773
3.282
Vocal fold medialization (any technique)†
4.477
4.126
4.986
3.763
3.505
4.022
>5
>5
>5
2.861
2.526
3.196
Laryngotracheal repair for stenosis (resection with reanastamosis or cartilage grafting)
>5
>5
>5
5.000
4.747
5.253
Myringotomy and tube placement
1.540
1.353
2.019
1.736
1.590
1.884
Tonsillectomy and adenoidectomy
1.676
1.483
2.070
1.895
1.767
2.022
Excision of thyroglossal duct cyst†
3.816
3.540
4.211
3.105
2.866
3.344
Bronchoscopy with removal of foreign body from trachea or mainstem bronchus
3.434
3.127
3.887
3.289
3.004
3.574
Esophagoscopy with removal foreign body (coin)
3.113
2.833
3.534
2.974
2.663
3.284
Pediatric tracheotomy (age >2 years old, 5
>5
>5
4.250
3.913
4.587
Laryngology
Videostroboscopy† Pediatric otolaryngology
Sleep Uvulopalatopharyngoplasty Genioglossus advancement†
Values given as >5 indicate that even at the PGY-5 level, the target of 90% of residents perceiving attainment of competence was not reached for the procedure. As such, range boundaries could not be determined. *Range boundaries refer to the intersections between the upper and lower 95% confidence interval curves calculated by logistic regression analysis from the resident response data for each procedure, and the 90% competence threshold line (see Fig. 1 for graphical representation). † Procedures for which 95% confidence interval for program director PGY mean and range boundaries for resident PGY 90% level do not overlap. MMF 5 maxillomandibular fixation; ORIF 5 open reduction and internal fixation; PGY 5postgraduate year; SCM 5 sternocleidomastoid muscle
Laryngoscope 125: February 2015
Chadwick et al.: Surgical Competence in Otolaryngology
335
these procedures by completion of training. We also found that facial plastic and reconstructive surgery revealed the greatest number of procedures for which resident and PD responses differed greatly, among all subspecialty areas. It is unclear if this is an artifact attributable to the specific procedures selected for this and other subspecialty areas, or if other differences in surgical experience or teaching methods exist. There are several limitations of this study that may cloud interpretation of the results. Survey data are inherently subjective in nature, and thus reflect the individual biases of the respondents. Residents were asked to assess their own competence, which was left to their interpretation of “competence” and their assessment of their own skills. Bias may exist if only more self-confident residents chose to participate, leading to earlier perceived PGY attainment of competence than if a better sampling of the resident cohort were achieved. PD responses may reflect not only their current program, but experiences from other programs in which they held prior positions or completed their training. In addition, PDs with charged opinions toward competency assessments (as some demonstrated by written comments made in the survey) may have been more likely to respond, and may be biased toward earlier or later resident attainment of competence. Although our analysis of program characteristics for resident and PD respondents showed no significant differences, the anonymity of the survey precluded matching resident and program director responses. Given the low and different response rates between resident (19.5%) and PD (37.9%) groups, these datasets likely represent different samplings of programs. As residency programs may have 25 or more residents but only one PD, the resident response data may disproportionately weight responses from larger programs compared to the PD data, where each program has only one set of responses. As discussed above, different samplings of programs in the two datasets may be impacted by the differing structure of individual residencies, such as rotation schedules, number and nature of subspecialty rotations, and overall resident case numbers. Also other programmatic differences may exist between resident and PD datasets that were not explored in the survey, such as region, setting (urban vs. rural), or presence of Veterans Administration hospitals. This potential for sampling bias may also impact the applicability of the results to all programs, even within the relatively small number of otolaryngology residencies in the United States. Despite its limitations, we feel the data provided here offer invaluable guidance in establishing milestones for assessment of resident surgical competence in otolaryngology. The systematic trend toward later perceived attainment of competence by residents suggests that relying only on resident determination for readiness for summative competency assessments may actually hinder resident progress. Potentially, residents perceived by
Laryngoscope 125: February 2015
336
their PD as ready to undergo formal assessment of competence should be encouraged to do so, despite their own hesitation. Pushing a resident slightly beyond their comfort zone while still in the supervised setting of residency may speed their skill development. Receiving affirmation and recognizing mastery of a particular procedure would then allow a resident to focus their skill development on areas in which more experience is still required.
CONCLUSION We provided estimates of the PGY at which otolaryngology residents attain competence in 32 common otolaryngologic procedures based on resident self-reports and PD perceptions. Residents may perceive attainment of competence at slightly later PGYs than PDs’ expectations, with the greatest disparities seen with office-based nonsurgical procedures, and the highest concentration of disparities seen in the area of facial plastic and reconstructive surgery. Among resident and program factors, gender had the most pervasive effect on resident selfreported attainment of competence, with male gender suggesting earlier perceived attainment of competence in almost one-third of procedures studied. These data may help guide the establishment of realistic training milestones for procedural skills to more effectively educate future generations of otolaryngologists.
BIBLIOGRAPHY 1. Tsue TT, Dugan JW, Burkey B. Assessment of surgical competency. Otolaryngol Clin North Am 2007;40:1237–1259. 2. Brown DJ, Thompson RE, Bhatti NI. Assessment of operative competency in otolaryngology residency: survey of US program directors. Laryngoscope 2008;118:1761–1764. 3. Carr MM. Program directors’ opinions about surgical competency in otolaryngology residents. Laryngoscope 2005;115:1208–1211. 4. Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope 2005;115:2127–2137. 5. Ahmed A, Ishman SL, Laeeq K, Bhatti NI. Assessment of improvement of trainee surgical skills in the operating room for tonsillectomy. Laryngoscope 2013;123:1639–1644. 6. Laeeq K, Lin SY, Varela DA, Lane AP, Reh D, Bhatti NI. Achievement of competency in endoscopic sinus surgery of otolaryngology residents. Laryngoscope 2013;123:2932–2934. 7. Ishman SL, Benke JR, Johnson KE, et al. Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy. Arch Otolaryngol Head Neck Surg 2012: 1–7. 8. Malik MU, Varela DA, Park E, et al. Determinants of resident competence in mastoidectomy: role of interest and deliberate practice. Laryngoscope 2013;123:3162–3167. 9. Trajkovski T, Veillette C, Backstein D, Wadey VM, Kraemer B. Resident self-assessment of operative experience in primary total knee and total hip arthroplasty: is it accurate? Can J Surg 2012;55:S153–S157. 10. Fronza JS, Prystowsky JP, DaRosa D, Fryer JP. Surgical residents’ perception of competence and relevance of the clinical curriculum to future practice. J Surg Educ 2012;69:792–797. 11. Minter RM, Gruppen LD, Napolitano KS, Gauger PG. Gender differences in the self-assessment of surgical residents. Am J Surg 2005;189: 647–650. 12. Lind DS, Rekkas S, Bui V, Lam T, Beierle E, Copeland EM III. Competency-based student self-assessment on a surgery rotation. J Surg Res 2002;105:31–34. 13. Blanch-Hartigan D. Medical students’ self-assessment of performance: results from three meta-analyses. Patient Educ Couns 2011;84:3–9. 14. Zirkle M, Taplin MA, Anthony R, Dubrowski A. Objective assessment of temporal bone drilling skills. Ann Otol Rhinol Laryngol 2007;116: 793–798.
Chadwick et al.: Surgical Competence in Otolaryngology
