Quality In-Training InitiativedA Solution to the Need for Education in Quality Improvement: Results from a Survey of Program Directors Rachel R Kelz, MD, MSCE, FACS, Morgan M Sellers, MD, Caroline E Reinke, Rachel L Medbery, MD, Jon Morris, MD, FACS, Clifford Ko, MD, MS, FACS
MD, MSHP,
The Next Accreditation System and the Clinical Learning Environment Review Program will emphasize practice-based learning and improvement and systems-based practice. We present the results of a survey of general surgery program directors to characterize the current state of quality improvement in graduate surgical education and introduce the Quality In-Training Initiative (QITI). STUDY DESIGN: In 2012, a 20-item survey was distributed to 118 surgical residency program directors from ACS NSQIP-affiliated hospitals. The survey content was developed in collaboration with the QITI to identify program director opinions regarding education in practice-based learning and improvement and systems-based practice, to investigate the status of quality improvement education in their respective programs, and to quantify the extent of resident participation in quality improvement. RESULTS: There was a 57% response rate. Eighty-five percent of program directors (n ¼ 57) reported that education in quality improvement is essential to future professional work in the field of surgery. Only 28% (n ¼ 18) of programs reported that at least 50% of their residents track and analyze their patient outcomes, compare them with norms/benchmarks/published standards, and identify opportunities to make practice improvements. CONCLUSIONS: Program directors recognize the importance of quality improvement efforts in surgical practice. Subpar participation in basic practice-based learning and improvement activities at the resident level reflects the need for support of these educational goals. The QITI will facilitate programmatic compliance with goals for quality improvement education. (J Am Coll Surg 2013;217:1126e1132. 2013 by the American College of Surgeons)
BACKGROUND:
The Accreditation Council for Graduate Medical Education (ACGME) has moved from a paradigm based on circumstantial training to a new educational model that tracks resident progression to competency using educational outcomes data, the Next Accreditation System (NAS) system.1-3 This system will include a series of essential tasks or milestones that residents must complete Disclosure Information: Nothing to disclose. Received May 10, 2013; Revised July 1, 2013; Accepted July 3, 2013. From the Department of Surgery, Perelman School of Medicine (Kelz, Sellers, Reinke, Morris) and the Leonard Davis Institute of Healthcare Economics, Wharton School (Kelz, Reinke), University of Pennsylvania, Philadelphia, PA; the Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL (Sellers, Ko); and the Departments of Surgery, University of California Los Angeles David Geffen School of Medicine and the VA Greater Los Angeles Healthcare System, Los Angeles, CA (Ko) and Emory University School of Medicine, Atlanta, GA (Medbery). Correspondence address: Rachel R Kelz, MD, MSCE, FACS, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104. email: [email protected]
ª 2013 by the American College of Surgeons Published by Elsevier Inc.
in order to successfully graduate from an accredited training program. Surgical residents will be required to demonstrate competence in the 6 core competencies including practice-based learning and improvement and systems-based practice. More recently, the ACGME launched the Clinical Learning Environmental Review (CLER) Program as a part of the 2011 Common Program Requirements.4 This program will review institutional resources and patient outcomes. The program will conduct site visits at sponsoring organizations to assess 6 focus areas including patient safety, quality improvement (QI), transitions in care, supervision, duty hours oversight, fatigue management and mitigation, and professionalism. The intended purpose is to move from duty hour monitoring to identification of key determinants of graduate medical education that will facilitate the production of high-quality physicians while ensuring that training programs adhere to the standards in patient care expected by the American public.
1126
ISSN 1072-7515/13/$36.00 http://dx.doi.org/10.1016/j.jamcollsurg.2013.07.395
Vol. 217, No. 6, December 2013
Abbreviations and Acronyms
ACGME ¼ Accreditation Council for Graduate Medical Education QI ¼ quality improvement QITI ¼ Quality In-Training Initiative
The Centers for Medicare and Medicaid Services are considering implementation of an outcomes-based graduate medical education funding plan that will tie financial support to performance-based standards.5 The standards have not yet been defined, but it has been suggested that programs that train residents in quality measurement and improvement, evidence-based medicine, and multidisciplinary teamwork will be rewarded. The Medicare Payment Advisory Commission (MedPAC) has identified these skills, among others, as vital to the success of future innovations to improve the value of our health care delivery system.6 In response to expert recommendations,2,7 programs will need to incorporate formal education in systems-based practice and practice-based learning and improvement in order to remain accredited. The multitude of new requirements designed to incorporate quality improvement skills into our educational paradigm are intended to make quality care sustainable. However, if we are to integrate quality into our care, it must become a part of the way that surgical residents work, in all aspects. Program directors control the content of the surgical education curricula. With the support of these organizational requirements, it is possible that surgical residency program directors hold the key to integrating quality into the surgical culture. To date, little is known about the educator perspectives on quality improvement and patient safety education. We present the results of a survey of program directors of general surgery residency programs that participate in the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) to characterize program director attributes related to quality improvement expertise, the current extent of resident participation in quality improvement, and opinions of the surgical educators regarding the educational priorities for the core competencies.
METHODS Survey design and sample In 2012, a 20-item survey was distributed via email to 118 surgical residency program directors from ACS NSQIPaffiliated hospitals. An introduction indicating that participation was voluntary and ensuring confidentiality was provided with each survey. Completion of the survey
Kelz et al
Quality Improvement Education
1127
constituted implied consent. The protocol was deemed exempt by the Institutional Review Board at the University of Pennsylvania Perelman School of Medicine. Survey instrument The survey was developed to investigate the extent of resident participation in quality improvement educational activities, as described in the literature, and to identify program director perceptions regarding the importance of introducing education in practice-based learning and improvement and systems-based practice to the curriculum. The survey content was developed in collaboration with the ACS NSQIP Quality In-Training Initiative (QITI). The QITI is a multidisciplinary collaborative of academic affiliates of the ACS NSQIP.8 Membership includes representation from surgical education, quality improvement, and clinical investigation. Members include licensed nurses, surgeons, and surgeons-in-training. The collaborative includes the voluntary expansion of each hospital’s ACS NSQIP team to include partners from within the graduate surgical education team. Using the ACS NSQIP expertise in quality science with partners in graduate medical education, the QITI aims to generate clinical outcomes reports for use in resident education (Appendix, online only), to develop a surgical quality improvement curriculum to facilitate the ability of program directors to teach quality improvement to their residents, and to disseminate the importance of a culture of quality care through the firm commitment of surgeons in academic centers to the achievement of optimal patient outcomes based on today’s standards using modern measurement tools. The items regarding program director perceptions were structured to reflect the concepts that emerged during QITI focus group sessions at national meetings (ACS NSQIP July 2011 and 2012, Association of Program Directors in Surgery April 2011 and 2012, ACS Clinical Congress October 2011 and 2012) and on QITI conference calls held on a monthly basis. A 13-item survey was pilot tested with 2 program directors to determine the readability and feasibility for program directors without particular expertise in quality improvement and patient safety. The final 20-item survey aimed at identifying program director experience with quality improvement activities and formal training, resident participation in activities related to practice-based learning and improvement, and program director attributes that may influence the educational program in quality improvement. Demographic items included program director sex, program director year of completion of general surgery residency training,
1128
Table 1.
Kelz et al
J Am Coll Surg
Quality Improvement Education
Attributes of Program Directors and General Surgery Training Programs in Hospitals Affiliated with the ACS NSQIP
Variable
Data
Year of completion of categorical general surgical training, median year (interquartile range, range) Program director involvement in quality improvement-related activities, n (%) Morbidity and mortality conference Root cause analysis (PD patient) Root cause analysis (not PD patient) Formal QI training (ie, Six Sigma, Lean, DMAIC, and PDSA, etc) Master’s degree in public health, clinical epidemiology, health policy or related field Other: Master’s degree in business administration Graduating chief residents, mean SD I consider education in quality improvement ____ to future professional work in the field of surgery, n (%) Very essential Essential Neutral Nonessential Very nonessential I am ______ that quality improvement initiatives can lead to better surgical care in the local setting, n (%) Not confident at all Not confident Unsure Confident Very confident
1996 (1987e2003, 1978e2009) 66 41 55 23 4 2 4.93
(100) (62) (83) (35) (6) (3) 2.07
27 30 8 2 0
(40) (45) (12) (3) (0)
2 7 10 32 15
(3) (11) (15) (49) (23)
PD, program director; QI, quality improvement.
program director job description, program director quality experience, and size of the residency program. Questions were divided into 3 domains: current status of quality education in the program, opinions regarding quality, and a ranking of educational priorities including practice-based learning and improvement. Ties regarding the importance of educational domains were permitted given the complexity of distinguishing among the competencies. A free text comment area was included given the importance of qualitative narration in opinion gathering. Study data were collected and managed using REDCap (Research Electronic Data Capture) electronic data capture tools hosted at the University of Pennsylvania.9 REDCap is a secure, web-based application designed to support data capture for research studies, providing an intuitive interface for validated data entry; audit trails for tracking data manipulation and export procedures; automated export procedures for seamless data downloads to common statistical packages; and procedures for importing data from external sources. Data analysis Descriptive statistics were performed. Analyses were conducted using STATA software, version 11 (STATA Institute Inc).
RESULTS Of the 118 program directors affiliated with the ACS NSQIP, 3 people could not be reached due to technical difficulties and 1 person opened the survey but responses were not recorded. Therefore, completed questionnaires were received from 66 of 115 (57%) of the program directors surveyed. The majority of respondents were male 53 (80%). Although all respondents described themselves as program directors (n ¼ 66; 100%), 20 (30%) reported their job description also included work as a clinical researcher, and 12 (18%) reported they also worked as a leader in quality improvement. There were no basic scientists in the group. The majority of program director respondents completed their general surgery residency before implementation of the 80-hour duty standards 49 (74%.) Twenty-five respondents reported completing subspecialty fellowship training (38%). All respondents reported involvement with morbidity and mortality conferences. Additional program director attributes are displayed in Table 1. Eighty-five percent of program director respondents (n ¼ 57) reported that education in quality improvement was essential to future professional work in the field of surgery. Seventy-two percent reported confidence that
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
1129
Table 2. Current Status of Quality Improvement Participation and Education in ACS NSQIP-Affiliated General Surgery Residency Programs Stem
The didactics for practice-based learning and improvement and/or systems based practice are taught by. Surgeon, leader in QI Surgeon, assigned, not QI leader Surgical housestaff member, QI track Surgical housestaff member, not QI track Nonsurgeon physician, leader in QI nonphysician QI representative, not in leadership position Don’t know Other Proportion of hospitals participating in the following national QI programs. ACS NSQIP NSQIP University HealthSystem Consortium Don’t know Other*
n
%
31 17 2 8 21 16 0 1
69 38 5 18 48 36 0 2
64 20 18 1 3
97 30 27 2 5
Forty-four program directors reported didactics for practice-based learning and improvement and/or systems-based practice. *American College of Surgeons (ACS) bariatric database, ACS National Accreditation Program for Breast Centers (NAPBC), United Network for Organ Sharing (UNOS), Centers for Medicare and Medicaid Services Core Measures, System-Based QI Database. QI, quality improvement.
quality improvement initiatives could lead to improved surgical care in the local setting (n ¼ 47). Sixty-seven percent of respondents reported didactics in practice-based learning and improvement and systems-based practice (n ¼ 44.) Fifty-eight percent of respondents reported that their residents worked with the quality improvement teams within their organizations (n ¼ 38). Didactics were taught by a broad array of personnel including housestaff (n ¼ 10) (Table 2). Twenty-eight percent (n¼18) of programs reported that at least 50% of their residents track and analyze their patient outcomes, compare them to norms/benchmarks/ published standards, and identify opportunities to make practice improvements. The reported percentage varied significantly by program (Fig. 1). The estimated median percentage of residents within each program that participated in a quality improvement project outside of morbidity and mortality conferences was 50% (range 0% to 100%) (Fig. 2). The median rank order of the core competencies by the program directors with respect to the educational priority that they should receive in graduate surgical education according to the respondents (1, highest priority; 6, lowest priority) was: 1-patient care, 2-medical knowledge, 2-professionalism, 3-interpersonal communication, 3-practice-based learning and improvement, and 4-systems-based practice. There was a general trend toward high priority for patient care and medical knowledge, with great variability in the rankings of the other core competencies.
Attitudes and opinions of program directors outside of the structured questions were captured through solicitation of free text comments. Themes observed among the comments included concern regarding attribution (n ¼ 4), frustration regarding educational mandates that reach outside of the domain of teaching (n ¼ 3), dissatisfaction with current QI educational programs (n ¼ 6), and the desire to continue the status quo (n ¼ 1).
Figure 1. Number of programs by the percent of residents who track and analyze their patients’ clinical outcomes, compare them with norms/benchmarks/published standards, and identify opportunities to make practice improvements.
1130
Kelz et al
Quality Improvement Education
Figure 2. Number of programs by the percent of residents that participate in quality improvement projects outside of morbidity and mortality conferences.
DISCUSSION In this study, we described the current state of resident education in quality science as reported by general surgery program directors affiliated with the ACS NSQIP. We found that in the majority of surgical residency programs sampled, fewer than 50% of residents track and analyze their patient outcomes, compare them with norms/ benchmarks/published standards, and formally identify opportunities to make practice improvements. Furthermore, we determined that the majority of residents rely on surgical morbidity and mortality conferences as the sole source of participation in a quality improvement project. Although many residents undoubtedly graduate without exposure and experience in formal quality improvement projects, the majority of training programs did report didactics targeting practice-based learning and improvement and systems-based practice. Our results likely reflect the relatively new emphasis on incorporation of these competencies into formal surgical education coupled with the educational priority that program directors place on the different competencies. The ACGME core competencies were adopted in 1999 and are slated for complete implementation by 2006.10 Given the introduction of the standard duty hours in 2003, program directors were saddled with the responsibility to adhere to the duty hours while developing a new approach to surgical education using the core competencies and continuing to train surgeons. Therefore, whether consciously or unconsciously, priorities had to be assigned to the educational importance of each of the core competencies in order to systematically approach the conversion process from an educational system based on experiential
J Am Coll Surg
accomplishments to one centered on competence-based outcomes. As demonstrated in our survey results, practice-based learning and improvement and systemsbased practice fell to the bottom of the priority list, with a higher educational urgency placed on competence in patient care, medical knowledge, and professionalism. Program directors are in the business of educating surgeons to provide patient care. The external pressures of public accountability for surgeon decision-making and health care outcomes have resulted in a paradigm shift from experiential accreditation to a system based on outcomes. Despite the phased implementation of the Next Accreditation System (NAS),2 the paradigm shift has created a lot of work for surgical educators without providing additional support. Some of the comments shared by the program directors clearly reflected frustration. However, despite some frustrations, the overwhelming majority of program directors affiliated with the ACS NSQIP believe that there is essential value in teaching residents the science of quality improvement. Several not-for-profit organizations, such as the Institute for Healthcare Improvement 11 and the University HealthSystem Consortium,12 have developed educational programs and databases for use in institutional quality improvement programs. To date, however, these programs are generic and are not surgery specific. The ACS NSQIP is devoted to the measurement and improvement of surgical care in the private sector.13 At the center of the program lies the most valid and reliable surgery-specific data registry available across surgical specialties.14,15 There are more than 400 hospitals currently enrolled. The ACS NSQIP data permit benchmarking of outcomes, generation of best practices guidelines, and an environment to foster collaboration and dissemination of information through frequent conference calls, newsletters, and a national meeting for sharing of information across institutions and disciplines. Given the enormous amount of work required for programs to shift training paradigms to the new accreditation system, collaboration across institutions to develop a thoughtful and comprehensive approach to education in quality science holds great promise. Institutional collaborations consisting of data feedback and training in continuous improvement techniques have effectively resulted in a reduction in mortality and morbidity, improvements in processes of care, and cost-savings across institutions.16,17 The majority of these collaborations have been regional or targeted at specific disease entities within surgery, and none to date have focused exclusively on surgical education. The ACS NSQIP Quality In-Training Initiative (QITI) is the first collaborative convened to address educational issues related to improving the quality of
Vol. 217, No. 6, December 2013
surgical care provided across the United States. Using the most robust surgical outcomes data available, the Quality In-Training Initiative will develop a data-centered educational program for advancement of quality improvement in surgery through indoctrination of our youngest surgeons. The program will be based on developing content that fits within the processes of patient care and allows for transmission of quality improvement knowledge to residents as a routine part of every clinical and educational activity. In so doing, program directors will have support to educate their housestaff in an environment that is steeped in quality improvement and committed to teaching each resident about the tools needed to provide high quality surgical care. Our study has several limitations. The survey instrument has not been previously validated. Additionally, the responses are opinion based and may not accurately capture the quantitative measures as suggested. However, our survey does contain vital information regarding the status of quality improvement education from the individuals responsible for the curriculum and therefore, regardless of the accuracy of the specific numbers reported, the overall themes expressed undoubtedly reflect the state of quality improvement education. An additional potential limitation of the study is that it surveyed ACS NSQIP program directors exclusively. Although ACS NSQIP hospitals have been shown to provide the highest quality surgical care,18 there are no data to suggest that enhanced collaboration between educational and quality improvement personnel or more rigorous quality improvement instruction exist at these hospitals when compared with non-ACS NSQIP hospitals. However, if the choice of surveying the ACS NSQIP program directors were to bias our results, we would expect the ACS NSQIP programs to have better QI education when compared with the other programs, which would suggest that we have overestimated the role that quality improvement education plays in modern surgical training across all accredited programs. If that were the case, then the need for an academic collaboration like the QITI would be even greater than stated by our findings because it would mean that fewer residents are receiving proper training in QI. Furthermore, the ACS NSQIP program directors represent 47% of the general surgery program directors and likely influence a larger portion of the overall number of surgical residents enrolled in residency programs due to the large size of many of the ACS NSQIP sponsor hospitals. Therefore, although the magnitude of our findings might differ if we surveyed all program directors, the need for programmatic support to achieve the QI goals of the new ACGME programs would remain substantial.
Kelz et al
Quality Improvement Education
1131
CONCLUSIONS The QITI is the first national surgical initiative aimed at infusing quality improvement education into the surgical training paradigm using clinical outcomes data. The QITI program will enable program directors to teach the quality improvement materials that they report are both essential to future professional work in the field of surgery and infrequently addressed outside of morbidity and mortality conferences and occasional didactic lectures. Through the local collaboration of quality and educational teams, the QITI hopes to lay the seeds for a future in which surgeons not only report outcomes but also analyze the data and develop tools to deliver optimal patient care as an expected part of the surgical culture. Author Contributions Study conception and design: Kelz, Sellers, Reinke, Morris, Ko Acquisition of data: Kelz, Sellers, Reinke, Morris Analysis and interpretation of data: Kelz, Sellers, Reinke, Medbery, Morris, Ko Drafting of manuscript: Kelz, Sellers, Reinke, Medbery, Morris Critical revision: Kelz, Sellers, Reinke, Medbery, Morris, Ko REFERENCES 1. Accreditation Council for Graduate Medical Education. Program requirements for graduate medical education in general surgery. 2012. Available at: http://www.acgme.org/ acgmeweb/Portals/0/PFAssets/ProgramRequirements/440_ general_surgery_01012008_07012012.pdf. Accessed May 1, 2012. 2. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation systemerationale and benefits. N Engl J Med 2012;366:1051e1056. 3. Sachdeva AK, Philibert I, Leach DC, et al. Patient safety curriculum for surgical residency programs: results of a national consensus conference. Surgery 2007;141:427e441. 4. Accreditation Council for Graduate Medical Education. Clinical Learning Environment Review (CLER) Program. Available at: http://www.acgme-nas.org/cler.html. Accessed April 15, 2013. 5. GME Commission. Graduate medical education financing: Focusing on educational priorities. In: Report to the Congress: Aligning Incentives in Medicare; 2010:103e125. 6. 111th United States Congress. America’s Affordable Health Choices Act of 2009. Washington, DC; 2009:HR 4872, Sec. 1505. Improving accountability for approved medical residency training. 7. Weinstein MD, Chair D. Ensuring an Effective Physician Workforce for the United States Recommendations for Graduate Medical Education to Meet the Needs of the Public. Atlanta, GA: Josiah Macy Jr Foundation; 2011. 8. Sellers M, Reinke CE, Kreider S, et al. ACS NSQIP quality in-training initiative pilot study. J Am Coll Surg 2013;217: 827e832.
1132
Kelz et al
Quality Improvement Education
9. Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)ea metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377e381. 10. Leach DC. Building and assessing competence: the potential for evidence-based graduate medical education. Qual Manage Health Care 2002;11:39e44. 11. Institute for Healthcare Improvement. Available at: http:// www.ihi.org. Accessed April 26, 2013. 12. University HealthSystem Consortium. Available at: www.uhc. edu. Accessed April 26, 2013. 13. American College of Surgeons National Surgical Quality Improvement Program. About ACS NSQIP: Program Overview. Available at: www.acsnsqip.org. Accessed December 19, 2011. 14. Fink AS, Campbell DA Jr, Mentzer RM Jr, et al. The National Surgical Quality Improvement Program in non-Veterans Administration hospitals: initial demonstration of feasibility. Ann Surg 2002;236:344e353; discussion 353e354.
J Am Coll Surg
15. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228:491e507. 16. Campbell DA Jr, Dellinger EP. Multihospital collaborations for surgical quality improvement. JAMA 2009;302: 1584e1585. 17. Guillamondegui OD, Gunter OL, Hines L, et al. Using the National Surgical Quality Improvement Program and the Tennessee Surgical Quality Collaborative to improve surgical outcomes. J Am Coll Surg 2012;214:709e714; discussion 714e716. 18. Hall BL, Hamilton BH, Richards K, et al. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 2009;250:363e376.
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
Figure 1. Sample of a Resident Report Currently Available for Use in Graduate Surgical Education
1132.e1
1132.e2
Kelz et al
Quality Improvement Education
J Am Coll Surg
Figure 2. Sample of the Proposed Resident Report after Centralization of the Reporting Process. Note. The report includes the following information: number of cases, the proportion of patients with 0 to 5þ risk factors, outcomes (readmissions, unplanned readmissions, readmissions related to the principal procedure, unplanned return to the operating room, death) and postoperative occurrences (wound occurrences, respiratory occurrences, urinary tract occurrences, central nervous system occurrences, cardiac occurrences, and other occurrences including transfusions, vein thrombosis, sepsis, and septic shock.) Comparison groups include the index resident, all QITI resident outcomes for the same postgraduate year (PGY) as the index resident, all residents from the index hospital at the same PGY as the index resident, and the index resident outcomes for the 6 reporting periods. All case details for the index resident will be included on the subsequent pages of the report.
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
Figure 3. Sample of the Team Report Currently Available for Use in Graduate Surgical Education
1132.e3
1132.e4
Kelz et al
Quality Improvement Education
J Am Coll Surg
Figure 4. Sample of the Proposed Team Report after Centralization of the Reporting Process. Note: The report includes the following information: number of cases, the proportion of patients with 0 to 5þ risk factors, outcomes (readmissions, unplanned readmissions, readmissions related to the principal procedure, unplanned return to the operating room, death) and postoperative occurrences (wound occurrences, respiratory occurrences, urinary tract occurrences, central nervous system occurrences, cardiac occurrences, and other occurrences including transfusions, vein thrombosis, sepsis, and septic shock.) Comparison group includes the team outcomes for the same service for the 6 months preceding the reporting period. All case details for the index team will be included on the subsequent pages of the report.
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
1132.e5
Figure 5. Sample of a Graphical Report. CVA, cerebrovascular accident; QITI, Quality In-Training Initiative; SSI, surgical site infection; UTI, urinary tract infection; VTE, venous thromboembolism.
MD, MSHP,
The Next Accreditation System and the Clinical Learning Environment Review Program will emphasize practice-based learning and improvement and systems-based practice. We present the results of a survey of general surgery program directors to characterize the current state of quality improvement in graduate surgical education and introduce the Quality In-Training Initiative (QITI). STUDY DESIGN: In 2012, a 20-item survey was distributed to 118 surgical residency program directors from ACS NSQIP-affiliated hospitals. The survey content was developed in collaboration with the QITI to identify program director opinions regarding education in practice-based learning and improvement and systems-based practice, to investigate the status of quality improvement education in their respective programs, and to quantify the extent of resident participation in quality improvement. RESULTS: There was a 57% response rate. Eighty-five percent of program directors (n ¼ 57) reported that education in quality improvement is essential to future professional work in the field of surgery. Only 28% (n ¼ 18) of programs reported that at least 50% of their residents track and analyze their patient outcomes, compare them with norms/benchmarks/published standards, and identify opportunities to make practice improvements. CONCLUSIONS: Program directors recognize the importance of quality improvement efforts in surgical practice. Subpar participation in basic practice-based learning and improvement activities at the resident level reflects the need for support of these educational goals. The QITI will facilitate programmatic compliance with goals for quality improvement education. (J Am Coll Surg 2013;217:1126e1132. 2013 by the American College of Surgeons)
BACKGROUND:
The Accreditation Council for Graduate Medical Education (ACGME) has moved from a paradigm based on circumstantial training to a new educational model that tracks resident progression to competency using educational outcomes data, the Next Accreditation System (NAS) system.1-3 This system will include a series of essential tasks or milestones that residents must complete Disclosure Information: Nothing to disclose. Received May 10, 2013; Revised July 1, 2013; Accepted July 3, 2013. From the Department of Surgery, Perelman School of Medicine (Kelz, Sellers, Reinke, Morris) and the Leonard Davis Institute of Healthcare Economics, Wharton School (Kelz, Reinke), University of Pennsylvania, Philadelphia, PA; the Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL (Sellers, Ko); and the Departments of Surgery, University of California Los Angeles David Geffen School of Medicine and the VA Greater Los Angeles Healthcare System, Los Angeles, CA (Ko) and Emory University School of Medicine, Atlanta, GA (Medbery). Correspondence address: Rachel R Kelz, MD, MSCE, FACS, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104. email: [email protected]
ª 2013 by the American College of Surgeons Published by Elsevier Inc.
in order to successfully graduate from an accredited training program. Surgical residents will be required to demonstrate competence in the 6 core competencies including practice-based learning and improvement and systems-based practice. More recently, the ACGME launched the Clinical Learning Environmental Review (CLER) Program as a part of the 2011 Common Program Requirements.4 This program will review institutional resources and patient outcomes. The program will conduct site visits at sponsoring organizations to assess 6 focus areas including patient safety, quality improvement (QI), transitions in care, supervision, duty hours oversight, fatigue management and mitigation, and professionalism. The intended purpose is to move from duty hour monitoring to identification of key determinants of graduate medical education that will facilitate the production of high-quality physicians while ensuring that training programs adhere to the standards in patient care expected by the American public.
1126
ISSN 1072-7515/13/$36.00 http://dx.doi.org/10.1016/j.jamcollsurg.2013.07.395
Vol. 217, No. 6, December 2013
Abbreviations and Acronyms
ACGME ¼ Accreditation Council for Graduate Medical Education QI ¼ quality improvement QITI ¼ Quality In-Training Initiative
The Centers for Medicare and Medicaid Services are considering implementation of an outcomes-based graduate medical education funding plan that will tie financial support to performance-based standards.5 The standards have not yet been defined, but it has been suggested that programs that train residents in quality measurement and improvement, evidence-based medicine, and multidisciplinary teamwork will be rewarded. The Medicare Payment Advisory Commission (MedPAC) has identified these skills, among others, as vital to the success of future innovations to improve the value of our health care delivery system.6 In response to expert recommendations,2,7 programs will need to incorporate formal education in systems-based practice and practice-based learning and improvement in order to remain accredited. The multitude of new requirements designed to incorporate quality improvement skills into our educational paradigm are intended to make quality care sustainable. However, if we are to integrate quality into our care, it must become a part of the way that surgical residents work, in all aspects. Program directors control the content of the surgical education curricula. With the support of these organizational requirements, it is possible that surgical residency program directors hold the key to integrating quality into the surgical culture. To date, little is known about the educator perspectives on quality improvement and patient safety education. We present the results of a survey of program directors of general surgery residency programs that participate in the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) to characterize program director attributes related to quality improvement expertise, the current extent of resident participation in quality improvement, and opinions of the surgical educators regarding the educational priorities for the core competencies.
METHODS Survey design and sample In 2012, a 20-item survey was distributed via email to 118 surgical residency program directors from ACS NSQIPaffiliated hospitals. An introduction indicating that participation was voluntary and ensuring confidentiality was provided with each survey. Completion of the survey
Kelz et al
Quality Improvement Education
1127
constituted implied consent. The protocol was deemed exempt by the Institutional Review Board at the University of Pennsylvania Perelman School of Medicine. Survey instrument The survey was developed to investigate the extent of resident participation in quality improvement educational activities, as described in the literature, and to identify program director perceptions regarding the importance of introducing education in practice-based learning and improvement and systems-based practice to the curriculum. The survey content was developed in collaboration with the ACS NSQIP Quality In-Training Initiative (QITI). The QITI is a multidisciplinary collaborative of academic affiliates of the ACS NSQIP.8 Membership includes representation from surgical education, quality improvement, and clinical investigation. Members include licensed nurses, surgeons, and surgeons-in-training. The collaborative includes the voluntary expansion of each hospital’s ACS NSQIP team to include partners from within the graduate surgical education team. Using the ACS NSQIP expertise in quality science with partners in graduate medical education, the QITI aims to generate clinical outcomes reports for use in resident education (Appendix, online only), to develop a surgical quality improvement curriculum to facilitate the ability of program directors to teach quality improvement to their residents, and to disseminate the importance of a culture of quality care through the firm commitment of surgeons in academic centers to the achievement of optimal patient outcomes based on today’s standards using modern measurement tools. The items regarding program director perceptions were structured to reflect the concepts that emerged during QITI focus group sessions at national meetings (ACS NSQIP July 2011 and 2012, Association of Program Directors in Surgery April 2011 and 2012, ACS Clinical Congress October 2011 and 2012) and on QITI conference calls held on a monthly basis. A 13-item survey was pilot tested with 2 program directors to determine the readability and feasibility for program directors without particular expertise in quality improvement and patient safety. The final 20-item survey aimed at identifying program director experience with quality improvement activities and formal training, resident participation in activities related to practice-based learning and improvement, and program director attributes that may influence the educational program in quality improvement. Demographic items included program director sex, program director year of completion of general surgery residency training,
1128
Table 1.
Kelz et al
J Am Coll Surg
Quality Improvement Education
Attributes of Program Directors and General Surgery Training Programs in Hospitals Affiliated with the ACS NSQIP
Variable
Data
Year of completion of categorical general surgical training, median year (interquartile range, range) Program director involvement in quality improvement-related activities, n (%) Morbidity and mortality conference Root cause analysis (PD patient) Root cause analysis (not PD patient) Formal QI training (ie, Six Sigma, Lean, DMAIC, and PDSA, etc) Master’s degree in public health, clinical epidemiology, health policy or related field Other: Master’s degree in business administration Graduating chief residents, mean SD I consider education in quality improvement ____ to future professional work in the field of surgery, n (%) Very essential Essential Neutral Nonessential Very nonessential I am ______ that quality improvement initiatives can lead to better surgical care in the local setting, n (%) Not confident at all Not confident Unsure Confident Very confident
1996 (1987e2003, 1978e2009) 66 41 55 23 4 2 4.93
(100) (62) (83) (35) (6) (3) 2.07
27 30 8 2 0
(40) (45) (12) (3) (0)
2 7 10 32 15
(3) (11) (15) (49) (23)
PD, program director; QI, quality improvement.
program director job description, program director quality experience, and size of the residency program. Questions were divided into 3 domains: current status of quality education in the program, opinions regarding quality, and a ranking of educational priorities including practice-based learning and improvement. Ties regarding the importance of educational domains were permitted given the complexity of distinguishing among the competencies. A free text comment area was included given the importance of qualitative narration in opinion gathering. Study data were collected and managed using REDCap (Research Electronic Data Capture) electronic data capture tools hosted at the University of Pennsylvania.9 REDCap is a secure, web-based application designed to support data capture for research studies, providing an intuitive interface for validated data entry; audit trails for tracking data manipulation and export procedures; automated export procedures for seamless data downloads to common statistical packages; and procedures for importing data from external sources. Data analysis Descriptive statistics were performed. Analyses were conducted using STATA software, version 11 (STATA Institute Inc).
RESULTS Of the 118 program directors affiliated with the ACS NSQIP, 3 people could not be reached due to technical difficulties and 1 person opened the survey but responses were not recorded. Therefore, completed questionnaires were received from 66 of 115 (57%) of the program directors surveyed. The majority of respondents were male 53 (80%). Although all respondents described themselves as program directors (n ¼ 66; 100%), 20 (30%) reported their job description also included work as a clinical researcher, and 12 (18%) reported they also worked as a leader in quality improvement. There were no basic scientists in the group. The majority of program director respondents completed their general surgery residency before implementation of the 80-hour duty standards 49 (74%.) Twenty-five respondents reported completing subspecialty fellowship training (38%). All respondents reported involvement with morbidity and mortality conferences. Additional program director attributes are displayed in Table 1. Eighty-five percent of program director respondents (n ¼ 57) reported that education in quality improvement was essential to future professional work in the field of surgery. Seventy-two percent reported confidence that
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
1129
Table 2. Current Status of Quality Improvement Participation and Education in ACS NSQIP-Affiliated General Surgery Residency Programs Stem
The didactics for practice-based learning and improvement and/or systems based practice are taught by. Surgeon, leader in QI Surgeon, assigned, not QI leader Surgical housestaff member, QI track Surgical housestaff member, not QI track Nonsurgeon physician, leader in QI nonphysician QI representative, not in leadership position Don’t know Other Proportion of hospitals participating in the following national QI programs. ACS NSQIP NSQIP University HealthSystem Consortium Don’t know Other*
n
%
31 17 2 8 21 16 0 1
69 38 5 18 48 36 0 2
64 20 18 1 3
97 30 27 2 5
Forty-four program directors reported didactics for practice-based learning and improvement and/or systems-based practice. *American College of Surgeons (ACS) bariatric database, ACS National Accreditation Program for Breast Centers (NAPBC), United Network for Organ Sharing (UNOS), Centers for Medicare and Medicaid Services Core Measures, System-Based QI Database. QI, quality improvement.
quality improvement initiatives could lead to improved surgical care in the local setting (n ¼ 47). Sixty-seven percent of respondents reported didactics in practice-based learning and improvement and systems-based practice (n ¼ 44.) Fifty-eight percent of respondents reported that their residents worked with the quality improvement teams within their organizations (n ¼ 38). Didactics were taught by a broad array of personnel including housestaff (n ¼ 10) (Table 2). Twenty-eight percent (n¼18) of programs reported that at least 50% of their residents track and analyze their patient outcomes, compare them to norms/benchmarks/ published standards, and identify opportunities to make practice improvements. The reported percentage varied significantly by program (Fig. 1). The estimated median percentage of residents within each program that participated in a quality improvement project outside of morbidity and mortality conferences was 50% (range 0% to 100%) (Fig. 2). The median rank order of the core competencies by the program directors with respect to the educational priority that they should receive in graduate surgical education according to the respondents (1, highest priority; 6, lowest priority) was: 1-patient care, 2-medical knowledge, 2-professionalism, 3-interpersonal communication, 3-practice-based learning and improvement, and 4-systems-based practice. There was a general trend toward high priority for patient care and medical knowledge, with great variability in the rankings of the other core competencies.
Attitudes and opinions of program directors outside of the structured questions were captured through solicitation of free text comments. Themes observed among the comments included concern regarding attribution (n ¼ 4), frustration regarding educational mandates that reach outside of the domain of teaching (n ¼ 3), dissatisfaction with current QI educational programs (n ¼ 6), and the desire to continue the status quo (n ¼ 1).
Figure 1. Number of programs by the percent of residents who track and analyze their patients’ clinical outcomes, compare them with norms/benchmarks/published standards, and identify opportunities to make practice improvements.
1130
Kelz et al
Quality Improvement Education
Figure 2. Number of programs by the percent of residents that participate in quality improvement projects outside of morbidity and mortality conferences.
DISCUSSION In this study, we described the current state of resident education in quality science as reported by general surgery program directors affiliated with the ACS NSQIP. We found that in the majority of surgical residency programs sampled, fewer than 50% of residents track and analyze their patient outcomes, compare them with norms/ benchmarks/published standards, and formally identify opportunities to make practice improvements. Furthermore, we determined that the majority of residents rely on surgical morbidity and mortality conferences as the sole source of participation in a quality improvement project. Although many residents undoubtedly graduate without exposure and experience in formal quality improvement projects, the majority of training programs did report didactics targeting practice-based learning and improvement and systems-based practice. Our results likely reflect the relatively new emphasis on incorporation of these competencies into formal surgical education coupled with the educational priority that program directors place on the different competencies. The ACGME core competencies were adopted in 1999 and are slated for complete implementation by 2006.10 Given the introduction of the standard duty hours in 2003, program directors were saddled with the responsibility to adhere to the duty hours while developing a new approach to surgical education using the core competencies and continuing to train surgeons. Therefore, whether consciously or unconsciously, priorities had to be assigned to the educational importance of each of the core competencies in order to systematically approach the conversion process from an educational system based on experiential
J Am Coll Surg
accomplishments to one centered on competence-based outcomes. As demonstrated in our survey results, practice-based learning and improvement and systemsbased practice fell to the bottom of the priority list, with a higher educational urgency placed on competence in patient care, medical knowledge, and professionalism. Program directors are in the business of educating surgeons to provide patient care. The external pressures of public accountability for surgeon decision-making and health care outcomes have resulted in a paradigm shift from experiential accreditation to a system based on outcomes. Despite the phased implementation of the Next Accreditation System (NAS),2 the paradigm shift has created a lot of work for surgical educators without providing additional support. Some of the comments shared by the program directors clearly reflected frustration. However, despite some frustrations, the overwhelming majority of program directors affiliated with the ACS NSQIP believe that there is essential value in teaching residents the science of quality improvement. Several not-for-profit organizations, such as the Institute for Healthcare Improvement 11 and the University HealthSystem Consortium,12 have developed educational programs and databases for use in institutional quality improvement programs. To date, however, these programs are generic and are not surgery specific. The ACS NSQIP is devoted to the measurement and improvement of surgical care in the private sector.13 At the center of the program lies the most valid and reliable surgery-specific data registry available across surgical specialties.14,15 There are more than 400 hospitals currently enrolled. The ACS NSQIP data permit benchmarking of outcomes, generation of best practices guidelines, and an environment to foster collaboration and dissemination of information through frequent conference calls, newsletters, and a national meeting for sharing of information across institutions and disciplines. Given the enormous amount of work required for programs to shift training paradigms to the new accreditation system, collaboration across institutions to develop a thoughtful and comprehensive approach to education in quality science holds great promise. Institutional collaborations consisting of data feedback and training in continuous improvement techniques have effectively resulted in a reduction in mortality and morbidity, improvements in processes of care, and cost-savings across institutions.16,17 The majority of these collaborations have been regional or targeted at specific disease entities within surgery, and none to date have focused exclusively on surgical education. The ACS NSQIP Quality In-Training Initiative (QITI) is the first collaborative convened to address educational issues related to improving the quality of
Vol. 217, No. 6, December 2013
surgical care provided across the United States. Using the most robust surgical outcomes data available, the Quality In-Training Initiative will develop a data-centered educational program for advancement of quality improvement in surgery through indoctrination of our youngest surgeons. The program will be based on developing content that fits within the processes of patient care and allows for transmission of quality improvement knowledge to residents as a routine part of every clinical and educational activity. In so doing, program directors will have support to educate their housestaff in an environment that is steeped in quality improvement and committed to teaching each resident about the tools needed to provide high quality surgical care. Our study has several limitations. The survey instrument has not been previously validated. Additionally, the responses are opinion based and may not accurately capture the quantitative measures as suggested. However, our survey does contain vital information regarding the status of quality improvement education from the individuals responsible for the curriculum and therefore, regardless of the accuracy of the specific numbers reported, the overall themes expressed undoubtedly reflect the state of quality improvement education. An additional potential limitation of the study is that it surveyed ACS NSQIP program directors exclusively. Although ACS NSQIP hospitals have been shown to provide the highest quality surgical care,18 there are no data to suggest that enhanced collaboration between educational and quality improvement personnel or more rigorous quality improvement instruction exist at these hospitals when compared with non-ACS NSQIP hospitals. However, if the choice of surveying the ACS NSQIP program directors were to bias our results, we would expect the ACS NSQIP programs to have better QI education when compared with the other programs, which would suggest that we have overestimated the role that quality improvement education plays in modern surgical training across all accredited programs. If that were the case, then the need for an academic collaboration like the QITI would be even greater than stated by our findings because it would mean that fewer residents are receiving proper training in QI. Furthermore, the ACS NSQIP program directors represent 47% of the general surgery program directors and likely influence a larger portion of the overall number of surgical residents enrolled in residency programs due to the large size of many of the ACS NSQIP sponsor hospitals. Therefore, although the magnitude of our findings might differ if we surveyed all program directors, the need for programmatic support to achieve the QI goals of the new ACGME programs would remain substantial.
Kelz et al
Quality Improvement Education
1131
CONCLUSIONS The QITI is the first national surgical initiative aimed at infusing quality improvement education into the surgical training paradigm using clinical outcomes data. The QITI program will enable program directors to teach the quality improvement materials that they report are both essential to future professional work in the field of surgery and infrequently addressed outside of morbidity and mortality conferences and occasional didactic lectures. Through the local collaboration of quality and educational teams, the QITI hopes to lay the seeds for a future in which surgeons not only report outcomes but also analyze the data and develop tools to deliver optimal patient care as an expected part of the surgical culture. Author Contributions Study conception and design: Kelz, Sellers, Reinke, Morris, Ko Acquisition of data: Kelz, Sellers, Reinke, Morris Analysis and interpretation of data: Kelz, Sellers, Reinke, Medbery, Morris, Ko Drafting of manuscript: Kelz, Sellers, Reinke, Medbery, Morris Critical revision: Kelz, Sellers, Reinke, Medbery, Morris, Ko REFERENCES 1. Accreditation Council for Graduate Medical Education. Program requirements for graduate medical education in general surgery. 2012. Available at: http://www.acgme.org/ acgmeweb/Portals/0/PFAssets/ProgramRequirements/440_ general_surgery_01012008_07012012.pdf. Accessed May 1, 2012. 2. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation systemerationale and benefits. N Engl J Med 2012;366:1051e1056. 3. Sachdeva AK, Philibert I, Leach DC, et al. Patient safety curriculum for surgical residency programs: results of a national consensus conference. Surgery 2007;141:427e441. 4. Accreditation Council for Graduate Medical Education. Clinical Learning Environment Review (CLER) Program. Available at: http://www.acgme-nas.org/cler.html. Accessed April 15, 2013. 5. GME Commission. Graduate medical education financing: Focusing on educational priorities. In: Report to the Congress: Aligning Incentives in Medicare; 2010:103e125. 6. 111th United States Congress. America’s Affordable Health Choices Act of 2009. Washington, DC; 2009:HR 4872, Sec. 1505. Improving accountability for approved medical residency training. 7. Weinstein MD, Chair D. Ensuring an Effective Physician Workforce for the United States Recommendations for Graduate Medical Education to Meet the Needs of the Public. Atlanta, GA: Josiah Macy Jr Foundation; 2011. 8. Sellers M, Reinke CE, Kreider S, et al. ACS NSQIP quality in-training initiative pilot study. J Am Coll Surg 2013;217: 827e832.
1132
Kelz et al
Quality Improvement Education
9. Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)ea metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377e381. 10. Leach DC. Building and assessing competence: the potential for evidence-based graduate medical education. Qual Manage Health Care 2002;11:39e44. 11. Institute for Healthcare Improvement. Available at: http:// www.ihi.org. Accessed April 26, 2013. 12. University HealthSystem Consortium. Available at: www.uhc. edu. Accessed April 26, 2013. 13. American College of Surgeons National Surgical Quality Improvement Program. About ACS NSQIP: Program Overview. Available at: www.acsnsqip.org. Accessed December 19, 2011. 14. Fink AS, Campbell DA Jr, Mentzer RM Jr, et al. The National Surgical Quality Improvement Program in non-Veterans Administration hospitals: initial demonstration of feasibility. Ann Surg 2002;236:344e353; discussion 353e354.
J Am Coll Surg
15. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228:491e507. 16. Campbell DA Jr, Dellinger EP. Multihospital collaborations for surgical quality improvement. JAMA 2009;302: 1584e1585. 17. Guillamondegui OD, Gunter OL, Hines L, et al. Using the National Surgical Quality Improvement Program and the Tennessee Surgical Quality Collaborative to improve surgical outcomes. J Am Coll Surg 2012;214:709e714; discussion 714e716. 18. Hall BL, Hamilton BH, Richards K, et al. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 2009;250:363e376.
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
Figure 1. Sample of a Resident Report Currently Available for Use in Graduate Surgical Education
1132.e1
1132.e2
Kelz et al
Quality Improvement Education
J Am Coll Surg
Figure 2. Sample of the Proposed Resident Report after Centralization of the Reporting Process. Note. The report includes the following information: number of cases, the proportion of patients with 0 to 5þ risk factors, outcomes (readmissions, unplanned readmissions, readmissions related to the principal procedure, unplanned return to the operating room, death) and postoperative occurrences (wound occurrences, respiratory occurrences, urinary tract occurrences, central nervous system occurrences, cardiac occurrences, and other occurrences including transfusions, vein thrombosis, sepsis, and septic shock.) Comparison groups include the index resident, all QITI resident outcomes for the same postgraduate year (PGY) as the index resident, all residents from the index hospital at the same PGY as the index resident, and the index resident outcomes for the 6 reporting periods. All case details for the index resident will be included on the subsequent pages of the report.
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
Figure 3. Sample of the Team Report Currently Available for Use in Graduate Surgical Education
1132.e3
1132.e4
Kelz et al
Quality Improvement Education
J Am Coll Surg
Figure 4. Sample of the Proposed Team Report after Centralization of the Reporting Process. Note: The report includes the following information: number of cases, the proportion of patients with 0 to 5þ risk factors, outcomes (readmissions, unplanned readmissions, readmissions related to the principal procedure, unplanned return to the operating room, death) and postoperative occurrences (wound occurrences, respiratory occurrences, urinary tract occurrences, central nervous system occurrences, cardiac occurrences, and other occurrences including transfusions, vein thrombosis, sepsis, and septic shock.) Comparison group includes the team outcomes for the same service for the 6 months preceding the reporting period. All case details for the index team will be included on the subsequent pages of the report.
Vol. 217, No. 6, December 2013
Kelz et al
Quality Improvement Education
1132.e5
Figure 5. Sample of a Graphical Report. CVA, cerebrovascular accident; QITI, Quality In-Training Initiative; SSI, surgical site infection; UTI, urinary tract infection; VTE, venous thromboembolism.
