558086
research-article2014
AJMXXX10.1177/1062860614558086American Journal of Medical QualityYanamadala et al
Article
Clinical Quality Improvement Curriculum for Faculty in an Academic Medical Center
American Journal of Medical Quality 1–8 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614558086 ajmq.sagepub.com
Mamata Yanamadala, MBBS1, Lisa G. Criscione-Schreiber, MD1, Jeffrey Hawley, BS, Mitchell T. Heflin, MD, MHS1, and Bimal R. Shah, MD, MBA1
Abstract Interested faculty enrolled in this 6-month-long quality improvement (QI) course to facilitate independent QI project work. The course included monthly 1.5-hour sessions: 20-minute presentations covering key QI concepts, then small group activities to facilitate project work. Faculty were required to identify, construct, and implement an independent QI project. They met individually with mentors twice during the course, with additional guidance offered virtually via phone or e-mail, and completed pretests and posttests of QI knowledge (maximum score = 15) and self-assessed confidence. A statistically significant difference in knowledge (pre-course mean = 7.75, standard deviation [SD] = 3.06; post-course mean = 11.75, SD = 3.28; P = .02) and self-assessed confidence (pre mean = 3.08, SD = 0.65; post mean = 4.5, SD = 0.68; P < .0001) was found. Of 8 faculty, 5 were able to conduct small tests of change; 3 studied the current processes and planned to run tests of change. Positive responses to this course helped obtain buy-in from leadership to develop a leadership program in QI. Keywords quality improvement, faculty development, curriculum, education Lack of health care provider expertise in quality improvement (QI) is one of the major barriers to successfully completing QI endeavors in health care settings.1,2 Many thought leaders, including professional organizations in medicine and nursing, have endorsed teaching QI to clinicians after recognizing the link between provider knowledge in QI and improvements in quality of care.3-6 Physician accreditation bodies require QI education as part of training and performance improvement modules to maintain certification. Despite these calls to action and a number of published educational innovations in this area, most medical schools and residency training programs lack quality and safety curricula. A key barrier to their development is the absence of faculty with the appropriate experiences and skills to teach these topics.7,8 Because the requirement for teaching QI skills during residency is recent, most current medical faculty members have received general exposure to QI only from clinical experience rather than as a part of their formal education and training, resulting in deficits in QI knowledge.9,10 To date, very few curricula for faculty development in QI have been tested and published.11-13 For busy faculty members to devote time to learning how to
conduct QI projects, curricula must have proven efficacy, make efficient use of their time investment, and must have a positive impact on their clinical practice and teaching. Therefore, it is critical that curricula in QI for faculty members be formally evaluated to understand their impact on these multiple domains. To date, the study institution has not offered a formal structured course for clinical faculty that focuses on improvement science. Therefore, the research team aimed to create and pilot a curriculum for teaching QI to faculty.
Methods The research team designed a pilot curriculum in improvement science for faculty. Enrollment in the program was limited to 8 faculty members from the Department of Medicine at Duke University; all volunteered for the 1
Duke University School of Medicine, Durham, NC
Corresponding Author: Mamata Yanamadala, MBBS, Division of Geriatrics, Department of Medicine, Duke University School of Medicine, DUMC Box 3003, Durham, NC 27710. Email: [email protected]
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
2
American Journal of Medical Quality
course based on either their role in leading or teaching QI or their interest in furthering their QI skills. A preevaluation and postevaluation of QI knowledge and selfassessed confidence was constructed as part of the program. The Duke University Institutional Review Board granted exemption for the evaluation of the program.
Curriculum Development Based in part on a program developed for residents and fellows,14 this curriculum aimed to provide a structured background to QI work, relevant literature readings, pertinent case studies, and independent project generation. After reviewing external resources and existing literature11-13 on faculty development programs in QI and consulting local and national experts, the curriculum was structured using content in the books Fundamentals of Health Care Improvement: A Guide to Improving Your Patients’ Care15 and The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.16 From these various sources, course leaders selectively prioritized topics for inclusion in the course based on the time allotted and the experience level of participants.
Curriculum Structure The curriculum was structured to combine didactic lectures, independent reading from both the aforementioned books and selected peer-reviewed journal articles, and development of an independent QI project in participants’ clinical areas (Table 1). The curriculum involved QI techniques using the principles of the Model for Improvement.16 The curriculum was delivered as 6 monthly, 1.5-hour sessions. Each session included a 15to 20-minute didactic presentation that covered key concepts in QI methodology, followed by small group content application activities to facilitate working on the learners’ QI projects (described further in the following section). The course was designed to facilitate sequential understanding of the approach to QI projects. Participants were assigned pre readings from the books and publications and tasks specific to their projects, such as define a local problem, write project charters, map existing processes, and identify areas for improvement.15,16
Project Development and Implementation To participate in the curriculum, participants agreed to identify, lead, and implement a QI project by completing one improvement cycle within their clinical setting. Participants were asked to identify relevant quality gaps in their practice area around clinical care delivery, process metrics, or performance measures. Participants brainstormed in groups during the first session about
project ideas and finalized one project concept to pursue during the course. Participants received guidance to refine their topic from the course directors (MY, BRS), who met with them individually twice during the course of the 6 months, with additional mentoring offered by course directors virtually through e-mail and telephone. Both course directors spent 3 half days over the 6 months of the course to meet with participants for mentoring and had protected time for mentoring. Participants presented their project ideas at session 3 and the results of the implementation or initial results at the end of the course to their coparticipants, course directors, and department leadership.
Evaluation A pre-post design was used to measure changes in selfassessed competence in the objectives of the course and knowledge of QI principles. Prior to attending the course, learners completed 3 assessments: (1) the previously validated Quality Improvement Knowledge Application Tool (QIKAT) pretest, (2) a baseline selfefficacy survey designed by the research team to measure self-efficacy regarding course objectives on a 5-point Likert scale, and (3) a multiple-choice questions (MCQs) knowledge test.17 The QIKAT, MCQs, and selfassessment were all administered during 15 minutes of the first session. The QIKAT has previously been validated in graduate medical education (GME) trainees and has been shown to discriminate between learners who have had training in QI and those who have not.17 It presents 3 scenarios, each of which is followed by 3 questions, including stating an aim, choosing measures, and choosing an intervention (the 3 questions in the Model for Improvement) associated with QI initiatives. Responses are scored from 0 to 5 for each scenario, giving a maximum score of 15.17 The open-ended format forces learners to generate answers spontaneously, avoiding cuing and enabling the assessment of problemsolving ability. Although QIKAT is a validated tool used widely to test QI knowledge and is structured to test knowledge in the Model for Improvement, it poses challenges in evaluating learner performance because of the qualitative nature of the responses. Therefore, a quantitative assessment with 7 MCQs targeting key concepts taught in the course was developed and used to supplement the QIKAT. The key concepts were identified for the MCQs using goals and objectives for the individual sessions. The postprogram summative evaluation assessed the effectiveness of the teaching modes and materials and course mentoring to plan future quality education programs. The post evaluation included a post-QIKAT, a self-efficacy survey, and MCQs and was e-mailed to the participants, who completed them outside of class.
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
3
Yanamadala et al Table 1. Course Structure. Session Title and Objectives
Session Activities
Follow-up Exercises
Session I: Introduction to QI •• None
•• Complete assigned readings •• Identify issues that might have a quality gap in your clinical setting •• Bring 2-3 clinically relevant issues (that you think are posing quality gaps in your practice area) that you think you can use for a QI project for classwork
•• Write your problems in one sentence and rate them if you have several of them •• Write an aim statement for your project
•• Complete assigned readings •• Conduct a literature search to find evidence about the topic and write a 1-paragraph background. Define the local problem in 1 paragraph •• Complete your project charter
•• Create a process model for your project •• Select measures for projects
•• Complete assigned readings •• Write QI proposal, be sure to include your background, local problem, stakeholders, process model, and measures •• Meet with mentors
•• Design changes for projects
•• Complete assigned readings •• Conduct first PDSA cycle
•• Recognize the value of analyzing data over time by using run charts and statistical process control charts •• Describe the difference between common cause variation and special cause variation •• Interpret run charts and statistical process control charts
•• Work on case examples for run charts
•• Analyze data from improvement cycles •• Prepare for final presentations
Session VI: Final presentations
•• Provide feedback to peer presentations
•• None
•• Define quality of health care •• State the current state of health care quality in the United States •• List various QI methodologies
Session II: Improvement science •• Review of QI methodology •• Write an effective aim statement for your improvement project •• Illustrate Model for Improvement for your project •• Define local problem
Session III: Process modeling and measures in QI •• Create a process model for your project •• Describe measures for your project
Session IV: Testing changes •• Identify barriers to developing changes •• State the principles of testing a change •• Identify principles of implementing change Session V: Understanding variation and data in QI
Abbreviations: PDSA, Plan-Do-Study-Act; QI, quality improvement.
Outcome Measures The primary outcome measures were the change in the QIKAT scores and implementation of the QI projects: (1) complete a current process analysis and (2) complete a test of change. The implementation of the QI projects was measured by reviewing participants’ presentations. Secondary outcome measures were the self-efficacy results. The number of GME trainees mentored for their QI projects by the participants also was tracked. A
program evaluation survey designed to gather formative feedback from the participants was analyzed.
Statistical Analysis Paired t tests were run for the change scores (pre-post) for QIKAT and self-efficacy surveys to determine whether there was a significant change in knowledge (QIKAT) and confidence (self-efficacy). The analysis was conducted using IBM SPSS Version 20 (IBM SPSS, Chicago,
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
4
American Journal of Medical Quality
Table 2. Knowledge and Self-efficacy Scores of the Faculty. Pre intervention
Post intervention
Evaluation
n
Mean
SD
Mean
SD
P Value (for Pre-Post) Change
QIKAT (score range = 0-15) Multiple choice test (total score = 7) Self-efficacy (scale = 1-5)
8 8 7
7.75 5.88 33.00
3.06 0.98 9.81
11.75 6.25 44.84
3.28 0.70 4.49
.02a .50 .008
Abbreviation: QIKAT, Quality Improvement Knowledge Application Tool; SD, standard deviation. a Paired sample t test P value based on test of percentage correct.
Illinois). Qualitative analysis of participant presentations was conducted for outcomes on their specific projects.
Results In all, 8 faculty members participated in the curriculum. There was a high degree of variability in age, clinical setting, academic rank, and experience with QI projects. Five of the participants were assistant professors, 2 were associate professors, and 1 was a medical instructor. The numbers of years in practice for the participants ranged between 1 and 25 years. Attendance at these sessions was high; all participants were present for 4 of the 6 sessions, 2 participants were absent for the session that included discussion on the process model, and 1 was absent for another session focused on data analysis. Aggregate attendance for all sessions was 94%.
Curriculum Evaluation Program evaluation results are displayed in Figure 2. All participants felt that the course format was useful to implement a project, and all responded that they would recommend the course to others. A majority of participants (5/8) felt that the pace and course content were appropriate. In general, participants consistently preferred group discussion and troubleshooting of one another’s project progress over topic-specific learning, as reflected in the following comments made by the participants in the course evaluation in response to the question, “What were the highlights of the course?” Collaborative learning, a mix of didactics, group work, individual assignments, and readings. Discussing/developing ideas for QI project with classmates.
Assessment of Participants’ Knowledge A statistically significant difference in knowledge was seen between the preprogram and postprogram QIKAT scores (maximum score = 15; P = .02; Table 2). A statistically significant difference was noted on self-assessed confidence on key QI concepts taught in the course (scale of 1-5; P < .0001; Table 2 and Figure 1). There was no significant difference in the mean scores on the multiplechoice assessment of knowledge (Table 2).
Project Implementation All participants were able to implement a QI project within the 6 months of the course period. As part of the project, they were able to define a local problem, write project charters, map existing processes, and identify areas for improvement with the help of mentoring inside the classroom and additional scheduled meetings with mentors. Of the 8 participants, 5 were able to implement their projects by conducting small tests of change within the 6-month period. The remaining 3 participants had an opportunity to review the current process and will run the tests of change after the 6 months. Project outcomes over the period of the course are outlined in Table 3.
Sharing QI projects and discussion about barriers and solutions.
One participant felt that the readings were lengthy and the didactics covered basic concepts only.
Discussion This 6-month faculty QI program was successful in improving QI knowledge and self-confidence in key QI components. This program was a unique opportunity to integrate a QI curriculum with a group of faculty who were interested in obtaining this training. Significant improvement was noted in the application of the QI concepts learned both on the QIKAT and during project work. The participants did better on QIKAT in comparison to the results reported from several other curricula.14 Faculty also reported improved confidence in proficiency of QI concepts but did not show any difference on the MCQs. Finally, all participants were successful in proposing and implementing a QI project. Most QI training programs in the published literature focus on multidisciplinary team training at a practice level.11 To the research team’s knowledge, there is only
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
5
Yanamadala et al
1. Idenfy gaps in health care quality 2. Describe model of quality improvement 3. Write an aim statement 4. Choose a process modeling method
Post-Test Mean
5. Describe one model for idenfying measures
Pre-Test Mean
6. Differenate common cause and special cause variaon 7.Interpret components of run charts 8.Apply Plan-Do-Study-Act cycle 9.Collaborate effecvely with other professionals 10. Make a change to improve health care 0
1
2
3
4
5
Figure 1. Self-efficacy in QI concepts measured before and after completion of the course. Confidence level: 1 = not confident at all; 2 = not confident; 3 = reasonably confident; 4 = confident; 5 = very confident. Abbreviations: QI, quality improvement.
one other published study that has focused on developing physician QI skills in an academic institution.12 That training program differed from the program discussed in this article with respect to the time spent and the mentorship the participants received. Their participants met biweekly for 2 hours and had one mentor assigned per participant who met face-to-face with the participants twice monthly. Although the curriculum presented here focused on the basics of identifying local QI problems and the process of developing and implementing local QI projects, it did not emphasize additional topics such as leadership or culture change. The expectation that participants would present their findings to leadership at the end of the program created a sense of importance, urgency, and validation of participants’ work in both programs. The shorter time period of the program presented in this article (6 vs 9 months) may have prevented individuals from completing multiple improvement cycles by the end of the course. Nevertheless, in this program, everyone identified a local problem or gap, designed a project, convened stakeholders, and collected crucial baseline data to start their projects.
Lessons Learned The experience from this pilot program will help the research team design future, similar programs. Individual
mentoring is key to the success of project work in QI as highlighted in prior studies.12,13 Participants in this pilot curriculum highly valued periodic group discussions of their project work and sharing systems knowledge and experiences and felt that the discussions facilitated their project work. Some participants felt the need for more advanced skills training, whereas others felt that the course content was adequate. In the future, the research team is considering adding a second course on more advanced skills in QI project implementation.
Limitations The pre-course and post-course scores on the MCQs were both very high without a statistically significant difference. This finding could indicate that the baseline QI knowledge of the faculty was high or could demonstrate a real gap between having fundamental QI knowledge versus applying QI knowledge to scenarios, given that faculty had a significant gain in the previously validated pre-course and post-course knowledge application test. Alternatively, the lack of change could indicate that the MCQs, as written, were unable to distinguish levels of performance. Additionally, because of the small sample of faculty, the study may have lacked the power to detect real change. The MCQs written by the research team must be edited and validated before further use.
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
6
American Journal of Medical Quality
Table 3. Participants’ Project Aim Statements, Project Outcomes in 6 Months, and Institute of Medicine (IOM)18 Aims Addressed by the Projects. Project Aim Statement
Preintervention Data
In the next 3 months, the QI team will reduce delays in reporting TEE results by 50% The clinic staff will decrease check-in and nurse triage time by 50% over the next 4 months The providers and transfer center will collaborate to improve the proportion of patients arriving with standardized salient clinical data by 25% over the next 3 months The outpatient clinic will increase the rates of follow-up appointments within 14 days of hospital discharge by 25% by July 2013 By July 2013, there will be a protocol for the medicine residents to follow before sending a patient they are evaluating over the phone to the ED, 50% of the time In the next 4 months, the clinic will improve total messages through patient portal by 30% The providers will address and document conformance to 3 cross-cutting quality measures in 80% of patients in 3 months In the next 12 months, we will increase rates of timely completion and transmission of discharge summaries for hospitalized patients on general medicine by 50%
Postintervention Data
n = 176 Weekday: 12/177 (7%) with reports in beyond 24 hours Weekend: 27/59 (46%) with reports in beyond 24 hours
n = 193 Weekday: 4/195 (2%) with reports in beyond 24 hours Weekend: 29/78 (37%) with reports in beyond 24 hours
Nurse triage time: 12 minutes for nondiabetic patients Nurse triage time: 18 minutes for diabetic patients No standard process exists to collect information on accepting patients for transfer
Detailed evaluation of the current process was done and areas for improvement identified for implementation Standardized process developed, piloted, modified based on feedback, and expanded. Data will be collected in next cycles 74% Had scheduled follow-up in 7 days 79% Had scheduled follow-up in 14 days 79% Had scheduled follow-up in 30 days New policy was made regarding phone calls before sending patients to the ED. Data will be collected in future months
14% Had scheduled follow-up in 7 days 46% Had scheduled follow-up in 14 days 61% Had scheduled follow-up in 30 days Residents’ and attendings’ attitudes were evaluated. Current process for phone call was studied Patients and staff attitudes were studied about the use of the patient portal Studied current practice Completed
research-article2014
AJMXXX10.1177/1062860614558086American Journal of Medical QualityYanamadala et al
Article
Clinical Quality Improvement Curriculum for Faculty in an Academic Medical Center
American Journal of Medical Quality 1–8 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614558086 ajmq.sagepub.com
Mamata Yanamadala, MBBS1, Lisa G. Criscione-Schreiber, MD1, Jeffrey Hawley, BS, Mitchell T. Heflin, MD, MHS1, and Bimal R. Shah, MD, MBA1
Abstract Interested faculty enrolled in this 6-month-long quality improvement (QI) course to facilitate independent QI project work. The course included monthly 1.5-hour sessions: 20-minute presentations covering key QI concepts, then small group activities to facilitate project work. Faculty were required to identify, construct, and implement an independent QI project. They met individually with mentors twice during the course, with additional guidance offered virtually via phone or e-mail, and completed pretests and posttests of QI knowledge (maximum score = 15) and self-assessed confidence. A statistically significant difference in knowledge (pre-course mean = 7.75, standard deviation [SD] = 3.06; post-course mean = 11.75, SD = 3.28; P = .02) and self-assessed confidence (pre mean = 3.08, SD = 0.65; post mean = 4.5, SD = 0.68; P < .0001) was found. Of 8 faculty, 5 were able to conduct small tests of change; 3 studied the current processes and planned to run tests of change. Positive responses to this course helped obtain buy-in from leadership to develop a leadership program in QI. Keywords quality improvement, faculty development, curriculum, education Lack of health care provider expertise in quality improvement (QI) is one of the major barriers to successfully completing QI endeavors in health care settings.1,2 Many thought leaders, including professional organizations in medicine and nursing, have endorsed teaching QI to clinicians after recognizing the link between provider knowledge in QI and improvements in quality of care.3-6 Physician accreditation bodies require QI education as part of training and performance improvement modules to maintain certification. Despite these calls to action and a number of published educational innovations in this area, most medical schools and residency training programs lack quality and safety curricula. A key barrier to their development is the absence of faculty with the appropriate experiences and skills to teach these topics.7,8 Because the requirement for teaching QI skills during residency is recent, most current medical faculty members have received general exposure to QI only from clinical experience rather than as a part of their formal education and training, resulting in deficits in QI knowledge.9,10 To date, very few curricula for faculty development in QI have been tested and published.11-13 For busy faculty members to devote time to learning how to
conduct QI projects, curricula must have proven efficacy, make efficient use of their time investment, and must have a positive impact on their clinical practice and teaching. Therefore, it is critical that curricula in QI for faculty members be formally evaluated to understand their impact on these multiple domains. To date, the study institution has not offered a formal structured course for clinical faculty that focuses on improvement science. Therefore, the research team aimed to create and pilot a curriculum for teaching QI to faculty.
Methods The research team designed a pilot curriculum in improvement science for faculty. Enrollment in the program was limited to 8 faculty members from the Department of Medicine at Duke University; all volunteered for the 1
Duke University School of Medicine, Durham, NC
Corresponding Author: Mamata Yanamadala, MBBS, Division of Geriatrics, Department of Medicine, Duke University School of Medicine, DUMC Box 3003, Durham, NC 27710. Email: [email protected]
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
2
American Journal of Medical Quality
course based on either their role in leading or teaching QI or their interest in furthering their QI skills. A preevaluation and postevaluation of QI knowledge and selfassessed confidence was constructed as part of the program. The Duke University Institutional Review Board granted exemption for the evaluation of the program.
Curriculum Development Based in part on a program developed for residents and fellows,14 this curriculum aimed to provide a structured background to QI work, relevant literature readings, pertinent case studies, and independent project generation. After reviewing external resources and existing literature11-13 on faculty development programs in QI and consulting local and national experts, the curriculum was structured using content in the books Fundamentals of Health Care Improvement: A Guide to Improving Your Patients’ Care15 and The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.16 From these various sources, course leaders selectively prioritized topics for inclusion in the course based on the time allotted and the experience level of participants.
Curriculum Structure The curriculum was structured to combine didactic lectures, independent reading from both the aforementioned books and selected peer-reviewed journal articles, and development of an independent QI project in participants’ clinical areas (Table 1). The curriculum involved QI techniques using the principles of the Model for Improvement.16 The curriculum was delivered as 6 monthly, 1.5-hour sessions. Each session included a 15to 20-minute didactic presentation that covered key concepts in QI methodology, followed by small group content application activities to facilitate working on the learners’ QI projects (described further in the following section). The course was designed to facilitate sequential understanding of the approach to QI projects. Participants were assigned pre readings from the books and publications and tasks specific to their projects, such as define a local problem, write project charters, map existing processes, and identify areas for improvement.15,16
Project Development and Implementation To participate in the curriculum, participants agreed to identify, lead, and implement a QI project by completing one improvement cycle within their clinical setting. Participants were asked to identify relevant quality gaps in their practice area around clinical care delivery, process metrics, or performance measures. Participants brainstormed in groups during the first session about
project ideas and finalized one project concept to pursue during the course. Participants received guidance to refine their topic from the course directors (MY, BRS), who met with them individually twice during the course of the 6 months, with additional mentoring offered by course directors virtually through e-mail and telephone. Both course directors spent 3 half days over the 6 months of the course to meet with participants for mentoring and had protected time for mentoring. Participants presented their project ideas at session 3 and the results of the implementation or initial results at the end of the course to their coparticipants, course directors, and department leadership.
Evaluation A pre-post design was used to measure changes in selfassessed competence in the objectives of the course and knowledge of QI principles. Prior to attending the course, learners completed 3 assessments: (1) the previously validated Quality Improvement Knowledge Application Tool (QIKAT) pretest, (2) a baseline selfefficacy survey designed by the research team to measure self-efficacy regarding course objectives on a 5-point Likert scale, and (3) a multiple-choice questions (MCQs) knowledge test.17 The QIKAT, MCQs, and selfassessment were all administered during 15 minutes of the first session. The QIKAT has previously been validated in graduate medical education (GME) trainees and has been shown to discriminate between learners who have had training in QI and those who have not.17 It presents 3 scenarios, each of which is followed by 3 questions, including stating an aim, choosing measures, and choosing an intervention (the 3 questions in the Model for Improvement) associated with QI initiatives. Responses are scored from 0 to 5 for each scenario, giving a maximum score of 15.17 The open-ended format forces learners to generate answers spontaneously, avoiding cuing and enabling the assessment of problemsolving ability. Although QIKAT is a validated tool used widely to test QI knowledge and is structured to test knowledge in the Model for Improvement, it poses challenges in evaluating learner performance because of the qualitative nature of the responses. Therefore, a quantitative assessment with 7 MCQs targeting key concepts taught in the course was developed and used to supplement the QIKAT. The key concepts were identified for the MCQs using goals and objectives for the individual sessions. The postprogram summative evaluation assessed the effectiveness of the teaching modes and materials and course mentoring to plan future quality education programs. The post evaluation included a post-QIKAT, a self-efficacy survey, and MCQs and was e-mailed to the participants, who completed them outside of class.
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
3
Yanamadala et al Table 1. Course Structure. Session Title and Objectives
Session Activities
Follow-up Exercises
Session I: Introduction to QI •• None
•• Complete assigned readings •• Identify issues that might have a quality gap in your clinical setting •• Bring 2-3 clinically relevant issues (that you think are posing quality gaps in your practice area) that you think you can use for a QI project for classwork
•• Write your problems in one sentence and rate them if you have several of them •• Write an aim statement for your project
•• Complete assigned readings •• Conduct a literature search to find evidence about the topic and write a 1-paragraph background. Define the local problem in 1 paragraph •• Complete your project charter
•• Create a process model for your project •• Select measures for projects
•• Complete assigned readings •• Write QI proposal, be sure to include your background, local problem, stakeholders, process model, and measures •• Meet with mentors
•• Design changes for projects
•• Complete assigned readings •• Conduct first PDSA cycle
•• Recognize the value of analyzing data over time by using run charts and statistical process control charts •• Describe the difference between common cause variation and special cause variation •• Interpret run charts and statistical process control charts
•• Work on case examples for run charts
•• Analyze data from improvement cycles •• Prepare for final presentations
Session VI: Final presentations
•• Provide feedback to peer presentations
•• None
•• Define quality of health care •• State the current state of health care quality in the United States •• List various QI methodologies
Session II: Improvement science •• Review of QI methodology •• Write an effective aim statement for your improvement project •• Illustrate Model for Improvement for your project •• Define local problem
Session III: Process modeling and measures in QI •• Create a process model for your project •• Describe measures for your project
Session IV: Testing changes •• Identify barriers to developing changes •• State the principles of testing a change •• Identify principles of implementing change Session V: Understanding variation and data in QI
Abbreviations: PDSA, Plan-Do-Study-Act; QI, quality improvement.
Outcome Measures The primary outcome measures were the change in the QIKAT scores and implementation of the QI projects: (1) complete a current process analysis and (2) complete a test of change. The implementation of the QI projects was measured by reviewing participants’ presentations. Secondary outcome measures were the self-efficacy results. The number of GME trainees mentored for their QI projects by the participants also was tracked. A
program evaluation survey designed to gather formative feedback from the participants was analyzed.
Statistical Analysis Paired t tests were run for the change scores (pre-post) for QIKAT and self-efficacy surveys to determine whether there was a significant change in knowledge (QIKAT) and confidence (self-efficacy). The analysis was conducted using IBM SPSS Version 20 (IBM SPSS, Chicago,
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
4
American Journal of Medical Quality
Table 2. Knowledge and Self-efficacy Scores of the Faculty. Pre intervention
Post intervention
Evaluation
n
Mean
SD
Mean
SD
P Value (for Pre-Post) Change
QIKAT (score range = 0-15) Multiple choice test (total score = 7) Self-efficacy (scale = 1-5)
8 8 7
7.75 5.88 33.00
3.06 0.98 9.81
11.75 6.25 44.84
3.28 0.70 4.49
.02a .50 .008
Abbreviation: QIKAT, Quality Improvement Knowledge Application Tool; SD, standard deviation. a Paired sample t test P value based on test of percentage correct.
Illinois). Qualitative analysis of participant presentations was conducted for outcomes on their specific projects.
Results In all, 8 faculty members participated in the curriculum. There was a high degree of variability in age, clinical setting, academic rank, and experience with QI projects. Five of the participants were assistant professors, 2 were associate professors, and 1 was a medical instructor. The numbers of years in practice for the participants ranged between 1 and 25 years. Attendance at these sessions was high; all participants were present for 4 of the 6 sessions, 2 participants were absent for the session that included discussion on the process model, and 1 was absent for another session focused on data analysis. Aggregate attendance for all sessions was 94%.
Curriculum Evaluation Program evaluation results are displayed in Figure 2. All participants felt that the course format was useful to implement a project, and all responded that they would recommend the course to others. A majority of participants (5/8) felt that the pace and course content were appropriate. In general, participants consistently preferred group discussion and troubleshooting of one another’s project progress over topic-specific learning, as reflected in the following comments made by the participants in the course evaluation in response to the question, “What were the highlights of the course?” Collaborative learning, a mix of didactics, group work, individual assignments, and readings. Discussing/developing ideas for QI project with classmates.
Assessment of Participants’ Knowledge A statistically significant difference in knowledge was seen between the preprogram and postprogram QIKAT scores (maximum score = 15; P = .02; Table 2). A statistically significant difference was noted on self-assessed confidence on key QI concepts taught in the course (scale of 1-5; P < .0001; Table 2 and Figure 1). There was no significant difference in the mean scores on the multiplechoice assessment of knowledge (Table 2).
Project Implementation All participants were able to implement a QI project within the 6 months of the course period. As part of the project, they were able to define a local problem, write project charters, map existing processes, and identify areas for improvement with the help of mentoring inside the classroom and additional scheduled meetings with mentors. Of the 8 participants, 5 were able to implement their projects by conducting small tests of change within the 6-month period. The remaining 3 participants had an opportunity to review the current process and will run the tests of change after the 6 months. Project outcomes over the period of the course are outlined in Table 3.
Sharing QI projects and discussion about barriers and solutions.
One participant felt that the readings were lengthy and the didactics covered basic concepts only.
Discussion This 6-month faculty QI program was successful in improving QI knowledge and self-confidence in key QI components. This program was a unique opportunity to integrate a QI curriculum with a group of faculty who were interested in obtaining this training. Significant improvement was noted in the application of the QI concepts learned both on the QIKAT and during project work. The participants did better on QIKAT in comparison to the results reported from several other curricula.14 Faculty also reported improved confidence in proficiency of QI concepts but did not show any difference on the MCQs. Finally, all participants were successful in proposing and implementing a QI project. Most QI training programs in the published literature focus on multidisciplinary team training at a practice level.11 To the research team’s knowledge, there is only
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
5
Yanamadala et al
1. Idenfy gaps in health care quality 2. Describe model of quality improvement 3. Write an aim statement 4. Choose a process modeling method
Post-Test Mean
5. Describe one model for idenfying measures
Pre-Test Mean
6. Differenate common cause and special cause variaon 7.Interpret components of run charts 8.Apply Plan-Do-Study-Act cycle 9.Collaborate effecvely with other professionals 10. Make a change to improve health care 0
1
2
3
4
5
Figure 1. Self-efficacy in QI concepts measured before and after completion of the course. Confidence level: 1 = not confident at all; 2 = not confident; 3 = reasonably confident; 4 = confident; 5 = very confident. Abbreviations: QI, quality improvement.
one other published study that has focused on developing physician QI skills in an academic institution.12 That training program differed from the program discussed in this article with respect to the time spent and the mentorship the participants received. Their participants met biweekly for 2 hours and had one mentor assigned per participant who met face-to-face with the participants twice monthly. Although the curriculum presented here focused on the basics of identifying local QI problems and the process of developing and implementing local QI projects, it did not emphasize additional topics such as leadership or culture change. The expectation that participants would present their findings to leadership at the end of the program created a sense of importance, urgency, and validation of participants’ work in both programs. The shorter time period of the program presented in this article (6 vs 9 months) may have prevented individuals from completing multiple improvement cycles by the end of the course. Nevertheless, in this program, everyone identified a local problem or gap, designed a project, convened stakeholders, and collected crucial baseline data to start their projects.
Lessons Learned The experience from this pilot program will help the research team design future, similar programs. Individual
mentoring is key to the success of project work in QI as highlighted in prior studies.12,13 Participants in this pilot curriculum highly valued periodic group discussions of their project work and sharing systems knowledge and experiences and felt that the discussions facilitated their project work. Some participants felt the need for more advanced skills training, whereas others felt that the course content was adequate. In the future, the research team is considering adding a second course on more advanced skills in QI project implementation.
Limitations The pre-course and post-course scores on the MCQs were both very high without a statistically significant difference. This finding could indicate that the baseline QI knowledge of the faculty was high or could demonstrate a real gap between having fundamental QI knowledge versus applying QI knowledge to scenarios, given that faculty had a significant gain in the previously validated pre-course and post-course knowledge application test. Alternatively, the lack of change could indicate that the MCQs, as written, were unable to distinguish levels of performance. Additionally, because of the small sample of faculty, the study may have lacked the power to detect real change. The MCQs written by the research team must be edited and validated before further use.
Downloaded from ajm.sagepub.com at Duke University Libraries on November 15, 2015
6
American Journal of Medical Quality
Table 3. Participants’ Project Aim Statements, Project Outcomes in 6 Months, and Institute of Medicine (IOM)18 Aims Addressed by the Projects. Project Aim Statement
Preintervention Data
In the next 3 months, the QI team will reduce delays in reporting TEE results by 50% The clinic staff will decrease check-in and nurse triage time by 50% over the next 4 months The providers and transfer center will collaborate to improve the proportion of patients arriving with standardized salient clinical data by 25% over the next 3 months The outpatient clinic will increase the rates of follow-up appointments within 14 days of hospital discharge by 25% by July 2013 By July 2013, there will be a protocol for the medicine residents to follow before sending a patient they are evaluating over the phone to the ED, 50% of the time In the next 4 months, the clinic will improve total messages through patient portal by 30% The providers will address and document conformance to 3 cross-cutting quality measures in 80% of patients in 3 months In the next 12 months, we will increase rates of timely completion and transmission of discharge summaries for hospitalized patients on general medicine by 50%
Postintervention Data
n = 176 Weekday: 12/177 (7%) with reports in beyond 24 hours Weekend: 27/59 (46%) with reports in beyond 24 hours
n = 193 Weekday: 4/195 (2%) with reports in beyond 24 hours Weekend: 29/78 (37%) with reports in beyond 24 hours
Nurse triage time: 12 minutes for nondiabetic patients Nurse triage time: 18 minutes for diabetic patients No standard process exists to collect information on accepting patients for transfer
Detailed evaluation of the current process was done and areas for improvement identified for implementation Standardized process developed, piloted, modified based on feedback, and expanded. Data will be collected in next cycles 74% Had scheduled follow-up in 7 days 79% Had scheduled follow-up in 14 days 79% Had scheduled follow-up in 30 days New policy was made regarding phone calls before sending patients to the ED. Data will be collected in future months
14% Had scheduled follow-up in 7 days 46% Had scheduled follow-up in 14 days 61% Had scheduled follow-up in 30 days Residents’ and attendings’ attitudes were evaluated. Current process for phone call was studied Patients and staff attitudes were studied about the use of the patient portal Studied current practice Completed
