bs_bs_banner
Journal of Evaluation in Clinical Practice ISSN 1365-2753
The feasibility of e-learning as a quality improvement tool Daniel Kobewka MD,1 Chantal Backman PhD MHA,6 Paul Hendry MD,8 Stanley J. Hamstra PhD,2,9,11 Kathryn N. Suh MD MSc,3,12 Catherine Code MD4 and Alan J. Forster MD MSc5,7,10,13,14 1 Lecturer, 2Research Director, 3Associate Professor, 4Assistant Professor, 5Professor, Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada 6 Manager, 7Scientific Director, Performance Measurement, The Ottawa Hospital, Ottawa, Canada 8 Professor of Cardiac Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Canada 9 Affiliate Investigator, 10Scientist, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada 11 Associate Director, Academy for Innovation in Medical Education, University of Ottawa, Ottawa, Canada 12 Associate Professor, 13Professor, Department of Epidemiology and Community Medicine, University of Ottawa 14 Scientist, Institute for Clinical Evaluative Sciences, Toronto, Canada
Keywords education, e-learning, hand washing, patient safety, pneumonia Correspondence Dr Alan Forster The Ottawa Hospital, Civic Campus Ottawa, ON K1Y 4E9, Canada E-mail: [email protected] Accepted for publication: 7 April 2014 doi:10.1111/jep.12169
Abstract Rational, aims and objectives Many quality problems exist in health care. We aim to investigate the feasibility and acceptability of using e-learning (defined as computer-based learning modules) to address gaps in quality of care. Methods We performed a qualitative evaluation of participants in a pilot e-learning program. Physician members of six medical teaching units (MTUs) at a multi-site tertiary care teaching hospital were asked to complete two e-learning modules addressing hand hygiene practices and management of community-acquired pneumonia (CAP). An e-learning design team created online modules that were made available to members of the six MTUs for 4 weeks using a password secured website. Use of the modules was voluntary. Participants’ perceptions of module content, mode of delivery, and suggestions for improvement were determined through focus groups. We then performed content analysis on the transcripts. We used system data to define patterns of module access. Results Out of 55 eligible users, 30 (55%) logged onto the system at least once. Residents (14/30, 47%) were less likely to use the system than medical students (9/14, 64%) and attending staff (7/11, 64%). Learners at all levels thought the modules were easy to use. Participants liked the knowledge-based material in the CAP module because it directly applied to their work. There were less favourable opinions of the hand hygiene module Conclusions Generating e-learning modules targeted at gaps in quality of care is feasible and acceptable to learners. Future studies should assess whether these approaches lead to desired changes in behavior.
Introduction Quality of care gaps are a major problem in health care, incurring significant financial and human health costs. McGlynn et al. found that ambulatory care patients receive recommended treatments in less than 50% of visits [1]. Others have shown that preventable adverse events occur in 2–8% of acute care hospital admission [2–6]. Although there are many contributing causes of these gaps, a major challenge is delivering targeted improvement strategies to health professionals to address specific performance and knowledge deficits [7]. For example, often an institution and/or health system develops guidelines or treatment protocols 606
that address newly identified quality problems. However, there are few methods for ensuring that new policies translate to practice. At a very basic level, it is uncertain whether doctors even receive information regarding such changes or proposals. e-Learning is a term to describe any system that uses a networklinked computer to deliver knowledge or skills training. It is commonly used in global industries to support ‘on-the-job’ education. In this setting, when a technology upgrade occurs in a plant or production line, workers must receive training to upgrade their skills and knowledge. e-Learning enables distribution of standard information, acknowledgement of receipt of the information, and even knowledge testing from a centralized location. These
Journal of Evaluation in Clinical Practice 20 (2014) 606–610 © 2014 John Wiley & Sons, Ltd.
D.M. Kobewka et al.
characteristics make e-learning an ideal tool to respond to newly identified or well-established quality problems. e-Learning has not been used previously in a quality improvement system that responds dynamically to specific quality problems; it has been used for curriculum delivery. Where e-learning has been used, educators have delivered materials designed to meet specific learning objectives for learners in undergraduate, postgraduate and continuing medical education contexts [8–11]. In general, these efforts have been effective and well received both by the educator and the learner [12–14]. In particular, learners appreciate that learning material is packaged into succinct modules related to a specific objective that can be reviewed at the learner’s convenience [10]. This study aims to determine if e-learning is a feasible method acceptable to doctors for delivering quality improvement curriculum. Given that previous uses of e-learning albeit for a different purpose or with a different target audience were successful, we felt it was likely that e-learning could be useful for delivering quality improvement curriculum. However, before widespread implementation at our institution, we wanted to explore the resources required to deliver the material and assess how this type of approach would be perceived.
E-learning as a quality improvement tool
Society of America/American Thoracic Society 2007 guidelines [16]. For this curriculum, we not only included text but also reinforced key concepts with multiple-choice self-testing, and interactive activities. The next step was to develop the technical capability to deliver the curriculum using an e-learning platform. For this, we partnered with the University of Ottawa’s medical school where there exists an e-learning team to support the undergraduate medical curriculum. This e-learning team is part of the Faculty of Medicine’s MedTech, which delivers technology services for students and faculty. The personnel who worked on our project included an instructional designer who laid out the module flow using PowerPoint and a multimedia technology designer who converted the presentations to an interactive web-based application. The modules were tested on all major browsers on OSX and Windows including Firefox, Chrome, Internet Explorer and Safari. Access to the site required an appropriate username and password which allowed us to track use. Our modules can be found at www.med.uottawa.ca/medweb/community_acquired_pneumnia/ and www.med.uottawa.ca/medweb/hand_hygiene/.
Roll-out strategy
Methods Setting and paradigm We performed a 1-month e-learning intervention on general medicine medical teaching units (MTUs) at a multi-site tertiary care teaching hospital in Ottawa, Ontario, Canada. The MTUs provide care to approximately 150 acute medicine patients at a time. There are six MTUs in total each consisting of one staff physician, one senior resident, four junior residents and three medical students. Following the intervention, we held focus groups. We then used content analysis of focus group transcripts to determine doctors’ attitudes and perceptions of e-learning. We also gathered recommendations on how to improve our e-learning system. The Hospital Research Ethics Board approved the project.
Design and development of e-learning modules To test our quality improvement approach, we developed e-learning modules to address two common quality problems: poor compliance with hand hygiene behaviours, and failure to adhere to treatment guidelines for community-acquired pneumonia (CAP). As in other settings, these two quality gaps existed in our institutions (data not shown but available upon request). We picked hand hygiene as one of the topics for our modules because it is a complex behavioural issue, whereas deficiencies in CAP management are often due to a lack of knowledge. Development of the modules involved curriculum development and technical implementation. We based our hand hygiene curriculum on Ajzen’s theory of planned behaviour [15], which helped us conceptualize why providers may not follow prescribed practices. The content included videos showing proper hand hygiene practices (including when and how to properly perform hand hygiene) and videos showing the personal testimony of actual patients who have been affected by potentially preventable hospital-acquired infections. We based our CAP curriculum on Infectious Disease
© 2014 John Wiley & Sons, Ltd.
We implemented e-learning modules on six general medicine MTUs over a 4-week block. When we performed the study, there were a total of 55 doctors and medical students working on the MTUs (including 11 staff internists, 6 senior residents, 24 junior residents and 14 medical students). At the beginning of the study, we invited all medical students, residents and staff to attend a lunch meeting where we explained the study. We asked willing participants to provide informed consent and to complete the modules at some point in the following 3 weeks. Participating team members signed consent forms.
Focus groups After 3 weeks, the participants were invited to attend a focus group. Junior residents and medical students were placed in separate groups from senior residents and staff to reduce social acceptance bias. A member of our research team with training in facilitating focus groups led all of the groups to ensure similar questions were asked. The purpose of the focus groups was to discover the participants’ general impressions of the e-learning modules including: the time commitment needed, perceptions of their content, opinions of whether e-learning could be used as a learning method, and suggestions for improvement. All focus group discussions were recorded, transcribed and analysed using content analysis.
Outcomes and analysis We collected module usage data, including total number of logins and number of distinct active users. We recorded and transcribed the focus group dialogue. Two independent reviewers analysed the transcripts using qualitative data analysis software ATLAS.ti (Berlin, Germany). Reviewers then met and came to a consensus on codes and themes in the focus group data. 607
E-learning as a quality improvement tool
D.M. Kobewka et al.
Table 1 Number of individuals who logged into each module separated according to position. Percent of total eligible participants in brackets
Medical students Residents Attending Total
Eligible to participate
Hand hygiene
CAP
Either module
14 30 11 55
6 (43) 11 (37) 7 (64) 24 (44)
9 (64) 13 (43) 7 (64) 29 (53)
9 (64) 14 (47) 7 (64) 30 (55)
CAP, community-acquired pneumonia.
Results We implemented the modules in May 2011. Twenty-four individuals (44%) used the hand hygiene module and 29 individuals (53%) used the CAP module (Table 1). Attending staff and medical students were more likely to access the modules than the residents. Individual users logged in to the hand hygiene module 32 times and the CAP module 36 times, indicating that several users accessed the system more than once. Nine medical students, two residents and two staff physicians participated in five different focus groups. Each focus group lasted between 13 and 30 minutes. One focus group participant did not complete the modules and did not provide a reason. All other participants completed both modules.
Time commitment Several medical students said the CAP module took them 30 minutes (group 1, line 45; group 1, line 57.) One resident and one staff said it took 15 minutes to complete (group 2, line 33; group 3, line 53).
Usability of the modules Overall, participants found the modules easy to use. Some comments from medical students to this effect were: They were easy to navigate and simple. They weren’t over cluttered and had simple steps and I really liked the summaries at the end – to just summarize. (Medical student group 5, line 5) I found that they were really straight forward. (Medical student group 5, line 9) Similarly, a resident commented that: . . . They were very straight forward, easy to follow along. The instructions were clear and they were quite intuitive. (Resident group 2, line 5) I thought they were very easy to use. They weren’t too complicated getting into them and navigating through them. (Resident group 2, line 119) Although overall people generally found the system easy to use, there were some issues. Some staff and medical students said they had technical problems with the hand hygiene module, including no volume for the videos and videos not playing at all (staff group 3, line 7; staff group 3, line 95; medical student group 5, line 78; medical student group 5, line 94). 608
Uptake of e-learning There were numerous positive comments about e-learning as a learning method: I thought they were really good. They were sort of similar to the CBL (case based learning) kind of cases that we did in our first two years of medical school, except they were really well organized and sort of short, to the point, gave you the information you needed. I thought they were pretty good. (Medical student group 5, line 7) I think that is a nice way to learn, you have active learning at night instead of reading a book, I feel like it sticks better. (Medical student group 1, line 99) I like the idea of e-learning. (Staff group 3, line 42) Individuals from each level of training appreciated that the modules could be done on their own time (resident group 2, line 123; medical student group 5, line 151). A staff physician said: In one sense I really like it because it allows you to do stuff on your own time. When a patient doesn’t show up in my clinic and I am not in my office so I didn’t bring other work to do, I can click through something and read it. (Staff group 3, line 127) Another staff physician did not think that her colleagues would do the modules without further incentive. She said: I think in terms of staff people, I don’t honestly know how many staff people would do the modules like this unless they were tied to getting credits for CME. There are so many CME things we can be doing. If it was tied to CME maybe staff people might do it. (Group 3, line 13) Some staff physicians thought that the e-learning modules would be helpful for medical students or residents saying: I think this is more something you could email out to residents like before you start your CTU [Clinical Teaching Unit] rotation, these are some things you may want to brush up on. (Staff group 3, line 23) and I don’t think that a lot of the staff would necessarily do them, but I think they would like to have them available for their house staff that they are working with. (Group 3, line 85)
Further e-learning Some medical students said they would like more learning modules on other internal medicine topics (medical student group 5 line 140, group 1 line 103). A staff also said that this would be helpful for house staff (staff group 3, line 85).
Positive comments about the pneumonia module All participants had positive comments about the content, presentation and relevance of the CAP module. One medical student said that: The pneumonia one was really good actually; it kind of helped me study for our exam. (Medical student group 5, line 13)
© 2014 John Wiley & Sons, Ltd.
D.M. Kobewka et al.
Another student said the module was very useful: It had good repetition; I think that was another strong point of it. (Medical student group 1, line 37) Similarly, a resident liked the module and explained: . . . there was a good explanation after as to why the answers that were provided were correct. I learned something from it. I thought the level of detail was good. (Resident group 2, line 21) A staff physician thought: . . . the pneumonia one was very good and really useful, especially useful for house staff to review if they are coming on CTU. (Staff group 3, line 7) There were no negative comments regarding the pneumonia module.
Positive comments about the hand hygiene module Participants thought the module was a good review saying: Hand washing was a really good learning and review, but I found it didn’t really tell me anything I didn’t already know. (Medical student group 1, line 3) I think it is good that it reinforces why you have to do it (hand washing). (Medical student group 1, line 71) and I still think that it was good for us to go through it. I mean as everybody has said, repetition is good. (Medical student group 5, line 88) Some users thought the filmed testimony of a patient affected by a hospital-acquired infection was poignant, saying: It left me with the feeling that I should be thinking about hand washing more because of the emotional impact at the beginning of it. (Resident group 2, line 77) A staff physician commented that the patient testimony was . . . quite compelling emotionally, the lady is clearly very upset. (Staff group 4, line 59)
Negative comments about the hand hygiene module Most medical students and a staff physician said that the hand hygiene module was redundant. One participant commented: It was just a reinforcement module. I didn’t learn a lot of extra information from it. (Medical student group 2, line 61) Another medical student also found it a difficult topic to engage with, saying: Hand hygiene – It is kind of a boring subject. It wasn’t something I particularly spent a lot of time on. The pneumonia one was more useful certainly to an internal medicine resident. I am always washing my hands anyways. (Medical student group 5, line 86) A staff physician thought that a system of public shaming for poor hand hygiene compliance would work better than e-learning and was sceptical about whether e-learning would change behaviour (staff group 3, line 7). A staff said: I am not that optimistic that just watching the video will change anything. I think even the resident and I both
© 2014 John Wiley & Sons, Ltd.
E-learning as a quality improvement tool
watching it sort of feel like we felt like okay this doesn’t really apply to me. I always wash my hands. (Staff group 3, line 105)
Ways to improve e-learning To improve the CAP module, some individuals suggested including a summary document that could be printed off or a link to current best practice guidelines (medical student group 1, line 116; staff group 3, line 71; staff group 3, line 127). For the hand hygiene module, one of the medical students suggested making it shorter (medical student group 5, line 112). A resident suggested that a quiz at the end of the module might be good, and a staff physician said that adding more data and evidence would improve it (resident group 2, line 103; staff group 3, line 95).
Discussion We developed and used e-learning modules designed to address two quality issues: CAP treatment and hand hygiene. More than half of eligible users logged on the system at least once. We obtained feedback from approximately half of the users in our focus groups. Learners at all levels thought the modules were easy to use with the exception of some technical problems with videos in the hand hygiene module. Participants liked the knowledge-based material in the CAP module because it directly applied to their work on the wards and for the medical students in particular because it helped them prepare for their examinations. Although uptake of our intervention was moderate with only 55% of those eligible completing at least one of the modules, we did not include active encouragement to do so. Use of the system was completely voluntary. We had limited advertisement of the program. Many house staff may not have attended our introductory session given competing priorities including planned absences (holidays or post-call) or being busy with patient care responsibilities. Thus, the uptake we observed may in fact be considered a very positive finding, especially when coupled with the positive comments of the users. Use of the e-learning system would likely be enhanced if completion was made mandatory for students and residents to pass the rotation or for staff to maintain hospital privileges. While using a system of rewards and punishments is tempting, especially when the information is vital for improving patient safety, it has been shown that an increased sense of autonomy and involvement in an educational endeavour is more likely to motivate people to learn and change [17,18]. To address this, designing a process to engage doctors in developing the hospital’s quality improvement education programs may yield better results compared with sanctions. There are three limitations in this work. Our e-learning program was dependent on access to web hosting and local expertise in curriculum development and website development that may not be available to all hospitals or health systems. This may not be an issue if a centralized library of learning modules were created. Second, relatively few people participated in the pilot. Despite this, we feel we have sufficient information to proceed for a wider roll-out in our institution. Third, we did not evaluate whether care 609
E-learning as a quality improvement tool
improved as a result of our intervention. Our future work will measure outcomes that result from quality improvement e-learning modules. If successful, this could justify creation of regional e-learning development groups to respond to quality issues in specific jurisdictions. We have shown that e-learning modules can be designed and implemented at our institution. It was very straightforward to develop two modules that were generally well received by a small group of users. Broader implementation may be more challenging to manage. In conclusion, e-learning approaches offer one potential tactic for improving quality of care especially when the problem is primarily a lack of knowledge. Future research should focus on measuring quality of care outcomes after an e-learning intervention.
References 1. McGlynn, E. A., Asch, S. M., Adams, J., et al. (2003) The quality of health care delivered to adults in the United States. The New England Journal of Medicine, 348 (26), 2635–2645. 2. Baker, G. R., Norton, P. G., Flintoft, V., et al. (2004) The Canadian adverse events study: the incidence of adverse events among hospital patients in Canada. CMAJ: Canadian Medical Association Journal = journal de l’Association medicale canadienne, 170 (11), 1678–1686. 3. Brennan, T. A., Leape, L. L., Laird, N. M., et al. (1991) Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard medical practice study I. The New England Journal of Medicine, 324 (6), 370–376. 4. Forster, A. J., Asmis, T. R., Clark, H. D., et al. (2004) Ottawa hospital patient safety study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. CMAJ: Canadian Medical Association journal = journal de l’Association medicale canadienne, 170 (8), 1235–1240. 5. Vincent, C., Neale, G. & Woloshynowych, M. (2001) Adverse events in British hospitals: preliminary retrospective record review. BMJ (Clinical Research Ed.), 322 (7285), 517–519. 6. Wilson, R. M., Runciman, W. B., Gibberd, R. W., Harrison, B. T., Newby, L. & Hamilton, J. D. (1995) The quality in Australian health care study. The Medical Journal of Australia, 163 (9), 458–471. 7. Shojania, K. G. & Grimshaw, J. M. (2005) Evidence-based quality improvement: the state of the science. Health Affairs (Project Hope), 24 (1), 138–150.
610
D.M. Kobewka et al.
8. Autti, T., Autti, H., Vehmas, T., Laitalainen, V. & Kivisaari, L. (2007) E-learning is a well-accepted tool in supplementary training among medical doctors: an experience of obligatory radiation protection training in healthcare. Acta Radiologica, 48 (5), 508–513. 9. Childs, S., Blenkinsopp, E., Hall, A. & Walton, G. (2005) Effective e-learning for health professionals and students – barriers and their solutions. A systematic review of the literature – findings from the HeXL project. Health Information and Libraries Journal, 22 (Suppl. 2), 20–32. 10. Wolbrink, T. A. & Burns, J. P. (2012) Internet-based learning and applications for critical care medicine. Journal of Intensive Care Medicine, 27 (5), 322–332. 11. Schermer, T. R., Akkermans, R. P., Crockett, A. J., et al. (2011) Effect of e-learning and repeated performance feedback on spirometry test quality in family practice: a cluster trial. Annals of Family Medicine, 9 (4), 330–336. 12. Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J. & Montori, V. M. (2008) Internet-based learning in the health professions: a meta-analysis. JAMA: The Journal of the American Medical Association, 300 (10), 1181–1196. 13. Wutoh, R., Boren, S. A. & Balas, E. A. (2004) E-learning: a review of Internet-based continuing medical education. The Journal of Continuing Education in the Health Professions, 24 (1), 20–30. 14. Laustsen, S., Bibby, B. M., Kristensen, B., Moller, J. K. & Thulstrup, A. M. (2009) E-learning may improve adherence to alcohol-based hand rubbing: a cohort study. American Journal of Infection Control, 37 (7), 565–568. 15. Ajzen, I. (1991) The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50 (2), 179–211. 16. Mandell, L. A., Wunderink, R. G., Anzueto, A., et al. (2007) Infectious diseases society of America/American thoracic society consensus guidelines on the management of community-acquired pneumonia in adults. Clinical Infectious Diseases : An Official Publication of the Infectious Diseases Society of America, 44 (Suppl. 2), S27–S72. 17. Barazzoni, F., Grilli, R., Amicosante, A. M., et al. (2002) Impact of end user involvement in implementing guidelines on routine pre-operative tests. International journal for quality in health care. Journal of the International Society for Quality in Health Care / ISQua, 14 (4), 321–327. 18. Pattal, E. A., Cooper, H. & Robinson, J. C. (2008) The effects of choice on intrinsic motivation and related outcomes: a metaanalysis of research findings. Psychological Bulletin, 134 (2), 270–300.
© 2014 John Wiley & Sons, Ltd.
Journal of Evaluation in Clinical Practice ISSN 1365-2753
The feasibility of e-learning as a quality improvement tool Daniel Kobewka MD,1 Chantal Backman PhD MHA,6 Paul Hendry MD,8 Stanley J. Hamstra PhD,2,9,11 Kathryn N. Suh MD MSc,3,12 Catherine Code MD4 and Alan J. Forster MD MSc5,7,10,13,14 1 Lecturer, 2Research Director, 3Associate Professor, 4Assistant Professor, 5Professor, Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada 6 Manager, 7Scientific Director, Performance Measurement, The Ottawa Hospital, Ottawa, Canada 8 Professor of Cardiac Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Canada 9 Affiliate Investigator, 10Scientist, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada 11 Associate Director, Academy for Innovation in Medical Education, University of Ottawa, Ottawa, Canada 12 Associate Professor, 13Professor, Department of Epidemiology and Community Medicine, University of Ottawa 14 Scientist, Institute for Clinical Evaluative Sciences, Toronto, Canada
Keywords education, e-learning, hand washing, patient safety, pneumonia Correspondence Dr Alan Forster The Ottawa Hospital, Civic Campus Ottawa, ON K1Y 4E9, Canada E-mail: [email protected] Accepted for publication: 7 April 2014 doi:10.1111/jep.12169
Abstract Rational, aims and objectives Many quality problems exist in health care. We aim to investigate the feasibility and acceptability of using e-learning (defined as computer-based learning modules) to address gaps in quality of care. Methods We performed a qualitative evaluation of participants in a pilot e-learning program. Physician members of six medical teaching units (MTUs) at a multi-site tertiary care teaching hospital were asked to complete two e-learning modules addressing hand hygiene practices and management of community-acquired pneumonia (CAP). An e-learning design team created online modules that were made available to members of the six MTUs for 4 weeks using a password secured website. Use of the modules was voluntary. Participants’ perceptions of module content, mode of delivery, and suggestions for improvement were determined through focus groups. We then performed content analysis on the transcripts. We used system data to define patterns of module access. Results Out of 55 eligible users, 30 (55%) logged onto the system at least once. Residents (14/30, 47%) were less likely to use the system than medical students (9/14, 64%) and attending staff (7/11, 64%). Learners at all levels thought the modules were easy to use. Participants liked the knowledge-based material in the CAP module because it directly applied to their work. There were less favourable opinions of the hand hygiene module Conclusions Generating e-learning modules targeted at gaps in quality of care is feasible and acceptable to learners. Future studies should assess whether these approaches lead to desired changes in behavior.
Introduction Quality of care gaps are a major problem in health care, incurring significant financial and human health costs. McGlynn et al. found that ambulatory care patients receive recommended treatments in less than 50% of visits [1]. Others have shown that preventable adverse events occur in 2–8% of acute care hospital admission [2–6]. Although there are many contributing causes of these gaps, a major challenge is delivering targeted improvement strategies to health professionals to address specific performance and knowledge deficits [7]. For example, often an institution and/or health system develops guidelines or treatment protocols 606
that address newly identified quality problems. However, there are few methods for ensuring that new policies translate to practice. At a very basic level, it is uncertain whether doctors even receive information regarding such changes or proposals. e-Learning is a term to describe any system that uses a networklinked computer to deliver knowledge or skills training. It is commonly used in global industries to support ‘on-the-job’ education. In this setting, when a technology upgrade occurs in a plant or production line, workers must receive training to upgrade their skills and knowledge. e-Learning enables distribution of standard information, acknowledgement of receipt of the information, and even knowledge testing from a centralized location. These
Journal of Evaluation in Clinical Practice 20 (2014) 606–610 © 2014 John Wiley & Sons, Ltd.
D.M. Kobewka et al.
characteristics make e-learning an ideal tool to respond to newly identified or well-established quality problems. e-Learning has not been used previously in a quality improvement system that responds dynamically to specific quality problems; it has been used for curriculum delivery. Where e-learning has been used, educators have delivered materials designed to meet specific learning objectives for learners in undergraduate, postgraduate and continuing medical education contexts [8–11]. In general, these efforts have been effective and well received both by the educator and the learner [12–14]. In particular, learners appreciate that learning material is packaged into succinct modules related to a specific objective that can be reviewed at the learner’s convenience [10]. This study aims to determine if e-learning is a feasible method acceptable to doctors for delivering quality improvement curriculum. Given that previous uses of e-learning albeit for a different purpose or with a different target audience were successful, we felt it was likely that e-learning could be useful for delivering quality improvement curriculum. However, before widespread implementation at our institution, we wanted to explore the resources required to deliver the material and assess how this type of approach would be perceived.
E-learning as a quality improvement tool
Society of America/American Thoracic Society 2007 guidelines [16]. For this curriculum, we not only included text but also reinforced key concepts with multiple-choice self-testing, and interactive activities. The next step was to develop the technical capability to deliver the curriculum using an e-learning platform. For this, we partnered with the University of Ottawa’s medical school where there exists an e-learning team to support the undergraduate medical curriculum. This e-learning team is part of the Faculty of Medicine’s MedTech, which delivers technology services for students and faculty. The personnel who worked on our project included an instructional designer who laid out the module flow using PowerPoint and a multimedia technology designer who converted the presentations to an interactive web-based application. The modules were tested on all major browsers on OSX and Windows including Firefox, Chrome, Internet Explorer and Safari. Access to the site required an appropriate username and password which allowed us to track use. Our modules can be found at www.med.uottawa.ca/medweb/community_acquired_pneumnia/ and www.med.uottawa.ca/medweb/hand_hygiene/.
Roll-out strategy
Methods Setting and paradigm We performed a 1-month e-learning intervention on general medicine medical teaching units (MTUs) at a multi-site tertiary care teaching hospital in Ottawa, Ontario, Canada. The MTUs provide care to approximately 150 acute medicine patients at a time. There are six MTUs in total each consisting of one staff physician, one senior resident, four junior residents and three medical students. Following the intervention, we held focus groups. We then used content analysis of focus group transcripts to determine doctors’ attitudes and perceptions of e-learning. We also gathered recommendations on how to improve our e-learning system. The Hospital Research Ethics Board approved the project.
Design and development of e-learning modules To test our quality improvement approach, we developed e-learning modules to address two common quality problems: poor compliance with hand hygiene behaviours, and failure to adhere to treatment guidelines for community-acquired pneumonia (CAP). As in other settings, these two quality gaps existed in our institutions (data not shown but available upon request). We picked hand hygiene as one of the topics for our modules because it is a complex behavioural issue, whereas deficiencies in CAP management are often due to a lack of knowledge. Development of the modules involved curriculum development and technical implementation. We based our hand hygiene curriculum on Ajzen’s theory of planned behaviour [15], which helped us conceptualize why providers may not follow prescribed practices. The content included videos showing proper hand hygiene practices (including when and how to properly perform hand hygiene) and videos showing the personal testimony of actual patients who have been affected by potentially preventable hospital-acquired infections. We based our CAP curriculum on Infectious Disease
© 2014 John Wiley & Sons, Ltd.
We implemented e-learning modules on six general medicine MTUs over a 4-week block. When we performed the study, there were a total of 55 doctors and medical students working on the MTUs (including 11 staff internists, 6 senior residents, 24 junior residents and 14 medical students). At the beginning of the study, we invited all medical students, residents and staff to attend a lunch meeting where we explained the study. We asked willing participants to provide informed consent and to complete the modules at some point in the following 3 weeks. Participating team members signed consent forms.
Focus groups After 3 weeks, the participants were invited to attend a focus group. Junior residents and medical students were placed in separate groups from senior residents and staff to reduce social acceptance bias. A member of our research team with training in facilitating focus groups led all of the groups to ensure similar questions were asked. The purpose of the focus groups was to discover the participants’ general impressions of the e-learning modules including: the time commitment needed, perceptions of their content, opinions of whether e-learning could be used as a learning method, and suggestions for improvement. All focus group discussions were recorded, transcribed and analysed using content analysis.
Outcomes and analysis We collected module usage data, including total number of logins and number of distinct active users. We recorded and transcribed the focus group dialogue. Two independent reviewers analysed the transcripts using qualitative data analysis software ATLAS.ti (Berlin, Germany). Reviewers then met and came to a consensus on codes and themes in the focus group data. 607
E-learning as a quality improvement tool
D.M. Kobewka et al.
Table 1 Number of individuals who logged into each module separated according to position. Percent of total eligible participants in brackets
Medical students Residents Attending Total
Eligible to participate
Hand hygiene
CAP
Either module
14 30 11 55
6 (43) 11 (37) 7 (64) 24 (44)
9 (64) 13 (43) 7 (64) 29 (53)
9 (64) 14 (47) 7 (64) 30 (55)
CAP, community-acquired pneumonia.
Results We implemented the modules in May 2011. Twenty-four individuals (44%) used the hand hygiene module and 29 individuals (53%) used the CAP module (Table 1). Attending staff and medical students were more likely to access the modules than the residents. Individual users logged in to the hand hygiene module 32 times and the CAP module 36 times, indicating that several users accessed the system more than once. Nine medical students, two residents and two staff physicians participated in five different focus groups. Each focus group lasted between 13 and 30 minutes. One focus group participant did not complete the modules and did not provide a reason. All other participants completed both modules.
Time commitment Several medical students said the CAP module took them 30 minutes (group 1, line 45; group 1, line 57.) One resident and one staff said it took 15 minutes to complete (group 2, line 33; group 3, line 53).
Usability of the modules Overall, participants found the modules easy to use. Some comments from medical students to this effect were: They were easy to navigate and simple. They weren’t over cluttered and had simple steps and I really liked the summaries at the end – to just summarize. (Medical student group 5, line 5) I found that they were really straight forward. (Medical student group 5, line 9) Similarly, a resident commented that: . . . They were very straight forward, easy to follow along. The instructions were clear and they were quite intuitive. (Resident group 2, line 5) I thought they were very easy to use. They weren’t too complicated getting into them and navigating through them. (Resident group 2, line 119) Although overall people generally found the system easy to use, there were some issues. Some staff and medical students said they had technical problems with the hand hygiene module, including no volume for the videos and videos not playing at all (staff group 3, line 7; staff group 3, line 95; medical student group 5, line 78; medical student group 5, line 94). 608
Uptake of e-learning There were numerous positive comments about e-learning as a learning method: I thought they were really good. They were sort of similar to the CBL (case based learning) kind of cases that we did in our first two years of medical school, except they were really well organized and sort of short, to the point, gave you the information you needed. I thought they were pretty good. (Medical student group 5, line 7) I think that is a nice way to learn, you have active learning at night instead of reading a book, I feel like it sticks better. (Medical student group 1, line 99) I like the idea of e-learning. (Staff group 3, line 42) Individuals from each level of training appreciated that the modules could be done on their own time (resident group 2, line 123; medical student group 5, line 151). A staff physician said: In one sense I really like it because it allows you to do stuff on your own time. When a patient doesn’t show up in my clinic and I am not in my office so I didn’t bring other work to do, I can click through something and read it. (Staff group 3, line 127) Another staff physician did not think that her colleagues would do the modules without further incentive. She said: I think in terms of staff people, I don’t honestly know how many staff people would do the modules like this unless they were tied to getting credits for CME. There are so many CME things we can be doing. If it was tied to CME maybe staff people might do it. (Group 3, line 13) Some staff physicians thought that the e-learning modules would be helpful for medical students or residents saying: I think this is more something you could email out to residents like before you start your CTU [Clinical Teaching Unit] rotation, these are some things you may want to brush up on. (Staff group 3, line 23) and I don’t think that a lot of the staff would necessarily do them, but I think they would like to have them available for their house staff that they are working with. (Group 3, line 85)
Further e-learning Some medical students said they would like more learning modules on other internal medicine topics (medical student group 5 line 140, group 1 line 103). A staff also said that this would be helpful for house staff (staff group 3, line 85).
Positive comments about the pneumonia module All participants had positive comments about the content, presentation and relevance of the CAP module. One medical student said that: The pneumonia one was really good actually; it kind of helped me study for our exam. (Medical student group 5, line 13)
© 2014 John Wiley & Sons, Ltd.
D.M. Kobewka et al.
Another student said the module was very useful: It had good repetition; I think that was another strong point of it. (Medical student group 1, line 37) Similarly, a resident liked the module and explained: . . . there was a good explanation after as to why the answers that were provided were correct. I learned something from it. I thought the level of detail was good. (Resident group 2, line 21) A staff physician thought: . . . the pneumonia one was very good and really useful, especially useful for house staff to review if they are coming on CTU. (Staff group 3, line 7) There were no negative comments regarding the pneumonia module.
Positive comments about the hand hygiene module Participants thought the module was a good review saying: Hand washing was a really good learning and review, but I found it didn’t really tell me anything I didn’t already know. (Medical student group 1, line 3) I think it is good that it reinforces why you have to do it (hand washing). (Medical student group 1, line 71) and I still think that it was good for us to go through it. I mean as everybody has said, repetition is good. (Medical student group 5, line 88) Some users thought the filmed testimony of a patient affected by a hospital-acquired infection was poignant, saying: It left me with the feeling that I should be thinking about hand washing more because of the emotional impact at the beginning of it. (Resident group 2, line 77) A staff physician commented that the patient testimony was . . . quite compelling emotionally, the lady is clearly very upset. (Staff group 4, line 59)
Negative comments about the hand hygiene module Most medical students and a staff physician said that the hand hygiene module was redundant. One participant commented: It was just a reinforcement module. I didn’t learn a lot of extra information from it. (Medical student group 2, line 61) Another medical student also found it a difficult topic to engage with, saying: Hand hygiene – It is kind of a boring subject. It wasn’t something I particularly spent a lot of time on. The pneumonia one was more useful certainly to an internal medicine resident. I am always washing my hands anyways. (Medical student group 5, line 86) A staff physician thought that a system of public shaming for poor hand hygiene compliance would work better than e-learning and was sceptical about whether e-learning would change behaviour (staff group 3, line 7). A staff said: I am not that optimistic that just watching the video will change anything. I think even the resident and I both
© 2014 John Wiley & Sons, Ltd.
E-learning as a quality improvement tool
watching it sort of feel like we felt like okay this doesn’t really apply to me. I always wash my hands. (Staff group 3, line 105)
Ways to improve e-learning To improve the CAP module, some individuals suggested including a summary document that could be printed off or a link to current best practice guidelines (medical student group 1, line 116; staff group 3, line 71; staff group 3, line 127). For the hand hygiene module, one of the medical students suggested making it shorter (medical student group 5, line 112). A resident suggested that a quiz at the end of the module might be good, and a staff physician said that adding more data and evidence would improve it (resident group 2, line 103; staff group 3, line 95).
Discussion We developed and used e-learning modules designed to address two quality issues: CAP treatment and hand hygiene. More than half of eligible users logged on the system at least once. We obtained feedback from approximately half of the users in our focus groups. Learners at all levels thought the modules were easy to use with the exception of some technical problems with videos in the hand hygiene module. Participants liked the knowledge-based material in the CAP module because it directly applied to their work on the wards and for the medical students in particular because it helped them prepare for their examinations. Although uptake of our intervention was moderate with only 55% of those eligible completing at least one of the modules, we did not include active encouragement to do so. Use of the system was completely voluntary. We had limited advertisement of the program. Many house staff may not have attended our introductory session given competing priorities including planned absences (holidays or post-call) or being busy with patient care responsibilities. Thus, the uptake we observed may in fact be considered a very positive finding, especially when coupled with the positive comments of the users. Use of the e-learning system would likely be enhanced if completion was made mandatory for students and residents to pass the rotation or for staff to maintain hospital privileges. While using a system of rewards and punishments is tempting, especially when the information is vital for improving patient safety, it has been shown that an increased sense of autonomy and involvement in an educational endeavour is more likely to motivate people to learn and change [17,18]. To address this, designing a process to engage doctors in developing the hospital’s quality improvement education programs may yield better results compared with sanctions. There are three limitations in this work. Our e-learning program was dependent on access to web hosting and local expertise in curriculum development and website development that may not be available to all hospitals or health systems. This may not be an issue if a centralized library of learning modules were created. Second, relatively few people participated in the pilot. Despite this, we feel we have sufficient information to proceed for a wider roll-out in our institution. Third, we did not evaluate whether care 609
E-learning as a quality improvement tool
improved as a result of our intervention. Our future work will measure outcomes that result from quality improvement e-learning modules. If successful, this could justify creation of regional e-learning development groups to respond to quality issues in specific jurisdictions. We have shown that e-learning modules can be designed and implemented at our institution. It was very straightforward to develop two modules that were generally well received by a small group of users. Broader implementation may be more challenging to manage. In conclusion, e-learning approaches offer one potential tactic for improving quality of care especially when the problem is primarily a lack of knowledge. Future research should focus on measuring quality of care outcomes after an e-learning intervention.
References 1. McGlynn, E. A., Asch, S. M., Adams, J., et al. (2003) The quality of health care delivered to adults in the United States. The New England Journal of Medicine, 348 (26), 2635–2645. 2. Baker, G. R., Norton, P. G., Flintoft, V., et al. (2004) The Canadian adverse events study: the incidence of adverse events among hospital patients in Canada. CMAJ: Canadian Medical Association Journal = journal de l’Association medicale canadienne, 170 (11), 1678–1686. 3. Brennan, T. A., Leape, L. L., Laird, N. M., et al. (1991) Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard medical practice study I. The New England Journal of Medicine, 324 (6), 370–376. 4. Forster, A. J., Asmis, T. R., Clark, H. D., et al. (2004) Ottawa hospital patient safety study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. CMAJ: Canadian Medical Association journal = journal de l’Association medicale canadienne, 170 (8), 1235–1240. 5. Vincent, C., Neale, G. & Woloshynowych, M. (2001) Adverse events in British hospitals: preliminary retrospective record review. BMJ (Clinical Research Ed.), 322 (7285), 517–519. 6. Wilson, R. M., Runciman, W. B., Gibberd, R. W., Harrison, B. T., Newby, L. & Hamilton, J. D. (1995) The quality in Australian health care study. The Medical Journal of Australia, 163 (9), 458–471. 7. Shojania, K. G. & Grimshaw, J. M. (2005) Evidence-based quality improvement: the state of the science. Health Affairs (Project Hope), 24 (1), 138–150.
610
D.M. Kobewka et al.
8. Autti, T., Autti, H., Vehmas, T., Laitalainen, V. & Kivisaari, L. (2007) E-learning is a well-accepted tool in supplementary training among medical doctors: an experience of obligatory radiation protection training in healthcare. Acta Radiologica, 48 (5), 508–513. 9. Childs, S., Blenkinsopp, E., Hall, A. & Walton, G. (2005) Effective e-learning for health professionals and students – barriers and their solutions. A systematic review of the literature – findings from the HeXL project. Health Information and Libraries Journal, 22 (Suppl. 2), 20–32. 10. Wolbrink, T. A. & Burns, J. P. (2012) Internet-based learning and applications for critical care medicine. Journal of Intensive Care Medicine, 27 (5), 322–332. 11. Schermer, T. R., Akkermans, R. P., Crockett, A. J., et al. (2011) Effect of e-learning and repeated performance feedback on spirometry test quality in family practice: a cluster trial. Annals of Family Medicine, 9 (4), 330–336. 12. Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J. & Montori, V. M. (2008) Internet-based learning in the health professions: a meta-analysis. JAMA: The Journal of the American Medical Association, 300 (10), 1181–1196. 13. Wutoh, R., Boren, S. A. & Balas, E. A. (2004) E-learning: a review of Internet-based continuing medical education. The Journal of Continuing Education in the Health Professions, 24 (1), 20–30. 14. Laustsen, S., Bibby, B. M., Kristensen, B., Moller, J. K. & Thulstrup, A. M. (2009) E-learning may improve adherence to alcohol-based hand rubbing: a cohort study. American Journal of Infection Control, 37 (7), 565–568. 15. Ajzen, I. (1991) The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50 (2), 179–211. 16. Mandell, L. A., Wunderink, R. G., Anzueto, A., et al. (2007) Infectious diseases society of America/American thoracic society consensus guidelines on the management of community-acquired pneumonia in adults. Clinical Infectious Diseases : An Official Publication of the Infectious Diseases Society of America, 44 (Suppl. 2), S27–S72. 17. Barazzoni, F., Grilli, R., Amicosante, A. M., et al. (2002) Impact of end user involvement in implementing guidelines on routine pre-operative tests. International journal for quality in health care. Journal of the International Society for Quality in Health Care / ISQua, 14 (4), 321–327. 18. Pattal, E. A., Cooper, H. & Robinson, J. C. (2008) The effects of choice on intrinsic motivation and related outcomes: a metaanalysis of research findings. Psychological Bulletin, 134 (2), 270–300.
© 2014 John Wiley & Sons, Ltd.
