509270
HPPXXX10.1177/1524839913509270He alth Promotion PracticeKenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 2013
On Your Time: Online Training for the Public Health Workforce Hope Worden Kenefick, MSW, PhD1 Sharon Ravid, MPH2 Kathleen MacVarish, REHS/RS, MS2 Jennifer Tsoi, MPH2 Kenny Weill, MS3 Elizabeth Faye, BA2 Anne Fidler, MS, ScD2
The need for competency-based training for the public health workforce is well documented. However, human and financial resource limitations within public health agencies often make it difficult for public health practitioners to attend classroom-based training programs. The Internet is an increasingly popular way of extending training beyond the workforce. Although research describes attributes of effective online learning modules, much of the available training delivered via the Internet does not incorporate such attributes. The authors describe the On Your Time training series, an effective distance education program and training model for public health practitioners, which includes a standardized process for development, review, evaluation, and continuous quality improvement. On Your Time is a series of awareness-level (i.e., addressing what practitioners should know), competency-based training modules that address topics related to regulatory responsibilities of public health practitioners (e.g., assuring compliance with codes and regulations governing housing, retail food safety, private water supplies, hazardous and solid waste, on-site wastewater systems, etc.), public health surveillance, case investigation, disease prevention, health promotion, and emergency preparedness. The replicable model incorporates what is known about best practices for online training and maximizes available resources in the interests of sustainability.
Health Promotion Practice March 2014 Vol. 15, Suppl 1 48S–55S DOI: 10.1177/1524839913509270 © 2014 Society for Public Health Education
Keywords: workforce development; training; distance education; technology; continuing education
1
HWK Consulting, Barrington, NH, USA Boston University School of Public Health, Boston, MA, USA 3 K. Weill Consulting Group, LLC, Brookline, MA, USA 2
Authors’ Note: This work is supported by the New England Alliance for Public Health Workforce Development, a Health Resources and Services Administration Public Health Training Center (Grant No. UB6HP20150), and by the Local Public Health Institute of Massachusetts, a Massachusetts Department of Public Health, Emergency Preparedness Bureau–funded entity supported by the Centers for Disease Control and Prevention’s Public Health Preparedness and Response for Bioterrorism (Grant No. 5U90TP116997-10). We would like to acknowledge Dr. Wayne LaMorte and Rob Schadt for their immense contributions to this project as well as Courtney Perdios for reviewing the article. Address correspondence to Sharon Ravid, Program Manager, New England Alliance for Public Health Workforce Development at Boston University School of Public Health, 715 Albany Street, Talbot Building W118C Boston, MA 02118, USA; e-mail: sravid@ bu.edu.
Supplement Note: This article is published in the supplement “Public Health Training Centers,” which was supported by the U.S. Department of Health and Human Services (HHS) Health Resources and Services Administration (HRSA) through a Cooperative Agreement (UBGHP20199) to Rollins School of Public Health, Emory University.
48S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Introduction >>
The need for competency-based training for the public health workforce has been well documented (Chandler, Qureshi, Gebbie, & Morse, 2008; Gebbie & Turnock, 2006; Harun, 2002; Koo & Miner, 2010; Wright, 2009). Recognizing the inherent challenge of maintaining and enhancing the skills of public health practitioners, the Institutes of Medicine (2003a, 2003b) has called for continuous, lifelong training for the public health workforce. This need, coupled with local health department staffing shortages that have expanded workloads and budgetary constraints restricting employee travel, has led to increased use of distance learning technology to deliver training. Although the attributes of effective online learning (e.g., structure, preferences of adult learners, evaluation protocols, etc.) have been studied, much of the available Internetbased training does not incorporate such attributes (Alur, Fatima, & Joseph, 2002; Chandler et al., 2008; Zusevics, Gilmore, Jecklin, & Swain, 2009). Ensuring that online trainings are up-to-date, accurate, effective, and user-friendly is an additional challenge, as it requires ongoing investment in content review, editing, and evaluation. This article describes a process for the development, review, evaluation, and improvement of distance education through the use of online, self-paced, awarenesslevel training modules for local public health practitioners. These modules address the learning needs of the public health workforce, incorporate what is known about best practices for online training, and maximize available resources in the interest of sustainability.
Background >>
A large portion of U.S. public health workers are employed in small organizations with limited human and financial resources (Gebbie, Merrill, & Tilson, 2002). This is the case in Massachusetts, where most local health agencies serve small populations (> Applications
Since January 2010, the Boston University School of Public Health has managed the Local Public Health Institute of Massachusetts, the state’s training institute for local public health practitioners. Institute staff work with an advisory committee, comprising local and state public health representatives, to pursue a critical mission: To provide and ensure a competent workforce by strengthening and sustaining the capacity of local boards of health to prepare for and respond to public health issues and emergencies and to promote the health of residents of the Commonwealth. The institute’s Competency Report (2010b) and unpublished Analysis of Gaps in Training for the Massachusetts Public Health Workforce (2010a) detail public health training needs across the state and identify 17 program areas and 10 crosscutting areas in which local public health practitioners should have competency (see Table 1). Additionally, ample literature (Ballew et al., 2013; Bryan et al., 2009; Chaney et al., 2009; Claus
et al., 2008; Claus et al., 2009; Giguere & Minotti, 2003; Koo & Miner, 2010; Zusevics et al., 2009) describe the effective attributes of online training programs. Based on this information, Boston University faculty and staff have been working in partnership with state and local governmental public health practitioners to design and develop On Your Time, a series of online, awareness-level training modules for the local public health workforce. When the series is complete (estimated to be in 2014), it will cover the areas identified by the Competency Report and training gap analysis. Moreover, the learning objectives of each module will be linked back to the report. According to the Institute for Community Health (2012), statewide, there are approximately 3,600 Board of Health members, local health department staff, and state public health and environmental protection staff. These groups comprise the target audience for On Your Time, although certain modules are also appropriate for the general public and the regulated community (e.g., tanning facilities, septic systems, lyme disease, etc.). The series is based on nine principles dictating the training modules: 1. Public health competency-based and prioritized to address the most pressing gaps in training as determined by the Local Public Health Institute of Massachusetts (2010a) 2. Evaluated for impact and satisfaction and subject to quality improvement as needed 3. Accessible any time, free of charge, via the Internet or CD on request (for those without high-speed Internet) 4. Based on best practices for distance education and adult learning (American Distance Education Consortium, 2002; Boettcher, 2007; Hanover Research Council, 2009), including being easy to navigate, using interactive links to additional resources, and displaying a good balance of words and graphics 5. Reviewed annually by subject matter experts to ensure up-to-date content 6. Useful in multiple ways (e.g., as an online course for an individual, as a reference manual for an individual, and as an instructor-led training for a group using the facilitator’s guide) 7. Supportive of public health credentialing by providing continuing education credits/contact hours 8. Resource-friendly, drawing from existing content and volunteer subject matter experts whenever possible to support sustainability 9. Organized according to a standard format for modules related to regulatory program areas (see Table 2), and following a consistent, specific order.
50S HEALTH PROMOTION PRACTICE / March 2014
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Table 1 Public Health Competencies in Massachusetts Program Area Competencies 1. Air Quality 2. Animal Control 3. Body Art 4. Disease Case Management 5. Disease Surveillance, Investigation, and Follow-Up 6. Drinking Water 7. Food Protection 8. Hazardous and Medical/Biologic Waste 9. Health Promotion and Disease Prevention 10. Housing 11. Nuisance Control and Noisome Trades 12. Recreational Camps for Children 13. Recreational Waters: Swimming Pools and Bathing Beaches 14. Solid Waste 15. Tanning Establishments 16. Vaccine Management 17. Wastewater Treatment Crosscutting Competencies 1. Advocacy 2. Analysis, Problem Solving, and Risk Management 3. Communication 4. Community/Public Health Assessment 5. Cultural Competence 6. Emergency Preparedness 7. Health Education 8. Leadership 9. Legal Issues 10. Project Development, Planning, and Management SOURCE: Local Public Health Institute of Massachusetts (2010a, 2010b).
Table 2 Standard Format for Regulatory Training Modules Title and credits Introduction and learning objectives Background Including science and health Administration and authority Regulation and laws Inspection tools and equipment Enforcement strategies Summary of standards Resources
The modules describe the main concepts of each topic with associated examples and illustrations and employ interactive activities—such as embedded video or audio clips, online photo albums, flash cards, or multiplechoice quizzes with feedback—to keep participants engaged. Users have two options to access information: they can view the module online with a download and print option for use as a reference manual or they can complete the module online—along with pretest, posttest, and evaluation—and earn continuing education credit. The latter option allows users to print a certificate of completion. For added flexibility and to meet the needs of adults who prefer face-to-face training, the modules are packaged with
Kenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 51S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Table 3 Development Process for On Your Time Training Modules Step 1
2
Elements Training topics identified and prioritized Subject matter experts identified
3
Module content received and reviewed
4
Module drafted with software
5 6
Interactivity integrated Facilitator’s guide drafted
7
Module draft reviewed and edited Additional module elements drafted Copy reviewed and edited Full package reviewed and finalized
9 10 12
14
Posting
15
Master CD produced
16 17
Evaluation Content review
Description Training topics come from the Competency Report (LPHI, 2010b); the advisory committee members, with feedback from constituents, prioritize 6 to 8 modules for development each year. Boston University identifies a module development team consisting of primary and contributing authors and reviewers; the university also identifies and shares any existing training materials or content available on the training topic. Authors follow a standard table of contents when drafting the content (as a Microsoft® Word document); university staff and faculty review and edit the document, which is returned for review to the development team; this step is repeated, as necessary. A Boston University e-learning specialist uploads the Word document to online publishing software; the layout conforms to a standard design outlined in the LPHI’s style guide. After content is loaded, interactive features are added to each module. Boston University staff draft a facilitator’s guide, using a template with standard instructions, teaching tips, facilitator notes, supplemental activities, and discussion topics. The development team vets the draft module and facilitator’s guide and Boston University staff incorporate edits. University staff draft pre- and posttest questions and map the crosscutting competencies. All elements are sent to an editor for review and Boston University staff incorporate edits. The LPHI advisory committee reviews the entire package (online module, facilitator’s guide, test questions, table of crosscutting competencies); Boston University staff incorporate final edits. University staff post and market the module and links to the LPHI’s learning management system. University provides a CD of the module if the trainee does not have access to highspeed Internet. Evaluator and university staff analyze evaluation data for quality improvement. University staff review and update modules annually, with help from students and committee members. Web links are systematically checked semi-annually.
NOTE: LPHI = Local Public Health Institute of Massachusetts.
a facilitator’s guide that contains additional discussion material and teaching tips and can be printed and used to guide classroom training. Modules are also available on CD for those without high-speed Internet capabilities, a known problem in certain parts of Massachusetts.
Development Costs and Progress >> to Date
Module development and review uses a multi step process (see Table 3) and team approach with
representatives from academia, the state’s departments of public health and environmental protection, local boards of health, local health departments, and state public health associations, including the Massachusetts Environmental Health Association, Massachusetts Health Officers Association, Massachusetts Association of Public Health Nurses, Massachusetts Association of Health Boards, and Western Massachusetts Public Health Association. Based on available resources, a typical module takes approximately 80 hours to develop over a 6- to 12-month period, a process that costs approximately $3,000 to
52S HEALTH PROMOTION PRACTICE / March 2014
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Table 4 Current On Your Time Module Titles 1. Orientation to Local Public Health in Massachusetts 2. Using the Health and Homeland Alert Network 3. Massachusetts Virtual Epidemiologic Network Introduction (MAVEN) 4. Recreational Waters: Swimming Pools 5. Lyme Disease 6. Dealing With Stress in Disasters: Building Psychological Resilience 7. Bed Bugs 8. MAVEN—TB 9. Tanning Facilities 10. Wastewater and Title 5 11. Public Health Law and Legal Issues in Massachusetts 12. Hoarding 13. Housing 14. Emergency Preparedness in Massachusetts 15. Food Protection 16. Body Art 17. Emergency Preparedness Begins at Home 18. Nuisance Control and Abatement 19. Public Health Workforce Protection 20. Recreational Camps for Children 21. Recreational Waters: Bathing Beaches 22. Drinking Water and Private Wells
$5,000, excluding software licenses and institute infrastructure costs. Graduate students often conduct background research and assist with computer programming, which gives them a valuable experience while containing development costs. Budget line items include payments for the primary author if a volunteer cannot be identified, e-learning specialist, technology and education experts, editor, administrator, and evaluator. The first online module was launched in January 2011. As of May 2013, 22 On Your Time modules have been completed (see Table 4), covering many identified programmatic and crosscutting training needs.
Evaluation Findings to Date >> The goal of On Your Time is to increase public health workforce expertise in the programmatic and crosscutting competencies identified in the Competency Report (Local Public Health Institute, 2010b). To assess this goal, we employ a mixed-methods evaluation using quantitative questions, supplemented by a few qualitative items. Collection of demographic and descriptive
data at the point of trainee registration enables assessment of the population of module users, intended to be our target audience of public health professionals across the state of Massachusetts. Changes in knowledge attributable to the trainings are assessed through online pre- and posttest multiple-choice quizzes. Additionally, we use 5-point Likert-type scales to assess perceived changes in trainees’ ability to perform module-specific learning objectives before and after training and satisfaction with various aspects of the modules. Openended questions solicit additional information about the value of the modules, areas for improvement, and suggestions for future module topics. As of May 2013, 730 trainees have completed modules and received certificates of completion. In addition, module webpages have had over 10,000 other hits and more than 3,755 returning users, demonstrating their utility as reference manuals and curricula for group/classroom training. As new modules are added and marketing efforts increased, the number of users is increasing dramatically. Of the 730 trainees who received certificates of completion, 70% are members of the local or state public health workforce, the target audience. Trainees come from all areas of the state, with the largest group (24.1%) residing in western Massachusetts, where access to classroom education historically has been impeded by the lost work time and expenses associated with off-site training. The smallest proportion of certificate earners (4.5%) work in Boston, where there is greater access to classroom learning opportunities. Analysis of evaluation data has been completed for the period January 2011 to June 2012, during which time 300 trainees completed the modules. Data show that the modules effectively increased trainee knowledge about module-specific content, and improved trainees’ perceptions of their ability to perform modulespecific learning objectives. More trainees answered 65 of the 75 quiz questions correctly at posttest than at pretest, with the average posttest score being 91%, compared with 61% pretest. Trainees rated their selfperceived ability to perform module-specific learning objectives using a 5-point Likert-type scale (1 = not at all prepared to 5 = extremely prepared) at pre- and posttest. Mean pretest ratings for the modules ranged from 2.17 to 2.97, and mean posttest ratings ranged from 2.76 to 4.75. The mean ratings for each individual module increased from pre- to posttest, indicating that trainees’ perceptions of their ability to perform module-specific learning objectives improved after training. A separate 5-point Likert-type scale was used to assess agreement with a number of statements about the training (1 = strong disagreement to 5 = strong agreement). These data indicate high rates of satisfaction with
Kenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 53S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
module-specific content, difficulty level and technical information, use of quiz questions to engage participants in learning, links to resources, and the overall learning experience. Trainees strongly agreed that they would recommend the modules to new and experienced public health practitioners. For all modules, mean scores for each satisfaction measure were 4.0 or greater. Examination of open-ended question responses showed that users particularly value the interactive nature of the modules and the ability to engage in training at a self-directed pace.
Next Steps and Sustainability >> The training institute plans to complete all 27 planned competency modules by 2014. The institute will also increase its marketing to state and local public health workers and municipal officials to spur greater use of the modules. The enthusiastic involvement of state and local practitioners and the ongoing engagement of statewide professional organizations ensure sustainability. Senior leaders at the state public health and environmental health departments have committed staff to serve as subject matter experts and content reviewers, and leaders of state public health associations have enlisted volunteers with content expertise to participate in the module development and review process. The advisory committee of the Local Public Health Institute of Massachusetts will continue to prioritize training needs for annual module planning and also take part in module development and review. The institute’s advisory committee identified 10 steps that will be implemented to sustain and grow On Your Time:
1. Secure and use additional training grant resources to support development of new modules and quality improvement of existing modules. 2. Work with state and local agency practitioners and university faculty, staff, and students to develop new modules and review existing modules. 3. Engage the training institute’s advisory committee and state professional associations in prioritizing module development and identifying subject matter experts for module review and revisions. 4. Evaluate knowledge gains and perceived ability to perform learning objectives as a result of training, as well as user satisfaction with each training module. 5. Engage subject matter experts to analyze evaluation data to inform quality improvement efforts. 6. Offer contact hours and certificates of completion. 7. Develop and post facilitator guides for alternative delivery methods (e.g., classroom or small group instruction).
8. Continue to emphasize the need for public health workforce development to state and local public health officials and municipal leaders, including town managers, mayors, and boards of selectmen. 9. Designate select On Your Time modules as prerequisites for performance-level trainings (e.g., Massa chusetts Public Health Inspector Trainings in Housing and Food Protection). 10. Market relevant modules to the regulated community (e.g., tanning facility operators).
Discussion >> After thoroughly researching available online public health trainings and reviewing the associated literature, the Local Public Health Institute of Massachusetts could find no comparable training modules anywhere in the country incorporating the full suite of On Your Time features and the institute’s standard, highly vetted methodology. Standardized On Your Time modules transcend the caliber and content preferences of any one instructor and assure high-quality training for all Massachusetts public health workers, whether in rural, suburban, or urban settings. By combining the resources of Boston University School of Public Health–funded training programs and forging partnerships with professional associations and state and local public health personnel who share a mission to train the workforce, the Local Public Health Institute of Massachusetts has been able to implement and expand a unique distance learning program with a modest budget. There were several lessons learned throughout the development process: 1. Subject matter experts should not only serve as onetime providers of materials and module content but also remain involved in the development and annual review process and vetting of the module. Their involvement ensures that the final product reflects the standard of practice and day-to-day work of local public health professionals. 2. It is beneficial to recruit one subject matter expert to act as the primary module author, usually a local or state public health practitioner. This individual is responsible for drafting module content and provides a relevant perspective on the application of public health laws and regulations and enforcement of practice standards. 3. The use of interactive features is important to keep module users engaged. 4. A standard table of contents and consistent format gives primary authors a template for drafting new modules and makes it easier for trainees to find information quickly.
54S HEALTH PROMOTION PRACTICE / March 2014
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
5. Evaluation and continuous quality improvement are crucial to ensure the effectiveness of the modules.
Conclusions >> In this article, we provide details to facilitate replication of the On Your Time model in other states, including the principles underpinning the design process, an outline of the module development process and costs, an overview of the evaluation process and findings to date, steps to ensure sustainability, and lessons learned. Sustainable, competency-based distance learning training models like the On Your Time series are critical to ensure public health practitioners are prepared to perform their mandated duties and to address persistent and emerging public health problems and opportunities. Other states interested in developing their own training series may conserve resources by employing the guidance provided in this article and adapting completed On Your Time modules—available at www. masslocalinstitute.org—to address their state-specific laws, regulations, and practices. References Alur, P., Fatima, K., & Joseph, R. (2002). Medical teaching websites: do they reflect the learning paradigm? Medical Teaching, 24, 422-424. American Distance Education Consortium. (2002). Guiding principles for distance learning. Retrieved from http://www.adec. edu/admin/papers/distance-learning_principles.html Ballew, P., Castro, S., Claus, J., Kittur, N., Brennan, L., & Brownson, R. C. (2013). Developing web-based training for public health practitioners: What can we learn from a review of five disciplines? Health Education Research, 28, 276-287. Boettcher, J. V. (2007). Ten core principles for designing effective learning environments: Insights from brain research and pedagogical theory. Retrieved from www.innovateonline.info/index. php?view=article&id=54 Bryan, R. L., Kreuter, M. W., & Brownson, R. C. (2009). Integrating adult learning principles into training for public health practice. Health Promotion Practices, 10, 557-563. Chandler, T., Qureshi, K., Gebbie, K. M., & Morse, S. S. (2008). Teaching emergency preparedness to public health workers: Use of blended learning in web-based training. Public Health Reports, 123, 676-680. Chaney, B. H., Eddy, J. M., Dorman, S. M., Glessner, L. L., Green, B. L., & Lara-Alecio, R. A. (2009). Primer on quality indicators of distance education. Health Promotion Practice, 10, 222-231.
Claus, J. M., Erickson, J., Brennan Ramirez, L. K., Baker, E. A., & Land, G. (2008). Missouri-evaluation of an evidence-based intervention planning website for public health practitioners. In D. J. Friedman & R. G. Parrish (Eds.), Issues in evaluating health department web-based data query systems: Working papers (pp. 77-90). Princeton, NJ: Robert Wood Johnson Foundation. Gebbie, K., Merrill, J., & Tilson, H. H. (2002). The public health workforce. Health Affairs, 21(6), 57-67. Gebbie, K. M., & Turnock, B. J. (2006). The public health workforce, 2006: New challenges. Health Affairs, 25, 923-933. Giguere, P., & Minotti, J. (2003). Developing high-quality webbased training for adult learners. Educational Technology, 43, 57-58. Hanover Research Council. (2009). Best practices in online teaching strategies. Washington, DC: Author. Retrieved from http:// www.uwec.edu/AcadAff/resources/edtech/upload/BestPractices-in-Online-Teaching-Strategies-Membership.pdf Harun, M. H. (2002). Integrating e-learning into the workplace. Internet and Higher Education, 4, 301-310. Hyde, J., & Tovar, A. (2006). Strengthening local public health in Massachusetts: A call to action. Results of a Statewide workforce assessment conducted for the Coalition for Local Public Health. Cambridge, MA: Institute for Community Health. Retrieved from http://mphaweb.org/resources/strength_lph_6_06.pdf Institute for Community Health. (2012). PHACTS study to shape public health departments in future. Institute for Community Health Newsletter, Spring, 4. Institutes of Medicine. (2003a). The future of the public’s health in the twenty-first century. Washington, DC: National Academies Press. Institutes of Medicine. (2003b). Who will keep the public healthy? Educating public health professionals for the twenty-first century. Washington, DC: National Academies Press. Koo, D., & Miner, K. (2010). Outcome-based workforce development and education in public health. Annual Review of Public Health, 31, 253-269. Local Public Health Institute of Massachusetts. (2010a). Analysis of gaps in training for the Massachusetts public health workforce. Unpublished report. Local Public Health Institute of Massachusetts. (2010b). Competency report. Retrieved from http://www.masslocalinstitute.org/wp-content/uploads/2012/04/MA-Competeny-Reportrev-Dec-2010.pdf Wright, K. (2009). Competency development in public health leadership. American Journal of Public Health, 90, 1202-1207. Zusevics, K. L., Gilmore, G. D., Jecklin, R. A., & Swain, G. R. (2009). Online education: The needs, interests, and capacities of Wisconsin public health professionals. MERLOT Journal of Online Learning and Teaching, 5, 531-545. Retrieved from http:// jolt.merlot.org/vol5no3/zusevics_0909.htm
Kenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 55S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
HPPXXX10.1177/1524839913509270He alth Promotion PracticeKenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 2013
On Your Time: Online Training for the Public Health Workforce Hope Worden Kenefick, MSW, PhD1 Sharon Ravid, MPH2 Kathleen MacVarish, REHS/RS, MS2 Jennifer Tsoi, MPH2 Kenny Weill, MS3 Elizabeth Faye, BA2 Anne Fidler, MS, ScD2
The need for competency-based training for the public health workforce is well documented. However, human and financial resource limitations within public health agencies often make it difficult for public health practitioners to attend classroom-based training programs. The Internet is an increasingly popular way of extending training beyond the workforce. Although research describes attributes of effective online learning modules, much of the available training delivered via the Internet does not incorporate such attributes. The authors describe the On Your Time training series, an effective distance education program and training model for public health practitioners, which includes a standardized process for development, review, evaluation, and continuous quality improvement. On Your Time is a series of awareness-level (i.e., addressing what practitioners should know), competency-based training modules that address topics related to regulatory responsibilities of public health practitioners (e.g., assuring compliance with codes and regulations governing housing, retail food safety, private water supplies, hazardous and solid waste, on-site wastewater systems, etc.), public health surveillance, case investigation, disease prevention, health promotion, and emergency preparedness. The replicable model incorporates what is known about best practices for online training and maximizes available resources in the interests of sustainability.
Health Promotion Practice March 2014 Vol. 15, Suppl 1 48S–55S DOI: 10.1177/1524839913509270 © 2014 Society for Public Health Education
Keywords: workforce development; training; distance education; technology; continuing education
1
HWK Consulting, Barrington, NH, USA Boston University School of Public Health, Boston, MA, USA 3 K. Weill Consulting Group, LLC, Brookline, MA, USA 2
Authors’ Note: This work is supported by the New England Alliance for Public Health Workforce Development, a Health Resources and Services Administration Public Health Training Center (Grant No. UB6HP20150), and by the Local Public Health Institute of Massachusetts, a Massachusetts Department of Public Health, Emergency Preparedness Bureau–funded entity supported by the Centers for Disease Control and Prevention’s Public Health Preparedness and Response for Bioterrorism (Grant No. 5U90TP116997-10). We would like to acknowledge Dr. Wayne LaMorte and Rob Schadt for their immense contributions to this project as well as Courtney Perdios for reviewing the article. Address correspondence to Sharon Ravid, Program Manager, New England Alliance for Public Health Workforce Development at Boston University School of Public Health, 715 Albany Street, Talbot Building W118C Boston, MA 02118, USA; e-mail: sravid@ bu.edu.
Supplement Note: This article is published in the supplement “Public Health Training Centers,” which was supported by the U.S. Department of Health and Human Services (HHS) Health Resources and Services Administration (HRSA) through a Cooperative Agreement (UBGHP20199) to Rollins School of Public Health, Emory University.
48S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Introduction >>
The need for competency-based training for the public health workforce has been well documented (Chandler, Qureshi, Gebbie, & Morse, 2008; Gebbie & Turnock, 2006; Harun, 2002; Koo & Miner, 2010; Wright, 2009). Recognizing the inherent challenge of maintaining and enhancing the skills of public health practitioners, the Institutes of Medicine (2003a, 2003b) has called for continuous, lifelong training for the public health workforce. This need, coupled with local health department staffing shortages that have expanded workloads and budgetary constraints restricting employee travel, has led to increased use of distance learning technology to deliver training. Although the attributes of effective online learning (e.g., structure, preferences of adult learners, evaluation protocols, etc.) have been studied, much of the available Internetbased training does not incorporate such attributes (Alur, Fatima, & Joseph, 2002; Chandler et al., 2008; Zusevics, Gilmore, Jecklin, & Swain, 2009). Ensuring that online trainings are up-to-date, accurate, effective, and user-friendly is an additional challenge, as it requires ongoing investment in content review, editing, and evaluation. This article describes a process for the development, review, evaluation, and improvement of distance education through the use of online, self-paced, awarenesslevel training modules for local public health practitioners. These modules address the learning needs of the public health workforce, incorporate what is known about best practices for online training, and maximize available resources in the interest of sustainability.
Background >>
A large portion of U.S. public health workers are employed in small organizations with limited human and financial resources (Gebbie, Merrill, & Tilson, 2002). This is the case in Massachusetts, where most local health agencies serve small populations (> Applications
Since January 2010, the Boston University School of Public Health has managed the Local Public Health Institute of Massachusetts, the state’s training institute for local public health practitioners. Institute staff work with an advisory committee, comprising local and state public health representatives, to pursue a critical mission: To provide and ensure a competent workforce by strengthening and sustaining the capacity of local boards of health to prepare for and respond to public health issues and emergencies and to promote the health of residents of the Commonwealth. The institute’s Competency Report (2010b) and unpublished Analysis of Gaps in Training for the Massachusetts Public Health Workforce (2010a) detail public health training needs across the state and identify 17 program areas and 10 crosscutting areas in which local public health practitioners should have competency (see Table 1). Additionally, ample literature (Ballew et al., 2013; Bryan et al., 2009; Chaney et al., 2009; Claus
et al., 2008; Claus et al., 2009; Giguere & Minotti, 2003; Koo & Miner, 2010; Zusevics et al., 2009) describe the effective attributes of online training programs. Based on this information, Boston University faculty and staff have been working in partnership with state and local governmental public health practitioners to design and develop On Your Time, a series of online, awareness-level training modules for the local public health workforce. When the series is complete (estimated to be in 2014), it will cover the areas identified by the Competency Report and training gap analysis. Moreover, the learning objectives of each module will be linked back to the report. According to the Institute for Community Health (2012), statewide, there are approximately 3,600 Board of Health members, local health department staff, and state public health and environmental protection staff. These groups comprise the target audience for On Your Time, although certain modules are also appropriate for the general public and the regulated community (e.g., tanning facilities, septic systems, lyme disease, etc.). The series is based on nine principles dictating the training modules: 1. Public health competency-based and prioritized to address the most pressing gaps in training as determined by the Local Public Health Institute of Massachusetts (2010a) 2. Evaluated for impact and satisfaction and subject to quality improvement as needed 3. Accessible any time, free of charge, via the Internet or CD on request (for those without high-speed Internet) 4. Based on best practices for distance education and adult learning (American Distance Education Consortium, 2002; Boettcher, 2007; Hanover Research Council, 2009), including being easy to navigate, using interactive links to additional resources, and displaying a good balance of words and graphics 5. Reviewed annually by subject matter experts to ensure up-to-date content 6. Useful in multiple ways (e.g., as an online course for an individual, as a reference manual for an individual, and as an instructor-led training for a group using the facilitator’s guide) 7. Supportive of public health credentialing by providing continuing education credits/contact hours 8. Resource-friendly, drawing from existing content and volunteer subject matter experts whenever possible to support sustainability 9. Organized according to a standard format for modules related to regulatory program areas (see Table 2), and following a consistent, specific order.
50S HEALTH PROMOTION PRACTICE / March 2014
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Table 1 Public Health Competencies in Massachusetts Program Area Competencies 1. Air Quality 2. Animal Control 3. Body Art 4. Disease Case Management 5. Disease Surveillance, Investigation, and Follow-Up 6. Drinking Water 7. Food Protection 8. Hazardous and Medical/Biologic Waste 9. Health Promotion and Disease Prevention 10. Housing 11. Nuisance Control and Noisome Trades 12. Recreational Camps for Children 13. Recreational Waters: Swimming Pools and Bathing Beaches 14. Solid Waste 15. Tanning Establishments 16. Vaccine Management 17. Wastewater Treatment Crosscutting Competencies 1. Advocacy 2. Analysis, Problem Solving, and Risk Management 3. Communication 4. Community/Public Health Assessment 5. Cultural Competence 6. Emergency Preparedness 7. Health Education 8. Leadership 9. Legal Issues 10. Project Development, Planning, and Management SOURCE: Local Public Health Institute of Massachusetts (2010a, 2010b).
Table 2 Standard Format for Regulatory Training Modules Title and credits Introduction and learning objectives Background Including science and health Administration and authority Regulation and laws Inspection tools and equipment Enforcement strategies Summary of standards Resources
The modules describe the main concepts of each topic with associated examples and illustrations and employ interactive activities—such as embedded video or audio clips, online photo albums, flash cards, or multiplechoice quizzes with feedback—to keep participants engaged. Users have two options to access information: they can view the module online with a download and print option for use as a reference manual or they can complete the module online—along with pretest, posttest, and evaluation—and earn continuing education credit. The latter option allows users to print a certificate of completion. For added flexibility and to meet the needs of adults who prefer face-to-face training, the modules are packaged with
Kenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 51S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Table 3 Development Process for On Your Time Training Modules Step 1
2
Elements Training topics identified and prioritized Subject matter experts identified
3
Module content received and reviewed
4
Module drafted with software
5 6
Interactivity integrated Facilitator’s guide drafted
7
Module draft reviewed and edited Additional module elements drafted Copy reviewed and edited Full package reviewed and finalized
9 10 12
14
Posting
15
Master CD produced
16 17
Evaluation Content review
Description Training topics come from the Competency Report (LPHI, 2010b); the advisory committee members, with feedback from constituents, prioritize 6 to 8 modules for development each year. Boston University identifies a module development team consisting of primary and contributing authors and reviewers; the university also identifies and shares any existing training materials or content available on the training topic. Authors follow a standard table of contents when drafting the content (as a Microsoft® Word document); university staff and faculty review and edit the document, which is returned for review to the development team; this step is repeated, as necessary. A Boston University e-learning specialist uploads the Word document to online publishing software; the layout conforms to a standard design outlined in the LPHI’s style guide. After content is loaded, interactive features are added to each module. Boston University staff draft a facilitator’s guide, using a template with standard instructions, teaching tips, facilitator notes, supplemental activities, and discussion topics. The development team vets the draft module and facilitator’s guide and Boston University staff incorporate edits. University staff draft pre- and posttest questions and map the crosscutting competencies. All elements are sent to an editor for review and Boston University staff incorporate edits. The LPHI advisory committee reviews the entire package (online module, facilitator’s guide, test questions, table of crosscutting competencies); Boston University staff incorporate final edits. University staff post and market the module and links to the LPHI’s learning management system. University provides a CD of the module if the trainee does not have access to highspeed Internet. Evaluator and university staff analyze evaluation data for quality improvement. University staff review and update modules annually, with help from students and committee members. Web links are systematically checked semi-annually.
NOTE: LPHI = Local Public Health Institute of Massachusetts.
a facilitator’s guide that contains additional discussion material and teaching tips and can be printed and used to guide classroom training. Modules are also available on CD for those without high-speed Internet capabilities, a known problem in certain parts of Massachusetts.
Development Costs and Progress >> to Date
Module development and review uses a multi step process (see Table 3) and team approach with
representatives from academia, the state’s departments of public health and environmental protection, local boards of health, local health departments, and state public health associations, including the Massachusetts Environmental Health Association, Massachusetts Health Officers Association, Massachusetts Association of Public Health Nurses, Massachusetts Association of Health Boards, and Western Massachusetts Public Health Association. Based on available resources, a typical module takes approximately 80 hours to develop over a 6- to 12-month period, a process that costs approximately $3,000 to
52S HEALTH PROMOTION PRACTICE / March 2014
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
Table 4 Current On Your Time Module Titles 1. Orientation to Local Public Health in Massachusetts 2. Using the Health and Homeland Alert Network 3. Massachusetts Virtual Epidemiologic Network Introduction (MAVEN) 4. Recreational Waters: Swimming Pools 5. Lyme Disease 6. Dealing With Stress in Disasters: Building Psychological Resilience 7. Bed Bugs 8. MAVEN—TB 9. Tanning Facilities 10. Wastewater and Title 5 11. Public Health Law and Legal Issues in Massachusetts 12. Hoarding 13. Housing 14. Emergency Preparedness in Massachusetts 15. Food Protection 16. Body Art 17. Emergency Preparedness Begins at Home 18. Nuisance Control and Abatement 19. Public Health Workforce Protection 20. Recreational Camps for Children 21. Recreational Waters: Bathing Beaches 22. Drinking Water and Private Wells
$5,000, excluding software licenses and institute infrastructure costs. Graduate students often conduct background research and assist with computer programming, which gives them a valuable experience while containing development costs. Budget line items include payments for the primary author if a volunteer cannot be identified, e-learning specialist, technology and education experts, editor, administrator, and evaluator. The first online module was launched in January 2011. As of May 2013, 22 On Your Time modules have been completed (see Table 4), covering many identified programmatic and crosscutting training needs.
Evaluation Findings to Date >> The goal of On Your Time is to increase public health workforce expertise in the programmatic and crosscutting competencies identified in the Competency Report (Local Public Health Institute, 2010b). To assess this goal, we employ a mixed-methods evaluation using quantitative questions, supplemented by a few qualitative items. Collection of demographic and descriptive
data at the point of trainee registration enables assessment of the population of module users, intended to be our target audience of public health professionals across the state of Massachusetts. Changes in knowledge attributable to the trainings are assessed through online pre- and posttest multiple-choice quizzes. Additionally, we use 5-point Likert-type scales to assess perceived changes in trainees’ ability to perform module-specific learning objectives before and after training and satisfaction with various aspects of the modules. Openended questions solicit additional information about the value of the modules, areas for improvement, and suggestions for future module topics. As of May 2013, 730 trainees have completed modules and received certificates of completion. In addition, module webpages have had over 10,000 other hits and more than 3,755 returning users, demonstrating their utility as reference manuals and curricula for group/classroom training. As new modules are added and marketing efforts increased, the number of users is increasing dramatically. Of the 730 trainees who received certificates of completion, 70% are members of the local or state public health workforce, the target audience. Trainees come from all areas of the state, with the largest group (24.1%) residing in western Massachusetts, where access to classroom education historically has been impeded by the lost work time and expenses associated with off-site training. The smallest proportion of certificate earners (4.5%) work in Boston, where there is greater access to classroom learning opportunities. Analysis of evaluation data has been completed for the period January 2011 to June 2012, during which time 300 trainees completed the modules. Data show that the modules effectively increased trainee knowledge about module-specific content, and improved trainees’ perceptions of their ability to perform modulespecific learning objectives. More trainees answered 65 of the 75 quiz questions correctly at posttest than at pretest, with the average posttest score being 91%, compared with 61% pretest. Trainees rated their selfperceived ability to perform module-specific learning objectives using a 5-point Likert-type scale (1 = not at all prepared to 5 = extremely prepared) at pre- and posttest. Mean pretest ratings for the modules ranged from 2.17 to 2.97, and mean posttest ratings ranged from 2.76 to 4.75. The mean ratings for each individual module increased from pre- to posttest, indicating that trainees’ perceptions of their ability to perform module-specific learning objectives improved after training. A separate 5-point Likert-type scale was used to assess agreement with a number of statements about the training (1 = strong disagreement to 5 = strong agreement). These data indicate high rates of satisfaction with
Kenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 53S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
module-specific content, difficulty level and technical information, use of quiz questions to engage participants in learning, links to resources, and the overall learning experience. Trainees strongly agreed that they would recommend the modules to new and experienced public health practitioners. For all modules, mean scores for each satisfaction measure were 4.0 or greater. Examination of open-ended question responses showed that users particularly value the interactive nature of the modules and the ability to engage in training at a self-directed pace.
Next Steps and Sustainability >> The training institute plans to complete all 27 planned competency modules by 2014. The institute will also increase its marketing to state and local public health workers and municipal officials to spur greater use of the modules. The enthusiastic involvement of state and local practitioners and the ongoing engagement of statewide professional organizations ensure sustainability. Senior leaders at the state public health and environmental health departments have committed staff to serve as subject matter experts and content reviewers, and leaders of state public health associations have enlisted volunteers with content expertise to participate in the module development and review process. The advisory committee of the Local Public Health Institute of Massachusetts will continue to prioritize training needs for annual module planning and also take part in module development and review. The institute’s advisory committee identified 10 steps that will be implemented to sustain and grow On Your Time:
1. Secure and use additional training grant resources to support development of new modules and quality improvement of existing modules. 2. Work with state and local agency practitioners and university faculty, staff, and students to develop new modules and review existing modules. 3. Engage the training institute’s advisory committee and state professional associations in prioritizing module development and identifying subject matter experts for module review and revisions. 4. Evaluate knowledge gains and perceived ability to perform learning objectives as a result of training, as well as user satisfaction with each training module. 5. Engage subject matter experts to analyze evaluation data to inform quality improvement efforts. 6. Offer contact hours and certificates of completion. 7. Develop and post facilitator guides for alternative delivery methods (e.g., classroom or small group instruction).
8. Continue to emphasize the need for public health workforce development to state and local public health officials and municipal leaders, including town managers, mayors, and boards of selectmen. 9. Designate select On Your Time modules as prerequisites for performance-level trainings (e.g., Massa chusetts Public Health Inspector Trainings in Housing and Food Protection). 10. Market relevant modules to the regulated community (e.g., tanning facility operators).
Discussion >> After thoroughly researching available online public health trainings and reviewing the associated literature, the Local Public Health Institute of Massachusetts could find no comparable training modules anywhere in the country incorporating the full suite of On Your Time features and the institute’s standard, highly vetted methodology. Standardized On Your Time modules transcend the caliber and content preferences of any one instructor and assure high-quality training for all Massachusetts public health workers, whether in rural, suburban, or urban settings. By combining the resources of Boston University School of Public Health–funded training programs and forging partnerships with professional associations and state and local public health personnel who share a mission to train the workforce, the Local Public Health Institute of Massachusetts has been able to implement and expand a unique distance learning program with a modest budget. There were several lessons learned throughout the development process: 1. Subject matter experts should not only serve as onetime providers of materials and module content but also remain involved in the development and annual review process and vetting of the module. Their involvement ensures that the final product reflects the standard of practice and day-to-day work of local public health professionals. 2. It is beneficial to recruit one subject matter expert to act as the primary module author, usually a local or state public health practitioner. This individual is responsible for drafting module content and provides a relevant perspective on the application of public health laws and regulations and enforcement of practice standards. 3. The use of interactive features is important to keep module users engaged. 4. A standard table of contents and consistent format gives primary authors a template for drafting new modules and makes it easier for trainees to find information quickly.
54S HEALTH PROMOTION PRACTICE / March 2014
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
5. Evaluation and continuous quality improvement are crucial to ensure the effectiveness of the modules.
Conclusions >> In this article, we provide details to facilitate replication of the On Your Time model in other states, including the principles underpinning the design process, an outline of the module development process and costs, an overview of the evaluation process and findings to date, steps to ensure sustainability, and lessons learned. Sustainable, competency-based distance learning training models like the On Your Time series are critical to ensure public health practitioners are prepared to perform their mandated duties and to address persistent and emerging public health problems and opportunities. Other states interested in developing their own training series may conserve resources by employing the guidance provided in this article and adapting completed On Your Time modules—available at www. masslocalinstitute.org—to address their state-specific laws, regulations, and practices. References Alur, P., Fatima, K., & Joseph, R. (2002). Medical teaching websites: do they reflect the learning paradigm? Medical Teaching, 24, 422-424. American Distance Education Consortium. (2002). Guiding principles for distance learning. Retrieved from http://www.adec. edu/admin/papers/distance-learning_principles.html Ballew, P., Castro, S., Claus, J., Kittur, N., Brennan, L., & Brownson, R. C. (2013). Developing web-based training for public health practitioners: What can we learn from a review of five disciplines? Health Education Research, 28, 276-287. Boettcher, J. V. (2007). Ten core principles for designing effective learning environments: Insights from brain research and pedagogical theory. Retrieved from www.innovateonline.info/index. php?view=article&id=54 Bryan, R. L., Kreuter, M. W., & Brownson, R. C. (2009). Integrating adult learning principles into training for public health practice. Health Promotion Practices, 10, 557-563. Chandler, T., Qureshi, K., Gebbie, K. M., & Morse, S. S. (2008). Teaching emergency preparedness to public health workers: Use of blended learning in web-based training. Public Health Reports, 123, 676-680. Chaney, B. H., Eddy, J. M., Dorman, S. M., Glessner, L. L., Green, B. L., & Lara-Alecio, R. A. (2009). Primer on quality indicators of distance education. Health Promotion Practice, 10, 222-231.
Claus, J. M., Erickson, J., Brennan Ramirez, L. K., Baker, E. A., & Land, G. (2008). Missouri-evaluation of an evidence-based intervention planning website for public health practitioners. In D. J. Friedman & R. G. Parrish (Eds.), Issues in evaluating health department web-based data query systems: Working papers (pp. 77-90). Princeton, NJ: Robert Wood Johnson Foundation. Gebbie, K., Merrill, J., & Tilson, H. H. (2002). The public health workforce. Health Affairs, 21(6), 57-67. Gebbie, K. M., & Turnock, B. J. (2006). The public health workforce, 2006: New challenges. Health Affairs, 25, 923-933. Giguere, P., & Minotti, J. (2003). Developing high-quality webbased training for adult learners. Educational Technology, 43, 57-58. Hanover Research Council. (2009). Best practices in online teaching strategies. Washington, DC: Author. Retrieved from http:// www.uwec.edu/AcadAff/resources/edtech/upload/BestPractices-in-Online-Teaching-Strategies-Membership.pdf Harun, M. H. (2002). Integrating e-learning into the workplace. Internet and Higher Education, 4, 301-310. Hyde, J., & Tovar, A. (2006). Strengthening local public health in Massachusetts: A call to action. Results of a Statewide workforce assessment conducted for the Coalition for Local Public Health. Cambridge, MA: Institute for Community Health. Retrieved from http://mphaweb.org/resources/strength_lph_6_06.pdf Institute for Community Health. (2012). PHACTS study to shape public health departments in future. Institute for Community Health Newsletter, Spring, 4. Institutes of Medicine. (2003a). The future of the public’s health in the twenty-first century. Washington, DC: National Academies Press. Institutes of Medicine. (2003b). Who will keep the public healthy? Educating public health professionals for the twenty-first century. Washington, DC: National Academies Press. Koo, D., & Miner, K. (2010). Outcome-based workforce development and education in public health. Annual Review of Public Health, 31, 253-269. Local Public Health Institute of Massachusetts. (2010a). Analysis of gaps in training for the Massachusetts public health workforce. Unpublished report. Local Public Health Institute of Massachusetts. (2010b). Competency report. Retrieved from http://www.masslocalinstitute.org/wp-content/uploads/2012/04/MA-Competeny-Reportrev-Dec-2010.pdf Wright, K. (2009). Competency development in public health leadership. American Journal of Public Health, 90, 1202-1207. Zusevics, K. L., Gilmore, G. D., Jecklin, R. A., & Swain, G. R. (2009). Online education: The needs, interests, and capacities of Wisconsin public health professionals. MERLOT Journal of Online Learning and Teaching, 5, 531-545. Retrieved from http:// jolt.merlot.org/vol5no3/zusevics_0909.htm
Kenefick et al. / ONLINE TRAINING FOR PUBLIC HEALTH WORKFORCE 55S
Downloaded from hpp.sagepub.com at UNIV CALGARY LIBRARY on May 24, 2015
