American Journal of Pharmaceutical Education 2015; 79 (1) Article 06.
RESEARCH A Qualitative Analysis of Common Concerns about Challenges Facing Pharmacy Experiential Education Programs Jennifer Danielson, PharmD, MBA,a Karen Craddick, PharmD,b Dayl Eccles, PharmD,c Abigail Kwasnik, PharmD,d Teresa A. O’Sullivan, PharmDa a
University of Washington School of Pharmacy, Seattle, Washington Cedars Sinai Medical Center, Los Angeles, California c Rite Aid Pharmacy, Yelm, Washington d St. Joseph Hospital Franciscan Health-System, Tacoma, Washington b
Submitted February 19, 2014; accepted November 10, 2014; published February 17, 2015.
Objective. To qualitatively analyze free-text responses gathered as part of a previously published survey in order to systematically identify common concerns facing pharmacy experiential education (EE) programs. Methods. In 2011, EE directors at all 118 accredited pharmacy schools in the US were asked in a survey to describe the most pressing issues facing their programs. Investigators performed qualitative, thematic analysis of responses and compared results against demographic data (institution type, class size, number of practice sites, number and type of EE faculty member/staff). Expert and novice investigators identified common themes via an iterative process. To check validity, additional expert and novice reviewers independently coded responses. The Cohen kappa coefficient was calculated and showed good agreement between investigators and reviewers. Results. Seventy-eight responses were received (66% response rate) representing 75% of publicly funded institutions and 71% of schools with class sizes 51-150. Themes identified as common concerns were site capacity, workload/financial support, quality assurance, preceptor development, preceptor stipends, assessment, onboarding, and support/recognition from administration. Good agreement (mean percent agreement 93%, ƙ range50.59-0.92) was found between investigators and reviewers. Conclusion. Site capacity for student placements continues to be the foremost concern for many experiential education programs. New concerns about preceptor development and procedures for placing and orienting students at individual practice sites (ie, “onboarding”) have emerged and must be addressed as new accreditation standards are implemented. Keywords: experiential learning, pharmacy education, clinical clerkship, qualitative research, capacity building, advanced pharmacy practice experiences, introductory pharmacy practice experiences, preceptor development, quality assurance
student placement capacity created by dramatic growth in number of pharmacy schools and student enrollments in recent years.2-8 Even though many schools have expanded and reorganized their EE programs,9-13 further reorganization may be necessary to realign job responsibilities with the new emphasis on quality improvement, preceptor development, and interprofessional education.14,15 Now is a good time take stock of all concerns facing EE programs so that appropriate changes can be made to meet new accreditation standards. In 2008, Haase and colleagues reported expert recommendations for ensuring quality in experiential education in a white paper sponsored by the American College of Clinical Pharmacy.16 This report described insights
INTRODUCTION With the release of the American Council for Pharmacy Education’s (ACPE) draft accreditation standards for pharmacy education,1 which emphasize preceptor development and assessment in experiential education (EE), colleges and schools of pharmacy will likely be making changes to accommodate these standards. Faculty members and administrators in EE programs will now be faced with new challenges, in addition to existing difficulties in Corresponding Author: Jennifer Danielson, University of Washington School of Pharmacy, Box 357630, Office of Professional Pharmacy Education, South Campus Center, Suite 244, 1601 NE Columbia Road Seattle, WA 98195. Tel: 206-543-1924. Fax: 206-221-2689. E-mail: [email protected]
1
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. from practicing clinical pharmacists, many of whom were pharmacy faculty members and preceptors. Additional insights from pharmacy preceptors about EE were reported in 2008 and 2013.17,18 Concerns from hospital pharmacists regarding capacity for placements in the hospital setting have also been published in the past 10 years.19-21 However, few reports have been published about what faculty members and staff directly administering EE programs perceive as concerns.22 In 2011, Danielson et al disseminated a survey analyzing the curricular, personnel, and financial characteristics of EE programs to EE directors across the United States.13 As part of the survey, respondents were asked to describe the most pressing issues facing their EE programs. The results of this question were subsequently analyzed and are reported in this paper, so that additional insight from faculty members directly involved in administering EE programs could be shared publicly as many seek strategies for implementing new accreditation standards.
through iterative, repeated comparison. They then independently coded the responses using the standard themes with the possibility of 1 response being categorized into multiple themes. Agreement between the investigators’ independent coding was calculated first, then common coding for the discrepancies was determined by consensus. To check validity, additional reviewers (again, an EE expert faculty member and a novice student) independently coded the responses using the standard themes created by the primary investigators.27 Consistency with investigators’ coding was first compared using percent agreement. The Cohen kappa coefficient (ƙ) was then calculated between the investigators’ and reviewers’ coding results. A ƙ.0.70 was considered satisfactory agreement in this analysis.28 Final results were compared to results of a similar survey conducted in 200113 to determine how concerns have changed over time.
RESULTS Eighty-one colleges and schools responded to the survey (69% response rate), and 78 responses were received for the question of interest (66% response rate). Comparison of respondents with the known population at the time of the survey29 showed that the proportions of responses received for publicly funded institutions (75%) and schools with class size 51-150 (medium 72%, large 71%) were adequate to represent these types of pharmacy colleges and schools. Since so few small schools exist anymore, results from the 1 small school that responded were combined with results for medium-sized schools. Thirty-three respondents (42%) reported being associated with an academic health center, which matches national statistics.29 Eight common themes were identified as concerns (see Table 1). Good agreement was found between the investigators’ initial independent coding (mean 96%, range 86-100%) (Table 1). Mean percent agreement between investigators and reviewers for all themes was 93% (range 78-100%). Good agreement (ƙ$0.7) was found between investigators and the expert reviewer for all themes except for the last 3 (assessment, onboarding, and support/recognition from the dean). Good agreement (ƙ$0.7) was found between the investigators and the novice reviewer for all themes. (Table 2). Site capacity remained as the highest concern over the past 10 years (Table 2). While slight differences were made in categorization between 2001 and 2011, concerns related to financial support for EE, workloads of EE faculty members/staff, preceptor stipends, and support/ recognition by school administration were consistently reported at similar frequencies across years. New issues identified included preceptor development (18%) and
METHODS In 2011, EE directors at all 118 accredited schools of pharmacy in the United States were surveyed using a webbased questionnaire (Catalyst Web Tools, University of Washington Information Technology, Seattle, WA). As part of the 35-item questionnaire, respondents were asked the following open-ended question: “What are the 2 or 3 most pressing issues your professional experience program is facing today?”13 Free-text answers were collected from May to December 2011 and downloaded to Microsoft Excel for analysis. Other data collected included type of institution (public or private) association with academic health centers, number of introductory pharmacy practice experiences (IPPEs) and advanced pharmacy practice experiences (APPEs), number and position of EE faculty members/staff, and class size. Class sizes were categorized as small (#50 students), medium (51-100 students), large (101-150 students), and very large (.150 students). Identifying information about the individual completing the survey was not collected. Goal response rate was 70% (confidence 95%, margin of error 5%).23-25 The project was approved as exempt by the University of Washington Human Subjects Division. The investigation team included 2 faculty members (each with more than 10 years of experience in EE and 1 with training in qualitative methods) and 3 fourth-year doctor of pharmacy students. Investigators performed content analysis of the free-text responses using thematic analysis methodology informed by grounded theory.26 First, the primary investigators, an EE expert (faculty member) and novice (student), identified standard themes 2
3
99
96
Quality Assurance
86
APPEs
Workload/Funding
91
IPPEs
100
99
Competition
Increasing Class Size
95
Investigator Agreement (%)
Site Capacity
Theme
Time, money/salary, personnel/manpower, burnout/turnover, or workload of faculty member/staff in EE departments mentioned. Issues such as having inadequate support staff or faculty member time to manage/coordinate the EE program or to provide teaching/scholarship in addition to program administration, strategic planning, or site visits/quality assurance are put into this category. Also included is funding in general or funding for the EE program. Does not include when someone specifically states funding is needed to pay preceptors or for preceptor stipends. Quality (quality assurance or improvement, or QA/AI) mentioned. This could be quality of sites/preceptors themselves or QA/QI responsibilities or activities that EE faculty members/staff have. Mention of improving standards or assuring consistency between sites/preceptors would also be included in this category.
Capacity issues concerning IPPE requirements or site recruitment for IPPEs is mentioned. Capacity issues for APPEs mentioned.
Capacity issues attributed to increasing enrollments or large class size either at the respondent’s school or at other schools.
Any mention at all of sites and the ability to find or maintain enough sites/preceptors (in amount or type) to support the program. Capacity issues are attributed to competition with other schools or the opening of new schools. Many of these responses actually use the word “competition.”
Description
Table 1. Common Themes Identified with Descriptions and Example Responses
(Continued)
Several responses mention quality control or assurance by name. “Quality assurance for a large number of sites” “Limited number of quality clinical programs available at acute care institutions” “Quality control between sites”
Several responses mention the acute care (or acute care/ gen med) and ambulatory care APPEs by name. “Adequate staff to support the Office of Experiential Education considering the percentage (30%) of the curriculum made up of experiential education.” “Low morale due to under-staffed condition” “Funding for more office/faculty support”
“Saturation of the market with multiple pharmacy schools” “Number of new schools of pharmacy and/or existing schools with expanding enrollments canabalizing preceptors and practice sites.” “Competition/availability of enough quality sites in all needed content areas” “Adequate capacity for institutional IPPE sites/ preceptors to cover our increasing class size” “How do we maintain quality in experiential education with the recent increase in schools and the “established” schools increasing their class sizes?” Several responses mention institutional IPPEs by name.
Examples
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06.
4 Refers to a lack of appreciation on the part of administration for type and the amount of work EE and its faculty members/ staff do. May also include mention that EE programs are not called out as their own recognized unit or do not have representation on the appropriate committees or stakeholder groups to be involved in decisions affecting it. This category does not refer to funding support (that would be under Workload/Manpower).
99
97
Onboarding Students
Support/ Recognition from Administration
97
100
Refers to the need to provide training or development to preceptors in general or as ACPE requires, as well as to increase clinical skills or raise standards at sites so as to have appropriate learning opportunities for students. Many mention “preceptor development” or “adjunct faculty training” by name. Refers to paying sites/preceptors, preceptor stipends (paid by their own school or others), or inadequate funding to pay preceptors/sites in general. This category does not refer to inadequate funding of the EE office/faculty members/staff itself. Mentions general “assessment” or specific assessment of students or collection of assessment data in some way. The response could refer to assessment of student performance or could mention general program assessment. Most of these responses include the word “assessment” or “assessing.” Refers to managing compliance requirements, paperwork/ documentation, onboarding or other prerotation check-offs needed to place students at sites.
Description
97
Investigator Agreement (%)
Assessment
Preceptor Stipends
Preceptor Development
Theme
Table 1. (Continued )
“Increasing compliance requirements for students in practice sites and trying to meet these with no increases in staff.” “Increased complexity of institutional placements (background, drug testing, training requirements).” “Increasing amount of overhead (from sites, accreditation, affiliation agreements, etc.) that we are required to do now that keep us from doing our ‘real’ work” “Recognition by Administration of the scope and responsibility of experiential education” “Having administration acknowledge the importance of experiential education” “...not treated as members of the Administrative Team and, therefore, do not have a voice in the direction of the program.”
“Standardizing assessment tools for ease in preceptor use so that we can maximize sites.” “Assessment of student learning” “Maintaining quality of assessment”
“Resources to pay for the work we expect from preceptors” “Paying for sites.” “Sites/Preceptors asking to be paid”
“Implementing and training adjunct faculty members on how to assess students’ outcomes from their experiences.” “Coordination of education for preceptor faculty” “Advancing our clinical sites”
Examples
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06.
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. Table 2. Agreement and Comparison of Themes Over Time 2001 Survey17 (n=67) Theme (ranked by frequency) 1. Finding/maintaining sites(73%) 2. Assessment/QA (34%) 3. Adequate financial/staff support (28%) 4. Faculty member workload(18%) 5. Payments to sites/preceptors (15%) 6. Desire for better understanding by dean (7%) 7. Dealing with student problems n(4%)
2011 Survey (n=78) Theme (ranked by frequency)
Investigators vs Expert Reviewer kappa (ƙ)
Investigators vs Novice Reviewer kappa (ƙ)
1. Site capacity, 58(73%)
0.80
0.87
2. Workload/funding for EE, 33 (42%) 3. Quality assurance, 27 (34%)
0.76 0.80
0.87 0.77
4. Preceptor development, 14 (18%)
0.70
0.79
5. Preceptor stipends, 10 (14%)
0.89
1
6. Assessment, 10(14%)
0.64
0.84
7. Onboarding students, 9 (11%)
0.59
0.79
8. Support/recognition from administration, 6 (8%)
0.65
0.92
procedures for placing and orienting students at individual practice sites—a process referred to as “onboarding” by many health systems (11%). Dealing with student problems was not mentioned at all in 2011. Private institutions reported concerns about capacity and faculty member/staff workload more often than public institutions (Table 3), even though the former reported having more APPE and IPPE sites than public schools. No difference was found in number of EE faculty member and staff between public and private schools.13 Large and very large schools seemed more concerned about preceptor development and stipends than medium schools, and very large schools were most concerned about quality assurance compared to others. The number of full-time equivalents and type of EE faculty member/staff seemed to be related to select concerns.
Schools with fewer full-time equivalents mentioned workload and financial support more often than schools that employ more faculty members and staff. Schools without associate or assistant dean positions mentioned concerns related to workload and financial support as well as recognition from administration more often than schools with these positions. Schools without an associate or assistant director reported workload/financial support as a concern more often than schools with such positions. Concerns related to site capacity were further divided into issues related to competition with other schools, increased class size, and difficulties with placing IPPEs or APPEs specifically (Table 1). Capacity difficulties related to competition with other schools were reported more often (40%) than difficulties associated with increased class size (12%), IPPEs (31%), or APPEs (26%).
Table 3. Themes by Type and Size of Institution Theme (ranked by frequency) 1. Site capacity 2.Workload/funding for EE 3. Quality assurance 4. Preceptor development 5. Preceptor stipends 6. Assessment 7. Onboarding students 8. Support/recognition from administration
Public, n=46(%) 32 22 15 8 6 5 5 4
(68) (23) (32) (17) (13) (11) (11) (9)
Private, n=31(%)
Medium (#100 students) n=35 (%)
26 (81) 11 (34) 12 (38) 6 (19) 4 (13) 5 (16) 4 (13) 2 (6)
26 (76) 20 (57) 12 (38) 4 (11) 1 (3) 4 (12) 5 (15) 4 (12)
5
Large (101-150 students) n=24 (%) 16 8 6 6 5 3 3 2
(67) (33) (25) (25) (21) (13) (13) (8)
Very Large (.150 students) n=18 (%) 14 (74) 3 (15) 8 (42) 4 (22) 4 (22) 3 (17) 1 (6) 0
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. (TJC) in 2011.30 Some studies proposed streamlining placement procedures so students complete many of their APPEs continuously within 1 system, which reduces paperwork and improves learning outcomes.20,31-35 These results would support use of block scheduling as a way to create efficiency. The results also suggest an important challenge for EE programs is preceptor development, explicitly required by the 2016 draft standards.1 Vos and Trewet suggested that a multifactorial approach of online, live, and print materials, combined with targeted preceptor development through site visits, improved preceptor performance.36 This approach, although logical, is resource-intensive. Combined with results suggesting EE workload and financial support may already be suboptimal, additional resources will be required to address this growing need in experiential education. Although the number of faculty members and staff in EE programs has grown over time,13 these results suggest that the current full-time equivalent allocation for EE may still not be enough to meet future requirements. That schools with an associate or assistant dean of experiential education reported fewer concerns about workload and financial support may indicate that individuals at this level are in a position to secure adequate resources for their EE departments. This trend would support recommendations made in 2005 that the ideal organization for experiential education include an associate or assistant dean as the department or program head.10 That schools without an assistant or associate director reported concerns related to workload would support that future recommendations include such a position in EE at all schools. Concern about EE workload and financial support seemed to be inversely related to class size. More than half of medium-sized schools cited this issue, while only 33% and 15% of large and very large schools mentioned it, respectively. This trend may indicate that economies of scale are reached at a class size of more than 100 students. On the other hand, issues related to quality assurance grew with class size, which suggests that large numbers of students and sites spread faculty members and staff too thin to thoroughly evaluate all preceptors/sites regularly. For instance, private schools perceived more concerns about capacity and faculty member/staff workload when they had more sites but had no more faculty members or staff to manage them. Lack of ability to evaluate preceptor or student performance could pose difficulties in assessing sites for minimum quality standards. Comparison of reported quality and capacity concerns with ratios of students to preceptors or ratios of students to sites could not be conducted because exact
When comparing results with number of practice sites claimed by each school, distinct differences became apparent. Those citing quality assurance as a concern reported more APPE (median 307) and IPPE (median 142) sites compared to those who did not (median 201 and 102, respectively). Responses expressing concerns about preceptor stipends reported higher numbers of APPE sites (median 380 versus 201). Contrary to expectation, those citing capacity as a concern reported higher numbers of APPE (median 535) and IPPE (median 150) sites as compared to those who did not mention this issue at all (median 212 and 100, respectively).
DISCUSSION While the overall goal response rate was not achieved, these results were sufficient to be generalized to public institutions, schools with medium to large class size, and institutions associated with academic health centers. Since public schools with medium to large class sizes represent 69% of the entire population,29 the results from this analysis can be generalized to many colleges and schools. Closer examination of the responses from underrepresented institutions did not reveal outlying differences. For example, the 1 small school respondent reported concerns consistent with other schools (ie, quality, adequate faculty members/staff in EE, and preceptor development). Many concerns identified in this analysis were consistent with concerns raised in 2001,22 even though analysis methods were chosen to minimize influence from investigators’ assumptions or previously published conclusions. Including novices who were naı¨ve about EE administrative functions and historical trends helped to mitigate inherent bias from the preconceived constructs that experienced EE faculty members brought to the analysis. One difference noted was that workload of faculty members/staff in EE was combined with funding/support for EE into the same theme, whereas it was reported separately in 2001.22 Upon further examination of responses, it seemed appropriate that these issues were combined because funding and financial support was usually mentioned in reference to lack of sufficient faculty members and staff to do necessary work as opposed to need for more funding in general. And yet, the consistency of these results over time confirms that these concerns continue to be “top of mind” for EE directors. One of the new concerns to emerge was the procedures required for placing students at practice sites, or onboarding, added significant paperwork and time constraints to EE staff when health systems began implementing new standards published by The Joint Commission 6
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. class size data were not collected. Further exploration of the potential connection between student/preceptor and student/ site ratios and concerns about quality should be conducted, especially since the new accreditation guidelines recommend ratios of 3:1 for IPPEs and 2:1 for APPEs.37 Many of our results are consistent with other published concerns. For instance, resources to support EE (eg, practice sites, financial support, personnel), preceptor development, and standardized assessment of EE— our second, fourth and sixth most frequently mentioned concerns—were listed in the top 20 recommendations for the draft standards by those attending the ACPE Conference on Advancing Quality in Pharmacy Education.14 Fifty-five percent of conference attendees claimed that competition with other schools (our most frequent response) was a problem, and 48% identified demands on EE staff and fiscal resources (our second most frequent response) as a problem.15 Such corroborating results help validate our analysis.17-18 The themes raised here are legitimate concerns that must be addressed as EE moves forward. One limitation to these results is that other questions in the survey could have impacted the responses to this question. For instance, this question contained the only free-text response and many others were related to quantitative report of the personnel employed in EE and number of sites used. Therefore, respondents may have been more likely to express concerns related to lack of the number of faculty members/staff or site capacity. Demographics of actual individuals responding to the survey were not collected and thus cannot be fully evaluated. However, the nature of the survey as a whole involved primarily reporting quantitative data about the EE program, so individual demographics would be expected to have little to no influence on responses. Finally, it is interesting that other issues such as integration of interprofessional education (IPE) and use of portfolios to demonstrate professional development, which have been highlighted in previous publications,38-39 were not mentioned at all in the responses. This difference could be related to one of the limitations of these results. The data was collected 3 years ago and, therefore, may not accurately reflect the current state of EE. Since the time of this survey, new educational outcomes were published by the Center for Advancement of Pharmacy Education (CAPE)40 and fully absorbed into the draft accreditation standards.1 The CAPE Outcomes emphasize the affective learning domain in addition to knowledge, skills, and abilities that have been traditionally assessed. The new accreditation standards require integration and assessment of IPE throughout the curriculum, use of validated assessment tools for APPEs, and recommend portfolios to
promote reflection about professional development. The themes identified in our results do not include consideration of the changes needed to incorporate these different assessment tools. The new standards also separate preceptor development and quality assurance of practice sites as entirely new standards. Even though preceptor development and quality assurance were identified as concerns, they might be even greater concerns if this survey were conducted now. Therefore, further investigation is needed about the challenges of implementing the new CAPE Outcomes and ACPE Accreditation Standards over time to determine their potential impact on EE.
CONCLUSION While site capacity continues to challenge pharmacy schools, concerns related to increased demands on EE faculty members/staff to provide preceptor development and to support onboarding procedures for students pose new problems. The consistency of these results with other evidence supports the fact that workloads of faculty members/staff, resources and financial support, and quality assurance in experiential education continue to challenge schools. Further investigation of these issues should be conducted so they are taken into account as new accreditation standards are implemented.
ACKNOWLEDGMENT The authors would like to recognize Lynne Robins, PhD, professor of biomedial informatics and medical education at the University of Washington School of Medicine, for her guidance and consultation on the qualitative methods involved in this project.
REFERENCES 1. Accreditation Council for Pharmacy Education. Accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree (Standards 2016). https:// www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed January 26, 2015. 2. Scheckelhff DJ, Bush CG, Flynn AA, et al. Capacity of hospitals to partner with academia to meet experiential education requirements for pharmacy students. Am J Pharm Educ. 2008;72(5):Article 117. 3. Brackett PD, Byrd DC, Duke LJ, et al. Barriers to expanding advanced pharmacy practice experience site availability in an experiential education consortium. Am J Pharm Educ. 2009;73(5): Article 82. 4. Danielson J, Ramirez J, Krueger J, et al. The capacity ratio as a measure of solvency in experiential education. Am J Pharm Educ. 2011;75(10):Article 198. 5. Haswell JL, Byrd DC, Foster S, Brown R. The use of capacity ratios in introductory pharmacy practice experiences. Am J Pharm Educ. 2012;76(9):Article 172. 6. McClellan HG, Byrd DC, Brown R. Capacity ratios to assess the solvency of a college’s advanced pharmacy practice experience program. Am J Pharm Educ. 2013;77(2):Article 28.
7
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. 25. Draugalis JR, Plaza CM. Best practices for survey research reports revisited: Implications of target population, probability sampling, and response rate. Am J Pharm Educ. 2009:73(8):Article 142. 26. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Thousand Oaks, CA: Sage Publications; 2002. 27. Anderson C. Presenting and evaluating qualitative research. Am J Pharm Educ. 2010;74(8):Article 141. 28. Viera AJ, Garrett JM. Understanding interobserver agreement: The kappa statistic. Fam Med. 2005;37(5):360-363. 29. Pharmacy Admissions Requirements Table 3. Institutional information and graduate degree programs. American Association of Colleges of Pharmacy. http://www.aacp.org/resources/research/ institutionalresearch/Documents/11_Enrollments.pdf Accessed January 26, 2015. 30. The Joint Commission. Human resource standards applicability to contracted and volunteer personnel. May 10, 2011. http://www. jointcommission.org/standards_information/jcfaqdetails.aspx? StandardsFAQId5344&StandardsFAQChapterId566. Accessed January 20, 2014. 31. Hatton RC, Weitzel KW. Complete-block scheduling for advanced pharmacy practice experiences. Am J Health Syst Pharm. 2013;70(23):2144-2151. 32. Mangino P, Adams T, Patterson B, et al. Enhancing pharmacy practice clerkship by use of block scheduling, patient case conferences, and self-taught modules. American Society of HealthSystem Pharmacists Midyear Clinical Meeting, Orlando, Florida, December 2004. 33. Lucarotti RL, Smith GB, Bugdalski-Stutrud C. Development of a triple-track approach to provision of advanced pharmacy practice experiences. American Association of Colleges of Pharmacy Annual Meeting, Boston, Massachusetts, July 2009. 34. Dennis VC, Britton ML, Wheeler RE, Carter SM. Practice experiences at a single institutional practice site to improve advanced pharmacy practice examination performance. Am J Pharm Educ. 2014;78(3):Article 60. 35. Taylor RZ, Wisneski SS, Kaun MA, Parteleno P, Williams J, Goldman MP. Sequential advanced pharmacy practice experinces at one institution for students from three pharmacy schools. Am J Health Syst Pharm. 2014;71(2):140-144. 36. Vos SS, Trewet CB. A comprehensive approach to preceptor development. Am J Pharm Educ. 2012;76(3):Article 47. 37. Accreditation Council for Pharmacy Education. Guidance for the accreditation standards and key elements for the professional program in pharmacy leading to the Doctor of Pharmacy degree. (Draft Guidance for Standards 2016). https://www.acpe-accredit.org/pdf/ GuidanceStandards2016DRAFTv60FIRSTRELEASEVERSION.pdf. Accessed May 3, 2014. 38. Skrabal MZ, Turner PD, Jones RM, Tilleman JA, Coover KL. Portfolio use and practices in US colleges and schools of pharmacy. Am J Pharm Educ. 2012:76(3):Article 46. 39. Jones KM, Blumenthal DK, Burke JM, et al. Interprofessional education in introductory pharmacy practice experiences at US Colleges and Schools of Pharmacy. Am J Pharm Educ. 2012;76(5): Article 80. 40. Medina MS, Plaza CM, Stowe CD, et al. Center for the Advancement of Pharmacy Education 2013 Educational Outcomes. Am J Pharm Educ. 2013;77(8);Article 162.
7. Accreditation Council for Pharmacy Education. Pre-accredited and Accredited Professional Programs of Colleges and Schools of Pharmacy. https://www.acpe-accredit.org/shared_info/ programsSecure.asp. Accessed December 23, 2013. 8. American Association of Colleges of Pharmacy. Fall 2011 Profile of Pharmacy Students: Enrollments. http://www.aacp.org/resources/ research/institutionalresearch/Pages/StudentApplications, EnrollmentsandDegreesConferred.aspx. Accessed December 23, 2013. 9. Lin AY, Altiere RJ, Harris WT, et al. Leadership: the nexus between challenge and opportunity: Reports of the 2002-03 Academic Affairs, Professional Affairs, and Research Graduate Affairs Committees. Am J Pharm Educ. 2003;67(3):Article S05. 10. Reynolds JR, Briceland LL, Carter JT, et al. Experiential education delivery—Ensuring success through support and development of the faculty and administrative team: Report of the 2004-2005 Professional Affairs Committee. Am J Pharm Educ. 2005;69(5):Article S9. 11. Plaza CM, Draugalis JR. Implications of advanced pharmacy practice experience placements: a 5-year update. Am J Pharm Educ. 2005;69(3):Article 45. 12. Baugh G, Ernthausen L, Young E. Experiential education workload and workforce survey. American Association of Colleges of Pharmacy Annual Meeting, Orlando, Florida, July 16, 2012. 13. Danielson J, Eccles D, Kwasnik A, Craddick K, Heinz AK, Harralson AR. Status of pharmacy practice experience education programs. Am J Pharm Educ. 2014;78(4):Article 72. 14. Beardsley RS, Zellmer WA, Vlasses PH. Charting accreditation’s future: recommendations for the next generation of accreditation standards for doctor of pharmacy education. Am J Pharm Educ. 2013;77(3):Article 45. 15. Beardsley RS, Zorek JA, Zellmer WA, Vlasses PH. Charting accreditation’s future: results of the pre-conference survey: ACPE invitational conference on advancing quality in pharmacy education. Am J Pharm Educ. 2013;77(3):Article 46. 16. Haase KK, Smythe MA, Orlando PL, Resman-Targoff BH, Smith LS. Ensuring quality experiential education. Pharmacotherapy 2008;28(12):1548-1551. 17. Skrabal MZ, Jones RM, Nemire RE. National survey of volunteer pharmacy preceptors. Am J Pharm Educ. 2008;72(5):Article 112. 18. Bond R, Godwin D, Thompson ME, Wittstrom K. Preceptor perceptions of the importance of experiential guidelines. Am J Pharm Educ. 2013;77(7):Article 144. 19. Scheckelhoff DJ, Bush CG, Flynn AA, et al. Capacity of hospitals to partner with academia to meet experiential education requirements for pharmacy students. Am J Health Syst Pharm. 2008;65(21):2045-2046. 20. Smith KM, Phelps PK, Mazur JE, May JR. Relationships between colleges of pharmacy and academic health centers. Am J Health Syst Pharm. 2008;65(18):1750-1754. 21. Traynor K. Experiential education requirements squeeze schools, rotation sites. Am J Health Syst Pharm. 2004;61(10):1537-1538. 22. Harralson AF. Financial, personnel, and curricular characteristics of advanced practice experience programs. Am J Pharm Educ. 2003;67(1):Article 17. 23. Fincham JE, Draugalis JR. The importance of survey research standards. Am J Pharm Educ. 2013:77(1):Article 4. 24. Meszaros K, Barnett MJ, Lenth RV, Knapp KK. Pharmacy school survey standards revisited. Am J Pharm Educ 2013;77(1):Article 3.
8
RESEARCH A Qualitative Analysis of Common Concerns about Challenges Facing Pharmacy Experiential Education Programs Jennifer Danielson, PharmD, MBA,a Karen Craddick, PharmD,b Dayl Eccles, PharmD,c Abigail Kwasnik, PharmD,d Teresa A. O’Sullivan, PharmDa a
University of Washington School of Pharmacy, Seattle, Washington Cedars Sinai Medical Center, Los Angeles, California c Rite Aid Pharmacy, Yelm, Washington d St. Joseph Hospital Franciscan Health-System, Tacoma, Washington b
Submitted February 19, 2014; accepted November 10, 2014; published February 17, 2015.
Objective. To qualitatively analyze free-text responses gathered as part of a previously published survey in order to systematically identify common concerns facing pharmacy experiential education (EE) programs. Methods. In 2011, EE directors at all 118 accredited pharmacy schools in the US were asked in a survey to describe the most pressing issues facing their programs. Investigators performed qualitative, thematic analysis of responses and compared results against demographic data (institution type, class size, number of practice sites, number and type of EE faculty member/staff). Expert and novice investigators identified common themes via an iterative process. To check validity, additional expert and novice reviewers independently coded responses. The Cohen kappa coefficient was calculated and showed good agreement between investigators and reviewers. Results. Seventy-eight responses were received (66% response rate) representing 75% of publicly funded institutions and 71% of schools with class sizes 51-150. Themes identified as common concerns were site capacity, workload/financial support, quality assurance, preceptor development, preceptor stipends, assessment, onboarding, and support/recognition from administration. Good agreement (mean percent agreement 93%, ƙ range50.59-0.92) was found between investigators and reviewers. Conclusion. Site capacity for student placements continues to be the foremost concern for many experiential education programs. New concerns about preceptor development and procedures for placing and orienting students at individual practice sites (ie, “onboarding”) have emerged and must be addressed as new accreditation standards are implemented. Keywords: experiential learning, pharmacy education, clinical clerkship, qualitative research, capacity building, advanced pharmacy practice experiences, introductory pharmacy practice experiences, preceptor development, quality assurance
student placement capacity created by dramatic growth in number of pharmacy schools and student enrollments in recent years.2-8 Even though many schools have expanded and reorganized their EE programs,9-13 further reorganization may be necessary to realign job responsibilities with the new emphasis on quality improvement, preceptor development, and interprofessional education.14,15 Now is a good time take stock of all concerns facing EE programs so that appropriate changes can be made to meet new accreditation standards. In 2008, Haase and colleagues reported expert recommendations for ensuring quality in experiential education in a white paper sponsored by the American College of Clinical Pharmacy.16 This report described insights
INTRODUCTION With the release of the American Council for Pharmacy Education’s (ACPE) draft accreditation standards for pharmacy education,1 which emphasize preceptor development and assessment in experiential education (EE), colleges and schools of pharmacy will likely be making changes to accommodate these standards. Faculty members and administrators in EE programs will now be faced with new challenges, in addition to existing difficulties in Corresponding Author: Jennifer Danielson, University of Washington School of Pharmacy, Box 357630, Office of Professional Pharmacy Education, South Campus Center, Suite 244, 1601 NE Columbia Road Seattle, WA 98195. Tel: 206-543-1924. Fax: 206-221-2689. E-mail: [email protected]
1
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. from practicing clinical pharmacists, many of whom were pharmacy faculty members and preceptors. Additional insights from pharmacy preceptors about EE were reported in 2008 and 2013.17,18 Concerns from hospital pharmacists regarding capacity for placements in the hospital setting have also been published in the past 10 years.19-21 However, few reports have been published about what faculty members and staff directly administering EE programs perceive as concerns.22 In 2011, Danielson et al disseminated a survey analyzing the curricular, personnel, and financial characteristics of EE programs to EE directors across the United States.13 As part of the survey, respondents were asked to describe the most pressing issues facing their EE programs. The results of this question were subsequently analyzed and are reported in this paper, so that additional insight from faculty members directly involved in administering EE programs could be shared publicly as many seek strategies for implementing new accreditation standards.
through iterative, repeated comparison. They then independently coded the responses using the standard themes with the possibility of 1 response being categorized into multiple themes. Agreement between the investigators’ independent coding was calculated first, then common coding for the discrepancies was determined by consensus. To check validity, additional reviewers (again, an EE expert faculty member and a novice student) independently coded the responses using the standard themes created by the primary investigators.27 Consistency with investigators’ coding was first compared using percent agreement. The Cohen kappa coefficient (ƙ) was then calculated between the investigators’ and reviewers’ coding results. A ƙ.0.70 was considered satisfactory agreement in this analysis.28 Final results were compared to results of a similar survey conducted in 200113 to determine how concerns have changed over time.
RESULTS Eighty-one colleges and schools responded to the survey (69% response rate), and 78 responses were received for the question of interest (66% response rate). Comparison of respondents with the known population at the time of the survey29 showed that the proportions of responses received for publicly funded institutions (75%) and schools with class size 51-150 (medium 72%, large 71%) were adequate to represent these types of pharmacy colleges and schools. Since so few small schools exist anymore, results from the 1 small school that responded were combined with results for medium-sized schools. Thirty-three respondents (42%) reported being associated with an academic health center, which matches national statistics.29 Eight common themes were identified as concerns (see Table 1). Good agreement was found between the investigators’ initial independent coding (mean 96%, range 86-100%) (Table 1). Mean percent agreement between investigators and reviewers for all themes was 93% (range 78-100%). Good agreement (ƙ$0.7) was found between investigators and the expert reviewer for all themes except for the last 3 (assessment, onboarding, and support/recognition from the dean). Good agreement (ƙ$0.7) was found between the investigators and the novice reviewer for all themes. (Table 2). Site capacity remained as the highest concern over the past 10 years (Table 2). While slight differences were made in categorization between 2001 and 2011, concerns related to financial support for EE, workloads of EE faculty members/staff, preceptor stipends, and support/ recognition by school administration were consistently reported at similar frequencies across years. New issues identified included preceptor development (18%) and
METHODS In 2011, EE directors at all 118 accredited schools of pharmacy in the United States were surveyed using a webbased questionnaire (Catalyst Web Tools, University of Washington Information Technology, Seattle, WA). As part of the 35-item questionnaire, respondents were asked the following open-ended question: “What are the 2 or 3 most pressing issues your professional experience program is facing today?”13 Free-text answers were collected from May to December 2011 and downloaded to Microsoft Excel for analysis. Other data collected included type of institution (public or private) association with academic health centers, number of introductory pharmacy practice experiences (IPPEs) and advanced pharmacy practice experiences (APPEs), number and position of EE faculty members/staff, and class size. Class sizes were categorized as small (#50 students), medium (51-100 students), large (101-150 students), and very large (.150 students). Identifying information about the individual completing the survey was not collected. Goal response rate was 70% (confidence 95%, margin of error 5%).23-25 The project was approved as exempt by the University of Washington Human Subjects Division. The investigation team included 2 faculty members (each with more than 10 years of experience in EE and 1 with training in qualitative methods) and 3 fourth-year doctor of pharmacy students. Investigators performed content analysis of the free-text responses using thematic analysis methodology informed by grounded theory.26 First, the primary investigators, an EE expert (faculty member) and novice (student), identified standard themes 2
3
99
96
Quality Assurance
86
APPEs
Workload/Funding
91
IPPEs
100
99
Competition
Increasing Class Size
95
Investigator Agreement (%)
Site Capacity
Theme
Time, money/salary, personnel/manpower, burnout/turnover, or workload of faculty member/staff in EE departments mentioned. Issues such as having inadequate support staff or faculty member time to manage/coordinate the EE program or to provide teaching/scholarship in addition to program administration, strategic planning, or site visits/quality assurance are put into this category. Also included is funding in general or funding for the EE program. Does not include when someone specifically states funding is needed to pay preceptors or for preceptor stipends. Quality (quality assurance or improvement, or QA/AI) mentioned. This could be quality of sites/preceptors themselves or QA/QI responsibilities or activities that EE faculty members/staff have. Mention of improving standards or assuring consistency between sites/preceptors would also be included in this category.
Capacity issues concerning IPPE requirements or site recruitment for IPPEs is mentioned. Capacity issues for APPEs mentioned.
Capacity issues attributed to increasing enrollments or large class size either at the respondent’s school or at other schools.
Any mention at all of sites and the ability to find or maintain enough sites/preceptors (in amount or type) to support the program. Capacity issues are attributed to competition with other schools or the opening of new schools. Many of these responses actually use the word “competition.”
Description
Table 1. Common Themes Identified with Descriptions and Example Responses
(Continued)
Several responses mention quality control or assurance by name. “Quality assurance for a large number of sites” “Limited number of quality clinical programs available at acute care institutions” “Quality control between sites”
Several responses mention the acute care (or acute care/ gen med) and ambulatory care APPEs by name. “Adequate staff to support the Office of Experiential Education considering the percentage (30%) of the curriculum made up of experiential education.” “Low morale due to under-staffed condition” “Funding for more office/faculty support”
“Saturation of the market with multiple pharmacy schools” “Number of new schools of pharmacy and/or existing schools with expanding enrollments canabalizing preceptors and practice sites.” “Competition/availability of enough quality sites in all needed content areas” “Adequate capacity for institutional IPPE sites/ preceptors to cover our increasing class size” “How do we maintain quality in experiential education with the recent increase in schools and the “established” schools increasing their class sizes?” Several responses mention institutional IPPEs by name.
Examples
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06.
4 Refers to a lack of appreciation on the part of administration for type and the amount of work EE and its faculty members/ staff do. May also include mention that EE programs are not called out as their own recognized unit or do not have representation on the appropriate committees or stakeholder groups to be involved in decisions affecting it. This category does not refer to funding support (that would be under Workload/Manpower).
99
97
Onboarding Students
Support/ Recognition from Administration
97
100
Refers to the need to provide training or development to preceptors in general or as ACPE requires, as well as to increase clinical skills or raise standards at sites so as to have appropriate learning opportunities for students. Many mention “preceptor development” or “adjunct faculty training” by name. Refers to paying sites/preceptors, preceptor stipends (paid by their own school or others), or inadequate funding to pay preceptors/sites in general. This category does not refer to inadequate funding of the EE office/faculty members/staff itself. Mentions general “assessment” or specific assessment of students or collection of assessment data in some way. The response could refer to assessment of student performance or could mention general program assessment. Most of these responses include the word “assessment” or “assessing.” Refers to managing compliance requirements, paperwork/ documentation, onboarding or other prerotation check-offs needed to place students at sites.
Description
97
Investigator Agreement (%)
Assessment
Preceptor Stipends
Preceptor Development
Theme
Table 1. (Continued )
“Increasing compliance requirements for students in practice sites and trying to meet these with no increases in staff.” “Increased complexity of institutional placements (background, drug testing, training requirements).” “Increasing amount of overhead (from sites, accreditation, affiliation agreements, etc.) that we are required to do now that keep us from doing our ‘real’ work” “Recognition by Administration of the scope and responsibility of experiential education” “Having administration acknowledge the importance of experiential education” “...not treated as members of the Administrative Team and, therefore, do not have a voice in the direction of the program.”
“Standardizing assessment tools for ease in preceptor use so that we can maximize sites.” “Assessment of student learning” “Maintaining quality of assessment”
“Resources to pay for the work we expect from preceptors” “Paying for sites.” “Sites/Preceptors asking to be paid”
“Implementing and training adjunct faculty members on how to assess students’ outcomes from their experiences.” “Coordination of education for preceptor faculty” “Advancing our clinical sites”
Examples
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06.
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. Table 2. Agreement and Comparison of Themes Over Time 2001 Survey17 (n=67) Theme (ranked by frequency) 1. Finding/maintaining sites(73%) 2. Assessment/QA (34%) 3. Adequate financial/staff support (28%) 4. Faculty member workload(18%) 5. Payments to sites/preceptors (15%) 6. Desire for better understanding by dean (7%) 7. Dealing with student problems n(4%)
2011 Survey (n=78) Theme (ranked by frequency)
Investigators vs Expert Reviewer kappa (ƙ)
Investigators vs Novice Reviewer kappa (ƙ)
1. Site capacity, 58(73%)
0.80
0.87
2. Workload/funding for EE, 33 (42%) 3. Quality assurance, 27 (34%)
0.76 0.80
0.87 0.77
4. Preceptor development, 14 (18%)
0.70
0.79
5. Preceptor stipends, 10 (14%)
0.89
1
6. Assessment, 10(14%)
0.64
0.84
7. Onboarding students, 9 (11%)
0.59
0.79
8. Support/recognition from administration, 6 (8%)
0.65
0.92
procedures for placing and orienting students at individual practice sites—a process referred to as “onboarding” by many health systems (11%). Dealing with student problems was not mentioned at all in 2011. Private institutions reported concerns about capacity and faculty member/staff workload more often than public institutions (Table 3), even though the former reported having more APPE and IPPE sites than public schools. No difference was found in number of EE faculty member and staff between public and private schools.13 Large and very large schools seemed more concerned about preceptor development and stipends than medium schools, and very large schools were most concerned about quality assurance compared to others. The number of full-time equivalents and type of EE faculty member/staff seemed to be related to select concerns.
Schools with fewer full-time equivalents mentioned workload and financial support more often than schools that employ more faculty members and staff. Schools without associate or assistant dean positions mentioned concerns related to workload and financial support as well as recognition from administration more often than schools with these positions. Schools without an associate or assistant director reported workload/financial support as a concern more often than schools with such positions. Concerns related to site capacity were further divided into issues related to competition with other schools, increased class size, and difficulties with placing IPPEs or APPEs specifically (Table 1). Capacity difficulties related to competition with other schools were reported more often (40%) than difficulties associated with increased class size (12%), IPPEs (31%), or APPEs (26%).
Table 3. Themes by Type and Size of Institution Theme (ranked by frequency) 1. Site capacity 2.Workload/funding for EE 3. Quality assurance 4. Preceptor development 5. Preceptor stipends 6. Assessment 7. Onboarding students 8. Support/recognition from administration
Public, n=46(%) 32 22 15 8 6 5 5 4
(68) (23) (32) (17) (13) (11) (11) (9)
Private, n=31(%)
Medium (#100 students) n=35 (%)
26 (81) 11 (34) 12 (38) 6 (19) 4 (13) 5 (16) 4 (13) 2 (6)
26 (76) 20 (57) 12 (38) 4 (11) 1 (3) 4 (12) 5 (15) 4 (12)
5
Large (101-150 students) n=24 (%) 16 8 6 6 5 3 3 2
(67) (33) (25) (25) (21) (13) (13) (8)
Very Large (.150 students) n=18 (%) 14 (74) 3 (15) 8 (42) 4 (22) 4 (22) 3 (17) 1 (6) 0
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. (TJC) in 2011.30 Some studies proposed streamlining placement procedures so students complete many of their APPEs continuously within 1 system, which reduces paperwork and improves learning outcomes.20,31-35 These results would support use of block scheduling as a way to create efficiency. The results also suggest an important challenge for EE programs is preceptor development, explicitly required by the 2016 draft standards.1 Vos and Trewet suggested that a multifactorial approach of online, live, and print materials, combined with targeted preceptor development through site visits, improved preceptor performance.36 This approach, although logical, is resource-intensive. Combined with results suggesting EE workload and financial support may already be suboptimal, additional resources will be required to address this growing need in experiential education. Although the number of faculty members and staff in EE programs has grown over time,13 these results suggest that the current full-time equivalent allocation for EE may still not be enough to meet future requirements. That schools with an associate or assistant dean of experiential education reported fewer concerns about workload and financial support may indicate that individuals at this level are in a position to secure adequate resources for their EE departments. This trend would support recommendations made in 2005 that the ideal organization for experiential education include an associate or assistant dean as the department or program head.10 That schools without an assistant or associate director reported concerns related to workload would support that future recommendations include such a position in EE at all schools. Concern about EE workload and financial support seemed to be inversely related to class size. More than half of medium-sized schools cited this issue, while only 33% and 15% of large and very large schools mentioned it, respectively. This trend may indicate that economies of scale are reached at a class size of more than 100 students. On the other hand, issues related to quality assurance grew with class size, which suggests that large numbers of students and sites spread faculty members and staff too thin to thoroughly evaluate all preceptors/sites regularly. For instance, private schools perceived more concerns about capacity and faculty member/staff workload when they had more sites but had no more faculty members or staff to manage them. Lack of ability to evaluate preceptor or student performance could pose difficulties in assessing sites for minimum quality standards. Comparison of reported quality and capacity concerns with ratios of students to preceptors or ratios of students to sites could not be conducted because exact
When comparing results with number of practice sites claimed by each school, distinct differences became apparent. Those citing quality assurance as a concern reported more APPE (median 307) and IPPE (median 142) sites compared to those who did not (median 201 and 102, respectively). Responses expressing concerns about preceptor stipends reported higher numbers of APPE sites (median 380 versus 201). Contrary to expectation, those citing capacity as a concern reported higher numbers of APPE (median 535) and IPPE (median 150) sites as compared to those who did not mention this issue at all (median 212 and 100, respectively).
DISCUSSION While the overall goal response rate was not achieved, these results were sufficient to be generalized to public institutions, schools with medium to large class size, and institutions associated with academic health centers. Since public schools with medium to large class sizes represent 69% of the entire population,29 the results from this analysis can be generalized to many colleges and schools. Closer examination of the responses from underrepresented institutions did not reveal outlying differences. For example, the 1 small school respondent reported concerns consistent with other schools (ie, quality, adequate faculty members/staff in EE, and preceptor development). Many concerns identified in this analysis were consistent with concerns raised in 2001,22 even though analysis methods were chosen to minimize influence from investigators’ assumptions or previously published conclusions. Including novices who were naı¨ve about EE administrative functions and historical trends helped to mitigate inherent bias from the preconceived constructs that experienced EE faculty members brought to the analysis. One difference noted was that workload of faculty members/staff in EE was combined with funding/support for EE into the same theme, whereas it was reported separately in 2001.22 Upon further examination of responses, it seemed appropriate that these issues were combined because funding and financial support was usually mentioned in reference to lack of sufficient faculty members and staff to do necessary work as opposed to need for more funding in general. And yet, the consistency of these results over time confirms that these concerns continue to be “top of mind” for EE directors. One of the new concerns to emerge was the procedures required for placing students at practice sites, or onboarding, added significant paperwork and time constraints to EE staff when health systems began implementing new standards published by The Joint Commission 6
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. class size data were not collected. Further exploration of the potential connection between student/preceptor and student/ site ratios and concerns about quality should be conducted, especially since the new accreditation guidelines recommend ratios of 3:1 for IPPEs and 2:1 for APPEs.37 Many of our results are consistent with other published concerns. For instance, resources to support EE (eg, practice sites, financial support, personnel), preceptor development, and standardized assessment of EE— our second, fourth and sixth most frequently mentioned concerns—were listed in the top 20 recommendations for the draft standards by those attending the ACPE Conference on Advancing Quality in Pharmacy Education.14 Fifty-five percent of conference attendees claimed that competition with other schools (our most frequent response) was a problem, and 48% identified demands on EE staff and fiscal resources (our second most frequent response) as a problem.15 Such corroborating results help validate our analysis.17-18 The themes raised here are legitimate concerns that must be addressed as EE moves forward. One limitation to these results is that other questions in the survey could have impacted the responses to this question. For instance, this question contained the only free-text response and many others were related to quantitative report of the personnel employed in EE and number of sites used. Therefore, respondents may have been more likely to express concerns related to lack of the number of faculty members/staff or site capacity. Demographics of actual individuals responding to the survey were not collected and thus cannot be fully evaluated. However, the nature of the survey as a whole involved primarily reporting quantitative data about the EE program, so individual demographics would be expected to have little to no influence on responses. Finally, it is interesting that other issues such as integration of interprofessional education (IPE) and use of portfolios to demonstrate professional development, which have been highlighted in previous publications,38-39 were not mentioned at all in the responses. This difference could be related to one of the limitations of these results. The data was collected 3 years ago and, therefore, may not accurately reflect the current state of EE. Since the time of this survey, new educational outcomes were published by the Center for Advancement of Pharmacy Education (CAPE)40 and fully absorbed into the draft accreditation standards.1 The CAPE Outcomes emphasize the affective learning domain in addition to knowledge, skills, and abilities that have been traditionally assessed. The new accreditation standards require integration and assessment of IPE throughout the curriculum, use of validated assessment tools for APPEs, and recommend portfolios to
promote reflection about professional development. The themes identified in our results do not include consideration of the changes needed to incorporate these different assessment tools. The new standards also separate preceptor development and quality assurance of practice sites as entirely new standards. Even though preceptor development and quality assurance were identified as concerns, they might be even greater concerns if this survey were conducted now. Therefore, further investigation is needed about the challenges of implementing the new CAPE Outcomes and ACPE Accreditation Standards over time to determine their potential impact on EE.
CONCLUSION While site capacity continues to challenge pharmacy schools, concerns related to increased demands on EE faculty members/staff to provide preceptor development and to support onboarding procedures for students pose new problems. The consistency of these results with other evidence supports the fact that workloads of faculty members/staff, resources and financial support, and quality assurance in experiential education continue to challenge schools. Further investigation of these issues should be conducted so they are taken into account as new accreditation standards are implemented.
ACKNOWLEDGMENT The authors would like to recognize Lynne Robins, PhD, professor of biomedial informatics and medical education at the University of Washington School of Medicine, for her guidance and consultation on the qualitative methods involved in this project.
REFERENCES 1. Accreditation Council for Pharmacy Education. Accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree (Standards 2016). https:// www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed January 26, 2015. 2. Scheckelhff DJ, Bush CG, Flynn AA, et al. Capacity of hospitals to partner with academia to meet experiential education requirements for pharmacy students. Am J Pharm Educ. 2008;72(5):Article 117. 3. Brackett PD, Byrd DC, Duke LJ, et al. Barriers to expanding advanced pharmacy practice experience site availability in an experiential education consortium. Am J Pharm Educ. 2009;73(5): Article 82. 4. Danielson J, Ramirez J, Krueger J, et al. The capacity ratio as a measure of solvency in experiential education. Am J Pharm Educ. 2011;75(10):Article 198. 5. Haswell JL, Byrd DC, Foster S, Brown R. The use of capacity ratios in introductory pharmacy practice experiences. Am J Pharm Educ. 2012;76(9):Article 172. 6. McClellan HG, Byrd DC, Brown R. Capacity ratios to assess the solvency of a college’s advanced pharmacy practice experience program. Am J Pharm Educ. 2013;77(2):Article 28.
7
American Journal of Pharmaceutical Education 2015; 79 (1) Article 06. 25. Draugalis JR, Plaza CM. Best practices for survey research reports revisited: Implications of target population, probability sampling, and response rate. Am J Pharm Educ. 2009:73(8):Article 142. 26. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Thousand Oaks, CA: Sage Publications; 2002. 27. Anderson C. Presenting and evaluating qualitative research. Am J Pharm Educ. 2010;74(8):Article 141. 28. Viera AJ, Garrett JM. Understanding interobserver agreement: The kappa statistic. Fam Med. 2005;37(5):360-363. 29. Pharmacy Admissions Requirements Table 3. Institutional information and graduate degree programs. American Association of Colleges of Pharmacy. http://www.aacp.org/resources/research/ institutionalresearch/Documents/11_Enrollments.pdf Accessed January 26, 2015. 30. The Joint Commission. Human resource standards applicability to contracted and volunteer personnel. May 10, 2011. http://www. jointcommission.org/standards_information/jcfaqdetails.aspx? StandardsFAQId5344&StandardsFAQChapterId566. Accessed January 20, 2014. 31. Hatton RC, Weitzel KW. Complete-block scheduling for advanced pharmacy practice experiences. Am J Health Syst Pharm. 2013;70(23):2144-2151. 32. Mangino P, Adams T, Patterson B, et al. Enhancing pharmacy practice clerkship by use of block scheduling, patient case conferences, and self-taught modules. American Society of HealthSystem Pharmacists Midyear Clinical Meeting, Orlando, Florida, December 2004. 33. Lucarotti RL, Smith GB, Bugdalski-Stutrud C. Development of a triple-track approach to provision of advanced pharmacy practice experiences. American Association of Colleges of Pharmacy Annual Meeting, Boston, Massachusetts, July 2009. 34. Dennis VC, Britton ML, Wheeler RE, Carter SM. Practice experiences at a single institutional practice site to improve advanced pharmacy practice examination performance. Am J Pharm Educ. 2014;78(3):Article 60. 35. Taylor RZ, Wisneski SS, Kaun MA, Parteleno P, Williams J, Goldman MP. Sequential advanced pharmacy practice experinces at one institution for students from three pharmacy schools. Am J Health Syst Pharm. 2014;71(2):140-144. 36. Vos SS, Trewet CB. A comprehensive approach to preceptor development. Am J Pharm Educ. 2012;76(3):Article 47. 37. Accreditation Council for Pharmacy Education. Guidance for the accreditation standards and key elements for the professional program in pharmacy leading to the Doctor of Pharmacy degree. (Draft Guidance for Standards 2016). https://www.acpe-accredit.org/pdf/ GuidanceStandards2016DRAFTv60FIRSTRELEASEVERSION.pdf. Accessed May 3, 2014. 38. Skrabal MZ, Turner PD, Jones RM, Tilleman JA, Coover KL. Portfolio use and practices in US colleges and schools of pharmacy. Am J Pharm Educ. 2012:76(3):Article 46. 39. Jones KM, Blumenthal DK, Burke JM, et al. Interprofessional education in introductory pharmacy practice experiences at US Colleges and Schools of Pharmacy. Am J Pharm Educ. 2012;76(5): Article 80. 40. Medina MS, Plaza CM, Stowe CD, et al. Center for the Advancement of Pharmacy Education 2013 Educational Outcomes. Am J Pharm Educ. 2013;77(8);Article 162.
7. Accreditation Council for Pharmacy Education. Pre-accredited and Accredited Professional Programs of Colleges and Schools of Pharmacy. https://www.acpe-accredit.org/shared_info/ programsSecure.asp. Accessed December 23, 2013. 8. American Association of Colleges of Pharmacy. Fall 2011 Profile of Pharmacy Students: Enrollments. http://www.aacp.org/resources/ research/institutionalresearch/Pages/StudentApplications, EnrollmentsandDegreesConferred.aspx. Accessed December 23, 2013. 9. Lin AY, Altiere RJ, Harris WT, et al. Leadership: the nexus between challenge and opportunity: Reports of the 2002-03 Academic Affairs, Professional Affairs, and Research Graduate Affairs Committees. Am J Pharm Educ. 2003;67(3):Article S05. 10. Reynolds JR, Briceland LL, Carter JT, et al. Experiential education delivery—Ensuring success through support and development of the faculty and administrative team: Report of the 2004-2005 Professional Affairs Committee. Am J Pharm Educ. 2005;69(5):Article S9. 11. Plaza CM, Draugalis JR. Implications of advanced pharmacy practice experience placements: a 5-year update. Am J Pharm Educ. 2005;69(3):Article 45. 12. Baugh G, Ernthausen L, Young E. Experiential education workload and workforce survey. American Association of Colleges of Pharmacy Annual Meeting, Orlando, Florida, July 16, 2012. 13. Danielson J, Eccles D, Kwasnik A, Craddick K, Heinz AK, Harralson AR. Status of pharmacy practice experience education programs. Am J Pharm Educ. 2014;78(4):Article 72. 14. Beardsley RS, Zellmer WA, Vlasses PH. Charting accreditation’s future: recommendations for the next generation of accreditation standards for doctor of pharmacy education. Am J Pharm Educ. 2013;77(3):Article 45. 15. Beardsley RS, Zorek JA, Zellmer WA, Vlasses PH. Charting accreditation’s future: results of the pre-conference survey: ACPE invitational conference on advancing quality in pharmacy education. Am J Pharm Educ. 2013;77(3):Article 46. 16. Haase KK, Smythe MA, Orlando PL, Resman-Targoff BH, Smith LS. Ensuring quality experiential education. Pharmacotherapy 2008;28(12):1548-1551. 17. Skrabal MZ, Jones RM, Nemire RE. National survey of volunteer pharmacy preceptors. Am J Pharm Educ. 2008;72(5):Article 112. 18. Bond R, Godwin D, Thompson ME, Wittstrom K. Preceptor perceptions of the importance of experiential guidelines. Am J Pharm Educ. 2013;77(7):Article 144. 19. Scheckelhoff DJ, Bush CG, Flynn AA, et al. Capacity of hospitals to partner with academia to meet experiential education requirements for pharmacy students. Am J Health Syst Pharm. 2008;65(21):2045-2046. 20. Smith KM, Phelps PK, Mazur JE, May JR. Relationships between colleges of pharmacy and academic health centers. Am J Health Syst Pharm. 2008;65(18):1750-1754. 21. Traynor K. Experiential education requirements squeeze schools, rotation sites. Am J Health Syst Pharm. 2004;61(10):1537-1538. 22. Harralson AF. Financial, personnel, and curricular characteristics of advanced practice experience programs. Am J Pharm Educ. 2003;67(1):Article 17. 23. Fincham JE, Draugalis JR. The importance of survey research standards. Am J Pharm Educ. 2013:77(1):Article 4. 24. Meszaros K, Barnett MJ, Lenth RV, Knapp KK. Pharmacy school survey standards revisited. Am J Pharm Educ 2013;77(1):Article 3.
8
