Original Research
Commitment to Change and Assessment of Confidence: Tools to Inform the Design and Evaluation of Interprofessional Education
JESSICA A. EVANS, BS; PAUL E. MAZMANIAN, PHD; ALAN W. DOW, MD, MSHA; KELLY S. LOCKEMAN, PHD; VICTOR A. YANCHICK, PHD Introduction: This study examines use of the commitment-to-change model (CTC) and explores the role of confidence in evaluating change associated with participation in an interprofessional education (IPE) symposium. Participants included students, faculty, and practitioners in the health professions. Methods: Satisfaction with the symposium and levels of commitment and confidence in implementing a change were assessed with a post-questionnaire and a follow-up questionnaire distributed 60 days later. Participants who reported changed behavior were compared with those who did not make a change. Independent sample t-tests determined whether there were differences between groups in their average level of commitment and/or confidence immediately following the symposium and at follow-up. Results: At post-symposium, attendees were satisfied with content and format. Sixty-eight percent said they would make a change in profession related activities. At 60 days, 53% indicated they had implemented a change. In comparison to those who reported no change, those who made a change reported higher levels of commitment and higher levels of confidence. Logistic regression suggested that the combination of commitment and confidence did not predict implementation in this sample; however, confidence had a higher odds ratio for predicting success than did commitment. Discussion: Confidence should be studied further in relation to commitment as a predictor of behavioral change associated with participation in an IPE symposium. Evaluators and instructional designers should consider use of follow-up support activities to improve learners’ confidence and likelihood of successful behavior change in the workplace. Key Words: interprofessional education, continuing professional development, commitment to change, collaborative practice, confidence, evaluation-educational intervention
Introduction Disclosures: The authors declare no conflicting interests. Jessica Evans is supported by a grant from the Donald W. Reynolds Foundation and by the Phil R. Manning Award of the Society for Academic Continuing Medical Education. Paul Mazmanian is supported by a grant award (UL1TR000058) from the NIH and also the Donald W. Reynolds Interdisciplinary Partnership in Geriatric Education. Alan Dow is supported by the Josiah H. Macy Jr. Foundation and the Donald W. Reynolds Foundation. Kelly Lockeman is supported by a grant from the Donald W. Reynolds Foundation. Ms. Evans: Research Assistant for Assessment and Evaluation Studies, School of Medicine, Virginia Commonwealth University; Dr. Mazmanian: Associate Dean for Assessment and Evaluation Studies, School of Medicine, and Director of Evaluation for the VCU Center for Clinical and Translational Research, Virginia Commonwealth University; Dr. Dow: Assistant Vice President of Health Sciences for Interprofessional Education and Collaborative Care and Assistant Dean of Medical Education, Department of Internal Medicine, Virginia Commonwealth University; Dr. Lockeman: Assistant Director of Research and Evaluation, Center for Interprofessional Education and Collaborative Care, Virginia Commonwealth University; Dr. Yanchick: Dean for the School of Pharmacy, and Professor with the Department of Pharmaceutics, Virginia Commonwealth University.
Commitment to change (CTC) is a widely accepted approach to promoting change and to measuring the effectiveness of education in the health professions.1,2 In the CTC model, learners indicate whether they will make a change in their practice as a result of attending an educational event.2 CTC follows specific steps: (1) a question that solicits a statement of change, (2) a Likert scale to rate the learner’s commitment to the change, (3) a follow-up reminder of the intended
Correspondence: Jessica Evans, Box 980466, Richmond, VA 23298-0565; e-mail: [email protected]. © 2014 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education. • Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/chp.21246
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS, 34(3):155–163, 2014
Evans et al.
change, (4) a question asking whether the intended change was made, (5) questions regarding barriers to change or reasons for nonimplementation of intended changes, and (6) additional questions regarding what facilitated the change.1 By requesting that a learner commit to a change, binding that individual to a behavioral act,3 an educational planner provides the learner with a path to reduce dissonance between actual and desired performance.4 Commitment-to-change studies include theoretical foundations,2,4–6 program evaluation,7–10 and experimentation to test conditions for change.11–14 As a rule, these studies recommend further examination of CTC to elaborate the properties of commitment and to validate its role in promoting or measuring change in education and care. Shershneva et al1 suggest participants’ pre- and post-event self-assessments of their awareness, knowledge, communication skills, and confidence, as related to the subject change should provide insights about learners’ progress throughout a learning-to-change continuum.1 Cash et al found confidence a facilitator of knowledge acquisition and knowledge mastery among physicians.15 Whereas, Bandura studied confidence as self-efficacy, or task specific confidence.16 We hypothesized that confidence might add further information to the CTC model by accounting for a learner’s perception of his or her capacity to implement change in his or her practice environment. In this study, we examine the use of the CTC model and explore the role of confidence in evaluating change resulting from participating in an interprofessional education (IPE) symposium. Our project expands, elaborates, and refines knowledge explaining CTC, without abandoning the core features of the CTC model.
Methods The Educational Intervention Interprofessional education (IPE) occurs when two or more health and social care professions are interactively learning about, from, and with each other.17 Collaborative practice in health care occurs when multiple health workers from different professional backgrounds provide comprehensive services by working with patients, their families, caregivers, and communities to deliver the highest quality of care across settings.17 Rigorous research is required to better understand the impact of different approaches to IPE on increasing collaborative practice and on improving outcomes.18,19 The Emswiller Interprofessional Symposium was held March 9, 2013, in Richmond, Virginia. Primary instructional objectives for the Symposium were to: (1) recognize and describe how the changing environment of health care will increase the need for interprofessional practice, (2) discuss how to utilize new approaches to increase interprofessional col156
laboration and education to improve patient care, and (3) apply lessons from innovative programs in interprofessional education and collaborative practice within education and practice environments. Participants included health care professionals, faculty, and students with an interest in interprofessional education and practice. Sessions included large group presentations with questions and answers, panel discussions, a poster session, and less formal dialogue among attendees. Study Design Commitment to change was used to track the impact of the Symposium on attitudes and behavior by counting the number of learners who made commitments to change and determining the percentage of learners who, in follow-up, made changes consistent with their stated commitment. We administered a post-questionnaire at the conclusion of the educational activity and distributed a follow-up questionnaire approximately 60 days following the event. At the end of the Symposium, the post-questionnaire asked attendees to evaluate the utility of this activity on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). It also asked participants to rate the content of the Symposium overall, to indicate whether its objectives were realized, and to note the value of the group presentations, panel discussions, and poster sessions. In addition, the post-questionnaire asked participants to identify a change they would make in their practice or profession-related activities as a result of their participation in the educational activity. For this change, they were asked to indicate on a 5-point scale (1 = lowest, 5 = highest) their level of: (1) commitment to the change, and (2) confidence regarding successful implementation of the stated change (EXHIBIT 1). Approximately 60 days following the Symposium, an electronic follow-up questionnaire was emailed to respondents who provided a valid e-mail address. The questionnaire invitations included respondents’ change statements verbatim. This questionnaire asked respondents to indicate whether they were successful in implementing the change they specified in the post-questionnaire. In addition, it asked respondents to describe barriers they faced while attempting to implement the change. Respondents who indicated they had not implemented the change or who reported uncertainty regarding implementation were asked follow-up questions about their current level of commitment and confidence about future implementation. Those individuals who did not respond to the initial e-mail solicitation were reminded about the questionnaire by e-mail up to three times. Two investigators (JE and PM) categorized intended changes by the type of change (attitudinal or behavioral) and aligned the changes with one of the Symposium’s three primary instructional objectives. Categorizations had to be unanimous among the adjudicators. We then matched
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
EXHIBIT 1. Commitment to Change, Confidence, and Barriers Questions from the Symposium Evaluation
responses from post-questionnaire with responses to followup questionnaire. For purposes of data analysis, we grouped respondents who were uncertain about having made a change with those respondents who indicated that they did not make a change. Participants who reported having implemented a change were compared with those who did not make a change in attitudes or behavior. As parametric statistics are accepted for use with data from Likert scales, self-reported levels of commitment and confidence were treated as continuous measures for analysis.20 Independent sample t-tests determined whether there were differences between these groups in their average level of commitment and/or confidence levels immediately following the event and at follow-up. Logistic regression was applied to investigate participants’ levels of com-
mitment and confidence and their likelihood of implementing a specified change.
Results One hundred thirty-eight registrants representing 7 major health professions including additional specializations gathered to discuss improving interprofessional education and practice. Of the 138 registrants, 120 (87%) completed the post-questionnaire. Of the 138 registrants, 20 self-identified as students/trainees (14%), 66 (48%) self-identified as educators/academics, 46 (33%) self-identified as practitioners, and 5 (4%) said they were administrators.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
157
Evans et al.
FIGURE 1. Post-questionnaire Demographics
There were 120 respondents to the post-questionnaire, of which 81 (68%) provided data regarding their level of experience (see FIGURE 1). Twenty-four of these 81 respondents (30%) were in the latter stages of their career, having completed between 31 and 50 years in professional practice. Thirty-seven (46%) had between 11 and 30 years of practice experience. The remaining 20 respondents (25%) had acquired up to 10 years of experience. Ninety-four (79%) of the 120 post-questionnaire respondents were female. Out of the 120 attendees who completed the postquestionnaire, 95 (79%) provided contact information and answered the qualifying question: “As a result of participating in this Symposium, will you make changes in your practice or profession-related activities?” Of these 95 respondents, 65 (68%) reported that they would make a change in their practice- or profession-related activities; the remaining 30 respondents (32%) indicated that they would not make a change (see FIGURE 2). Sixty-two (95%) of the 65 attendees who indicated they would make a change specified a change they would make. These 62 respondents were eligible to partake in the follow-up portion of the study because they provided both a valid e-mail address and a change statement. Fifty-eight (93%) of the 62 respondents who specified a change also indicated a commitment level associated with implementation of this change, and 56 of these respondents (90%) reported a confidence level. Similarly, 56 of the 62 respondents who designated a change (90%) indicated both a commitment and confidence level for the change they stated they would make. TABLE 1 presents a sample of attendees’ responses to the follow-up statement: “If yes, please specify one change you will make in your practice or profession-related activities.” All 62 change statements aligned with at least 1 of
158
the Symposium’s instructional objectives. Objective 3 was associated with 58 total intended changes, 54 of which were behavioral and 4 of which were attitudinal (see TABLE 1). Objective 1 was associated with 4 intended changes, 1 attitudinal and 3 behavioral. Objective 2 related to instructional design during the Symposium and was not associated with any of the change statements. Overall, 57 of the 62 participants (92%) indicated that they would make a behavioral change, and the remaining 5 (8%) stated they would make an attitudinal change. Thirty-four (55%) of the 62 attendees who provided a valid e-mail and made a commitment to change completed the follow-up questionnaire. Out of this group, 18 (53%) indicated that they had implemented a change. When those respondents who were unsuccessful at implementing a change were asked, “What has prevented you from successful implementation of the specified change?” Five of the 15 respondents (33%) stated they lacked the time required to implement the change, 2 (13%) cited a lack of funds, and 2 (13%) indicated competing priorities. One person mentioned that others did not respond to their inquiries or requests to meet, another individual stated they lacked the necessary occupational standing within the university to implement a change, and 4 (27%) maintained that they were still in the planning phase. Commitment levels of those who later succeeded and those who failed to implement the change revealed no significant difference (mean = 4.79, N = 14 versus mean = 4.53, N = 15, respectively, p = .081; FIGURE 3). In contrast, those who succeeded in implementing the change reported higher levels of confidence immediately following the Symposium than those who did not implement a change (mean = 4.54, N = 13 versus mean = 3.87, N = 15 respectively,
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
FIGURE 2. CTC Evaluation Model: Tracking Attendees’ Behavioral Changes (See EXHIBIT 1 for question text) TABLE 1. Examples of Changes Symposium Attendees Indicated They Would Integrate Into Their Practice or Profession-Related Activities
Role
Behavioral change
Educator/Academic
●
Attitudinal change
Creation of elective courses in community based IPE/IPP
●
Think more strategically about IP education
Practitioner
●
Utilize staff meetings and safety hurdles to emphasize collaboration in care and to promote better communication among medical nursing team
●
Be optimistic in practice
Student
●
Participate in the student organization regarding interprofessional collaboration
●
Be on the lookout for even more opportunities to interact with students and faculty in other disciplines
Total N = 62; behavioral change (N) = 57; attitudinal change (N) = 5.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
159
Evans et al.
FIGURE 3. Comparison of Confidence for Change and Commitment to Change for Implementers Versus Nonimplementers at the Post-Symposium Time Point
p = .014; FIGURE 3). Furthermore, nonimplementers became less confident in their ability to make the stated change at follow-up in comparison to their level of confidence immediately after the Symposium (mean = 3.87, N = 15 versus mean = 3.20; N = 16, p = .004; FIGURE 4). Similarly, nonimplementers had less commitment to the change at followup (mean = 4.53, N = 15 after the Symposium versus mean = 3.67, N = 16 at follow-up, p = .002; FIGURE 4). Additionally, a chi-square test of goodness-of-fit was performed to determine whether gender influenced success in implementing a change. Respondents’ decision to make a change to their profession or practice related activities did not differ by gender (𝜒 2 (1, N = 34) = .551, p = .458). While levels of commitment and confidence following the Symposium correlated with Symposium satisfaction ratings (r = .30, p = .01 and r = .23, p = .05, respectively), Symposium satisfaction levels of those who implemented a change and those who did not implement a change revealed no significant difference (4.50 versus 4.56, respectively, p = .299). Each attendee’s commitment level was also highly correlated with his/her confidence level immediately after the Symposium (r = .49, p = .01). However, confi-
160
dence and commitment of nonimplementers were not significantly correlated at the follow-up time point (r = .31, p = .251). Logistic regression analysis was conducted to test whether commitment to change and confidence in successful implementation were effective at predicting an individual’s likelihood of implementing the change, after the Symposium. Negelkerke’s R2 for this regression model was .225, indicating that self-reported levels of commitment and confidence among this sample only accounted for about 23% of the variance in whether a participant implemented or did not implement an intended change. Neither predictor was statistically significant, but confidence (𝛽 = 1.21, OR = 3.36, p = .081) had a higher odds ratio for predicting success than commitment (𝛽 = –.11, OR = .90, p = .922). Discussion To expand the evidence base and theoretical foundations of CTC,1,2,4–13 we conducted independent sample t-tests to explore the influence of confidence and commitment in relation to behavioral change following the Symposium. Within
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
FIGURE 4. Comparison of Confidence for Change and Commitment to Change for Nonimplementers at the Post-Symposium Versus 60 Day Follow-up Time Points
the CTC model, levels of commitment following the Symposium were not related to attendees’ implementation behavior. However, participants reporting higher levels of confidence after the Symposium were more likely to implement a change than those with less confidence. These results suggest that confidence may be a better predictor of change than commitment in select education and practice settings. Furthermore, participants who were unsuccessful implementing change noted decreasing levels of commitment and of confidence with time, suggesting that individuals who do not implement a change may become discouraged by unsuccessful attempts to implement the practice changes inspired by IPE activities. Consistent with findings from prior studies,7,10 insufficient time was a frequently cited barrier in our study. Because goals must be acknowledged and fully understood to be realized,21 individuals who did not implement a change may not have accepted or fully comprehended the goal, thereby preventing its actualization. Follow-up support activities, particularly for participants who lack the confidence to succeed, may help clarify goals, solidify commitment, and translate CPD learning into changed attitudes and be-
havior. Possible enhancements to learning activities include crafting content with better information regarding how to change or how to lead change,22 assisting in the identification and utilization of resources,4 and helping to specify and implement strategies for increasing confidence to overcome barriers to change,10 including those found in our study: insufficient time, insufficient resources, competing priorities, lack of occupational standing, and nonresponsiveness from others. Potential strategies for increasing selfefficacy (task-specific confidence) could include those that require a learner to practice newly learned skills and model others.3
Limitations Like many studies in the social and behavioral sciences, a major limitation of our study is the small sample size. Thirty-five attendees (20%) did not complete the postquestionnaire, and attrition at follow-up further compounded the problem. Nevertheless, confidence had a higher odds ratio for predicting success than did commitment, suggesting
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
161
Evans et al.
that confidence should be studied further in relation to behavior change. To increase the response rate, future researchers might consider building incentives into the study design, although careful thought must be given to the question of whether response rate is being increased or the natural mechanisms of CTC are being influenced by an added incentive. Another limitation of our data is the potential for selfreport bias. For example, a lack of variance in self-reported satisfaction made it impossible to determine whether satisfaction was related to implementation of change, and some attendees may have exhibited attribution bias or social desirability bias by ascribing their lack of success in implementing a change to external forces, rather than to their own self-agency. These concerns are not uncommon to CTC. The gap between perceived and actual competence among learners requires continued study and further investigation to better understand the follow-up component of the current CTC model, and the possible value of confidence in learning activities designed to overcome barriers to change. Reconciling these disparities is a direction for future research to better define the concept of confidence, including any relationship to the idea of commitment and how these two concepts may be operationalized across different types of education and practice settings. Conclusion An interprofessional Symposium was associated with changes in practice that were more related to attendees’ confidence in change than in their level of commitment to change. Our results suggest that further investigation is important to define the role confidence plays in relation to commitment. This study provides an avenue for expanding our theoretical understanding of the CTC framework and it provides guidance for improved planning and evaluation of interprofessional education in CPD. Acknowledgments We are grateful to Moshe Feldman, PhD, Assistant Director of Research and Evaluation, Assistant Professor, Assessment and Evaluation Studies, School of Medicine; to Deborah DiazGranados, PhD, Program Evaluator, Center for Clinical and Translational Research, Assistant Professor, Assessment and Evaluation Studies, School of Medicine; and to Antoinette B. Coe, PharmD, Graduate Research Assistant, Assessment and Evaluation Studies, Graduate Student, School of Pharmacy, School of Medicine, for their review and advice on early drafts of the questionnaire. 162
Lessons for Practice ●
●
●
The commitment-to-change model can be used with assessments of confidence to inform the design and evaluation of interprofessional education for CPD. Confidence should be explored in conjunction with commitment as a predictor of successful implementation of behavioral change. Evaluators and instructional designers should consider use of follow-up support activities to improve learners’ confidence and likelihood of successful behavior change in the workplace.
References 1. Shershneva MB, Wang MF, Lindeman GC, Savoy JN, Olson CA. Commitment to practice change: an evaluator’s perspective. Eval Health Prof. 2010;33(3):256–275. 2. Overton GK, MacVicar R. Requesting a commitment to change: conditions that produce behavioral or attitudinal commitment. J Contin Educ Health Prof. 2008;28(2):60–66. 3. Locke EA, Latham GP, Erez M. The determinants of goal commitment. Acad Manage Rev. 1988;13(1):23–39. 4. Mazmanian PE, Mazmanian PM. Commitment to change: theoretical foundations, methods, and outcomes. J Contin Educ Health Prof. 1999;19(4):200–207. 5. Purkis IE. Commitment to change: an instrument for evaluating CME course. J Med Educ. 1982;57:61–63. 6. Mazmanian PE, Waugh JL, Mazmanian, PM. Commitment to change: ideational roots, empirical evidence, and ethical implications. J Contin Educ Health Prof. 1997;17(3):133–140. 7. Dolcourt DL. Commitment to change: a strategy for promoting educational effectiveness. J Contin Educ Health Prof. 2000;20(3):156–163. 8. Parker FW, Mazmanian PE. Commitments, learning contracts, and seminars in hospital-based CME: change in knowledge and behavior. J Contin Educ Health Prof. 1992;12(1):49–63. 9. White MI, Grzybowski S, Broudo M. Commitment to change instrument enhances program planning, implementation, and evaluation. J Contin Educ Health Prof. 2004;24(3):153–162. 10. Martin KO, Mazmanian PM. Anticipated and encountered barriers to change in CME: Tools for planning and evaluation. J Contin Educ Health Prof. 1991;11:301–318. 11. Mazmanian PE, Johnson RE, Zhang A, Boothby J, Yeatts EJ. Effects of a signature on rates of change: a randomized controlled trial involving continuing education and the commitment-to-change model. J Contin Educ Health Prof. 2001;76(6):642–646. 12. Wakefield JG. Commitment to change: exploring its role in changing physician behavior through continuing education. J Contin Educ Health Prof. 2004;24(4):197–204. 13. Domino FJ, Chopra S, Seligman M, Sullivan K, Quirk M. The impact on medical practice of commitments to change following CME
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
14.
15.
16. 17.
lectures: a randomized controlled trial. Med Teach. 2011;33:e495– e500. Pereles L, Lockyer J, Hogan D, Gondocz T. Effectiveness of commitment contracts in facilitating change in continuing medical education interventions. J Contin Educ Health Prof. 1997;17(1):27–31. Cash B, Mitchner NA, Ravyn D. Confidence-based learning CME: overcoming barriers in irritable bowel syndrome with constipation. J Contin Educ Health Prof. 2011;31(3):157–164. Bandura A. Self-Efficacy: The Exercise of Control. New York, NY: W. H. Freeman and Company; 1997. World Health Organization. A WHO report: framework for action on interprofessional education and collaborative practice. J Allied Health. 2010;39(Supplement 1):196–197.
18. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013;3: CD002213. 19. National Research Council. Using Science as Evidence in Public Policy. Washington, DC: National Academies Press; 2012. 20. Norman G. Likert scales, levels of measurement and the “laws” of statistics. Adv in Health Sci Educ. 2010;15:625–632. 21. Locke E, Latham G. New directions in goal-setting theory. Clin Psychol Sci. 2006;15(5):265–268. 22. Dow AW, DiazGranados DG, Mazmanian PE, Retchin SM. Applying organizational science to health care: a framework for collaborative practice. Acad Med. 2013;88(7):952–957.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
163
Commitment to Change and Assessment of Confidence: Tools to Inform the Design and Evaluation of Interprofessional Education
JESSICA A. EVANS, BS; PAUL E. MAZMANIAN, PHD; ALAN W. DOW, MD, MSHA; KELLY S. LOCKEMAN, PHD; VICTOR A. YANCHICK, PHD Introduction: This study examines use of the commitment-to-change model (CTC) and explores the role of confidence in evaluating change associated with participation in an interprofessional education (IPE) symposium. Participants included students, faculty, and practitioners in the health professions. Methods: Satisfaction with the symposium and levels of commitment and confidence in implementing a change were assessed with a post-questionnaire and a follow-up questionnaire distributed 60 days later. Participants who reported changed behavior were compared with those who did not make a change. Independent sample t-tests determined whether there were differences between groups in their average level of commitment and/or confidence immediately following the symposium and at follow-up. Results: At post-symposium, attendees were satisfied with content and format. Sixty-eight percent said they would make a change in profession related activities. At 60 days, 53% indicated they had implemented a change. In comparison to those who reported no change, those who made a change reported higher levels of commitment and higher levels of confidence. Logistic regression suggested that the combination of commitment and confidence did not predict implementation in this sample; however, confidence had a higher odds ratio for predicting success than did commitment. Discussion: Confidence should be studied further in relation to commitment as a predictor of behavioral change associated with participation in an IPE symposium. Evaluators and instructional designers should consider use of follow-up support activities to improve learners’ confidence and likelihood of successful behavior change in the workplace. Key Words: interprofessional education, continuing professional development, commitment to change, collaborative practice, confidence, evaluation-educational intervention
Introduction Disclosures: The authors declare no conflicting interests. Jessica Evans is supported by a grant from the Donald W. Reynolds Foundation and by the Phil R. Manning Award of the Society for Academic Continuing Medical Education. Paul Mazmanian is supported by a grant award (UL1TR000058) from the NIH and also the Donald W. Reynolds Interdisciplinary Partnership in Geriatric Education. Alan Dow is supported by the Josiah H. Macy Jr. Foundation and the Donald W. Reynolds Foundation. Kelly Lockeman is supported by a grant from the Donald W. Reynolds Foundation. Ms. Evans: Research Assistant for Assessment and Evaluation Studies, School of Medicine, Virginia Commonwealth University; Dr. Mazmanian: Associate Dean for Assessment and Evaluation Studies, School of Medicine, and Director of Evaluation for the VCU Center for Clinical and Translational Research, Virginia Commonwealth University; Dr. Dow: Assistant Vice President of Health Sciences for Interprofessional Education and Collaborative Care and Assistant Dean of Medical Education, Department of Internal Medicine, Virginia Commonwealth University; Dr. Lockeman: Assistant Director of Research and Evaluation, Center for Interprofessional Education and Collaborative Care, Virginia Commonwealth University; Dr. Yanchick: Dean for the School of Pharmacy, and Professor with the Department of Pharmaceutics, Virginia Commonwealth University.
Commitment to change (CTC) is a widely accepted approach to promoting change and to measuring the effectiveness of education in the health professions.1,2 In the CTC model, learners indicate whether they will make a change in their practice as a result of attending an educational event.2 CTC follows specific steps: (1) a question that solicits a statement of change, (2) a Likert scale to rate the learner’s commitment to the change, (3) a follow-up reminder of the intended
Correspondence: Jessica Evans, Box 980466, Richmond, VA 23298-0565; e-mail: [email protected]. © 2014 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education. • Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/chp.21246
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS, 34(3):155–163, 2014
Evans et al.
change, (4) a question asking whether the intended change was made, (5) questions regarding barriers to change or reasons for nonimplementation of intended changes, and (6) additional questions regarding what facilitated the change.1 By requesting that a learner commit to a change, binding that individual to a behavioral act,3 an educational planner provides the learner with a path to reduce dissonance between actual and desired performance.4 Commitment-to-change studies include theoretical foundations,2,4–6 program evaluation,7–10 and experimentation to test conditions for change.11–14 As a rule, these studies recommend further examination of CTC to elaborate the properties of commitment and to validate its role in promoting or measuring change in education and care. Shershneva et al1 suggest participants’ pre- and post-event self-assessments of their awareness, knowledge, communication skills, and confidence, as related to the subject change should provide insights about learners’ progress throughout a learning-to-change continuum.1 Cash et al found confidence a facilitator of knowledge acquisition and knowledge mastery among physicians.15 Whereas, Bandura studied confidence as self-efficacy, or task specific confidence.16 We hypothesized that confidence might add further information to the CTC model by accounting for a learner’s perception of his or her capacity to implement change in his or her practice environment. In this study, we examine the use of the CTC model and explore the role of confidence in evaluating change resulting from participating in an interprofessional education (IPE) symposium. Our project expands, elaborates, and refines knowledge explaining CTC, without abandoning the core features of the CTC model.
Methods The Educational Intervention Interprofessional education (IPE) occurs when two or more health and social care professions are interactively learning about, from, and with each other.17 Collaborative practice in health care occurs when multiple health workers from different professional backgrounds provide comprehensive services by working with patients, their families, caregivers, and communities to deliver the highest quality of care across settings.17 Rigorous research is required to better understand the impact of different approaches to IPE on increasing collaborative practice and on improving outcomes.18,19 The Emswiller Interprofessional Symposium was held March 9, 2013, in Richmond, Virginia. Primary instructional objectives for the Symposium were to: (1) recognize and describe how the changing environment of health care will increase the need for interprofessional practice, (2) discuss how to utilize new approaches to increase interprofessional col156
laboration and education to improve patient care, and (3) apply lessons from innovative programs in interprofessional education and collaborative practice within education and practice environments. Participants included health care professionals, faculty, and students with an interest in interprofessional education and practice. Sessions included large group presentations with questions and answers, panel discussions, a poster session, and less formal dialogue among attendees. Study Design Commitment to change was used to track the impact of the Symposium on attitudes and behavior by counting the number of learners who made commitments to change and determining the percentage of learners who, in follow-up, made changes consistent with their stated commitment. We administered a post-questionnaire at the conclusion of the educational activity and distributed a follow-up questionnaire approximately 60 days following the event. At the end of the Symposium, the post-questionnaire asked attendees to evaluate the utility of this activity on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). It also asked participants to rate the content of the Symposium overall, to indicate whether its objectives were realized, and to note the value of the group presentations, panel discussions, and poster sessions. In addition, the post-questionnaire asked participants to identify a change they would make in their practice or profession-related activities as a result of their participation in the educational activity. For this change, they were asked to indicate on a 5-point scale (1 = lowest, 5 = highest) their level of: (1) commitment to the change, and (2) confidence regarding successful implementation of the stated change (EXHIBIT 1). Approximately 60 days following the Symposium, an electronic follow-up questionnaire was emailed to respondents who provided a valid e-mail address. The questionnaire invitations included respondents’ change statements verbatim. This questionnaire asked respondents to indicate whether they were successful in implementing the change they specified in the post-questionnaire. In addition, it asked respondents to describe barriers they faced while attempting to implement the change. Respondents who indicated they had not implemented the change or who reported uncertainty regarding implementation were asked follow-up questions about their current level of commitment and confidence about future implementation. Those individuals who did not respond to the initial e-mail solicitation were reminded about the questionnaire by e-mail up to three times. Two investigators (JE and PM) categorized intended changes by the type of change (attitudinal or behavioral) and aligned the changes with one of the Symposium’s three primary instructional objectives. Categorizations had to be unanimous among the adjudicators. We then matched
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
EXHIBIT 1. Commitment to Change, Confidence, and Barriers Questions from the Symposium Evaluation
responses from post-questionnaire with responses to followup questionnaire. For purposes of data analysis, we grouped respondents who were uncertain about having made a change with those respondents who indicated that they did not make a change. Participants who reported having implemented a change were compared with those who did not make a change in attitudes or behavior. As parametric statistics are accepted for use with data from Likert scales, self-reported levels of commitment and confidence were treated as continuous measures for analysis.20 Independent sample t-tests determined whether there were differences between these groups in their average level of commitment and/or confidence levels immediately following the event and at follow-up. Logistic regression was applied to investigate participants’ levels of com-
mitment and confidence and their likelihood of implementing a specified change.
Results One hundred thirty-eight registrants representing 7 major health professions including additional specializations gathered to discuss improving interprofessional education and practice. Of the 138 registrants, 120 (87%) completed the post-questionnaire. Of the 138 registrants, 20 self-identified as students/trainees (14%), 66 (48%) self-identified as educators/academics, 46 (33%) self-identified as practitioners, and 5 (4%) said they were administrators.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
157
Evans et al.
FIGURE 1. Post-questionnaire Demographics
There were 120 respondents to the post-questionnaire, of which 81 (68%) provided data regarding their level of experience (see FIGURE 1). Twenty-four of these 81 respondents (30%) were in the latter stages of their career, having completed between 31 and 50 years in professional practice. Thirty-seven (46%) had between 11 and 30 years of practice experience. The remaining 20 respondents (25%) had acquired up to 10 years of experience. Ninety-four (79%) of the 120 post-questionnaire respondents were female. Out of the 120 attendees who completed the postquestionnaire, 95 (79%) provided contact information and answered the qualifying question: “As a result of participating in this Symposium, will you make changes in your practice or profession-related activities?” Of these 95 respondents, 65 (68%) reported that they would make a change in their practice- or profession-related activities; the remaining 30 respondents (32%) indicated that they would not make a change (see FIGURE 2). Sixty-two (95%) of the 65 attendees who indicated they would make a change specified a change they would make. These 62 respondents were eligible to partake in the follow-up portion of the study because they provided both a valid e-mail address and a change statement. Fifty-eight (93%) of the 62 respondents who specified a change also indicated a commitment level associated with implementation of this change, and 56 of these respondents (90%) reported a confidence level. Similarly, 56 of the 62 respondents who designated a change (90%) indicated both a commitment and confidence level for the change they stated they would make. TABLE 1 presents a sample of attendees’ responses to the follow-up statement: “If yes, please specify one change you will make in your practice or profession-related activities.” All 62 change statements aligned with at least 1 of
158
the Symposium’s instructional objectives. Objective 3 was associated with 58 total intended changes, 54 of which were behavioral and 4 of which were attitudinal (see TABLE 1). Objective 1 was associated with 4 intended changes, 1 attitudinal and 3 behavioral. Objective 2 related to instructional design during the Symposium and was not associated with any of the change statements. Overall, 57 of the 62 participants (92%) indicated that they would make a behavioral change, and the remaining 5 (8%) stated they would make an attitudinal change. Thirty-four (55%) of the 62 attendees who provided a valid e-mail and made a commitment to change completed the follow-up questionnaire. Out of this group, 18 (53%) indicated that they had implemented a change. When those respondents who were unsuccessful at implementing a change were asked, “What has prevented you from successful implementation of the specified change?” Five of the 15 respondents (33%) stated they lacked the time required to implement the change, 2 (13%) cited a lack of funds, and 2 (13%) indicated competing priorities. One person mentioned that others did not respond to their inquiries or requests to meet, another individual stated they lacked the necessary occupational standing within the university to implement a change, and 4 (27%) maintained that they were still in the planning phase. Commitment levels of those who later succeeded and those who failed to implement the change revealed no significant difference (mean = 4.79, N = 14 versus mean = 4.53, N = 15, respectively, p = .081; FIGURE 3). In contrast, those who succeeded in implementing the change reported higher levels of confidence immediately following the Symposium than those who did not implement a change (mean = 4.54, N = 13 versus mean = 3.87, N = 15 respectively,
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
FIGURE 2. CTC Evaluation Model: Tracking Attendees’ Behavioral Changes (See EXHIBIT 1 for question text) TABLE 1. Examples of Changes Symposium Attendees Indicated They Would Integrate Into Their Practice or Profession-Related Activities
Role
Behavioral change
Educator/Academic
●
Attitudinal change
Creation of elective courses in community based IPE/IPP
●
Think more strategically about IP education
Practitioner
●
Utilize staff meetings and safety hurdles to emphasize collaboration in care and to promote better communication among medical nursing team
●
Be optimistic in practice
Student
●
Participate in the student organization regarding interprofessional collaboration
●
Be on the lookout for even more opportunities to interact with students and faculty in other disciplines
Total N = 62; behavioral change (N) = 57; attitudinal change (N) = 5.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
159
Evans et al.
FIGURE 3. Comparison of Confidence for Change and Commitment to Change for Implementers Versus Nonimplementers at the Post-Symposium Time Point
p = .014; FIGURE 3). Furthermore, nonimplementers became less confident in their ability to make the stated change at follow-up in comparison to their level of confidence immediately after the Symposium (mean = 3.87, N = 15 versus mean = 3.20; N = 16, p = .004; FIGURE 4). Similarly, nonimplementers had less commitment to the change at followup (mean = 4.53, N = 15 after the Symposium versus mean = 3.67, N = 16 at follow-up, p = .002; FIGURE 4). Additionally, a chi-square test of goodness-of-fit was performed to determine whether gender influenced success in implementing a change. Respondents’ decision to make a change to their profession or practice related activities did not differ by gender (𝜒 2 (1, N = 34) = .551, p = .458). While levels of commitment and confidence following the Symposium correlated with Symposium satisfaction ratings (r = .30, p = .01 and r = .23, p = .05, respectively), Symposium satisfaction levels of those who implemented a change and those who did not implement a change revealed no significant difference (4.50 versus 4.56, respectively, p = .299). Each attendee’s commitment level was also highly correlated with his/her confidence level immediately after the Symposium (r = .49, p = .01). However, confi-
160
dence and commitment of nonimplementers were not significantly correlated at the follow-up time point (r = .31, p = .251). Logistic regression analysis was conducted to test whether commitment to change and confidence in successful implementation were effective at predicting an individual’s likelihood of implementing the change, after the Symposium. Negelkerke’s R2 for this regression model was .225, indicating that self-reported levels of commitment and confidence among this sample only accounted for about 23% of the variance in whether a participant implemented or did not implement an intended change. Neither predictor was statistically significant, but confidence (𝛽 = 1.21, OR = 3.36, p = .081) had a higher odds ratio for predicting success than commitment (𝛽 = –.11, OR = .90, p = .922). Discussion To expand the evidence base and theoretical foundations of CTC,1,2,4–13 we conducted independent sample t-tests to explore the influence of confidence and commitment in relation to behavioral change following the Symposium. Within
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
FIGURE 4. Comparison of Confidence for Change and Commitment to Change for Nonimplementers at the Post-Symposium Versus 60 Day Follow-up Time Points
the CTC model, levels of commitment following the Symposium were not related to attendees’ implementation behavior. However, participants reporting higher levels of confidence after the Symposium were more likely to implement a change than those with less confidence. These results suggest that confidence may be a better predictor of change than commitment in select education and practice settings. Furthermore, participants who were unsuccessful implementing change noted decreasing levels of commitment and of confidence with time, suggesting that individuals who do not implement a change may become discouraged by unsuccessful attempts to implement the practice changes inspired by IPE activities. Consistent with findings from prior studies,7,10 insufficient time was a frequently cited barrier in our study. Because goals must be acknowledged and fully understood to be realized,21 individuals who did not implement a change may not have accepted or fully comprehended the goal, thereby preventing its actualization. Follow-up support activities, particularly for participants who lack the confidence to succeed, may help clarify goals, solidify commitment, and translate CPD learning into changed attitudes and be-
havior. Possible enhancements to learning activities include crafting content with better information regarding how to change or how to lead change,22 assisting in the identification and utilization of resources,4 and helping to specify and implement strategies for increasing confidence to overcome barriers to change,10 including those found in our study: insufficient time, insufficient resources, competing priorities, lack of occupational standing, and nonresponsiveness from others. Potential strategies for increasing selfefficacy (task-specific confidence) could include those that require a learner to practice newly learned skills and model others.3
Limitations Like many studies in the social and behavioral sciences, a major limitation of our study is the small sample size. Thirty-five attendees (20%) did not complete the postquestionnaire, and attrition at follow-up further compounded the problem. Nevertheless, confidence had a higher odds ratio for predicting success than did commitment, suggesting
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
161
Evans et al.
that confidence should be studied further in relation to behavior change. To increase the response rate, future researchers might consider building incentives into the study design, although careful thought must be given to the question of whether response rate is being increased or the natural mechanisms of CTC are being influenced by an added incentive. Another limitation of our data is the potential for selfreport bias. For example, a lack of variance in self-reported satisfaction made it impossible to determine whether satisfaction was related to implementation of change, and some attendees may have exhibited attribution bias or social desirability bias by ascribing their lack of success in implementing a change to external forces, rather than to their own self-agency. These concerns are not uncommon to CTC. The gap between perceived and actual competence among learners requires continued study and further investigation to better understand the follow-up component of the current CTC model, and the possible value of confidence in learning activities designed to overcome barriers to change. Reconciling these disparities is a direction for future research to better define the concept of confidence, including any relationship to the idea of commitment and how these two concepts may be operationalized across different types of education and practice settings. Conclusion An interprofessional Symposium was associated with changes in practice that were more related to attendees’ confidence in change than in their level of commitment to change. Our results suggest that further investigation is important to define the role confidence plays in relation to commitment. This study provides an avenue for expanding our theoretical understanding of the CTC framework and it provides guidance for improved planning and evaluation of interprofessional education in CPD. Acknowledgments We are grateful to Moshe Feldman, PhD, Assistant Director of Research and Evaluation, Assistant Professor, Assessment and Evaluation Studies, School of Medicine; to Deborah DiazGranados, PhD, Program Evaluator, Center for Clinical and Translational Research, Assistant Professor, Assessment and Evaluation Studies, School of Medicine; and to Antoinette B. Coe, PharmD, Graduate Research Assistant, Assessment and Evaluation Studies, Graduate Student, School of Pharmacy, School of Medicine, for their review and advice on early drafts of the questionnaire. 162
Lessons for Practice ●
●
●
The commitment-to-change model can be used with assessments of confidence to inform the design and evaluation of interprofessional education for CPD. Confidence should be explored in conjunction with commitment as a predictor of successful implementation of behavioral change. Evaluators and instructional designers should consider use of follow-up support activities to improve learners’ confidence and likelihood of successful behavior change in the workplace.
References 1. Shershneva MB, Wang MF, Lindeman GC, Savoy JN, Olson CA. Commitment to practice change: an evaluator’s perspective. Eval Health Prof. 2010;33(3):256–275. 2. Overton GK, MacVicar R. Requesting a commitment to change: conditions that produce behavioral or attitudinal commitment. J Contin Educ Health Prof. 2008;28(2):60–66. 3. Locke EA, Latham GP, Erez M. The determinants of goal commitment. Acad Manage Rev. 1988;13(1):23–39. 4. Mazmanian PE, Mazmanian PM. Commitment to change: theoretical foundations, methods, and outcomes. J Contin Educ Health Prof. 1999;19(4):200–207. 5. Purkis IE. Commitment to change: an instrument for evaluating CME course. J Med Educ. 1982;57:61–63. 6. Mazmanian PE, Waugh JL, Mazmanian, PM. Commitment to change: ideational roots, empirical evidence, and ethical implications. J Contin Educ Health Prof. 1997;17(3):133–140. 7. Dolcourt DL. Commitment to change: a strategy for promoting educational effectiveness. J Contin Educ Health Prof. 2000;20(3):156–163. 8. Parker FW, Mazmanian PE. Commitments, learning contracts, and seminars in hospital-based CME: change in knowledge and behavior. J Contin Educ Health Prof. 1992;12(1):49–63. 9. White MI, Grzybowski S, Broudo M. Commitment to change instrument enhances program planning, implementation, and evaluation. J Contin Educ Health Prof. 2004;24(3):153–162. 10. Martin KO, Mazmanian PM. Anticipated and encountered barriers to change in CME: Tools for planning and evaluation. J Contin Educ Health Prof. 1991;11:301–318. 11. Mazmanian PE, Johnson RE, Zhang A, Boothby J, Yeatts EJ. Effects of a signature on rates of change: a randomized controlled trial involving continuing education and the commitment-to-change model. J Contin Educ Health Prof. 2001;76(6):642–646. 12. Wakefield JG. Commitment to change: exploring its role in changing physician behavior through continuing education. J Contin Educ Health Prof. 2004;24(4):197–204. 13. Domino FJ, Chopra S, Seligman M, Sullivan K, Quirk M. The impact on medical practice of commitments to change following CME
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
Commitment, Confidence, and Interprofessional CPD
14.
15.
16. 17.
lectures: a randomized controlled trial. Med Teach. 2011;33:e495– e500. Pereles L, Lockyer J, Hogan D, Gondocz T. Effectiveness of commitment contracts in facilitating change in continuing medical education interventions. J Contin Educ Health Prof. 1997;17(1):27–31. Cash B, Mitchner NA, Ravyn D. Confidence-based learning CME: overcoming barriers in irritable bowel syndrome with constipation. J Contin Educ Health Prof. 2011;31(3):157–164. Bandura A. Self-Efficacy: The Exercise of Control. New York, NY: W. H. Freeman and Company; 1997. World Health Organization. A WHO report: framework for action on interprofessional education and collaborative practice. J Allied Health. 2010;39(Supplement 1):196–197.
18. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013;3: CD002213. 19. National Research Council. Using Science as Evidence in Public Policy. Washington, DC: National Academies Press; 2012. 20. Norman G. Likert scales, levels of measurement and the “laws” of statistics. Adv in Health Sci Educ. 2010;15:625–632. 21. Locke E, Latham G. New directions in goal-setting theory. Clin Psychol Sci. 2006;15(5):265–268. 22. Dow AW, DiazGranados DG, Mazmanian PE, Retchin SM. Applying organizational science to health care: a framework for collaborative practice. Acad Med. 2013;88(7):952–957.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(3), 2014 DOI: 10.1002/chp
163
