Patient Education and Counseling 98 (2015) 753–761
Contents lists available at ScienceDirect
Patient Education and Counseling journal homepage: www.elsevier.com/locate/pateducou
Assessment
Using the patient centered observation form: Evaluation of an online training program Misbah Keen a,b, Jeanne Cawse-Lucas a, Jan Carline a,b, Larry Mauksch a,* a b
Department of Family Medicine, University of Washington, School of Medicine, Seattle, USA Department of BioMedical Informatics and Medical Education, University of Washington, School of Medicine, Seattle, USA
A R T I C L E I N F O
A B S T R A C T
Article history: Received 14 August 2014 Received in revised form 17 February 2015 Accepted 7 March 2015
Objective: The Patient Centered Observation Form (PCOF) helps trainees identify and describe specific communication skills and enhance self-awareness about skill use. We studied the effectiveness and ease of use of the Improving Communication Assessment Program (ICAP), an online module that prepares trainees to use the PCOF. Methods: Students, residents and medical educators viewed two videos (common and better skill use) of the same interaction and rated each video using the PCOF. Video sequence was randomized. We assessed agreement with experts, ease of use, concepts learned, and areas of confusion. Results: Trainees (211) achieved strong agreement (.83) with experts and were highly satisfied (mean 4.18 out of 5). Viewing the common video first produced higher agreement (.87 vs .79; ES = .4) with experts and greater satisfaction (4.36 vs 4.02, ES .4) than viewing the better video first. Trainees reported diverse areas of learning and minimal confusion. Conclusion: ICAP training to use the PCOF may facilitate teaching and assessment of communication skills and enrich training through peer observation and feedback. We offer several educational strategies. Practice implications: Learning to use the PCOF via the ICAP module may accelerate communication training for medical students, residents, medical educators and practicing clinicians. ß 2015 Elsevier Ireland Ltd. All rights reserved.
1. Introduction Effective communication is an essential physician skill to build trust, convey caring, and provide education and treatment [1–4]. Most medical schools teach and evaluate communication skills in the first and second years through didactic efforts, simulation, and limited practice with selected patients. However, medical schools are challenged to integrate communication skill training during the clinical phase of training [5]. Faculty often lack the time, training or language to train senior students or residents [6–8]. Lacking an articulate vocabulary means that educators struggle to reveal their tacit knowledge [8–10] and instead may rely solely on role modeling without explaining their behavior. A limited vocabulary leads to teachers and learners struggling to teach and articulate questions [7,8]. The Patient Centered Observation Form (PCOF) was designed to help observers and those being observed identify and describe
* Corresponding author at: University of Washington, School of Medicine, Department of Family Medicine, 6026 30th Ave NE, Seattle, WA 98115, USA. Tel.: +1 206 920 1245. E-mail addresses: [email protected], [email protected] (L. Mauksch). http://dx.doi.org/10.1016/j.pec.2015.03.005 0738-3991/ß 2015 Elsevier Ireland Ltd. All rights reserved.
specific skills, enhance self-awareness [11] and structure coaching during formative assessment. As such our definition of potential trainees includes medical students, residents, nurses, faculty physicians and other medical educators observing others during real patient encounters or on video. 1.1. Form development The process of creating the tool began in 2004. The first version of this tool used only narrative feedback about categories in the Kalamazoo consensus statement of essential communication skills [2]. In 2005 behavioral anchors were added corresponding to skills and categories described in the Kalamazoo consensus statement and in 13 communication assessment instruments [12]. The experts contributing to these sources consistently named the selected categories and skills. The tool has been revised several times since 2005 in response to feedback from users. The PCOF (Fig. 1 PCOF 2011 version) categorizes competence, and identifies skills, at the intersection of time management and quality of care [13]. Addressing trainee time management concerns is essential to reduce trainee anxiety that impedes learning [5,14]. To avoid promoting a grading mentality the PCOF does not use numerical
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
754
scoring but instead focuses on recording evidence of skill use using a checklist with space to add narrative commentary. Participants categorize the communication skills by checking off one or more
elements within each of thirteen PCOF categories. The PCOF requires users to count elements in a skill category to determine competence. Element sums within the range listed in the left-hand
Patient Centered Observation Form Trainee name______________________ Observer__________________Obsrvn#____Date__________ Directions; Track behaviors in left column. Then, mark one box per row: a, b or c. Competent skill use is in one of the right two columns. Record important provider / patient comments and verbal / non-verbal cues in the notes. Use form to enhance your learning, vocabulary, and self-awareness. Ratings can be for individual interviews or to summarize several interactions. If requested, use this form to guide verbal feedback to someone you observe.
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
.
Establishes Rapport Introduces self Warm greeting Acknowledges all in the room by name Uses eye contact Humor or non medical interaction
1a. Uses 0-2 elements
1b.Uses 3 elements.
1c.Uses ≥ 4 elements
2a. Uses 0-1 elements
2b. Uses 2 elements
2c. Uses 3 or more elements
3a. Uses 0-1 elements
3b. Uses 2 elements
3c. Uses ≥ 3 elements
4a. Uses 0 elements
4b. Uses 1 element
4c. Uses 2 or more elements
5a. Uses 0-1 elements
5b. Uses 2 elements
5c. Uses 3 or more elements
6b. Uses 1 element
6c. Uses 2 or more elements
Notes:
Maintains Relationship Throughout the Visit Strong verbal or non-verbal empathy Listens well using continuer phrases (“um hmm”) Repeats important verbal content; Demonstrates mindfulness through curiosity, selfreflection, or presence Notes:
Collaborative upfront agenda setting Additional elicitation- “something else?”each elicitation counts as a new element Acknowledges agenda items from other team member (eg MA) or from EMR. Confirms what is most important to patient? Note patient concerns here:
Maintains Efficiency through transparent (out loud) thinking: about visit time use / visit organization about problem priorities about problem solving Notes:
Gathering Information Uses open-ended question X____ Uses reflecting statement X____ Uses summary/clarifying statement X____ Count each time the skill is used as one element_ Notes:
.
Assessing Patient or Family Perspective on Health Acknowledges patient verbal or non-verbal cues. Explores patient beliefs or feelings Explores contextual influences: family, cultural spiritual. Number of patient verbal / non-verbal cues___
6a. Uses 0 elements
Notes:
© University of Washington Department of Family Medicine, May, 2011 Contact Larry Mauksch for further information Fig. 1.
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
755
Patient Centered Observation Form Trainee name______________________ Observer__________________Obsrvn#____Date__________
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
Electronic Medical Record Use Regularly describes use of EMR to patient Maintains eye contact with patient during majority of time while using EMR. Positions monitor to be viewed by patient Points to screen
7a. Uses 0 or 1 elements.
7b. Uses 2 elements
7c. Uses 3 or 4 elements
8a. Never
8b. some of the time, up to half the time
8c. Most of the time
9a. Uses 0-1 elements
9b. Uses 2 elements
9c. Uses 3 or more elements
10c. Uses 0-1 elements or lectures patient
10b. Uses 2-3 elements
10c. Uses 4 or more elements
11a. Use 0 elements
11b. Uses 1-2 elements
11c. Uses 3-4 elements.
12a. Use 0-1 element
12b. Uses 2 elements
13a. Uses 0-1 element
13b. Uses 2 elements
Notes:
Physical Exam Prepares patient before physical exam actions and describes exam findings during the exam (“I am going to ___ ” then “your lungs sound healthy”) Notes:
Sharing Information Avoids or explains medical jargon Summaries cover biomedical concerns Summaries cover psychosocial concerns. Invites Q/A Notes:
Behavior Change Discussions Explores pt knowledge about behaviors Explores pros and cons of behavior change Scales confidence or importance Asks permission to give advice Reflects or summarizes patient thoughts and feelings Creates a plan aligned with patient’s readiness Affirms behavior change effort or success Notes:
Co-creating a plan Informed Decision Making (when appropriate) Shares evidence(when available) behind recommendations Describes alternative options Examines pros and cons Describes uncertainties of viable options Shared Decision Making Plan respects patient’s biomedical goals / values. Plan respects patient’s psychosocial goals and values. Asks for patient preferences Asks for patient input and, if needed, modifies plan
12c. Uses
3 elements
Notes:
Closure and Follow-up Asks for questions about today’s topics. Prints After Visit Summary Uses Teachback. = Asking the patient to explain his/her understanding of the plan
13c. Uses 3 elements
Notes:
© University of Washington Department of Family Medicine, May, 2011 Contact Larry Mauksch for further information Fig. 1. (Continued ).
column denotes sub-competent performance while element counts in the ranges in either of the two right-hand columns denote competent performance, hence the thick vertical line separating the left hand column from the middle and right-hand
column. Not all encounters require element sums in the right hand column. Research [15,16] suggests that the training needed to master communication assessment tools requires several hours and is the
756
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
Table 1 PCOF elements. Establishing report [2] Maintaining relationship [24,25] Collaborative agenda setting [18] Maintaining efficiency [26] Gathering information [2] Assessing patient perspective [27–29] Electronic medical record use [30] Physical exam skills [31] Sharing information [2] Behavioral change discussion [32] Co-creating a plan [33] Closure and follow-up [34]
province of small numbers of highly trained faculty or researchers. However, the PCOF is designed for all trainees including a wide range of medical educators. Therefore we needed a training methodology that is efficient, effective and attractive. In 2007 the Improving Communication Assessment Program (ICAP) was created to provide online asynchronous PCOF training to help users at all developmental levels identify and name core communication skills (http://uwfamilymedicine.org/pcof). The Improving Communication Assessment Program has trainees use the PCOF to rate physician-patient interaction in two videos. One video demonstrates COMMON skill use characterized by the absence of selected skills listed in Table 1. The second, BETTER video, recreates the same interaction but with the physician demonstrating selected skills that were absent in the COMMON version. The trainee downloads a paper version of the PCOF, records observed skills, transfers ratings to ICAP and then submits these ratings. The next ICAP screen shows how experts rated the video1 with an explanation for each rating allowing the user to compare and learn. The training takes approximately 40 min. Using COMMON and BETTER videos helps make communication skills tangible by creating contrast and has been shown to be effective in prior educational trials [5,17–19]. The ICAP program is an integral component of the Patient Centered Communication Curriculum embedded within the required six-week, third year family medicine clerkship at the University of Washington School of Medicine (UWSOM). Faculty at one of 51 clerkship sites across our five state region use the ICAP to learn the PCOF as do all the clerkship students (approx. 230/year). Each week clerkship faculty and students are expected to observe one another using the PCOF in an outpatient setting and discuss communication skill use [5]. The ICAP program is also used to train other health care providers in various roles and at various levels of professional development including residents, medical educators, community physicians and nurses. Additionally, It is used for an advanced communication skills course that was disseminated to several medical schools [14]. While we have used consecutive versions of the ICAP for several years, we have not assessed its effectiveness or appeal to users. Therefore, we studied the following questions: 1. Is the Improving Communication Assessment Program effective in training users to use the PCOF? a. Is there adequate agreement between trainee and expert ratings? b. Is the training easy to understand and complete? 1 Two behavioral scientist faculties (Larry Mauksch, M.Ed and Valerie Ross, MS.) and two physician faculties (Tom Greer, MD., MPH and Fredrick Chen, MD, MPH) with expertise in communication rated the videos separately and then met to achieve consensus and agree on wording to describe reasoning for each rating.
c. Is there a correlation between ease of use and agreement with experts? 2. Does the order of viewing the ICAP videos make a significant difference in agreement with the experts and ease of use? 3. What are the most important concepts that trainees learn? 4. What concepts confuse trainees?
2. Methods ICAP programing randomized users to view either the COMMON or BETTER videos first. All responses were anonymous. Prior to viewing videos participants were asked to enter (through typing) age, gender and professional role. After rating both videos participants were asked to describe the ‘‘most important concept/skill learned’’ and concepts ‘‘that were confusing’’, and rate the ease of using the ICAP training from 1 (extremely difficult to understand and complete) to 5 (very easy to understand and complete). Analysis of narrative responses used a grounded theoretical approach [20]. Grounded theory is a qualitative research process in which the definitions of codes are refined throughout an iterative analytic process. Three authors (MK, LM and JCL) developed a coding scheme by separately working through respondent answers until we reached saturation. Then we compared codes to reach consensus on a final coding scheme. Most of the codes directly corresponded to the skills on PCOF and additional codes were added for answers that did not directly relate to the skills. Over 80% agreement was obtained on initial coding. The authors then met to reach consensus on items with coding discrepancies. All numerical items including the frequencies for codes were analyzed using SPSS version 19, IBM Inc. Simple descriptive statistics were calculated for all variables, and comparisons were made using T-tests and One Way Analyses of Variance. The main focus of the analyses was the agreement of participant codes with those provided by the experts and the potential effects of video order on agreement. For analytic purposes element scores were collapsed into dichotomous categories of the left hand column or the right two columns. A score of 0 was given to an element if the participant rating did not agree with the expert, and 1 if the participant rating agreed with the expert. The total score for each video ranged from 0 to 13. Additional analyses focused on the relationship of participant professional role with agreement with expert ratings, and qualitative measures of learning value in terms of concepts perceived to have been understood or still considered to be confusing by the participants. 3. Results A total of 211 trainees completed the ICAP training between October 2011 and June 2012. Sixty-five percent were female. Most users (64.5%) were below 30 years of age. The majority were medical students (72%), 8% were Nurses, 4.7% Behavioral Scientists, 4.3% faculty physicians, 1.9% Medical Residents and 7.6% Other. No statistically significant differences were found for comparisons of agreement scores by gender, age (below 30 years of age vs older), and professional role (medical student vs all other roles combined). Further analyses were conducted for the participants as a whole. Trainees achieved a high degree of agreement (83%) with the experts regardless of the order of video viewing (Table 2). Trainees who saw the common case first had significantly (p < .05; effect size = .4) higher agreement with the experts (87.1% vs 78.8%) on both cases and the combined score. For both approaches (better or common case first) 87.41% found the training very easy or mostly easy to understand and complete
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
757
Table 2 Agreement with experts on common and better cases by the order in which they were seen. Case
Better then common (N = 106) Mean (SD)
Common then Better (N = 105) Mean (SD)
Combined (N = 211) Mean (SD)
F value
Sig
Effect size
Common case Better case Combined
9.83 (4.38) 10.67 (1.87) 20.50 (6.56)
11.02 (3.34) 11.63 (1.87) 22.65 (4.65)
10.42 (3.93) 11.15 (2.49) 21.56 (5.78)
8.11 4.91 7.51
.005 .028 .007
.39 .30 .36
and only 3.8% found the training somewhat or extremely difficult to understand and complete, mean 4.18(SD .85). Trainees who saw the common version first found the training easier to understand and complete than those who saw the better version first (4.36 vs 4.02; effect size = .4). The rated ease of use did not correlate with expert agreement on either case or the combined case scores. Analysis of participants’ responses to the questions about ‘most important concept learned’ and ‘concepts that were confusing’ yielded 21 themes, most of which were mentioned in both categories (Table 3). Some participants mentioned more than one important skill learned for a total of 311 responses, and more than one confusing concept for a total of 269 responses. Trainees reported that the most important concept learned was ‘‘value of educational model’’ (24.6%) followed by ‘‘collaborative upfront agenda setting’’ (19.4%) and ‘‘EMR use’’ (17.5%). Trainees reported that the part that confused them the most was ‘‘none’’ or ‘‘nothing’’ (47.4%) followed by ‘‘value of educational model’’ (10.9%) and ‘‘EMR Use’’ (8%). In writing about the value of the educational model, trainees commented that ‘‘effective communication is a learned skill,’’ while others highlighted the importance of practice both in observation and feedback. The trainees who were confused about the educational model felt that the tool was unwieldy and wondered if ‘‘awareness of the ratings can lead to productive changes.’’ Use of the electronic medical record was another area of learning and confusion. Trainees were interested in learning to use the ‘‘EMR in a way that is constructive in the encounter and not a hindrance’’ and commented on how powerful it was to observe a physician use the EMR in a way that was and was not patientcentered. ‘‘The concept of how to use an EMR effectively was very
Table 3 Concepts that were learned or confusing.
None or nothing Value of educational model EMR use Perspective on health Collaborative agenda setting Co-creating a plan Practice Efficiency—transparent thinking Time management Behavior change discussion Shared decision making Maintain relationship Informed decision making Patient centered Establish rapport Gathering information Closure and follow-up Other Physical exam Sharing information Awareness
Most confusing concepts
Most important concepts learned
Percent reporting
Percent reporting
47.3 10.9 8.0 6.1 5.6 5.6 5.6 5.2 5.2 4.7 4.2 3.7 2.8 2.3 1.8 1.8 1.4 1.4 .9 .9 .9
0 24.6 17.5 8.0 19.4 7.5 2.3 9.0 3.7 4.2 4.7 8.5 0 15.1 12.3 6.1 5.6 5.6 5.2 1.8 8.5
beneficial, as it is easy to walk in to a patient’s room and not realize how you appear to them. Watching the physician walk into the room and immediately start typing was uncomfortable, and it was nice to contrast the two videos.’’ 4. Discussion The Improving Communication Assessment Project (ICAP) (http://uwfamilymedicine.org/pcof) was effective in helping trainees learn to use the Patient Centered Observation Form (PCOF). We found strong agreement between trainee ratings and expert ratings. Users found this online training easy to understand and complete. However, we found no relationship between ease of use and agreement with experts. Training effectiveness appears stronger when rating a less sophisticated skill demonstration before rating a more sophisticated skill demonstration, rather than the other way around. Trainee comments highlight the value of the educational model, agenda setting, EHR use, being patient centered, and several other skills. One half of the users indicated that there was nothing confusing about the training process with only a small minority expressing confusion about any aspect of the training. These results combined with data from earlier studies using the COMMON and BETTER model suggests that the ICAP design has value that is at least partially independent of PCOF usability. User ratings of the training design were not related to agreement with experts. Half of users report no confusion with the training and the most common attribute of the training was the educational model itself. Earlier studies [5,17–19] using COMMON and BETTER videos of the same encounters to teach communication skills were associated with improved outcomes. Showing the COMMON video before the BETTER video probably increases the impact of the ICAP design. The ICAP model may help dissolve significant barriers to teaching communication during years of clinical training. To our knowledge this is the first study showing that a communication assessment tool (PCOF) can be learned online and in a relatively short period of time. The asynchronous design of ICAP allows faculty to learn skills on their time schedule. ICAP training may help medical educators deconstruct their own tacit knowledge facilitating more frequent, specific formative feedback [6,9]. On a recent survey (unpublished), the University of Washington family medicine clerkship site directors throughout the WWAMI region rate the ICAP training very highly and comment that this training is an extremely important part of the communication curriculum. Just as ICAP training may help overcome barriers to faculty development, it may help overcome barriers to training for medical students and residents. Once trainees begin regular patient care, assessment should shift from a ‘‘shows how’’ focus as measured with standardized patients to a ‘‘does’’ focus to determine if desired skill use exists in the care of real patients [21]. However, many faculty struggle to find the time to offer assessment to trainees during actual patient encounters. Moreover, we must address a formidable, well-documented ‘‘hidden curriculum’’ [22] where supervising resident and faculty role models countermand the development of respectful relationships. Educational methodology that enables learners to give feedback to one another increases the number of times
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
758
learners can receive structured feedback. Our results suggest that ICAP training may accelerate learning. Once a communication assessment tool is learned many educational methods become
available to strengthen curricular designs. Third year clerkship students [5] and fourth year students taking an advanced communication elective [14] report significant benefit from
Patient Centered Observation Form–Clinician Version Trainee name______________________ Observer__________________Obsrvn#____Date__________ Directions; Track behaviors in left column. Then, mark one box per row: a, b or c. Competent skill use is in one of the right two right side columns. Record important provider / patient comments and verbal / non-verbal cues in the notes. Use form to enhance your learning, vocabulary, and self-awareness. Ratings can be for individual interviews or to summarize several interactions. If requested, use this form to guide verbal feedback to someone you observe.
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
.
Establishes Rapport Introduces self Warm greeting Acknowledges all in the room by name Uses eye contact Humor or non medical interaction
1a. Uses 0-2 elements
1b.Uses 3 elements.
1c.Uses ≥ 4 elements
2a. Uses 0-1 elements
2b. Uses 2 elements
2c. Uses 3 or more elements
3a. Uses 0-1 elements
3b. Uses 2 elements
3c. Uses ≥ 3 elements
4a. Uses 0 elements
4b. Uses 1 element
4c. Uses 2 or more elements
5a. Uses 0-1 elements
5b. Uses 2 elements
5c. Uses 3 or more elements
6b. Uses 1 element
6c. Uses 2 or more elements
Notes:
Maintains Relationship Throughout the Visit Uses verbal or non-verbal empathy during discussions or during the exam Uses continuer phrases (“um hmm”) Repeats important verbal content Demonstrates mindfulness through presence, curiosity, intent focus, not seeming “rushed” or acknowledging distractions Notes:
Collaborative upfront agenda setting Additional elicitation- “something else?” * X______ * each elicitation counts as a new element Acknowledges agenda items from other team member (eg MA) or from EMR. Asks or confirms what is most important to patient. Note patient concerns here:
Maintains Efficiency using transparent (out loud) thinking and respectful interruption: Talks about visit time use / visit organization Talks about problem priorities Talks about problem solving strategies Respectful interruption/redirection using EEE: Excuse your self, Empathize/validate issue being interrupted, Explain the reason for interruption ( eg, for Topic tracking) Notes:
Gathering Information Uses open-ended question X____ Uses reflecting statement X____ Uses summary/clarifying statement X____ Count each time the skill is used as one element_ Notes:
.
Assessing Patient or Family Perspective on Health Acknowledges patient verbal or non-verbal cues. Explores patient beliefs or feelings Explores contextual influences: family, cultural, spiritual. Number of patient verbal / non-verbal cues___
6a. Uses 0 elements
Notes:
© University of Washington Department of Family Medicine, September, 2013 Contact Larry Mauksch for further information Fig. 2.
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
759
Patient Centered Observation Form – Clinician Version Trainee name______________________ Observer__________________Obsrvn#____Date__________
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
Electronic Medical Record Use Regularly describes use of EMR to patient Maintains eye contact with patient during majority of time while using EMR. Positions monitor to be viewed by patient Points to screen
7a. Uses 0 or 1 elements.
7b. Uses 2 elements
7c. Uses 3 or 4 elements
8a. 0-1 exam elements (eg., lungs)
8b. 2 exam elements (eg, heart, lung)
8c. > 2 exam elements (eg, heart, lung, ears)
9a. Uses 0-1 elements
9b. Uses 2 elements
9c. Uses 3 or more elements
10c. Uses 0-1 elements or lectures patient
10b. Uses 2-3 elements
10c. Uses 4 or more elements
11a. Use 0-2 element
11b. Uses 3-4 elements
11c. Uses ≥ 5 elements
12a. Uses 0-1 element
12b. Uses 2 elements
12c. Uses 3 elements
Notes:
Physical Exam Prepares patient before physical exam actions and describes exam findings during the exam (“I am going to ___ ” then “your lungs sound healthy”) Notes:
Sharing Information Avoids or explains medical jargon Summaries cover biomedical concerns Summaries cover psychosocial concerns. Invites Q/A Notes:
Behavior Change Discussions Explores pt knowledge about behaviors Explores pros and cons of behavior change Scales importance of or confidence in change (1- 10) Asks permission to give advice Reflects comments about: desire, ability, reason, need, or commitment to change (respects ambivalence) Creates a plan aligned with patient’s readiness ( see MA/nurse version of PCOF Affirms behavior change effort or success Notes:
Co-creating a plan Assesses patient preferred decision making role States the clinical issue or decision to be made Describes options Discusses pros and cons Discusses uncertainties with the decision Assesses patient understanding Asks for patient preferences Identifies and resolves decisional differences Plan respects patients goals and values
Notes:
Closure Asks for questions about today’s topics. Co-creates and prints a readable After Visit Summary Uses Teachback. = Asking the patient to explain his/her understanding of the plan Combines Teachback and AVS creation while sharing the screen or notepad. (Counts for 3 elements) Notes:
© University of Washington Department of Family Medicine, September, 2013 Contact Larry Mauksch for further information Fig. 2. (Continued ).
observing and being observed. In Table 4 we review pros and cons of assorted educational strategies for using the PCOF and associated faculty time demands.
One of the ACGME core competencies, practice based learning and improvement, includes building the capacity to self assess learning needs and actively pursue needed information and
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
760
Table 4 Applications of the PCOF: faculty time demand, advantages and risks.
Direct observation in the exam room Video review Closed circuit
Faculty time demand
Advantages
Risks
High—loss of income or other activity High—loss of income or other activity Moderate—some loss of income or other activity
Immediate feedback and an opportunity for the trainee to refine skills with the next patient Trainee observes self, can monitor time use, discuss skills Faculty can document medical records while observing, discuss observations with other faculty to enhance learning [35] Increased numbers of observations, promotes trainee reflection and learning Teaching through role modeling; trainees reflect on observing faculty. Faculty are prompted to be on the their best behavior
Trainee initially self conscious Risk of faculty upstaging relationship Delayed practice Requires technical expertise and expense Faculty are vulnerable to distraction, CCTV requires added upfront expense
Peer
Very low
Faculty are observed
Low
training. This ability is often called self regulation [23] and is a key ingredient for life long learning. ICAP training and subsequent practice using the PCOF may assist learners to develop an ‘‘observer self’’, [14] a key component of self-regulation. While the focus of this report is on the ICAP online module, the success of the module is partially dependent on the design of the tool upon which training is focused. Our study may contribute to the limited psychometric data on the PCOF and help improve the PCOF design. In a separate study, Chesser et al. evaluated interrater reliability of the 2011 version of the PCOF and found an overall inter-rater reliability of .67 with four raters across 13 resident encounters. Their assessment used a higher bar than our study, inconsistent with the PCOF design, by treating answers in all three PCOF columns separately rather than collapsing the two right-hand columns into a single score. Using our scoring method would almost certainly improve their inter-rater results. Our results provide additional validity for the PCOF. The construction of the video cases included features related to skills included in the PCOF and so establishes content validity of the categories in the current instrument. The ability of the experts and trainees to consistently categorize various aspects of the physician behaviors in the videos combined with the data showing minimal confusion and ease of use provides evidence of PCOF face validity. We combined feedback from the two years of ICAP users with data from the Chesser et al. study and new skills we want to teach to refine several categories on the PCOF (Fig. 2 PCOF 2013 version http://courses.washington.edu/pove/files/PCOF_9_27_2013_ clinician.pdf) including maintaining relationship throughout the visit, behavior change discussions, co-creating a plan and closure. A new ICAP module for the 2013 version of the PCOF will be completed by early 2016. This study has limitations. Our data comes from combining the use of the PCOF with common and better versions of the same encounter. Agreement with experts may not have been as high if trainees viewed a series of unrelated encounters. Our study does not provide psychometric data to support using the PCOF for summative assessment. Our results do not predict the accuracy, probability of use, or ease of use for the PCOF after ICAP training. 5. Conclusion The Improving Communication Assessment Project helped trainees effectively and efficiently identify and name specific communication skills on the Patient Centered Observation Form. To our knowledge no other communication assessment tool has been evaluated for trainee use or widespread adoption by medical educators. Regularly using the PCOF to observe colleagues during the later years of medical school through residency and into practice may help physicians address long standing struggles to become effective communicators with their patients. Future
Peer feedback value is limited by lower level of trainee experience Passive trainee role
research should examine if ICAP training to use the PCOF combined with subsequent PCOF use, during real patient encounters, produces lasting changes in physician behavior, time use, and if these changes affect physician satisfaction, patient satisfaction, and health outcomes.
Role of funding None.
Conflict of interest statement Mr. Mauksch receives fees and honoraria for providing training and consultation on communication skills to educational institutions and health care organizations. Misbah Keen, Jan Carline and Jeanne Cawse-Lucas—None. Acknowledgements The authors wish to thank Valerie Ross, MS, Tom Greer, MD, MPH and Frederick Chen MD, MPH for their help creating expert ratings of the ICAP videos. References [1] Stewart M, Brown J. Patient-centered medicine: transforming the clinical method. In: Transforming the clinical method. 2nd ed. Abingdon: Radcliffe Medical Press; 2003. [2] Makoul G. Essential elements of communication in medical encounters: the Kalamazoo consensus statement. Acad Med 2001;76:390–3. [3] Epstein RM, Fiscella K, Lesser CS, Stange KC. Why the nation needs a policy push on patient-centered health care. Health Aff (Millwood) 2010;29:1489– 95. http://dx.doi.org/10.1377/hlthaff.2009.0888 (29/8/1489 [pii]). [4] Cuff P, Vanselow N, editors. Improving medical education: Enhancing the behavioral and social science content of medical school curricula. Washington, DC: Institute of Medicine, Committee on Behavioral and Social Sciences in Medical School Curriclula; 2004. [5] Egnew TR, Mauksch LB, Greer T, Farber SJ. Integrating communication training into a required family medicine clerkship. Acad Med 2004;79:737–43 (79/8/ 737 [pii]). [6] Beckman TJ. Lessons learned from a peer review of bedside teaching. Acad Med 2004;79:343–6. [7] Egnew TR, Wilson HJ. Role modeling the doctor–patient relationship in the clinical curriculum. Fam Med 2011;43:99–105. [8] Weissmann PF, Branch WT, Gracey CF, Haidet P, Frankel RM. Role modeling humanistic behavior: learning bedside manner from the experts. Acad Med 2006;81:661–7. http://dx.doi.org/10.1097/01.ACM.0000232423.81299.fe (00001888-200607000-00017 [pii]). [9] Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med 2011;86:846–52. http://dx.doi.org/10.1097/ ACM.0b013e31821db1bc. [10] Egnew TR, Wilson HJ. Faculty and medical students’ perceptions of teaching and learning about the doctor–patient relationship. Patient Educ Couns 2010;79:199–206. http://dx.doi.org/10.1016/j.pec.2009.08.012.
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761 [11] Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: a challenge for medical educators. J Contin Educ Health Prof 2008;28:5–13. http://dx.doi.org/10.1002/chp.149. [12] Schirmer JM, Mauksch L, Lang F, et al. Assessing communication competence: a review of current tools. Fam Med 2005;37:184–92. [13] Mauksch LB, Dugdale DC, Dodson S, Epstein R. Relationship, communication, and efficiency in the medical encounter: creating a clinical model from a literature review. Arch Intern Med 2008;168:1387–95. http://dx.doi.org/ 10.1001/archinte.168.13.1387 (168/13/1387 [pii]). [14] Mauksch L, Farber S, Greer HT. Design, dissemination, and evaluation of an advanced communication elective at seven U.S. medical schools. Acad Med 2013;88:843–51. http://dx.doi.org/10.1097/ACM.0b013e31828fd5ed. [15] Lang F, McCord R, Harvill L, Anderson DS. Communication assessment using the common ground instrument: psychometric properties. Fam Med 2004;36:189–98. [16] Krupat E, Frankel R, Stein T, Irish J. The four habits coding scheme: validation of an instrument to assess clinicians’ communication behavior. Patient Educ Couns 2006;62:38–45. S0738-3991(05)00130-8 [pii]. [17] Losh DP, Mauksch LB, Arnold RW, et al. Teaching inpatient communication skills to medical students: an innovative strategy. Acad Med 2005;80:118–24. [18] Brock DM, Mauksch LB, Witteborn S, Hummel J, Nagasawa P, Robins LS. Effectiveness of intensive physician training in upfront agenda setting. J Gen Intern Med 2011;26:1317–23. http://dx.doi.org/10.1007/s11606-0111773-y. [19] Mauksch LB, Hillenburg L, Robins L. The establishing focus protocol: training for collaborative agenda setting and time management in the medical interview. Fam Syst Health 2001;19:147–57. [20] Corbin J, Strauss A. Grounded theory research: procedures, canons, and evaluative criteria. Qual Sociol 1990;13:3–21. http://dx.doi.org/10.1007/ BF00988593. [21] Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63–7. [22] Hafferty FW. Beyond curriculum reform: confronting medicine’s hidden curriculum. Acad Med 1998;73:403–7.
761
[23] Brydges R, Butler D. A reflective analysis of medical education research on selfregulation in learning and practice. Med Educ 2012;46:71–9. http:// dx.doi.org/10.1111/j. 1365-2923.2011.04100.x. [24] Halpern J. What is clinical empathy? J Gen Intern Med 2003;18:670–4 (jgi21017 [pii]). [25] Epstein RM. Mindful practice. J Amer Med Assoc 1999;282:833–9 (jsc90157 [pii]). [26] Robins L, Witteborn S, Miner L, Mauksch L, Edwards K, Brock D. Identifying transparency in physician communication. Patient Educ Couns 2011;83:73–9. http://dx.doi.org/10.1016/j.pec.2010.05.004 (S07383991(10)00285-5 [pii]). [27] Levinson W, Gorawara-Bhat R, Lamb J. A study of patient clues and physician responses in primary care and surgical settings. J Amer Med Assoc 2000;284:1021–7. [28] Lang F, Floyd MR, Beine KL. Clues to patients’ explanations and concerns about their illnesses. A call for active listening. Arch Fam Med 2000;9:222–7. [29] Lang F, Floyd MR, Beine KL, Buck P. Sequenced questioning to elicit the patient’s perspective on illness: effects on information disclosure, patient satisfaction, and time expenditure. Fam Med 2002;34:325–30. [30] Ventres W, Kooienga S, Marlin R. EHRs in the exam room: tips on patientcentered care. Fam Pract Manage 2006;13:45–7. [31] Arnold RW, Losh DP, Mauksch LB, et al. Lexicon creation to promote faculty development in medical communication. Patient Educ Couns 2009;74:179– 83. http://dx.doi.org/10.1016/j.pec.2008.08.016 (S0738-3991(08)00460-6 [pii]). [32] Rollnick S, Mason P, Butler C. Health behavior change. Edinburgh: Churchill Livingstone; 1999. [33] Braddock 3rd CH, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. J Amer Med Assoc 1999;282:2313–20. [34] Schillinger D, Piette J, Grumbach K, et al. Closing the loop: physician communication with diabetic patients who have low health literacy. Arch Intern Med 2003;163:83–90 (ioi10753 [pii]). [35] Ross V, Mauksch L, Huntington J, Beard JM. Interdisciplinary direct observation: impact on precepting, residents, and faculty. Fam Med 2012;44:318–24.
Contents lists available at ScienceDirect
Patient Education and Counseling journal homepage: www.elsevier.com/locate/pateducou
Assessment
Using the patient centered observation form: Evaluation of an online training program Misbah Keen a,b, Jeanne Cawse-Lucas a, Jan Carline a,b, Larry Mauksch a,* a b
Department of Family Medicine, University of Washington, School of Medicine, Seattle, USA Department of BioMedical Informatics and Medical Education, University of Washington, School of Medicine, Seattle, USA
A R T I C L E I N F O
A B S T R A C T
Article history: Received 14 August 2014 Received in revised form 17 February 2015 Accepted 7 March 2015
Objective: The Patient Centered Observation Form (PCOF) helps trainees identify and describe specific communication skills and enhance self-awareness about skill use. We studied the effectiveness and ease of use of the Improving Communication Assessment Program (ICAP), an online module that prepares trainees to use the PCOF. Methods: Students, residents and medical educators viewed two videos (common and better skill use) of the same interaction and rated each video using the PCOF. Video sequence was randomized. We assessed agreement with experts, ease of use, concepts learned, and areas of confusion. Results: Trainees (211) achieved strong agreement (.83) with experts and were highly satisfied (mean 4.18 out of 5). Viewing the common video first produced higher agreement (.87 vs .79; ES = .4) with experts and greater satisfaction (4.36 vs 4.02, ES .4) than viewing the better video first. Trainees reported diverse areas of learning and minimal confusion. Conclusion: ICAP training to use the PCOF may facilitate teaching and assessment of communication skills and enrich training through peer observation and feedback. We offer several educational strategies. Practice implications: Learning to use the PCOF via the ICAP module may accelerate communication training for medical students, residents, medical educators and practicing clinicians. ß 2015 Elsevier Ireland Ltd. All rights reserved.
1. Introduction Effective communication is an essential physician skill to build trust, convey caring, and provide education and treatment [1–4]. Most medical schools teach and evaluate communication skills in the first and second years through didactic efforts, simulation, and limited practice with selected patients. However, medical schools are challenged to integrate communication skill training during the clinical phase of training [5]. Faculty often lack the time, training or language to train senior students or residents [6–8]. Lacking an articulate vocabulary means that educators struggle to reveal their tacit knowledge [8–10] and instead may rely solely on role modeling without explaining their behavior. A limited vocabulary leads to teachers and learners struggling to teach and articulate questions [7,8]. The Patient Centered Observation Form (PCOF) was designed to help observers and those being observed identify and describe
* Corresponding author at: University of Washington, School of Medicine, Department of Family Medicine, 6026 30th Ave NE, Seattle, WA 98115, USA. Tel.: +1 206 920 1245. E-mail addresses: [email protected], [email protected] (L. Mauksch). http://dx.doi.org/10.1016/j.pec.2015.03.005 0738-3991/ß 2015 Elsevier Ireland Ltd. All rights reserved.
specific skills, enhance self-awareness [11] and structure coaching during formative assessment. As such our definition of potential trainees includes medical students, residents, nurses, faculty physicians and other medical educators observing others during real patient encounters or on video. 1.1. Form development The process of creating the tool began in 2004. The first version of this tool used only narrative feedback about categories in the Kalamazoo consensus statement of essential communication skills [2]. In 2005 behavioral anchors were added corresponding to skills and categories described in the Kalamazoo consensus statement and in 13 communication assessment instruments [12]. The experts contributing to these sources consistently named the selected categories and skills. The tool has been revised several times since 2005 in response to feedback from users. The PCOF (Fig. 1 PCOF 2011 version) categorizes competence, and identifies skills, at the intersection of time management and quality of care [13]. Addressing trainee time management concerns is essential to reduce trainee anxiety that impedes learning [5,14]. To avoid promoting a grading mentality the PCOF does not use numerical
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
754
scoring but instead focuses on recording evidence of skill use using a checklist with space to add narrative commentary. Participants categorize the communication skills by checking off one or more
elements within each of thirteen PCOF categories. The PCOF requires users to count elements in a skill category to determine competence. Element sums within the range listed in the left-hand
Patient Centered Observation Form Trainee name______________________ Observer__________________Obsrvn#____Date__________ Directions; Track behaviors in left column. Then, mark one box per row: a, b or c. Competent skill use is in one of the right two columns. Record important provider / patient comments and verbal / non-verbal cues in the notes. Use form to enhance your learning, vocabulary, and self-awareness. Ratings can be for individual interviews or to summarize several interactions. If requested, use this form to guide verbal feedback to someone you observe.
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
.
Establishes Rapport Introduces self Warm greeting Acknowledges all in the room by name Uses eye contact Humor or non medical interaction
1a. Uses 0-2 elements
1b.Uses 3 elements.
1c.Uses ≥ 4 elements
2a. Uses 0-1 elements
2b. Uses 2 elements
2c. Uses 3 or more elements
3a. Uses 0-1 elements
3b. Uses 2 elements
3c. Uses ≥ 3 elements
4a. Uses 0 elements
4b. Uses 1 element
4c. Uses 2 or more elements
5a. Uses 0-1 elements
5b. Uses 2 elements
5c. Uses 3 or more elements
6b. Uses 1 element
6c. Uses 2 or more elements
Notes:
Maintains Relationship Throughout the Visit Strong verbal or non-verbal empathy Listens well using continuer phrases (“um hmm”) Repeats important verbal content; Demonstrates mindfulness through curiosity, selfreflection, or presence Notes:
Collaborative upfront agenda setting Additional elicitation- “something else?”each elicitation counts as a new element Acknowledges agenda items from other team member (eg MA) or from EMR. Confirms what is most important to patient? Note patient concerns here:
Maintains Efficiency through transparent (out loud) thinking: about visit time use / visit organization about problem priorities about problem solving Notes:
Gathering Information Uses open-ended question X____ Uses reflecting statement X____ Uses summary/clarifying statement X____ Count each time the skill is used as one element_ Notes:
.
Assessing Patient or Family Perspective on Health Acknowledges patient verbal or non-verbal cues. Explores patient beliefs or feelings Explores contextual influences: family, cultural spiritual. Number of patient verbal / non-verbal cues___
6a. Uses 0 elements
Notes:
© University of Washington Department of Family Medicine, May, 2011 Contact Larry Mauksch for further information Fig. 1.
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
755
Patient Centered Observation Form Trainee name______________________ Observer__________________Obsrvn#____Date__________
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
Electronic Medical Record Use Regularly describes use of EMR to patient Maintains eye contact with patient during majority of time while using EMR. Positions monitor to be viewed by patient Points to screen
7a. Uses 0 or 1 elements.
7b. Uses 2 elements
7c. Uses 3 or 4 elements
8a. Never
8b. some of the time, up to half the time
8c. Most of the time
9a. Uses 0-1 elements
9b. Uses 2 elements
9c. Uses 3 or more elements
10c. Uses 0-1 elements or lectures patient
10b. Uses 2-3 elements
10c. Uses 4 or more elements
11a. Use 0 elements
11b. Uses 1-2 elements
11c. Uses 3-4 elements.
12a. Use 0-1 element
12b. Uses 2 elements
13a. Uses 0-1 element
13b. Uses 2 elements
Notes:
Physical Exam Prepares patient before physical exam actions and describes exam findings during the exam (“I am going to ___ ” then “your lungs sound healthy”) Notes:
Sharing Information Avoids or explains medical jargon Summaries cover biomedical concerns Summaries cover psychosocial concerns. Invites Q/A Notes:
Behavior Change Discussions Explores pt knowledge about behaviors Explores pros and cons of behavior change Scales confidence or importance Asks permission to give advice Reflects or summarizes patient thoughts and feelings Creates a plan aligned with patient’s readiness Affirms behavior change effort or success Notes:
Co-creating a plan Informed Decision Making (when appropriate) Shares evidence(when available) behind recommendations Describes alternative options Examines pros and cons Describes uncertainties of viable options Shared Decision Making Plan respects patient’s biomedical goals / values. Plan respects patient’s psychosocial goals and values. Asks for patient preferences Asks for patient input and, if needed, modifies plan
12c. Uses
3 elements
Notes:
Closure and Follow-up Asks for questions about today’s topics. Prints After Visit Summary Uses Teachback. = Asking the patient to explain his/her understanding of the plan
13c. Uses 3 elements
Notes:
© University of Washington Department of Family Medicine, May, 2011 Contact Larry Mauksch for further information Fig. 1. (Continued ).
column denotes sub-competent performance while element counts in the ranges in either of the two right-hand columns denote competent performance, hence the thick vertical line separating the left hand column from the middle and right-hand
column. Not all encounters require element sums in the right hand column. Research [15,16] suggests that the training needed to master communication assessment tools requires several hours and is the
756
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
Table 1 PCOF elements. Establishing report [2] Maintaining relationship [24,25] Collaborative agenda setting [18] Maintaining efficiency [26] Gathering information [2] Assessing patient perspective [27–29] Electronic medical record use [30] Physical exam skills [31] Sharing information [2] Behavioral change discussion [32] Co-creating a plan [33] Closure and follow-up [34]
province of small numbers of highly trained faculty or researchers. However, the PCOF is designed for all trainees including a wide range of medical educators. Therefore we needed a training methodology that is efficient, effective and attractive. In 2007 the Improving Communication Assessment Program (ICAP) was created to provide online asynchronous PCOF training to help users at all developmental levels identify and name core communication skills (http://uwfamilymedicine.org/pcof). The Improving Communication Assessment Program has trainees use the PCOF to rate physician-patient interaction in two videos. One video demonstrates COMMON skill use characterized by the absence of selected skills listed in Table 1. The second, BETTER video, recreates the same interaction but with the physician demonstrating selected skills that were absent in the COMMON version. The trainee downloads a paper version of the PCOF, records observed skills, transfers ratings to ICAP and then submits these ratings. The next ICAP screen shows how experts rated the video1 with an explanation for each rating allowing the user to compare and learn. The training takes approximately 40 min. Using COMMON and BETTER videos helps make communication skills tangible by creating contrast and has been shown to be effective in prior educational trials [5,17–19]. The ICAP program is an integral component of the Patient Centered Communication Curriculum embedded within the required six-week, third year family medicine clerkship at the University of Washington School of Medicine (UWSOM). Faculty at one of 51 clerkship sites across our five state region use the ICAP to learn the PCOF as do all the clerkship students (approx. 230/year). Each week clerkship faculty and students are expected to observe one another using the PCOF in an outpatient setting and discuss communication skill use [5]. The ICAP program is also used to train other health care providers in various roles and at various levels of professional development including residents, medical educators, community physicians and nurses. Additionally, It is used for an advanced communication skills course that was disseminated to several medical schools [14]. While we have used consecutive versions of the ICAP for several years, we have not assessed its effectiveness or appeal to users. Therefore, we studied the following questions: 1. Is the Improving Communication Assessment Program effective in training users to use the PCOF? a. Is there adequate agreement between trainee and expert ratings? b. Is the training easy to understand and complete? 1 Two behavioral scientist faculties (Larry Mauksch, M.Ed and Valerie Ross, MS.) and two physician faculties (Tom Greer, MD., MPH and Fredrick Chen, MD, MPH) with expertise in communication rated the videos separately and then met to achieve consensus and agree on wording to describe reasoning for each rating.
c. Is there a correlation between ease of use and agreement with experts? 2. Does the order of viewing the ICAP videos make a significant difference in agreement with the experts and ease of use? 3. What are the most important concepts that trainees learn? 4. What concepts confuse trainees?
2. Methods ICAP programing randomized users to view either the COMMON or BETTER videos first. All responses were anonymous. Prior to viewing videos participants were asked to enter (through typing) age, gender and professional role. After rating both videos participants were asked to describe the ‘‘most important concept/skill learned’’ and concepts ‘‘that were confusing’’, and rate the ease of using the ICAP training from 1 (extremely difficult to understand and complete) to 5 (very easy to understand and complete). Analysis of narrative responses used a grounded theoretical approach [20]. Grounded theory is a qualitative research process in which the definitions of codes are refined throughout an iterative analytic process. Three authors (MK, LM and JCL) developed a coding scheme by separately working through respondent answers until we reached saturation. Then we compared codes to reach consensus on a final coding scheme. Most of the codes directly corresponded to the skills on PCOF and additional codes were added for answers that did not directly relate to the skills. Over 80% agreement was obtained on initial coding. The authors then met to reach consensus on items with coding discrepancies. All numerical items including the frequencies for codes were analyzed using SPSS version 19, IBM Inc. Simple descriptive statistics were calculated for all variables, and comparisons were made using T-tests and One Way Analyses of Variance. The main focus of the analyses was the agreement of participant codes with those provided by the experts and the potential effects of video order on agreement. For analytic purposes element scores were collapsed into dichotomous categories of the left hand column or the right two columns. A score of 0 was given to an element if the participant rating did not agree with the expert, and 1 if the participant rating agreed with the expert. The total score for each video ranged from 0 to 13. Additional analyses focused on the relationship of participant professional role with agreement with expert ratings, and qualitative measures of learning value in terms of concepts perceived to have been understood or still considered to be confusing by the participants. 3. Results A total of 211 trainees completed the ICAP training between October 2011 and June 2012. Sixty-five percent were female. Most users (64.5%) were below 30 years of age. The majority were medical students (72%), 8% were Nurses, 4.7% Behavioral Scientists, 4.3% faculty physicians, 1.9% Medical Residents and 7.6% Other. No statistically significant differences were found for comparisons of agreement scores by gender, age (below 30 years of age vs older), and professional role (medical student vs all other roles combined). Further analyses were conducted for the participants as a whole. Trainees achieved a high degree of agreement (83%) with the experts regardless of the order of video viewing (Table 2). Trainees who saw the common case first had significantly (p < .05; effect size = .4) higher agreement with the experts (87.1% vs 78.8%) on both cases and the combined score. For both approaches (better or common case first) 87.41% found the training very easy or mostly easy to understand and complete
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
757
Table 2 Agreement with experts on common and better cases by the order in which they were seen. Case
Better then common (N = 106) Mean (SD)
Common then Better (N = 105) Mean (SD)
Combined (N = 211) Mean (SD)
F value
Sig
Effect size
Common case Better case Combined
9.83 (4.38) 10.67 (1.87) 20.50 (6.56)
11.02 (3.34) 11.63 (1.87) 22.65 (4.65)
10.42 (3.93) 11.15 (2.49) 21.56 (5.78)
8.11 4.91 7.51
.005 .028 .007
.39 .30 .36
and only 3.8% found the training somewhat or extremely difficult to understand and complete, mean 4.18(SD .85). Trainees who saw the common version first found the training easier to understand and complete than those who saw the better version first (4.36 vs 4.02; effect size = .4). The rated ease of use did not correlate with expert agreement on either case or the combined case scores. Analysis of participants’ responses to the questions about ‘most important concept learned’ and ‘concepts that were confusing’ yielded 21 themes, most of which were mentioned in both categories (Table 3). Some participants mentioned more than one important skill learned for a total of 311 responses, and more than one confusing concept for a total of 269 responses. Trainees reported that the most important concept learned was ‘‘value of educational model’’ (24.6%) followed by ‘‘collaborative upfront agenda setting’’ (19.4%) and ‘‘EMR use’’ (17.5%). Trainees reported that the part that confused them the most was ‘‘none’’ or ‘‘nothing’’ (47.4%) followed by ‘‘value of educational model’’ (10.9%) and ‘‘EMR Use’’ (8%). In writing about the value of the educational model, trainees commented that ‘‘effective communication is a learned skill,’’ while others highlighted the importance of practice both in observation and feedback. The trainees who were confused about the educational model felt that the tool was unwieldy and wondered if ‘‘awareness of the ratings can lead to productive changes.’’ Use of the electronic medical record was another area of learning and confusion. Trainees were interested in learning to use the ‘‘EMR in a way that is constructive in the encounter and not a hindrance’’ and commented on how powerful it was to observe a physician use the EMR in a way that was and was not patientcentered. ‘‘The concept of how to use an EMR effectively was very
Table 3 Concepts that were learned or confusing.
None or nothing Value of educational model EMR use Perspective on health Collaborative agenda setting Co-creating a plan Practice Efficiency—transparent thinking Time management Behavior change discussion Shared decision making Maintain relationship Informed decision making Patient centered Establish rapport Gathering information Closure and follow-up Other Physical exam Sharing information Awareness
Most confusing concepts
Most important concepts learned
Percent reporting
Percent reporting
47.3 10.9 8.0 6.1 5.6 5.6 5.6 5.2 5.2 4.7 4.2 3.7 2.8 2.3 1.8 1.8 1.4 1.4 .9 .9 .9
0 24.6 17.5 8.0 19.4 7.5 2.3 9.0 3.7 4.2 4.7 8.5 0 15.1 12.3 6.1 5.6 5.6 5.2 1.8 8.5
beneficial, as it is easy to walk in to a patient’s room and not realize how you appear to them. Watching the physician walk into the room and immediately start typing was uncomfortable, and it was nice to contrast the two videos.’’ 4. Discussion The Improving Communication Assessment Project (ICAP) (http://uwfamilymedicine.org/pcof) was effective in helping trainees learn to use the Patient Centered Observation Form (PCOF). We found strong agreement between trainee ratings and expert ratings. Users found this online training easy to understand and complete. However, we found no relationship between ease of use and agreement with experts. Training effectiveness appears stronger when rating a less sophisticated skill demonstration before rating a more sophisticated skill demonstration, rather than the other way around. Trainee comments highlight the value of the educational model, agenda setting, EHR use, being patient centered, and several other skills. One half of the users indicated that there was nothing confusing about the training process with only a small minority expressing confusion about any aspect of the training. These results combined with data from earlier studies using the COMMON and BETTER model suggests that the ICAP design has value that is at least partially independent of PCOF usability. User ratings of the training design were not related to agreement with experts. Half of users report no confusion with the training and the most common attribute of the training was the educational model itself. Earlier studies [5,17–19] using COMMON and BETTER videos of the same encounters to teach communication skills were associated with improved outcomes. Showing the COMMON video before the BETTER video probably increases the impact of the ICAP design. The ICAP model may help dissolve significant barriers to teaching communication during years of clinical training. To our knowledge this is the first study showing that a communication assessment tool (PCOF) can be learned online and in a relatively short period of time. The asynchronous design of ICAP allows faculty to learn skills on their time schedule. ICAP training may help medical educators deconstruct their own tacit knowledge facilitating more frequent, specific formative feedback [6,9]. On a recent survey (unpublished), the University of Washington family medicine clerkship site directors throughout the WWAMI region rate the ICAP training very highly and comment that this training is an extremely important part of the communication curriculum. Just as ICAP training may help overcome barriers to faculty development, it may help overcome barriers to training for medical students and residents. Once trainees begin regular patient care, assessment should shift from a ‘‘shows how’’ focus as measured with standardized patients to a ‘‘does’’ focus to determine if desired skill use exists in the care of real patients [21]. However, many faculty struggle to find the time to offer assessment to trainees during actual patient encounters. Moreover, we must address a formidable, well-documented ‘‘hidden curriculum’’ [22] where supervising resident and faculty role models countermand the development of respectful relationships. Educational methodology that enables learners to give feedback to one another increases the number of times
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
758
learners can receive structured feedback. Our results suggest that ICAP training may accelerate learning. Once a communication assessment tool is learned many educational methods become
available to strengthen curricular designs. Third year clerkship students [5] and fourth year students taking an advanced communication elective [14] report significant benefit from
Patient Centered Observation Form–Clinician Version Trainee name______________________ Observer__________________Obsrvn#____Date__________ Directions; Track behaviors in left column. Then, mark one box per row: a, b or c. Competent skill use is in one of the right two right side columns. Record important provider / patient comments and verbal / non-verbal cues in the notes. Use form to enhance your learning, vocabulary, and self-awareness. Ratings can be for individual interviews or to summarize several interactions. If requested, use this form to guide verbal feedback to someone you observe.
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
.
Establishes Rapport Introduces self Warm greeting Acknowledges all in the room by name Uses eye contact Humor or non medical interaction
1a. Uses 0-2 elements
1b.Uses 3 elements.
1c.Uses ≥ 4 elements
2a. Uses 0-1 elements
2b. Uses 2 elements
2c. Uses 3 or more elements
3a. Uses 0-1 elements
3b. Uses 2 elements
3c. Uses ≥ 3 elements
4a. Uses 0 elements
4b. Uses 1 element
4c. Uses 2 or more elements
5a. Uses 0-1 elements
5b. Uses 2 elements
5c. Uses 3 or more elements
6b. Uses 1 element
6c. Uses 2 or more elements
Notes:
Maintains Relationship Throughout the Visit Uses verbal or non-verbal empathy during discussions or during the exam Uses continuer phrases (“um hmm”) Repeats important verbal content Demonstrates mindfulness through presence, curiosity, intent focus, not seeming “rushed” or acknowledging distractions Notes:
Collaborative upfront agenda setting Additional elicitation- “something else?” * X______ * each elicitation counts as a new element Acknowledges agenda items from other team member (eg MA) or from EMR. Asks or confirms what is most important to patient. Note patient concerns here:
Maintains Efficiency using transparent (out loud) thinking and respectful interruption: Talks about visit time use / visit organization Talks about problem priorities Talks about problem solving strategies Respectful interruption/redirection using EEE: Excuse your self, Empathize/validate issue being interrupted, Explain the reason for interruption ( eg, for Topic tracking) Notes:
Gathering Information Uses open-ended question X____ Uses reflecting statement X____ Uses summary/clarifying statement X____ Count each time the skill is used as one element_ Notes:
.
Assessing Patient or Family Perspective on Health Acknowledges patient verbal or non-verbal cues. Explores patient beliefs or feelings Explores contextual influences: family, cultural, spiritual. Number of patient verbal / non-verbal cues___
6a. Uses 0 elements
Notes:
© University of Washington Department of Family Medicine, September, 2013 Contact Larry Mauksch for further information Fig. 2.
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
759
Patient Centered Observation Form – Clinician Version Trainee name______________________ Observer__________________Obsrvn#____Date__________
Skill Set and elements Check only what you see or hear. Avoid giving the beneit of the doubt.
Provider Centered Biomedical Focus
Patient Centered Biopsychosocial Focus
Electronic Medical Record Use Regularly describes use of EMR to patient Maintains eye contact with patient during majority of time while using EMR. Positions monitor to be viewed by patient Points to screen
7a. Uses 0 or 1 elements.
7b. Uses 2 elements
7c. Uses 3 or 4 elements
8a. 0-1 exam elements (eg., lungs)
8b. 2 exam elements (eg, heart, lung)
8c. > 2 exam elements (eg, heart, lung, ears)
9a. Uses 0-1 elements
9b. Uses 2 elements
9c. Uses 3 or more elements
10c. Uses 0-1 elements or lectures patient
10b. Uses 2-3 elements
10c. Uses 4 or more elements
11a. Use 0-2 element
11b. Uses 3-4 elements
11c. Uses ≥ 5 elements
12a. Uses 0-1 element
12b. Uses 2 elements
12c. Uses 3 elements
Notes:
Physical Exam Prepares patient before physical exam actions and describes exam findings during the exam (“I am going to ___ ” then “your lungs sound healthy”) Notes:
Sharing Information Avoids or explains medical jargon Summaries cover biomedical concerns Summaries cover psychosocial concerns. Invites Q/A Notes:
Behavior Change Discussions Explores pt knowledge about behaviors Explores pros and cons of behavior change Scales importance of or confidence in change (1- 10) Asks permission to give advice Reflects comments about: desire, ability, reason, need, or commitment to change (respects ambivalence) Creates a plan aligned with patient’s readiness ( see MA/nurse version of PCOF Affirms behavior change effort or success Notes:
Co-creating a plan Assesses patient preferred decision making role States the clinical issue or decision to be made Describes options Discusses pros and cons Discusses uncertainties with the decision Assesses patient understanding Asks for patient preferences Identifies and resolves decisional differences Plan respects patients goals and values
Notes:
Closure Asks for questions about today’s topics. Co-creates and prints a readable After Visit Summary Uses Teachback. = Asking the patient to explain his/her understanding of the plan Combines Teachback and AVS creation while sharing the screen or notepad. (Counts for 3 elements) Notes:
© University of Washington Department of Family Medicine, September, 2013 Contact Larry Mauksch for further information Fig. 2. (Continued ).
observing and being observed. In Table 4 we review pros and cons of assorted educational strategies for using the PCOF and associated faculty time demands.
One of the ACGME core competencies, practice based learning and improvement, includes building the capacity to self assess learning needs and actively pursue needed information and
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761
760
Table 4 Applications of the PCOF: faculty time demand, advantages and risks.
Direct observation in the exam room Video review Closed circuit
Faculty time demand
Advantages
Risks
High—loss of income or other activity High—loss of income or other activity Moderate—some loss of income or other activity
Immediate feedback and an opportunity for the trainee to refine skills with the next patient Trainee observes self, can monitor time use, discuss skills Faculty can document medical records while observing, discuss observations with other faculty to enhance learning [35] Increased numbers of observations, promotes trainee reflection and learning Teaching through role modeling; trainees reflect on observing faculty. Faculty are prompted to be on the their best behavior
Trainee initially self conscious Risk of faculty upstaging relationship Delayed practice Requires technical expertise and expense Faculty are vulnerable to distraction, CCTV requires added upfront expense
Peer
Very low
Faculty are observed
Low
training. This ability is often called self regulation [23] and is a key ingredient for life long learning. ICAP training and subsequent practice using the PCOF may assist learners to develop an ‘‘observer self’’, [14] a key component of self-regulation. While the focus of this report is on the ICAP online module, the success of the module is partially dependent on the design of the tool upon which training is focused. Our study may contribute to the limited psychometric data on the PCOF and help improve the PCOF design. In a separate study, Chesser et al. evaluated interrater reliability of the 2011 version of the PCOF and found an overall inter-rater reliability of .67 with four raters across 13 resident encounters. Their assessment used a higher bar than our study, inconsistent with the PCOF design, by treating answers in all three PCOF columns separately rather than collapsing the two right-hand columns into a single score. Using our scoring method would almost certainly improve their inter-rater results. Our results provide additional validity for the PCOF. The construction of the video cases included features related to skills included in the PCOF and so establishes content validity of the categories in the current instrument. The ability of the experts and trainees to consistently categorize various aspects of the physician behaviors in the videos combined with the data showing minimal confusion and ease of use provides evidence of PCOF face validity. We combined feedback from the two years of ICAP users with data from the Chesser et al. study and new skills we want to teach to refine several categories on the PCOF (Fig. 2 PCOF 2013 version http://courses.washington.edu/pove/files/PCOF_9_27_2013_ clinician.pdf) including maintaining relationship throughout the visit, behavior change discussions, co-creating a plan and closure. A new ICAP module for the 2013 version of the PCOF will be completed by early 2016. This study has limitations. Our data comes from combining the use of the PCOF with common and better versions of the same encounter. Agreement with experts may not have been as high if trainees viewed a series of unrelated encounters. Our study does not provide psychometric data to support using the PCOF for summative assessment. Our results do not predict the accuracy, probability of use, or ease of use for the PCOF after ICAP training. 5. Conclusion The Improving Communication Assessment Project helped trainees effectively and efficiently identify and name specific communication skills on the Patient Centered Observation Form. To our knowledge no other communication assessment tool has been evaluated for trainee use or widespread adoption by medical educators. Regularly using the PCOF to observe colleagues during the later years of medical school through residency and into practice may help physicians address long standing struggles to become effective communicators with their patients. Future
Peer feedback value is limited by lower level of trainee experience Passive trainee role
research should examine if ICAP training to use the PCOF combined with subsequent PCOF use, during real patient encounters, produces lasting changes in physician behavior, time use, and if these changes affect physician satisfaction, patient satisfaction, and health outcomes.
Role of funding None.
Conflict of interest statement Mr. Mauksch receives fees and honoraria for providing training and consultation on communication skills to educational institutions and health care organizations. Misbah Keen, Jan Carline and Jeanne Cawse-Lucas—None. Acknowledgements The authors wish to thank Valerie Ross, MS, Tom Greer, MD, MPH and Frederick Chen MD, MPH for their help creating expert ratings of the ICAP videos. References [1] Stewart M, Brown J. Patient-centered medicine: transforming the clinical method. In: Transforming the clinical method. 2nd ed. Abingdon: Radcliffe Medical Press; 2003. [2] Makoul G. Essential elements of communication in medical encounters: the Kalamazoo consensus statement. Acad Med 2001;76:390–3. [3] Epstein RM, Fiscella K, Lesser CS, Stange KC. Why the nation needs a policy push on patient-centered health care. Health Aff (Millwood) 2010;29:1489– 95. http://dx.doi.org/10.1377/hlthaff.2009.0888 (29/8/1489 [pii]). [4] Cuff P, Vanselow N, editors. Improving medical education: Enhancing the behavioral and social science content of medical school curricula. Washington, DC: Institute of Medicine, Committee on Behavioral and Social Sciences in Medical School Curriclula; 2004. [5] Egnew TR, Mauksch LB, Greer T, Farber SJ. Integrating communication training into a required family medicine clerkship. Acad Med 2004;79:737–43 (79/8/ 737 [pii]). [6] Beckman TJ. Lessons learned from a peer review of bedside teaching. Acad Med 2004;79:343–6. [7] Egnew TR, Wilson HJ. Role modeling the doctor–patient relationship in the clinical curriculum. Fam Med 2011;43:99–105. [8] Weissmann PF, Branch WT, Gracey CF, Haidet P, Frankel RM. Role modeling humanistic behavior: learning bedside manner from the experts. Acad Med 2006;81:661–7. http://dx.doi.org/10.1097/01.ACM.0000232423.81299.fe (00001888-200607000-00017 [pii]). [9] Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med 2011;86:846–52. http://dx.doi.org/10.1097/ ACM.0b013e31821db1bc. [10] Egnew TR, Wilson HJ. Faculty and medical students’ perceptions of teaching and learning about the doctor–patient relationship. Patient Educ Couns 2010;79:199–206. http://dx.doi.org/10.1016/j.pec.2009.08.012.
M. Keen et al. / Patient Education and Counseling 98 (2015) 753–761 [11] Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: a challenge for medical educators. J Contin Educ Health Prof 2008;28:5–13. http://dx.doi.org/10.1002/chp.149. [12] Schirmer JM, Mauksch L, Lang F, et al. Assessing communication competence: a review of current tools. Fam Med 2005;37:184–92. [13] Mauksch LB, Dugdale DC, Dodson S, Epstein R. Relationship, communication, and efficiency in the medical encounter: creating a clinical model from a literature review. Arch Intern Med 2008;168:1387–95. http://dx.doi.org/ 10.1001/archinte.168.13.1387 (168/13/1387 [pii]). [14] Mauksch L, Farber S, Greer HT. Design, dissemination, and evaluation of an advanced communication elective at seven U.S. medical schools. Acad Med 2013;88:843–51. http://dx.doi.org/10.1097/ACM.0b013e31828fd5ed. [15] Lang F, McCord R, Harvill L, Anderson DS. Communication assessment using the common ground instrument: psychometric properties. Fam Med 2004;36:189–98. [16] Krupat E, Frankel R, Stein T, Irish J. The four habits coding scheme: validation of an instrument to assess clinicians’ communication behavior. Patient Educ Couns 2006;62:38–45. S0738-3991(05)00130-8 [pii]. [17] Losh DP, Mauksch LB, Arnold RW, et al. Teaching inpatient communication skills to medical students: an innovative strategy. Acad Med 2005;80:118–24. [18] Brock DM, Mauksch LB, Witteborn S, Hummel J, Nagasawa P, Robins LS. Effectiveness of intensive physician training in upfront agenda setting. J Gen Intern Med 2011;26:1317–23. http://dx.doi.org/10.1007/s11606-0111773-y. [19] Mauksch LB, Hillenburg L, Robins L. The establishing focus protocol: training for collaborative agenda setting and time management in the medical interview. Fam Syst Health 2001;19:147–57. [20] Corbin J, Strauss A. Grounded theory research: procedures, canons, and evaluative criteria. Qual Sociol 1990;13:3–21. http://dx.doi.org/10.1007/ BF00988593. [21] Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63–7. [22] Hafferty FW. Beyond curriculum reform: confronting medicine’s hidden curriculum. Acad Med 1998;73:403–7.
761
[23] Brydges R, Butler D. A reflective analysis of medical education research on selfregulation in learning and practice. Med Educ 2012;46:71–9. http:// dx.doi.org/10.1111/j. 1365-2923.2011.04100.x. [24] Halpern J. What is clinical empathy? J Gen Intern Med 2003;18:670–4 (jgi21017 [pii]). [25] Epstein RM. Mindful practice. J Amer Med Assoc 1999;282:833–9 (jsc90157 [pii]). [26] Robins L, Witteborn S, Miner L, Mauksch L, Edwards K, Brock D. Identifying transparency in physician communication. Patient Educ Couns 2011;83:73–9. http://dx.doi.org/10.1016/j.pec.2010.05.004 (S07383991(10)00285-5 [pii]). [27] Levinson W, Gorawara-Bhat R, Lamb J. A study of patient clues and physician responses in primary care and surgical settings. J Amer Med Assoc 2000;284:1021–7. [28] Lang F, Floyd MR, Beine KL. Clues to patients’ explanations and concerns about their illnesses. A call for active listening. Arch Fam Med 2000;9:222–7. [29] Lang F, Floyd MR, Beine KL, Buck P. Sequenced questioning to elicit the patient’s perspective on illness: effects on information disclosure, patient satisfaction, and time expenditure. Fam Med 2002;34:325–30. [30] Ventres W, Kooienga S, Marlin R. EHRs in the exam room: tips on patientcentered care. Fam Pract Manage 2006;13:45–7. [31] Arnold RW, Losh DP, Mauksch LB, et al. Lexicon creation to promote faculty development in medical communication. Patient Educ Couns 2009;74:179– 83. http://dx.doi.org/10.1016/j.pec.2008.08.016 (S0738-3991(08)00460-6 [pii]). [32] Rollnick S, Mason P, Butler C. Health behavior change. Edinburgh: Churchill Livingstone; 1999. [33] Braddock 3rd CH, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. J Amer Med Assoc 1999;282:2313–20. [34] Schillinger D, Piette J, Grumbach K, et al. Closing the loop: physician communication with diabetic patients who have low health literacy. Arch Intern Med 2003;163:83–90 (ioi10753 [pii]). [35] Ross V, Mauksch L, Huntington J, Beard JM. Interdisciplinary direct observation: impact on precepting, residents, and faculty. Fam Med 2012;44:318–24.
