Patient Education and Counseling 98 (2015) 748–752
Contents lists available at ScienceDirect
Patient Education and Counseling journal homepage: www.elsevier.com/locate/pateducou
Medical Education
Breaking bad news is a teachable skill in pediatric residents: A feasibility study of an educational intervention Suzanne Reed a,*, Karyn Kassis a, Rollin Nagel b, Nicole Verbeck b, John D. Mahan a, Richard Shell a a b
Department of Pediatrics, The Ohio State University College of Medicine, Nationwide Children’s Hospital, Columbus, USA The Ohio State University College of Medicine, Columbus, USA
A R T I C L E I N F O
A B S T R A C T
Article history: Received 8 October 2014 Received in revised form 20 January 2015 Accepted 15 February 2015
Objective: Patients and physicians identify communication of bad news as a skill in need of improvement. Our objectives were to measure change in performance of first-year pediatric residents in the delivery of bad news after an educational intervention and to measure if changes in performance were sustained over time. Methods: Communication skills of 29 residents were assessed via videotaped standardized patient (SP) encounters at 3 time points: baseline, immediately post-intervention, and 3 months post-intervention. Educational intervention used was the previously published ‘‘GRIEV_ING Death Notification Protocol.’’ Results: The intraclass correlation coefficient demonstrated substantial inter-rater agreement with the assessment tool. Performance scores significantly improved from baseline to immediate postintervention. Performance at 3 months post-intervention showed no change in two subscales and small improvement in one subscale. Conclusions: We concluded that breaking bad news is a complex and teachable skill that can be developed in pediatric residents. Improvement was sustained over time, indicating the utility of this educational intervention. Practice implications: This study brings attention to the need for improved communication training, and the feasibility of an education intervention in a large training program. Further work in development of comprehensive communication curricula is necessary in pediatric graduate medical education programs. ß 2015 Elsevier Ireland Ltd. All rights reserved.
Keywords: Medical education Breaking bad news Pediatrics
1. Introduction Good communication skills are critical to the practice of excellent and effective medicine [1–3]. This includes the delicate and complex skill of breaking bad news. These skills are so critical that breaking bad news poorly can negatively affect patient relationships and have lasting impact on patients and families [1,2,4–6]. It is clear that patients [3,6–9] and physicians [1,9–14] still feel that many physicians struggle to achieve competence in these skills. Even for experienced clinicians, delivering bad news is a source of considerable stress [9,10,15]. Physiologic stress
* Corresponding author at: Nationwide Children’s Hospital, Division of Hematology/Oncology/BMT, 700 Children’s Drive, Columbus, OH 43205, USA. Tel.: +1 614 722 6349; fax: +1 614 722 3559. E-mail address: [email protected] (S. Reed). http://dx.doi.org/10.1016/j.pec.2015.02.015 0738-3991/ß 2015 Elsevier Ireland Ltd. All rights reserved.
responses have even been noted in both novice and experienced physicians in simulated patient encounters that involve breaking bad news [15,16]. Competence in breaking bad news is not always associated with increased experience. One study compared communication competency in medical students, residents, and attending physicians. Interestingly, while novice medical students had the lowest competence, no difference was found among more senior students, residents, and attending physicians [17]. Another study of novice and experienced physicians during a simulated bad news encounter found experience was inversely related to stress response, but performance was inversely related to burnout and fatigue, not inexperience alone [18]. Many residency and fellowship programs lack formal instruction on the delivery of bad news [19–22]. Traditional medical education focused on mastery of pathology and disease treatment may result in inadequate teaching and evaluation of less tangible
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
skills (e.g., communication) necessary for excellent patient care. While the Accreditation Council for Graduate Medical Education (ACGME) clearly prioritizes interpersonal skills and communication as one of the core competencies, and requires residents to achieve competence in this area [23–26], programs define ‘‘competence’’ differently, creating inconsistent education and evaluation. Much of the literature evaluating the education of physicians on the delivery of bad news is based on attitudes [13,9,27,28]. Most publications report that trainees do feel more comfortable with delivering bad news after an educational intervention [27]. There are few studies, however, that evaluate the objective skill of trainees in the delivery of bad news and/or the effectiveness of an educational intervention to improve bad news delivery. A few studies demonstrate improvement in this skill after an educational intervention for oncology trainees [29–31] and family practice trainees [32], but there is little evidence in pediatric trainees. Residents have indicated less preparation and training in breaking bad news in the pediatric setting compared to adult patient encounters [11]. The objectives for this study were to evaluate the acquisition and retention of breaking bad news skills, specifically telling a parent or grandparent that a child has died, in pediatric residents after an educational intervention. 2. Methods 2.1. Trainee population All first-year pediatrics and combined internal medicine/ pediatrics residents at our institution, a free-standing tertiary care children’s hospital, were invited to participate. There were no specific selection criteria, only availability and willingness to participate. Consent was obtained prior to baseline assessment. Institutional Review Board approval was obtained prior to standardized patient (SP) encounters. 2.2. Rationale for communication curriculum selected The teaching intervention selected for this study was ‘‘The GRIEV_ING Death Notification Protocol,’’ designed and published by Cherri Hobgood et al. [33] This curriculum is a 2-h educational experience, which includes short didactics, small group discussion, and role-play. Also included in the protocol are SP encounters, with case scenarios and detailed SP instructions (complete protocol can be accessed publicly, www.acep.org). This protocol was selected for several reasons. First, it is a complete curriculum with SP encounters and assessment tools, ready to use. Second, it includes a pediatric-specific SP encounter and SP training instructions. Third, this protocol focuses on only one type of very challenging ‘‘bad news:’’ death notification. Finally, the GRIEV_ING Protocol has demonstrated effectiveness in emergency medicine residents, medical students, and emergency medical service providers [33–35], and is supported as a resource by the ACGME Outcome Project [36]. Further details of the complete educational intervention will be provided on request. 2.3. SP encounters and delivery of educational intervention Residents skills in delivering bad news were assessed with an SP encounter at three separate time points: before teaching intervention (pre-test), immediately post-intervention (post-test), and 3 months post-intervention (post-post-test). Trained standardized patients were utilized from the Clinical Skills Education and Assessment Center at the Ohio State University College of Medicine (OSUCOM). Compensation for SPs was provided by a small OSUCOM grant. All simulated encounters were videotaped in unused patient rooms.
749
Baseline skills in delivering bad news were assessed with an SP encounter 4 months into the first year of pediatrics training. Next, the didactics, small group discussion, and role-playing of the GRIEV_ING curriculum were delivered at a resident retreat. The second encounter (post-test) occurred 1–2 weeks later and was identical in structure to the pre-test SP encounter, with a different death notification scenario. The final SP encounter (post-post-test) was completed approximately 3 months after the intervention using a different death notification scenario. Four investigators independently scored the pre-test videotaped encounters; two of the four investigators independently scored post-test and postpost-test videotaped encounters. 2.4. The cases The GRIEV_ING Protocol includes death notification cases for SP encounters and SP training instructions. We used one case exactly as written and modified the other two cases to present pediatric patients. 2.5. The assessments The GRIEV_ING Protocol also includes an assessment tool to measure competence. This 27-item instrument was developed to cover ten competency areas resulting from a literature review of death notification and were components of the GRIEV_ING educational intervention [33]. Additional published validity evidence for this instrument is limited: there was significant improvement for pre to post-intervention for emergency medicine residents using the full assessment tool [33] and significant improvement in medical students using an abbreviated 12-item version [34]. For the 12-item version the internal consistency was 0.56 and 0.65 and correlated significantly with interpersonal communication [34]. After preliminary evaluation of baseline assessments, we modified the assessment process by dividing items into subscales: Preparation (9 items, including clear introduction of self and role, and determining parent/grandparent’s level of knowledge of the event prior to news disclosure), bad news delivery (12 items, such as use of clear language without medical jargon, and use of silence to allow parent/grandparent reaction), and Wrap-up (8 items, such as addressing specific details of next steps (like organ donation), and providing personal availability for follow-up questions/needs). This determination of subscales evolved from faculty expert consensus and Pearson coefficients of pre-test scores indicating that subscales appeared to measure different components of the breaking bad news skill. All items of the original assessment tool were retained in the division into subscales. 2.6. Data analysis The intraclass correlation coefficient (ICC) was used to assess rater reliability. The ICC reflects both the degree of correspondence and agreement between raters and ranges from 0.00 to 1.00 (1.00 being perfect agreement). ICC Model 2,1 was used to analyze confidence of results from a randomly chosen rater. ICC Model 2,4 and Model 2,2 were calculated to clarify changes in reliability when using means across four and two raters, respectively (as opposed to a single randomly chosen rater), to assess performance. Each rater scored each assessment item on a yes/no basis (‘‘1’’ = performed item, ‘‘0’’ = did not perform item). Scores were calculated based on percentage of items performed correctly. Pearson’s correlation coefficients were used to calculate the association between subscale scores. Evidence of discriminant validity requires that subscales not correlate too highly if measuring different things [37,38]. Repeated measures ANOVAs
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
750
Subscale Means (Preparation, Bad News Delivery, and Wrap-up) over Assessments (N = 29) 100 90 80
% Correct
were used to compare trainee scores across subscales for each assessment period and across assessment periods for each subscale. For significant within-subject effects, post hoc contrast effects were calculated to determine significance of within-subject factor differences for each subscale or assessment period. 3. Results 3.1. Participation
70 60
Preparaon
50 40
Bad News Delivery Wrap-up
30
Of 44 first-year residents (34 pediatrics, 10 medicine-pediatrics), 29 (27 pediatrics, 2 medicine-pediatrics) participated in all three phases of the study. No residents voluntarily declined, however limitations in availability due to clinical obligations prevented some residents from participating in one or more SP encounters.
20 10 0 Pre-test
Post-test
Post-post-test
Fig. 1. Subscale means (Preparation, Bad News Delivery, and Wrap-up) over assessments (N = 29) % correct.
3.2. Inter-rater reliability Four raters independently assessed the pre-test SP encounters. The ICC 2,1 values, to evaluate inter-rater reliability of a randomly selected single rater, were 0.46 for Preparation, 0.40 for Bad News Delivery, and 0.36 for Wrap-up indicating at best a moderate correlation (0.41–0.60) or fair correlation (0.21–0.40), per Landis and Koch’s characterization [39]. The ICC 2,4 values, to evaluate the inter-rater reliability when all four raters’ scores were averaged, were 0.78 for Preparation, 0.73 for Bad News Delivery, and 0.70 for Wrap-up indicating substantial correlations (per Landis and Koch: 0.61–0.80) for all three subscales. Using the average of two raters continued to provide substantial correlations for each of the pretest SP encounter subscales, with ICC’s 2,2 of 0.63 for Preparation, 0.63 for Bad News Delivery, and 0.64 for Wrap-up. These two raters then independently scored all of the remaining encounters. Using the average of these two raters scores typically yielded ICC’s 2,2 in the substantial range (for Preparation and Bad News Delivery) and even the perfect range (0.81–1.00) for Wrap-up. Only the post-test ICC 2,2 for Bad News Delivery was only fair (0.39). 3.3. Subscale scores: pre-test Pre-test scores were low on all three subscales, with residents consistently scoring below 50% (see Table 1). Comparisons between subscales (F = 67.39, p < .001) and contrast tests demonstrated significant differences between each subscale (p < .001) with Wrap-up yielding the lowest scores (M = 18.53, SD = 10.10). Pearson coefficients demonstrated a lack of association among subscales ((r = 0.28, p = 0.15) between Preparation and Bad News Delivery, (r = .0.04, p = .84) between Preparation and Wrap-up, Table 1 Mean percent of items performed correctly with differences between and within subscales and assessments (N = 29). Assessment
Pre-testy Post-testy Post-post-testy
Subscale Preparation* Mean (SD)
Bad News Delivery* Mean (SD)
Wrap-up* Mean (SD)
38.31A,1 (12.45) 67.62B,1,a (13.11) 69.73B,1 (12.82)
52.44A,2 (12.67) 73.99B,1,b (7.93) 77.73B,2,c (7.97)
18.53A,3 (10.09) 43.97B,3 (18.26) 42.89B,3 (16.93)
Note: Post hoc tests: Means in columns with different capital superscripts are significantly different at p < .01. Means in rows with different numerical superscripts are significantly different at p < .01. Means in columns with different small superscripts are significantly different at p < .05. Means in rows with different small superscripts are significantly different at p < .05. * p < .001. Repeated measures ANOVA between assessments for the indicated subscale. y p < .001. Repeated measures ANOVA between subscales for the indicated assessment.
and (r = 0.03, p = .86) between Bad News Delivery and Wrap-up), further supporting the use of subscale scores to detect real differences in performance. 3.4. Post-test and post-post-test scores Post-test and post-post-test scores were calculated for each subscale (Table 1). Repeated measure ANOVA tests indicated significant differences between subscales for both the post-test (F = 38.20, p < .001) and post-post-test (F = 60.57, p < .001). Contrast tests for the post-subtests demonstrated significant differences between Preparation and Bad News Delivery (p = .015), Bad News Delivery and Wrap-up (p < .001), and Preparation and Wrap-up (p < .001). Contrast tests for the postpost-subtests demonstrated similar significant differences between subscales: Preparation and Bad News Delivery (p = .002), Bad News Delivery and Wrap-up (p < .001), and Preparation and Wrap-up (p < .001). Correlations between subscales for each test administration continued to demonstrate a lack of correlation (post-test: Preparation and Bad News Delivery (r = 0.28, p = .14, Bad News Delivery and Wrap-up (r = 0.13, p = .51, Preparation and Wrap-up (r = 0.05, p = .79); post-post-test: Preparation and Bad News Delivery (r = 0.33, p = .08, Bad News Delivery and Wrapup (r = 0.21, p = .27, Preparation and Wrap-up (r = 0.12, p = .54). 3.5. Comparisons across test administrations Repeated measures ANOVAs for each subscale across test administrations demonstrated differences for the Preparation (F = 63.46, p < .001), Bad News Delivery (F = 67.49, p < .001), and Wrap-up (F = 26.80, p < .001) subscales. Contrast tests demonstrated significant post-test improvement for all three subscales (all p’s < .001) (see Table 1). Post-test scores demonstrated percent improvement from pre-test scores of 41% (Bad News Delivery) to 137% (Wrap-up). The 3-month post-test subscale scores continued to demonstrate significant improvement compared to pre-test scores (all p’s < .001). Performance on the post-post-test Bad News Delivery demonstrated significant improvement (p = .035) compared to post-test scores, however, no differences between post-test and post-post-tests were evident for the Preparation (p = .50) or Wrap-up (p = .76) scales (see Fig. 1). 4. Discussion and conclusions 4.1. Discussion This study demonstrated that first-year pediatric residents’ communication skills when notifying parents or grandparents of a
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
child’s death significantly improved following an educational intervention based on the GRIEV_ING Protocol. The residents maintained this performance improvement for 3 months. These results suggest this skill is teachable and retainable, and illustrate the need and value of increased education and development of this complex skill. Our results suggest it would be short-sighted to rely on inherent communication skills alone for proficiency in death notification communication. This study also supplies evidence for the validity of the assessment tool and its division into three subscales. Subscale analysis at the pre-test, post-test, and 3-month follow-up assessments demonstrated discriminant validity, performance differences and lack of association between subscales. Future studies with larger sample sizes could use factor analysis to further assess subscales. The discreteness of the three subscales suggests that communication skills needed for death notification are complex and varied. This provides the opportunity for a more individualized approach to death notification education, and offers a future direction for enhanced bad news training. This is the first study examining death notification skills in pediatric residents. The improvement in performance and retention of skills after an educational intervention are encouraging. While this particular intervention focused on death notification specifically, it included concepts and techniques that can be applied to delivery of other types of bad news. The primary care pediatrician is not likely to deliver death news, however the delivery of bad news is a responsibility that crosses all general pediatric practices and subspecialties. Our results suggest that a modest investment by a residency program (in time, effort, and cost) can produce meaningful results and retention of widely applicable skills. Pediatric physician communication is somewhat unique, relative to other specialties. The communication exchange may have a wide range of patient involvement, and breaking bad news can be more sensitive for both patients and families. The relative infrequency of opportunities for trainees to practice breaking bad news is an added complication. Limited exposure for pediatrics trainees underscores the need for simulated training and experiential learning to supplement these less common real-life learning experiences. Our study demonstrated improved, but not perfect, communication skills after an educational intervention. While we did not set a specific ‘‘competence’’ score, on the post-test almost onethird of the residents scored below 70% on Preparation and Bad News Delivery, and most failed to reach 70% for Wrap-up. A need remains for additional skill improvement among trainees. This raises the question of ‘‘dose’’ and ‘‘effect.’’ Perhaps a larger ‘‘dose’’ of education (a longer workshop, more role-play, etc.) or repeated interventions (refresher workshops, etc.) could increase the degree of improvement from baseline. Future studies will be required to answer these important questions. There are several limitations of our study, including small sample size, single institution, and lack of control group. We could not account for differences in trainees’ previous experiences with breaking bad news and/or death notification. There is evidence of some medical student training in breaking bad news [33,34,40], suggesting that at least some medical students have exposure to this subject, though these experiences are poorly standardized. We did not survey our trainees about past bad news delivery training. Another limitation is that we did not account for general professional growth that typically occurs during the first year of residency. Exposure to another physician communicating death news is possible in the first year of residency, and it is unknown the effect, if any, of these observational experiences.
751
4.2. Conclusions In summary, this feasibility study implementing a modified intervention for pediatric trainees demonstrates that death notification is a difficult and complex, but teachable, skill. These pilot data support the need for further investigation. Our data also suggest that this complex skill can be broken down into more specific components, creating the potential for individualized training, to allow each trainee to reach his or her maximum potential in this skill. 4.3. Practice implications This project successfully applied the GRIEV_ING Protocol to teach communication skills for breaking bad news to pediatric residents, and provides preliminary data to inform and support more comprehensive communication curricula in pediatric graduate medical education programs. Future studies should focus on assessment of long-term retention of skills and on teaching other aspects of breaking bad news, with the goal of developing pediatricians with high levels of skill in communication. Financial disclosure The authors have no financial relationships relevant to this article to disclose. Conflict of interest The other authors have no conflicts of interest to disclose. References [1] Munoz Sastre MT, Sorum PC, Mullet E. Breaking bad news: the patient’s viewpoint. Health Commun 2011;26:649–55. [2] Ptacek JT, Ellison NM. Health care providers’ perspectives on breaking bad news to patients. Crit Care Nurs Q 2000;23:51–9. [3] Trudel JG, Leduc N, Dumont S. Perceived communication between physicians and breast cancer patients as a predicting factor of patients’ health-related quality of life: a longitudinal analysis. Psychooncology 2013. [4] Ptacek JT, Eberhardt TL. Breaking bad news. A review of the literature. J Am Med Assoc 1996;276:496–502. [5] Ptacek JT, Fries EA, Eberhardt TL, Ptacek JJ. Breaking bad news to patients:[ physicians’ perceptions of the process. Support Care Cancer 1999;7:113–20. [6] Ptacek JT, Ptacek JJ. Patients’ perceptions of receiving bad news about cancer. J Clin Oncol 2001;19:4160–4. [7] Mack JW, Hilden JM, Watterson J, Moore C, Turner B, Grier HE, et al. Parent and physician perspectives on quality of care at the end of life in children with cancer. J Clin Oncol 2005;23:9155–61. [8] Mazor KM, Simon SR, Yood RA, Martinson BC, Gunter MJ, Reed GW, et al. Health plan members’ views on forgiving medical errors. Am J Manag Care 2005;11: 49–52. [9] Orgel E, McCarter R, Jacobs S. A failing medical educational model: a selfassessment by physicians at all levels of training of ability and comfort to deliver bad news. J Palliat Med 2010;13:677–83. [10] Birkeland AL, Dahlgren L, Ha¨gglo¨f B, Rydberg A. Breaking bad news: an interview study of paediatric cardiologists. Cardiol Young 2011;21:286–91. [11] Dube CE, LaMonica A, Boyle W, Fuller B, Burkholder GJ. Self-assessment of communication skills preparedness: adult versus pediatric skills. Ambul Pediatr 2003;3:137–41. [12] Durall A, Zurakowski D, Wolfe J. Barriers to conducting advance care discussions for children with life-threatening conditions. Pediatrics 2012;129: e975–82. [13] Harrison ME, Walling A. What do we know about giving bad news? A review. Clin Pediatr (Phila) 2010;49:619–26. [14] Sweeny K, Shepperd JA, Han PK. The goals of communicating bad news in health care: do physicians and patients agree? Health Expect 2011. [15] Hulsman RL, Pranger S, Koot S, Fabriek M, Karemaker JM, Smets EM. How stressful is doctor-patient communication? Physiological and psychological stress of medical students in simulated history taking and bad-news consultations. Int J Psychophysiol 2010;77:26–34. [16] Meunier J, Merckaert I, Libert Y, Delvaux N, Etienne AM, Lie´nard A, et al. The effect of communication skills training on residents’ physiological arousal in a breaking bad news simulated task. Patient Educ Couns 2013;93:40–7. [17] Wouda JC, van de Wiel HB. The communication competency of medical students: residents and consultants. Patient Educ Couns 2012;86:57–62.
752
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
[18] Brown R, Dunn S, Byrnes K, Morris R, Heinrich P, Shaw J. Doctors’ stress responses and poor communication performance in simulated bad-news consultations. Acad Med 2009;84:1595–602. [19] Hebert HD, Butera JN, Castillo J, Mega AE. Are we training our fellows adequately in delivering bad news to patients? A survey of hematology/ oncology program directors. J Palliat Med 2009;12:1119–24. [20] File W, Bylund CL, Kesselheim J, Leonard D, Leavey P. Do pediatric hematology/ oncology (PHO) fellows receive communication training? Pediatr Blood Cancer 2013. [21] Kersun L, Gyi L, Morrison WE. Training in difficult conversations: a national survey of pediatric hematology–oncology and pediatric critical care physicians. J Palliat Med 2009;12:525–30. [22] Turner DA, Mink RB, Lee KJ, Winkler MK, Ross SL, Hornik CP, et al. Are pediatric critical care medicine fellowships teaching and evaluating communication and professionalism? Pediatr Crit Care Med 2013;14:454–61. [23] Green ML, Holmboe E. Perspective: the ACGME toolbox: half empty or half full? Acad Med 2010;85:787–90. [24] Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach 2007;29:648–54. [25] Hicks PJ, Schumacher DJ, Benson BJ, Burke AE, Englander R, Guralnick S, et al. The pediatrics milestones: conceptual framework, guiding principles, and approach to development. J Grad Med Educ 2010;2:410–8. [26] Rushton JL, Hicks PJ, Carraccio CL. The next phase of pediatric residency education: the partnership of the Milestones Project. Acad Pediatr 2010;10:91–2. [27] Rosenbaum ME, Ferguson KJ, Lobas JG. Teaching medical students and residents skills for delivering bad news: a review of strategies. Acad Med 2004;79: 107–17. [28] Rosenbaum ME, Wilson JF, Sloan DA. Clinical instruction for delivering bad news. Acad Med 1996;71:529. [29] Back AL, Arnold RM, Baile WF, Fryer-Edwards KA, Alexander SC, Barley GE, et al. Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Arch Intern Med 2007;167:453–60.
[30] Eid A, Petty M, Hutchins L, Thompson R. Breaking bad news: standardized patient intervention improves communication skills for hematologyoncology fellows and advanced practice nurses. J Cancer Educ 2009;24: 154–9. [31] Lienard A, Merckaert I, Libert Y, Bragard I, Delvaux N, Etienne AM, et al. Is it possible to improve residents breaking bad news skills? A randomised study assessing the efficacy of a communication skills training program. Br J Cancer 2010;103:171–7. [32] Amiel GE, Ungar L, Alperin M, Baharier Z, Cohen R, Reis S. Ability of primary care physician’s to break bad news: a performance based assessment of an educational intervention. Patient Educ Couns 2006;60:10–5. [33] Hobgood C, Harward D, Newton K, Davis W. The educational intervention GRIEV_ING improves the death notification skills of residents. Acad Emerg Med 2005;12:296–301. [34] Hobgood CD, Tamayo-Sarver JH, Hollar Jr DW, Sawning S. Griev_Ing: death notification skills and applications for fourth-year medical students. Teach Learn Med 2009;21:207–19. [35] Hobgood C, Mathew D, Woodyard DJ, Shofer FS, Brice JH. Death in the field: teaching paramedics to deliver effective death notifications using the educational intervention GRIEV_ING. Prehosp Emerg Care 2013;17: 501–10. [36] http://dconnect.acgme.org/outcome/implement/interperComSkills.pdf. [37] Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 1959;56:81–105. [38] Campbell DT. Recommendations for APA test standards regarding construct, trait, or discriminant validity. Am Psychol 1960;15:546–53. [39] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. [40] Schildmann J, Kupfer S, Burchardi N, Vollmann J. Teaching and evaluating breaking bad news: a pre-post evaluation study of a teaching intervention for medical students and a comparative analysis of different measurement instruments and raters. Patient Educ Couns 2012;86:210–9.
Contents lists available at ScienceDirect
Patient Education and Counseling journal homepage: www.elsevier.com/locate/pateducou
Medical Education
Breaking bad news is a teachable skill in pediatric residents: A feasibility study of an educational intervention Suzanne Reed a,*, Karyn Kassis a, Rollin Nagel b, Nicole Verbeck b, John D. Mahan a, Richard Shell a a b
Department of Pediatrics, The Ohio State University College of Medicine, Nationwide Children’s Hospital, Columbus, USA The Ohio State University College of Medicine, Columbus, USA
A R T I C L E I N F O
A B S T R A C T
Article history: Received 8 October 2014 Received in revised form 20 January 2015 Accepted 15 February 2015
Objective: Patients and physicians identify communication of bad news as a skill in need of improvement. Our objectives were to measure change in performance of first-year pediatric residents in the delivery of bad news after an educational intervention and to measure if changes in performance were sustained over time. Methods: Communication skills of 29 residents were assessed via videotaped standardized patient (SP) encounters at 3 time points: baseline, immediately post-intervention, and 3 months post-intervention. Educational intervention used was the previously published ‘‘GRIEV_ING Death Notification Protocol.’’ Results: The intraclass correlation coefficient demonstrated substantial inter-rater agreement with the assessment tool. Performance scores significantly improved from baseline to immediate postintervention. Performance at 3 months post-intervention showed no change in two subscales and small improvement in one subscale. Conclusions: We concluded that breaking bad news is a complex and teachable skill that can be developed in pediatric residents. Improvement was sustained over time, indicating the utility of this educational intervention. Practice implications: This study brings attention to the need for improved communication training, and the feasibility of an education intervention in a large training program. Further work in development of comprehensive communication curricula is necessary in pediatric graduate medical education programs. ß 2015 Elsevier Ireland Ltd. All rights reserved.
Keywords: Medical education Breaking bad news Pediatrics
1. Introduction Good communication skills are critical to the practice of excellent and effective medicine [1–3]. This includes the delicate and complex skill of breaking bad news. These skills are so critical that breaking bad news poorly can negatively affect patient relationships and have lasting impact on patients and families [1,2,4–6]. It is clear that patients [3,6–9] and physicians [1,9–14] still feel that many physicians struggle to achieve competence in these skills. Even for experienced clinicians, delivering bad news is a source of considerable stress [9,10,15]. Physiologic stress
* Corresponding author at: Nationwide Children’s Hospital, Division of Hematology/Oncology/BMT, 700 Children’s Drive, Columbus, OH 43205, USA. Tel.: +1 614 722 6349; fax: +1 614 722 3559. E-mail address: [email protected] (S. Reed). http://dx.doi.org/10.1016/j.pec.2015.02.015 0738-3991/ß 2015 Elsevier Ireland Ltd. All rights reserved.
responses have even been noted in both novice and experienced physicians in simulated patient encounters that involve breaking bad news [15,16]. Competence in breaking bad news is not always associated with increased experience. One study compared communication competency in medical students, residents, and attending physicians. Interestingly, while novice medical students had the lowest competence, no difference was found among more senior students, residents, and attending physicians [17]. Another study of novice and experienced physicians during a simulated bad news encounter found experience was inversely related to stress response, but performance was inversely related to burnout and fatigue, not inexperience alone [18]. Many residency and fellowship programs lack formal instruction on the delivery of bad news [19–22]. Traditional medical education focused on mastery of pathology and disease treatment may result in inadequate teaching and evaluation of less tangible
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
skills (e.g., communication) necessary for excellent patient care. While the Accreditation Council for Graduate Medical Education (ACGME) clearly prioritizes interpersonal skills and communication as one of the core competencies, and requires residents to achieve competence in this area [23–26], programs define ‘‘competence’’ differently, creating inconsistent education and evaluation. Much of the literature evaluating the education of physicians on the delivery of bad news is based on attitudes [13,9,27,28]. Most publications report that trainees do feel more comfortable with delivering bad news after an educational intervention [27]. There are few studies, however, that evaluate the objective skill of trainees in the delivery of bad news and/or the effectiveness of an educational intervention to improve bad news delivery. A few studies demonstrate improvement in this skill after an educational intervention for oncology trainees [29–31] and family practice trainees [32], but there is little evidence in pediatric trainees. Residents have indicated less preparation and training in breaking bad news in the pediatric setting compared to adult patient encounters [11]. The objectives for this study were to evaluate the acquisition and retention of breaking bad news skills, specifically telling a parent or grandparent that a child has died, in pediatric residents after an educational intervention. 2. Methods 2.1. Trainee population All first-year pediatrics and combined internal medicine/ pediatrics residents at our institution, a free-standing tertiary care children’s hospital, were invited to participate. There were no specific selection criteria, only availability and willingness to participate. Consent was obtained prior to baseline assessment. Institutional Review Board approval was obtained prior to standardized patient (SP) encounters. 2.2. Rationale for communication curriculum selected The teaching intervention selected for this study was ‘‘The GRIEV_ING Death Notification Protocol,’’ designed and published by Cherri Hobgood et al. [33] This curriculum is a 2-h educational experience, which includes short didactics, small group discussion, and role-play. Also included in the protocol are SP encounters, with case scenarios and detailed SP instructions (complete protocol can be accessed publicly, www.acep.org). This protocol was selected for several reasons. First, it is a complete curriculum with SP encounters and assessment tools, ready to use. Second, it includes a pediatric-specific SP encounter and SP training instructions. Third, this protocol focuses on only one type of very challenging ‘‘bad news:’’ death notification. Finally, the GRIEV_ING Protocol has demonstrated effectiveness in emergency medicine residents, medical students, and emergency medical service providers [33–35], and is supported as a resource by the ACGME Outcome Project [36]. Further details of the complete educational intervention will be provided on request. 2.3. SP encounters and delivery of educational intervention Residents skills in delivering bad news were assessed with an SP encounter at three separate time points: before teaching intervention (pre-test), immediately post-intervention (post-test), and 3 months post-intervention (post-post-test). Trained standardized patients were utilized from the Clinical Skills Education and Assessment Center at the Ohio State University College of Medicine (OSUCOM). Compensation for SPs was provided by a small OSUCOM grant. All simulated encounters were videotaped in unused patient rooms.
749
Baseline skills in delivering bad news were assessed with an SP encounter 4 months into the first year of pediatrics training. Next, the didactics, small group discussion, and role-playing of the GRIEV_ING curriculum were delivered at a resident retreat. The second encounter (post-test) occurred 1–2 weeks later and was identical in structure to the pre-test SP encounter, with a different death notification scenario. The final SP encounter (post-post-test) was completed approximately 3 months after the intervention using a different death notification scenario. Four investigators independently scored the pre-test videotaped encounters; two of the four investigators independently scored post-test and postpost-test videotaped encounters. 2.4. The cases The GRIEV_ING Protocol includes death notification cases for SP encounters and SP training instructions. We used one case exactly as written and modified the other two cases to present pediatric patients. 2.5. The assessments The GRIEV_ING Protocol also includes an assessment tool to measure competence. This 27-item instrument was developed to cover ten competency areas resulting from a literature review of death notification and were components of the GRIEV_ING educational intervention [33]. Additional published validity evidence for this instrument is limited: there was significant improvement for pre to post-intervention for emergency medicine residents using the full assessment tool [33] and significant improvement in medical students using an abbreviated 12-item version [34]. For the 12-item version the internal consistency was 0.56 and 0.65 and correlated significantly with interpersonal communication [34]. After preliminary evaluation of baseline assessments, we modified the assessment process by dividing items into subscales: Preparation (9 items, including clear introduction of self and role, and determining parent/grandparent’s level of knowledge of the event prior to news disclosure), bad news delivery (12 items, such as use of clear language without medical jargon, and use of silence to allow parent/grandparent reaction), and Wrap-up (8 items, such as addressing specific details of next steps (like organ donation), and providing personal availability for follow-up questions/needs). This determination of subscales evolved from faculty expert consensus and Pearson coefficients of pre-test scores indicating that subscales appeared to measure different components of the breaking bad news skill. All items of the original assessment tool were retained in the division into subscales. 2.6. Data analysis The intraclass correlation coefficient (ICC) was used to assess rater reliability. The ICC reflects both the degree of correspondence and agreement between raters and ranges from 0.00 to 1.00 (1.00 being perfect agreement). ICC Model 2,1 was used to analyze confidence of results from a randomly chosen rater. ICC Model 2,4 and Model 2,2 were calculated to clarify changes in reliability when using means across four and two raters, respectively (as opposed to a single randomly chosen rater), to assess performance. Each rater scored each assessment item on a yes/no basis (‘‘1’’ = performed item, ‘‘0’’ = did not perform item). Scores were calculated based on percentage of items performed correctly. Pearson’s correlation coefficients were used to calculate the association between subscale scores. Evidence of discriminant validity requires that subscales not correlate too highly if measuring different things [37,38]. Repeated measures ANOVAs
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
750
Subscale Means (Preparation, Bad News Delivery, and Wrap-up) over Assessments (N = 29) 100 90 80
% Correct
were used to compare trainee scores across subscales for each assessment period and across assessment periods for each subscale. For significant within-subject effects, post hoc contrast effects were calculated to determine significance of within-subject factor differences for each subscale or assessment period. 3. Results 3.1. Participation
70 60
Preparaon
50 40
Bad News Delivery Wrap-up
30
Of 44 first-year residents (34 pediatrics, 10 medicine-pediatrics), 29 (27 pediatrics, 2 medicine-pediatrics) participated in all three phases of the study. No residents voluntarily declined, however limitations in availability due to clinical obligations prevented some residents from participating in one or more SP encounters.
20 10 0 Pre-test
Post-test
Post-post-test
Fig. 1. Subscale means (Preparation, Bad News Delivery, and Wrap-up) over assessments (N = 29) % correct.
3.2. Inter-rater reliability Four raters independently assessed the pre-test SP encounters. The ICC 2,1 values, to evaluate inter-rater reliability of a randomly selected single rater, were 0.46 for Preparation, 0.40 for Bad News Delivery, and 0.36 for Wrap-up indicating at best a moderate correlation (0.41–0.60) or fair correlation (0.21–0.40), per Landis and Koch’s characterization [39]. The ICC 2,4 values, to evaluate the inter-rater reliability when all four raters’ scores were averaged, were 0.78 for Preparation, 0.73 for Bad News Delivery, and 0.70 for Wrap-up indicating substantial correlations (per Landis and Koch: 0.61–0.80) for all three subscales. Using the average of two raters continued to provide substantial correlations for each of the pretest SP encounter subscales, with ICC’s 2,2 of 0.63 for Preparation, 0.63 for Bad News Delivery, and 0.64 for Wrap-up. These two raters then independently scored all of the remaining encounters. Using the average of these two raters scores typically yielded ICC’s 2,2 in the substantial range (for Preparation and Bad News Delivery) and even the perfect range (0.81–1.00) for Wrap-up. Only the post-test ICC 2,2 for Bad News Delivery was only fair (0.39). 3.3. Subscale scores: pre-test Pre-test scores were low on all three subscales, with residents consistently scoring below 50% (see Table 1). Comparisons between subscales (F = 67.39, p < .001) and contrast tests demonstrated significant differences between each subscale (p < .001) with Wrap-up yielding the lowest scores (M = 18.53, SD = 10.10). Pearson coefficients demonstrated a lack of association among subscales ((r = 0.28, p = 0.15) between Preparation and Bad News Delivery, (r = .0.04, p = .84) between Preparation and Wrap-up, Table 1 Mean percent of items performed correctly with differences between and within subscales and assessments (N = 29). Assessment
Pre-testy Post-testy Post-post-testy
Subscale Preparation* Mean (SD)
Bad News Delivery* Mean (SD)
Wrap-up* Mean (SD)
38.31A,1 (12.45) 67.62B,1,a (13.11) 69.73B,1 (12.82)
52.44A,2 (12.67) 73.99B,1,b (7.93) 77.73B,2,c (7.97)
18.53A,3 (10.09) 43.97B,3 (18.26) 42.89B,3 (16.93)
Note: Post hoc tests: Means in columns with different capital superscripts are significantly different at p < .01. Means in rows with different numerical superscripts are significantly different at p < .01. Means in columns with different small superscripts are significantly different at p < .05. Means in rows with different small superscripts are significantly different at p < .05. * p < .001. Repeated measures ANOVA between assessments for the indicated subscale. y p < .001. Repeated measures ANOVA between subscales for the indicated assessment.
and (r = 0.03, p = .86) between Bad News Delivery and Wrap-up), further supporting the use of subscale scores to detect real differences in performance. 3.4. Post-test and post-post-test scores Post-test and post-post-test scores were calculated for each subscale (Table 1). Repeated measure ANOVA tests indicated significant differences between subscales for both the post-test (F = 38.20, p < .001) and post-post-test (F = 60.57, p < .001). Contrast tests for the post-subtests demonstrated significant differences between Preparation and Bad News Delivery (p = .015), Bad News Delivery and Wrap-up (p < .001), and Preparation and Wrap-up (p < .001). Contrast tests for the postpost-subtests demonstrated similar significant differences between subscales: Preparation and Bad News Delivery (p = .002), Bad News Delivery and Wrap-up (p < .001), and Preparation and Wrap-up (p < .001). Correlations between subscales for each test administration continued to demonstrate a lack of correlation (post-test: Preparation and Bad News Delivery (r = 0.28, p = .14, Bad News Delivery and Wrap-up (r = 0.13, p = .51, Preparation and Wrap-up (r = 0.05, p = .79); post-post-test: Preparation and Bad News Delivery (r = 0.33, p = .08, Bad News Delivery and Wrapup (r = 0.21, p = .27, Preparation and Wrap-up (r = 0.12, p = .54). 3.5. Comparisons across test administrations Repeated measures ANOVAs for each subscale across test administrations demonstrated differences for the Preparation (F = 63.46, p < .001), Bad News Delivery (F = 67.49, p < .001), and Wrap-up (F = 26.80, p < .001) subscales. Contrast tests demonstrated significant post-test improvement for all three subscales (all p’s < .001) (see Table 1). Post-test scores demonstrated percent improvement from pre-test scores of 41% (Bad News Delivery) to 137% (Wrap-up). The 3-month post-test subscale scores continued to demonstrate significant improvement compared to pre-test scores (all p’s < .001). Performance on the post-post-test Bad News Delivery demonstrated significant improvement (p = .035) compared to post-test scores, however, no differences between post-test and post-post-tests were evident for the Preparation (p = .50) or Wrap-up (p = .76) scales (see Fig. 1). 4. Discussion and conclusions 4.1. Discussion This study demonstrated that first-year pediatric residents’ communication skills when notifying parents or grandparents of a
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
child’s death significantly improved following an educational intervention based on the GRIEV_ING Protocol. The residents maintained this performance improvement for 3 months. These results suggest this skill is teachable and retainable, and illustrate the need and value of increased education and development of this complex skill. Our results suggest it would be short-sighted to rely on inherent communication skills alone for proficiency in death notification communication. This study also supplies evidence for the validity of the assessment tool and its division into three subscales. Subscale analysis at the pre-test, post-test, and 3-month follow-up assessments demonstrated discriminant validity, performance differences and lack of association between subscales. Future studies with larger sample sizes could use factor analysis to further assess subscales. The discreteness of the three subscales suggests that communication skills needed for death notification are complex and varied. This provides the opportunity for a more individualized approach to death notification education, and offers a future direction for enhanced bad news training. This is the first study examining death notification skills in pediatric residents. The improvement in performance and retention of skills after an educational intervention are encouraging. While this particular intervention focused on death notification specifically, it included concepts and techniques that can be applied to delivery of other types of bad news. The primary care pediatrician is not likely to deliver death news, however the delivery of bad news is a responsibility that crosses all general pediatric practices and subspecialties. Our results suggest that a modest investment by a residency program (in time, effort, and cost) can produce meaningful results and retention of widely applicable skills. Pediatric physician communication is somewhat unique, relative to other specialties. The communication exchange may have a wide range of patient involvement, and breaking bad news can be more sensitive for both patients and families. The relative infrequency of opportunities for trainees to practice breaking bad news is an added complication. Limited exposure for pediatrics trainees underscores the need for simulated training and experiential learning to supplement these less common real-life learning experiences. Our study demonstrated improved, but not perfect, communication skills after an educational intervention. While we did not set a specific ‘‘competence’’ score, on the post-test almost onethird of the residents scored below 70% on Preparation and Bad News Delivery, and most failed to reach 70% for Wrap-up. A need remains for additional skill improvement among trainees. This raises the question of ‘‘dose’’ and ‘‘effect.’’ Perhaps a larger ‘‘dose’’ of education (a longer workshop, more role-play, etc.) or repeated interventions (refresher workshops, etc.) could increase the degree of improvement from baseline. Future studies will be required to answer these important questions. There are several limitations of our study, including small sample size, single institution, and lack of control group. We could not account for differences in trainees’ previous experiences with breaking bad news and/or death notification. There is evidence of some medical student training in breaking bad news [33,34,40], suggesting that at least some medical students have exposure to this subject, though these experiences are poorly standardized. We did not survey our trainees about past bad news delivery training. Another limitation is that we did not account for general professional growth that typically occurs during the first year of residency. Exposure to another physician communicating death news is possible in the first year of residency, and it is unknown the effect, if any, of these observational experiences.
751
4.2. Conclusions In summary, this feasibility study implementing a modified intervention for pediatric trainees demonstrates that death notification is a difficult and complex, but teachable, skill. These pilot data support the need for further investigation. Our data also suggest that this complex skill can be broken down into more specific components, creating the potential for individualized training, to allow each trainee to reach his or her maximum potential in this skill. 4.3. Practice implications This project successfully applied the GRIEV_ING Protocol to teach communication skills for breaking bad news to pediatric residents, and provides preliminary data to inform and support more comprehensive communication curricula in pediatric graduate medical education programs. Future studies should focus on assessment of long-term retention of skills and on teaching other aspects of breaking bad news, with the goal of developing pediatricians with high levels of skill in communication. Financial disclosure The authors have no financial relationships relevant to this article to disclose. Conflict of interest The other authors have no conflicts of interest to disclose. References [1] Munoz Sastre MT, Sorum PC, Mullet E. Breaking bad news: the patient’s viewpoint. Health Commun 2011;26:649–55. [2] Ptacek JT, Ellison NM. Health care providers’ perspectives on breaking bad news to patients. Crit Care Nurs Q 2000;23:51–9. [3] Trudel JG, Leduc N, Dumont S. Perceived communication between physicians and breast cancer patients as a predicting factor of patients’ health-related quality of life: a longitudinal analysis. Psychooncology 2013. [4] Ptacek JT, Eberhardt TL. Breaking bad news. A review of the literature. J Am Med Assoc 1996;276:496–502. [5] Ptacek JT, Fries EA, Eberhardt TL, Ptacek JJ. Breaking bad news to patients:[ physicians’ perceptions of the process. Support Care Cancer 1999;7:113–20. [6] Ptacek JT, Ptacek JJ. Patients’ perceptions of receiving bad news about cancer. J Clin Oncol 2001;19:4160–4. [7] Mack JW, Hilden JM, Watterson J, Moore C, Turner B, Grier HE, et al. Parent and physician perspectives on quality of care at the end of life in children with cancer. J Clin Oncol 2005;23:9155–61. [8] Mazor KM, Simon SR, Yood RA, Martinson BC, Gunter MJ, Reed GW, et al. Health plan members’ views on forgiving medical errors. Am J Manag Care 2005;11: 49–52. [9] Orgel E, McCarter R, Jacobs S. A failing medical educational model: a selfassessment by physicians at all levels of training of ability and comfort to deliver bad news. J Palliat Med 2010;13:677–83. [10] Birkeland AL, Dahlgren L, Ha¨gglo¨f B, Rydberg A. Breaking bad news: an interview study of paediatric cardiologists. Cardiol Young 2011;21:286–91. [11] Dube CE, LaMonica A, Boyle W, Fuller B, Burkholder GJ. Self-assessment of communication skills preparedness: adult versus pediatric skills. Ambul Pediatr 2003;3:137–41. [12] Durall A, Zurakowski D, Wolfe J. Barriers to conducting advance care discussions for children with life-threatening conditions. Pediatrics 2012;129: e975–82. [13] Harrison ME, Walling A. What do we know about giving bad news? A review. Clin Pediatr (Phila) 2010;49:619–26. [14] Sweeny K, Shepperd JA, Han PK. The goals of communicating bad news in health care: do physicians and patients agree? Health Expect 2011. [15] Hulsman RL, Pranger S, Koot S, Fabriek M, Karemaker JM, Smets EM. How stressful is doctor-patient communication? Physiological and psychological stress of medical students in simulated history taking and bad-news consultations. Int J Psychophysiol 2010;77:26–34. [16] Meunier J, Merckaert I, Libert Y, Delvaux N, Etienne AM, Lie´nard A, et al. The effect of communication skills training on residents’ physiological arousal in a breaking bad news simulated task. Patient Educ Couns 2013;93:40–7. [17] Wouda JC, van de Wiel HB. The communication competency of medical students: residents and consultants. Patient Educ Couns 2012;86:57–62.
752
S. Reed et al. / Patient Education and Counseling 98 (2015) 748–752
[18] Brown R, Dunn S, Byrnes K, Morris R, Heinrich P, Shaw J. Doctors’ stress responses and poor communication performance in simulated bad-news consultations. Acad Med 2009;84:1595–602. [19] Hebert HD, Butera JN, Castillo J, Mega AE. Are we training our fellows adequately in delivering bad news to patients? A survey of hematology/ oncology program directors. J Palliat Med 2009;12:1119–24. [20] File W, Bylund CL, Kesselheim J, Leonard D, Leavey P. Do pediatric hematology/ oncology (PHO) fellows receive communication training? Pediatr Blood Cancer 2013. [21] Kersun L, Gyi L, Morrison WE. Training in difficult conversations: a national survey of pediatric hematology–oncology and pediatric critical care physicians. J Palliat Med 2009;12:525–30. [22] Turner DA, Mink RB, Lee KJ, Winkler MK, Ross SL, Hornik CP, et al. Are pediatric critical care medicine fellowships teaching and evaluating communication and professionalism? Pediatr Crit Care Med 2013;14:454–61. [23] Green ML, Holmboe E. Perspective: the ACGME toolbox: half empty or half full? Acad Med 2010;85:787–90. [24] Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach 2007;29:648–54. [25] Hicks PJ, Schumacher DJ, Benson BJ, Burke AE, Englander R, Guralnick S, et al. The pediatrics milestones: conceptual framework, guiding principles, and approach to development. J Grad Med Educ 2010;2:410–8. [26] Rushton JL, Hicks PJ, Carraccio CL. The next phase of pediatric residency education: the partnership of the Milestones Project. Acad Pediatr 2010;10:91–2. [27] Rosenbaum ME, Ferguson KJ, Lobas JG. Teaching medical students and residents skills for delivering bad news: a review of strategies. Acad Med 2004;79: 107–17. [28] Rosenbaum ME, Wilson JF, Sloan DA. Clinical instruction for delivering bad news. Acad Med 1996;71:529. [29] Back AL, Arnold RM, Baile WF, Fryer-Edwards KA, Alexander SC, Barley GE, et al. Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Arch Intern Med 2007;167:453–60.
[30] Eid A, Petty M, Hutchins L, Thompson R. Breaking bad news: standardized patient intervention improves communication skills for hematologyoncology fellows and advanced practice nurses. J Cancer Educ 2009;24: 154–9. [31] Lienard A, Merckaert I, Libert Y, Bragard I, Delvaux N, Etienne AM, et al. Is it possible to improve residents breaking bad news skills? A randomised study assessing the efficacy of a communication skills training program. Br J Cancer 2010;103:171–7. [32] Amiel GE, Ungar L, Alperin M, Baharier Z, Cohen R, Reis S. Ability of primary care physician’s to break bad news: a performance based assessment of an educational intervention. Patient Educ Couns 2006;60:10–5. [33] Hobgood C, Harward D, Newton K, Davis W. The educational intervention GRIEV_ING improves the death notification skills of residents. Acad Emerg Med 2005;12:296–301. [34] Hobgood CD, Tamayo-Sarver JH, Hollar Jr DW, Sawning S. Griev_Ing: death notification skills and applications for fourth-year medical students. Teach Learn Med 2009;21:207–19. [35] Hobgood C, Mathew D, Woodyard DJ, Shofer FS, Brice JH. Death in the field: teaching paramedics to deliver effective death notifications using the educational intervention GRIEV_ING. Prehosp Emerg Care 2013;17: 501–10. [36] http://dconnect.acgme.org/outcome/implement/interperComSkills.pdf. [37] Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 1959;56:81–105. [38] Campbell DT. Recommendations for APA test standards regarding construct, trait, or discriminant validity. Am Psychol 1960;15:546–53. [39] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. [40] Schildmann J, Kupfer S, Burchardi N, Vollmann J. Teaching and evaluating breaking bad news: a pre-post evaluation study of a teaching intervention for medical students and a comparative analysis of different measurement instruments and raters. Patient Educ Couns 2012;86:210–9.
