EDUCATION AND TRAINING
Standardized Patient and Standardized Interdisciplinary Team Meeting: Validation of a New Performance-Based Assessment Tool Misuzu Yuasa, MD,* Michael Nagoshi, MD,*† Celeste Oshiro-Wong, MPH,† Maung Tin, MD,* Aida Wen, MD,* and Kamal Masaki, MD*
The interdisciplinary team (IDT) approach is critical in the care of elderly adults. Performance-based tools to assess IDT skills have not been well validated. A novel assessment tool, the standardized patient (SP) and standardized interdisciplinary team meeting (SIDTM), consisting of two stations, was developed. First, trainees evaluate a SP hospitalized after a fall. Second, trainees play the role of the physician in a standardized IDT meeting with a standardized registered nurse (SRN) and standardized medical social worker (SMSW) for discharge planning. The SP-SIDTM was administered to 52 fourth-year medical students (MS4s) and six geriatric medicine fellows (GMFs) in 2011/12. The SP, SRN, and SMSW scored trainee performance on dichotomous checklists of clinical tasks and Likert scales of communication skills, which were compared according to level of training using t-tests. Trainees rated the SP-SIDTM experience as moderately difficult, length of time about right, and believability moderate to high. Reliability was high for both cases (Cronbach a = 0.73–0.87). Interobserver correlation between SRN and SMSW checklist scores (correlation coefficient (r) = 0.82, P < .001) and total scores (r = 0.69, P < .001) were high. The overall score on the SP-SIDTM case was significantly higher for GMF (75) than for MS4 (65, P = .002). These observations support the validity of this novel assessment tool. J Am Geriatr Soc 62:171–174, 2014.
Key words: geriatric education; clinical skills evaluation; standardized patients; interdisciplinary team; validation of assessment tool
T
wenty percent of the U.S. population will be aged 65 and older by 2030, creating multiple challenges for the healthcare system. Multiple chronic conditions coupled with acute illness; diverse living arrangements; and a variable range of economic, physical, and cognitive abilities makes care of older adults complex and time intensive. This complexity of care requires the expertise of multiple disciplines working in teams to improve the health outcomes of these individuals.1,2 Medical schools and postgraduate training programs must ensure that trainees have achieved the appropriate competencies to function well as members of an interdisciplinary team (IDT).3 At the John A. Burns School of Medicine, University of Hawaii, Honolulu, Hawaii, required geriatrics educational activities were added to the curriculum in all 4 years of medical school and all residency training programs as part of the Donald W. Reynolds training grant in 2001 to 2006. Since the 2005/06 academic year, the 4th-year medical student curriculum has culminated in a required 4-week rotation in geriatrics and palliative medicine. As part of a new grant from the Donald W. Reynolds Foundation, these curricula are being modified to include principles of IDT care. This article describes validation of a new performance-based assessment tool for measuring the ability of trainees to function in an IDT.
METHODS Curriculum
From the *Department of Geriatric Medicine, John A. Hartford Foundation Center of Excellence in Geriatrics, and †Center for Clinical Skills, Office of Medical Education, John A. Burns School of Medicine, University of Hawaii, Honolulu, Hawaii. Address correspondence to Michael Nagoshi, 347 N. Kuakini St., HPM-9, Honolulu, HI 96817, Hawaii. E-mail: [email protected] DOI: 10.1111/jgs.12604
JAGS 62:171–174, 2014 © 2013, Copyright the Authors Journal compilation © 2013, The American Geriatrics Society
The curriculum for the 4th-year medical students (MS4s) geriatrics and palliative medicine rotation consists of didactic sessions and clinical site experiences. Students are assessed with a geriatrics standardized patient examination (GSPX) and a final multiple-choice written examination. At clinical sites, students participate in IDT geriatric care and are required to use the Geriatric Interdisciplinary Care Summary (GICS) as a framework for recording their history and physical examinations on older adults.4 The GICS
0002-8614/14/$15.00
172
YUASA ET AL.
assists students in developing a systemic and individualized approach to geriatric syndromes and in managing individuals with multiple complex problems. Geriatric medicine fellows (GMFs) are also required to complete a GSPX midway through their first year of training.
Curriculum Development and Validation To assess a student’s ability to function as the physician on an interdisciplinary team, a new evaluation tool was developed consisting of two consecutive cases: a standardized patient (SP) case and a standardized interdisciplinary team meeting (SIDTM) to discuss care planning related to the first case. The following requirements guided development of the SP-SIDTM: Each student must interview and examine the individual they would be discussing in the IDT, the number of “team” members must be kept small to decrease training and administration costs, the stations must fit in with the logistics of the GSPX, the situation created would be one that most students encounter in training, the experience must feel “real” to the student, and the interaction between IDT members must be perceived as being authentic. The SP-SIDTM was developed as two consecutive 15-minute stations with accompanying 10-minute written interstation exercises (ISEs). The two stations were formatted to be incorporated within a multistation SP-based clinical skills evaluation. In the first station, trainees evaluate a SP hospitalized after a fall. The second station requires trainees to discuss discharge planning for the individual seen in the previous station in a standardized IDT meeting with a standardized registered nurse (SRN) and standardized medical social worker (SMSW). The trainee is evaluated in the role of the physician. SPs rated components of examinees’ interpersonal and communication skills on a seven-item, 5-point Likert-type scale. History taking, physical examination skills, counseling, and interdisciplinary team tasks were recorded on carefully constructed dichotomous checklists (CHKLST). The SRN and SMSW were instructed to avoid discussing examinee performance and to rate each student independently. Geriatricians developed the SP checklists using a case-development process previously described.5 An experienced SP trainer trained five pairs of volunteers to play the SRN and SMSW using standardized training materials. None of the volunteers were actual practicing nurses or social workers. After approximately 1 hour of onsite coaching, the volunteers were asked to memorize their roles before returning several days later for a second training session. During the second session, they performed their roles with a faculty member playing the role of the student until accuracy of portrayal and checklist completion were satisfactory. The checklists and training materials are available by request through The Portal of Geriatrics Online Education (www.pogoe.org) website.6 The SIDTM checklists and training materials accounted for interaction between the examinee and each of the team members and interaction between team members. Care was taken to ensure that facilitating discussion was measured rather than the simple presentation of information. For example, the trainee would receive credit for addressing pain management only if the SRN was asked for her
JANUARY 2014–VOL. 62, NO. 1
JAGS
opinion. The ISE after the SP encounter measured the examinee’s ability to synthesize clinical information related to falls. After the SIDTM station, examinees were required to construct a care plan using a modified version of the Geriatric Interdisciplinary Care Summary (GICS) template.4 The GICS was developed at the College of Medicine, University of Florida, and is a grid that learners use to integrate their knowledge and skills into an overall management plan for a medically complex older adult. The GICS grid consists of eight major care domains: physical medicine and rehabilitation, cognitive, emotional, medical and surgical, nutritional, environmental, social and caregiver, and economic. Using the GICS, learners identify health disciplines addressing these problems and formulate achievable treatment goals. Faculty members scored both ISEs against predetermined objective grading criteria. During the 2011/12 academic year, the SP-SIDTM was administered to 52 MS4s and six GMFs as part of their required GSPX. The students completed the GSPX at the end of their 4th-year geriatrics rotation. The fellows completed the GSPX in a single administration midway through their first year of fellowship. Two sets of six trainees completed the examination on each day. The students received aggregated scores across all stations as part of their evaluation for the rotation but were not provided specific feedback on their performance in each individual station. No attempts were made to prevent the students from discussing the stations with the students who had not yet taken the examination. For the SP and SIDTM stations, component scores were calculated as the percentage of correct items achieved. The total score for each station was calculated as the weighted average of component scores (CHKLST 50%, ISE or GICS 30%, IPS/COMM 20%). Combined SP-SIDTM total and component scores were calculated as the average of scores for both stations. Immediately after each station, examinees rated the difficulty of the case (minimal to extreme), case length (too short to too long), and the believability of the SP-SIDTM experience (not at all believable to extremely believable) on 7-point Likerttype scales. They also rated their prior experience with similar patients on a 5-point scale (never to ≥6 times).
Statistical Methods Reliability of checklists and rating scales was measured using Cronbach alpha. Differences between the groups of examinees were measured using the t-test for independent samples. Correlations between variables were measured using Pearson correlation coefficients. The University of Hawaii committee on human studies approved use of the de-identified learner data for this research as exempt from federal regulations pertaining to the protection of human research participants.
RESULTS The mean of total scores were higher for GMFs than MS4s on the SP station (72 vs 67, P = .13) and the SIDTM station (75 vs 65, P = .003), although the difference was statistically significant only for the SIDTM station (Table 1). The average component scores (CHKLST,
JAGS
JANUARY 2014–VOL. 62, NO. 1
VALIDATION OF A STANDARDIZED IDT MEETING
Table 1. Combined Scores According to Level of Training: Standardized Patient (SP) Only, Standardized Interdisciplinary Team Meeting (SIDTM) Only and SP-SIDTM
Checklista
Communication Skillsb
Patient Notec or Geriatric Interdisciplinary Care Summary
Total Scored
Mean Standard Deviation
Tool SP only MS4s 73 10 GMFs 77 6 P-valuee .38 SP-SIDTM only MS4s 70 12 GMFs 82 10 .03 P-valuee SP-SIDTM combinedf MS4s 72 9 GMFs 79 7 .04 P-valuee
88 7 94 3 .06
43 14 49 4 .36
67 7 72 4 .13
85 10 92 7 .09
43 10 54 13 .02
65 8 75 9 .003
85 10 92 7 .09
55 6 63 8 .006
66 5 74 6 .002
a
Percentage of checklist items correctly achieved. Percentage of points received on 15 communication items (maximum 4 points each). c Percentage of graded items achieved. d Weighted average of component scores (50% checklist + 30% patient note + 20% communication skills). e P-Values were calculated using t-tests comparing mean scores of 4th-year medical students (M4s; n = 52) and geriatric medicine fellows (GMFs; n = 6). f Average of SP and SIDTM scores. b
ISE or GICS, IPS/COMM) for the SIDTM station were significantly higher for GMFs (82, 54, 92, respectively) than for MS4s (70, 43, 85, respectively). The average component scores for the SP station were higher for GMFs than MS4s, but the differences were not statistically significant. The combined total score (Table 2) was significantly higher for GMFs than for MS4s (74 vs 66, P = .002). The combined checklist (79 vs 72, P = .04) and ISE/GICS (63 vs 55, P = .006) scores were also significantly higher for GMFs. The IPS/COMM scores were higher for GMFs, but
Table 2. Examinee Rating of Standardized Patient Standardized Interdisciplinary Team Meeting Experience Characteristics According to Level of Training Fourth-Year Medical Students Characteristic
Case difficultya Time allottedb Believabilityc Previous similar “real” experienced
Geriatric Medicine Fellows
Mean Standard Deviation
4.7 4.5 5.6 3.0
1.0 0.8 1.4 1.2
3.8 4.0 5.5 4.2
0.8 0.0 1.4 0.4
1 = minimal to 7 = extreme (4 = moderate). 1 = too short to 7 = too long (4 = about right). c 1 = not at all to 7 = extremely (4 = moderately). d 1 = never, 2 = once, 3 = twice, 4 = 3–5 times, 5 = ≥6 times. a
b
P-Value
.06 .13 .92 .03
173
the difference was not statistically significant (92 vs 85, P = .09). There was no difference in responses according to level of training for perception of case difficulty, time allotted for each case, or believability (Table 2). Overall, trainees rated the SP-SIDTM experience as moderately difficult, length of time as about right, and believability as moderate to extreme. More than 80% of trainees had at least one similar real patient experience. As expected, GMFs rated themselves as having significantly more previous similar real experience than MS4s. Reliability measured as Cronbach a was 0.780 for the SP station, 0.866 for the SIDTM station, and 0.871 for all graded items across both stations. Interobserver correlation between SRN and SMSW checklist scores (r = 0.82, P < .001) and total scores (r = 0.69, P < .001) were high. Correlation between SRN and SMSW communication skills scores was weaker (r = 0.45, P = .003).
DISCUSSION Medical schools and postgraduate training programs are integrating IDT skills within their curricula. Effective methods have been developed for teaching these skills, such as the Geriatric Interdisciplinary Team Training Program of the John A. Hartford Foundation.7–9 Methods to assess the acquisition of IDT skills are important to trainee feedback, curriculum evaluation, and public accountability. Validated instruments are available to assess interdisciplinary team knowledge, attitudes, and self-efficacy and have been used for programmatic evaluation.10–14 A previous study15 demonstrated the reliability and validity of a survey instrument for assessing interdisciplinary team performance in long-term care settings and in the Program of All-Inclusive Care of the Elderly program. Educational interventions with standardized patients or human patient simulators have been shown to improve interdisciplinary team attitudes and skills. The SP provides trainees with a patient to evaluate before collaborating on an interdisciplinary care plan. After the encounter, faculty observers and the SPs themselves provide feedback.16–20 Ideally, summative evaluation of IDT skills would require each examinee to participate individually in a standardized team meeting. The authors believe that this is the first report on the validation of a performance-based assessment of IDT skills using standardized team members from different disciplines. The SP-SIDTM achieved adequate statistical reliability as measured according to Cronbach alpha, and the SIDTM checklist showed good interobserver correlation. These findings suggest that IDT behaviors can be consistently measured and that overall student performance is likely to be stable with retesting. The lower interobserver correlation for communication skills checklist items was expected, because the SPs were not specifically trained to be consistent on these observations. Previous experience suggested that standardized patient examinations may not adequately distinguish levels of expertise.5 Scores on the SP component of the SP-SIDTM were not statistically different for GMFs and MS4s, although the observation that GMFs achieved significantly higher overall SP-SIDTM scores
174
YUASA ET AL.
provides evidence of construct validity and suggests that the SP-SIDTM distinguishes levels of learners. An IDT experience allowing trainees to evaluate a patient and subsequently play the role of the physician in an IDT setting was created. There were probably fewer team members than students encounter in actual clinical environments, but the small team allowed administrative costs (recruitment, training, and SP stipends) to be optimized. The students rated the time allotted for the SP-SIDTM as about right, and the difficulty of the case as moderate, suggesting that an assessment that readily fit into the current multistation SP-based GSPX was successfully created. The IDT scenario created appears to be one that most students encounter in training, as evidenced by the majority of MS4s and GMFs responding that they had had at least one similar experience in an actual patient care situation. In addition, MS4s and GMFs rated the experience as highly believable, suggesting that an experience that was authentic and felt “real” to the examinees was successfully created. Examinees with more training rated the examination as less difficult. Clinical experience with patients similar to the SP-SIDTM encountered on the examination increased with training. These observations lend additional credibility and support to the belief that the SP-SIDTM appears to evaluate what is intended. The lack of a criterion standard for measuring competence with these skills makes any further conclusions difficult. An important concern is whether the SIDTM checklist measures trainees’ facilitative skills as opposed to the general ability to discuss clinical problems in a group setting. The additional years of training between medical school and fellowship, allowing more time to develop these general abilities, could account for the difference in performance between GMFs and MS4s. Administering the SP-SIDTM to individuals in other disciplines who have specific training in facilitative skills may help to clarify this question. Further study of this question is planned. The number of trainees evaluated in this study was small. Whether the findings will hold up with subsequent MS4 and GMF cohorts or in other institutional settings remains to be seen, but the SP-SIDTM appears to be a promising innovative tool for the assessment of IDT skills.
CONCLUSION Trainees positively rated the SP-SIDTM, which generated scores with high reliability. Observations recorded on checklists showed excellent interobserver correlation. GMFs scored significantly higher than 4th-year medical students. These observations support the validity of this novel assessment tool.
ACKNOWLEDGMENTS Data from this research were presented at the American Geriatrics Society National Meeting, Grapevine, Texas, May 2013. Conflict of Interest: The editor in chief has reviewed the conflict of interest checklist provided by the authors and has determined that the authors have no financial or any other kind of personal conflicts with this paper. Drs. Masaki, Nagoshi, and Wen received grant funding
JANUARY 2014–VOL. 62, NO. 1
JAGS
from The Donald W. Reynolds Foundation Next Steps in Physicians’ Training in Geriatrics Grant. Dr. Masaki received grant funding from The John A. Hartford Foundation Center of Excellence in Geriatrics grant and the Pacific Islands Geriatric Education Center Health Resources and Services Administration Grant UB4HP19065. Author Contributions: Study concept and design: Nagoshi, Yuasa, Oshiro-Wong, Wen, Masaki. Acquisition of data: Nagoshi, Oshiro-Wong. Analysis and interpretation of data: Yuasa, Nagoshi, Masaki. Preparation of manuscript: Yuasa, Nagoshi, Masaki. Critical revision of manuscript: Yuasa, Nagoshi, Oshiro-Wong, Tin, Wen, Masaki. Sponsor’s Role: The funding sources had no role in the analysis or preparation of this manuscript.
REFERENCES 1. Mion L, Odegard PS, Resnick B et al. Interdisciplinary care for older adults with complex needs: American Geriatrics Society position statement. J Am Geriatr Soc 2006;54:849–852. 2. Position Statement on Interdisciplinary Team Training in Geriatrics: An Essential Component of Quality Healthcare for Older Adults. 2011 [online]. Available at www.americangeriatrics.org/files/documents/pha/PHA_ IDTstatement.pdf Accessed May 21, 2013. 3. Leipzig RM, Granville L, Simpson D et al. Keeping granny safe on July 1: A consensus on minimum geriatrics competencies for graduating medical students. Acad Med 2009;84:604–610. 4. Bautista MK, Meuleman JR, Shorr RI et al. Description and students’ perceptions of a required geriatric clerkship in postacute rehabilitative care. J Am Geriatr Soc 2009;57:1685–1691. 5. Nagoshi M, Williams S, Kasuya R et al. Using standardized patients to assess the geriatrics medicine skills of medical students, internal medicine residents, and geriatrics medicine fellows. Acad Med 2004;79:698–702. 6. Nagoshi M, Wong C, Tin MM. Standardized Interdisciplinary Team Meeting Case (Product 21230) [on-line]. Available at www.pogoe.org Accessed July 12, 2013. 7. Fulmer T, Flaherty E, Hyer K. The geriatric interdisciplinary team training (GITT) program. Gerontol Geriatr Educ 2003;24:3–12. 8. Fulmer T, Hyer K, Flaherty E et al. Geriatric interdisciplinary team training program: Evaluation results. J Aging Health 2005;17:443–470. 9. Djukic M, Fulmer T, Adams JG et al. NYU3T: Teaching, technology, teamwork: A model for interprofessional education scalability and sustainability. Nurs Clin North Am 2012;47:333–346. 10. Hyer K, Fairchild S, Abraham I et al. Measuring attitudes related to interdisciplinary training: Revisiting the Heinemann, Schmitt and Farrell ‘Attitudes Toward Health Care Teams’ Scale. J Interprof Care 2000; 14:249–258. 11. Clark PG, Leinhaas MM, Filinson R. Developing and evaluating an interdisciplinary clinical team training program: Lessons taught and lessons learned. Educ Gerontol 2002;28:491–510. 12. Clark PG. Evaluating an interdisciplinary team training institute in geriatrics: Implications for teaching teamwork theory and practice. Educ Gerontol 2002;28:511–528. 13. Luecht RM, Madsen MK, Taugher MP et al. Assessing professional perceptions: Design and validation of an Interdisciplinary Education Perception Scale. J Allied Health 1990;19:181–191. 14. McFadyen AK, Maclaren WM, Webster VS. The Interdisciplinary Education Perception Scale (IEPS): An alternative remodelled sub-scale structure and its reliability. J Interprof Care 2007;21:433–443. 15. Temkin-Greener H, Gross D, Kunitz SJ et al. Measuring interdisciplinary team performance in a long-term care setting. Med Care 2004;42:472–481. 16. Westberg SM, Adams J, Thiede K et al. An interprofessional activity using standardized patients. Am J Pharm Educ 2006;70:34. 17. Shrader S, McRae L, King WM 4th et al. A simulated interprofessional rounding experience in a clinical assessment course. Am J Pharm Educ 2011;75:61. 18. Vyas D, McCulloh R, Dyer C et al. An interprofessional course using human patient simulation to teach patient safety and teamwork skills. Am J Pharm Educ 2012;76:71. 19. MacDonnell CP, Rege SV, Misto K et al. An introductory interprofessional exercise for healthcare students. Am J Pharm Educ 2012;76:154. 20. Solomon P, Salfi J. Evaluation of an interprofessional education communication skills initiative. Educ Health (Abingdon, England) 2011;24:616.
Standardized Patient and Standardized Interdisciplinary Team Meeting: Validation of a New Performance-Based Assessment Tool Misuzu Yuasa, MD,* Michael Nagoshi, MD,*† Celeste Oshiro-Wong, MPH,† Maung Tin, MD,* Aida Wen, MD,* and Kamal Masaki, MD*
The interdisciplinary team (IDT) approach is critical in the care of elderly adults. Performance-based tools to assess IDT skills have not been well validated. A novel assessment tool, the standardized patient (SP) and standardized interdisciplinary team meeting (SIDTM), consisting of two stations, was developed. First, trainees evaluate a SP hospitalized after a fall. Second, trainees play the role of the physician in a standardized IDT meeting with a standardized registered nurse (SRN) and standardized medical social worker (SMSW) for discharge planning. The SP-SIDTM was administered to 52 fourth-year medical students (MS4s) and six geriatric medicine fellows (GMFs) in 2011/12. The SP, SRN, and SMSW scored trainee performance on dichotomous checklists of clinical tasks and Likert scales of communication skills, which were compared according to level of training using t-tests. Trainees rated the SP-SIDTM experience as moderately difficult, length of time about right, and believability moderate to high. Reliability was high for both cases (Cronbach a = 0.73–0.87). Interobserver correlation between SRN and SMSW checklist scores (correlation coefficient (r) = 0.82, P < .001) and total scores (r = 0.69, P < .001) were high. The overall score on the SP-SIDTM case was significantly higher for GMF (75) than for MS4 (65, P = .002). These observations support the validity of this novel assessment tool. J Am Geriatr Soc 62:171–174, 2014.
Key words: geriatric education; clinical skills evaluation; standardized patients; interdisciplinary team; validation of assessment tool
T
wenty percent of the U.S. population will be aged 65 and older by 2030, creating multiple challenges for the healthcare system. Multiple chronic conditions coupled with acute illness; diverse living arrangements; and a variable range of economic, physical, and cognitive abilities makes care of older adults complex and time intensive. This complexity of care requires the expertise of multiple disciplines working in teams to improve the health outcomes of these individuals.1,2 Medical schools and postgraduate training programs must ensure that trainees have achieved the appropriate competencies to function well as members of an interdisciplinary team (IDT).3 At the John A. Burns School of Medicine, University of Hawaii, Honolulu, Hawaii, required geriatrics educational activities were added to the curriculum in all 4 years of medical school and all residency training programs as part of the Donald W. Reynolds training grant in 2001 to 2006. Since the 2005/06 academic year, the 4th-year medical student curriculum has culminated in a required 4-week rotation in geriatrics and palliative medicine. As part of a new grant from the Donald W. Reynolds Foundation, these curricula are being modified to include principles of IDT care. This article describes validation of a new performance-based assessment tool for measuring the ability of trainees to function in an IDT.
METHODS Curriculum
From the *Department of Geriatric Medicine, John A. Hartford Foundation Center of Excellence in Geriatrics, and †Center for Clinical Skills, Office of Medical Education, John A. Burns School of Medicine, University of Hawaii, Honolulu, Hawaii. Address correspondence to Michael Nagoshi, 347 N. Kuakini St., HPM-9, Honolulu, HI 96817, Hawaii. E-mail: [email protected] DOI: 10.1111/jgs.12604
JAGS 62:171–174, 2014 © 2013, Copyright the Authors Journal compilation © 2013, The American Geriatrics Society
The curriculum for the 4th-year medical students (MS4s) geriatrics and palliative medicine rotation consists of didactic sessions and clinical site experiences. Students are assessed with a geriatrics standardized patient examination (GSPX) and a final multiple-choice written examination. At clinical sites, students participate in IDT geriatric care and are required to use the Geriatric Interdisciplinary Care Summary (GICS) as a framework for recording their history and physical examinations on older adults.4 The GICS
0002-8614/14/$15.00
172
YUASA ET AL.
assists students in developing a systemic and individualized approach to geriatric syndromes and in managing individuals with multiple complex problems. Geriatric medicine fellows (GMFs) are also required to complete a GSPX midway through their first year of training.
Curriculum Development and Validation To assess a student’s ability to function as the physician on an interdisciplinary team, a new evaluation tool was developed consisting of two consecutive cases: a standardized patient (SP) case and a standardized interdisciplinary team meeting (SIDTM) to discuss care planning related to the first case. The following requirements guided development of the SP-SIDTM: Each student must interview and examine the individual they would be discussing in the IDT, the number of “team” members must be kept small to decrease training and administration costs, the stations must fit in with the logistics of the GSPX, the situation created would be one that most students encounter in training, the experience must feel “real” to the student, and the interaction between IDT members must be perceived as being authentic. The SP-SIDTM was developed as two consecutive 15-minute stations with accompanying 10-minute written interstation exercises (ISEs). The two stations were formatted to be incorporated within a multistation SP-based clinical skills evaluation. In the first station, trainees evaluate a SP hospitalized after a fall. The second station requires trainees to discuss discharge planning for the individual seen in the previous station in a standardized IDT meeting with a standardized registered nurse (SRN) and standardized medical social worker (SMSW). The trainee is evaluated in the role of the physician. SPs rated components of examinees’ interpersonal and communication skills on a seven-item, 5-point Likert-type scale. History taking, physical examination skills, counseling, and interdisciplinary team tasks were recorded on carefully constructed dichotomous checklists (CHKLST). The SRN and SMSW were instructed to avoid discussing examinee performance and to rate each student independently. Geriatricians developed the SP checklists using a case-development process previously described.5 An experienced SP trainer trained five pairs of volunteers to play the SRN and SMSW using standardized training materials. None of the volunteers were actual practicing nurses or social workers. After approximately 1 hour of onsite coaching, the volunteers were asked to memorize their roles before returning several days later for a second training session. During the second session, they performed their roles with a faculty member playing the role of the student until accuracy of portrayal and checklist completion were satisfactory. The checklists and training materials are available by request through The Portal of Geriatrics Online Education (www.pogoe.org) website.6 The SIDTM checklists and training materials accounted for interaction between the examinee and each of the team members and interaction between team members. Care was taken to ensure that facilitating discussion was measured rather than the simple presentation of information. For example, the trainee would receive credit for addressing pain management only if the SRN was asked for her
JANUARY 2014–VOL. 62, NO. 1
JAGS
opinion. The ISE after the SP encounter measured the examinee’s ability to synthesize clinical information related to falls. After the SIDTM station, examinees were required to construct a care plan using a modified version of the Geriatric Interdisciplinary Care Summary (GICS) template.4 The GICS was developed at the College of Medicine, University of Florida, and is a grid that learners use to integrate their knowledge and skills into an overall management plan for a medically complex older adult. The GICS grid consists of eight major care domains: physical medicine and rehabilitation, cognitive, emotional, medical and surgical, nutritional, environmental, social and caregiver, and economic. Using the GICS, learners identify health disciplines addressing these problems and formulate achievable treatment goals. Faculty members scored both ISEs against predetermined objective grading criteria. During the 2011/12 academic year, the SP-SIDTM was administered to 52 MS4s and six GMFs as part of their required GSPX. The students completed the GSPX at the end of their 4th-year geriatrics rotation. The fellows completed the GSPX in a single administration midway through their first year of fellowship. Two sets of six trainees completed the examination on each day. The students received aggregated scores across all stations as part of their evaluation for the rotation but were not provided specific feedback on their performance in each individual station. No attempts were made to prevent the students from discussing the stations with the students who had not yet taken the examination. For the SP and SIDTM stations, component scores were calculated as the percentage of correct items achieved. The total score for each station was calculated as the weighted average of component scores (CHKLST 50%, ISE or GICS 30%, IPS/COMM 20%). Combined SP-SIDTM total and component scores were calculated as the average of scores for both stations. Immediately after each station, examinees rated the difficulty of the case (minimal to extreme), case length (too short to too long), and the believability of the SP-SIDTM experience (not at all believable to extremely believable) on 7-point Likerttype scales. They also rated their prior experience with similar patients on a 5-point scale (never to ≥6 times).
Statistical Methods Reliability of checklists and rating scales was measured using Cronbach alpha. Differences between the groups of examinees were measured using the t-test for independent samples. Correlations between variables were measured using Pearson correlation coefficients. The University of Hawaii committee on human studies approved use of the de-identified learner data for this research as exempt from federal regulations pertaining to the protection of human research participants.
RESULTS The mean of total scores were higher for GMFs than MS4s on the SP station (72 vs 67, P = .13) and the SIDTM station (75 vs 65, P = .003), although the difference was statistically significant only for the SIDTM station (Table 1). The average component scores (CHKLST,
JAGS
JANUARY 2014–VOL. 62, NO. 1
VALIDATION OF A STANDARDIZED IDT MEETING
Table 1. Combined Scores According to Level of Training: Standardized Patient (SP) Only, Standardized Interdisciplinary Team Meeting (SIDTM) Only and SP-SIDTM
Checklista
Communication Skillsb
Patient Notec or Geriatric Interdisciplinary Care Summary
Total Scored
Mean Standard Deviation
Tool SP only MS4s 73 10 GMFs 77 6 P-valuee .38 SP-SIDTM only MS4s 70 12 GMFs 82 10 .03 P-valuee SP-SIDTM combinedf MS4s 72 9 GMFs 79 7 .04 P-valuee
88 7 94 3 .06
43 14 49 4 .36
67 7 72 4 .13
85 10 92 7 .09
43 10 54 13 .02
65 8 75 9 .003
85 10 92 7 .09
55 6 63 8 .006
66 5 74 6 .002
a
Percentage of checklist items correctly achieved. Percentage of points received on 15 communication items (maximum 4 points each). c Percentage of graded items achieved. d Weighted average of component scores (50% checklist + 30% patient note + 20% communication skills). e P-Values were calculated using t-tests comparing mean scores of 4th-year medical students (M4s; n = 52) and geriatric medicine fellows (GMFs; n = 6). f Average of SP and SIDTM scores. b
ISE or GICS, IPS/COMM) for the SIDTM station were significantly higher for GMFs (82, 54, 92, respectively) than for MS4s (70, 43, 85, respectively). The average component scores for the SP station were higher for GMFs than MS4s, but the differences were not statistically significant. The combined total score (Table 2) was significantly higher for GMFs than for MS4s (74 vs 66, P = .002). The combined checklist (79 vs 72, P = .04) and ISE/GICS (63 vs 55, P = .006) scores were also significantly higher for GMFs. The IPS/COMM scores were higher for GMFs, but
Table 2. Examinee Rating of Standardized Patient Standardized Interdisciplinary Team Meeting Experience Characteristics According to Level of Training Fourth-Year Medical Students Characteristic
Case difficultya Time allottedb Believabilityc Previous similar “real” experienced
Geriatric Medicine Fellows
Mean Standard Deviation
4.7 4.5 5.6 3.0
1.0 0.8 1.4 1.2
3.8 4.0 5.5 4.2
0.8 0.0 1.4 0.4
1 = minimal to 7 = extreme (4 = moderate). 1 = too short to 7 = too long (4 = about right). c 1 = not at all to 7 = extremely (4 = moderately). d 1 = never, 2 = once, 3 = twice, 4 = 3–5 times, 5 = ≥6 times. a
b
P-Value
.06 .13 .92 .03
173
the difference was not statistically significant (92 vs 85, P = .09). There was no difference in responses according to level of training for perception of case difficulty, time allotted for each case, or believability (Table 2). Overall, trainees rated the SP-SIDTM experience as moderately difficult, length of time as about right, and believability as moderate to extreme. More than 80% of trainees had at least one similar real patient experience. As expected, GMFs rated themselves as having significantly more previous similar real experience than MS4s. Reliability measured as Cronbach a was 0.780 for the SP station, 0.866 for the SIDTM station, and 0.871 for all graded items across both stations. Interobserver correlation between SRN and SMSW checklist scores (r = 0.82, P < .001) and total scores (r = 0.69, P < .001) were high. Correlation between SRN and SMSW communication skills scores was weaker (r = 0.45, P = .003).
DISCUSSION Medical schools and postgraduate training programs are integrating IDT skills within their curricula. Effective methods have been developed for teaching these skills, such as the Geriatric Interdisciplinary Team Training Program of the John A. Hartford Foundation.7–9 Methods to assess the acquisition of IDT skills are important to trainee feedback, curriculum evaluation, and public accountability. Validated instruments are available to assess interdisciplinary team knowledge, attitudes, and self-efficacy and have been used for programmatic evaluation.10–14 A previous study15 demonstrated the reliability and validity of a survey instrument for assessing interdisciplinary team performance in long-term care settings and in the Program of All-Inclusive Care of the Elderly program. Educational interventions with standardized patients or human patient simulators have been shown to improve interdisciplinary team attitudes and skills. The SP provides trainees with a patient to evaluate before collaborating on an interdisciplinary care plan. After the encounter, faculty observers and the SPs themselves provide feedback.16–20 Ideally, summative evaluation of IDT skills would require each examinee to participate individually in a standardized team meeting. The authors believe that this is the first report on the validation of a performance-based assessment of IDT skills using standardized team members from different disciplines. The SP-SIDTM achieved adequate statistical reliability as measured according to Cronbach alpha, and the SIDTM checklist showed good interobserver correlation. These findings suggest that IDT behaviors can be consistently measured and that overall student performance is likely to be stable with retesting. The lower interobserver correlation for communication skills checklist items was expected, because the SPs were not specifically trained to be consistent on these observations. Previous experience suggested that standardized patient examinations may not adequately distinguish levels of expertise.5 Scores on the SP component of the SP-SIDTM were not statistically different for GMFs and MS4s, although the observation that GMFs achieved significantly higher overall SP-SIDTM scores
174
YUASA ET AL.
provides evidence of construct validity and suggests that the SP-SIDTM distinguishes levels of learners. An IDT experience allowing trainees to evaluate a patient and subsequently play the role of the physician in an IDT setting was created. There were probably fewer team members than students encounter in actual clinical environments, but the small team allowed administrative costs (recruitment, training, and SP stipends) to be optimized. The students rated the time allotted for the SP-SIDTM as about right, and the difficulty of the case as moderate, suggesting that an assessment that readily fit into the current multistation SP-based GSPX was successfully created. The IDT scenario created appears to be one that most students encounter in training, as evidenced by the majority of MS4s and GMFs responding that they had had at least one similar experience in an actual patient care situation. In addition, MS4s and GMFs rated the experience as highly believable, suggesting that an experience that was authentic and felt “real” to the examinees was successfully created. Examinees with more training rated the examination as less difficult. Clinical experience with patients similar to the SP-SIDTM encountered on the examination increased with training. These observations lend additional credibility and support to the belief that the SP-SIDTM appears to evaluate what is intended. The lack of a criterion standard for measuring competence with these skills makes any further conclusions difficult. An important concern is whether the SIDTM checklist measures trainees’ facilitative skills as opposed to the general ability to discuss clinical problems in a group setting. The additional years of training between medical school and fellowship, allowing more time to develop these general abilities, could account for the difference in performance between GMFs and MS4s. Administering the SP-SIDTM to individuals in other disciplines who have specific training in facilitative skills may help to clarify this question. Further study of this question is planned. The number of trainees evaluated in this study was small. Whether the findings will hold up with subsequent MS4 and GMF cohorts or in other institutional settings remains to be seen, but the SP-SIDTM appears to be a promising innovative tool for the assessment of IDT skills.
CONCLUSION Trainees positively rated the SP-SIDTM, which generated scores with high reliability. Observations recorded on checklists showed excellent interobserver correlation. GMFs scored significantly higher than 4th-year medical students. These observations support the validity of this novel assessment tool.
ACKNOWLEDGMENTS Data from this research were presented at the American Geriatrics Society National Meeting, Grapevine, Texas, May 2013. Conflict of Interest: The editor in chief has reviewed the conflict of interest checklist provided by the authors and has determined that the authors have no financial or any other kind of personal conflicts with this paper. Drs. Masaki, Nagoshi, and Wen received grant funding
JANUARY 2014–VOL. 62, NO. 1
JAGS
from The Donald W. Reynolds Foundation Next Steps in Physicians’ Training in Geriatrics Grant. Dr. Masaki received grant funding from The John A. Hartford Foundation Center of Excellence in Geriatrics grant and the Pacific Islands Geriatric Education Center Health Resources and Services Administration Grant UB4HP19065. Author Contributions: Study concept and design: Nagoshi, Yuasa, Oshiro-Wong, Wen, Masaki. Acquisition of data: Nagoshi, Oshiro-Wong. Analysis and interpretation of data: Yuasa, Nagoshi, Masaki. Preparation of manuscript: Yuasa, Nagoshi, Masaki. Critical revision of manuscript: Yuasa, Nagoshi, Oshiro-Wong, Tin, Wen, Masaki. Sponsor’s Role: The funding sources had no role in the analysis or preparation of this manuscript.
REFERENCES 1. Mion L, Odegard PS, Resnick B et al. Interdisciplinary care for older adults with complex needs: American Geriatrics Society position statement. J Am Geriatr Soc 2006;54:849–852. 2. Position Statement on Interdisciplinary Team Training in Geriatrics: An Essential Component of Quality Healthcare for Older Adults. 2011 [online]. Available at www.americangeriatrics.org/files/documents/pha/PHA_ IDTstatement.pdf Accessed May 21, 2013. 3. Leipzig RM, Granville L, Simpson D et al. Keeping granny safe on July 1: A consensus on minimum geriatrics competencies for graduating medical students. Acad Med 2009;84:604–610. 4. Bautista MK, Meuleman JR, Shorr RI et al. Description and students’ perceptions of a required geriatric clerkship in postacute rehabilitative care. J Am Geriatr Soc 2009;57:1685–1691. 5. Nagoshi M, Williams S, Kasuya R et al. Using standardized patients to assess the geriatrics medicine skills of medical students, internal medicine residents, and geriatrics medicine fellows. Acad Med 2004;79:698–702. 6. Nagoshi M, Wong C, Tin MM. Standardized Interdisciplinary Team Meeting Case (Product 21230) [on-line]. Available at www.pogoe.org Accessed July 12, 2013. 7. Fulmer T, Flaherty E, Hyer K. The geriatric interdisciplinary team training (GITT) program. Gerontol Geriatr Educ 2003;24:3–12. 8. Fulmer T, Hyer K, Flaherty E et al. Geriatric interdisciplinary team training program: Evaluation results. J Aging Health 2005;17:443–470. 9. Djukic M, Fulmer T, Adams JG et al. NYU3T: Teaching, technology, teamwork: A model for interprofessional education scalability and sustainability. Nurs Clin North Am 2012;47:333–346. 10. Hyer K, Fairchild S, Abraham I et al. Measuring attitudes related to interdisciplinary training: Revisiting the Heinemann, Schmitt and Farrell ‘Attitudes Toward Health Care Teams’ Scale. J Interprof Care 2000; 14:249–258. 11. Clark PG, Leinhaas MM, Filinson R. Developing and evaluating an interdisciplinary clinical team training program: Lessons taught and lessons learned. Educ Gerontol 2002;28:491–510. 12. Clark PG. Evaluating an interdisciplinary team training institute in geriatrics: Implications for teaching teamwork theory and practice. Educ Gerontol 2002;28:511–528. 13. Luecht RM, Madsen MK, Taugher MP et al. Assessing professional perceptions: Design and validation of an Interdisciplinary Education Perception Scale. J Allied Health 1990;19:181–191. 14. McFadyen AK, Maclaren WM, Webster VS. The Interdisciplinary Education Perception Scale (IEPS): An alternative remodelled sub-scale structure and its reliability. J Interprof Care 2007;21:433–443. 15. Temkin-Greener H, Gross D, Kunitz SJ et al. Measuring interdisciplinary team performance in a long-term care setting. Med Care 2004;42:472–481. 16. Westberg SM, Adams J, Thiede K et al. An interprofessional activity using standardized patients. Am J Pharm Educ 2006;70:34. 17. Shrader S, McRae L, King WM 4th et al. A simulated interprofessional rounding experience in a clinical assessment course. Am J Pharm Educ 2011;75:61. 18. Vyas D, McCulloh R, Dyer C et al. An interprofessional course using human patient simulation to teach patient safety and teamwork skills. Am J Pharm Educ 2012;76:71. 19. MacDonnell CP, Rege SV, Misto K et al. An introductory interprofessional exercise for healthcare students. Am J Pharm Educ 2012;76:154. 20. Solomon P, Salfi J. Evaluation of an interprofessional education communication skills initiative. Educ Health (Abingdon, England) 2011;24:616.
