AAIM Perspectives
AAIM Perspectives AAIM is the largest academically focused specialty organization representing departments of internal medicine at medical schools and teaching hospitals in the United States and Canada. As a consortium of five organizations, AAIM represents department chairs and chiefs; clerkship, residency, and fellowship program directors; division chiefs; and academic and business administrators as well as other faculty and staff in departments of internal medicine and their divisions.
The Next Accreditation System: A Strategy for Implementing New Reporting Standards Using a Hematology/Oncology Model Frances Collichio, MD,a Karen Kayoumi, BA,b Lyndsey Sierra, BA,c Charles P. Clayton, BA,b Marilyn Raymond, PT, PhD,c Elaine A. Muchmore, MDd a c
The University of North Carolina, Chapel Hill; bAmerican Society of Hematology, Washington, DC; American Society of Clinical Oncology, Alexandria, Va; dDepartment of Medicine, University of California, San Diego.
The Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME) began phased implementation in July of 2013.1 The NAS was developed, in part, to align the education of residents and fellows with 21st century medical care. In the report from the Carnegie Foundation regarding the current status of medical education, standardization of progressive learning outcomes was one of the major recommendations cited as necessary to improve medical training.2 By changing accreditation standards, this alignment is addressed. Key definitions of the NAS are included in Table 1.3-5 Starting in July of 2014, Internal Medicine (IM) Fellowship Programs and other “phase II” programs were included in the NAS, which requires new evaluation standards for fellows expanding on the original ACGME 6 competencies.6 Funding: American Society of Hematology and American Society of Clinical Oncology. Conflict of Interest: None. Authorship: All authors had access to the data and played a role in writing this manuscript. Requests for reprints should be addressed to Elaine A. Muchmore, MD, Department of Medicine, University of California, San Diego, VA Medical Center, 3350 La Jolla Village Dr, 11-E, San Diego, CA 92161. E-mail address: [email protected]
The IM subspecialty societies were asked to relate the distinctive qualities of their professions to the “IM Subspecialty Reporting Milestones.” Hematology and medical oncology work groups, sponsored by the American Society of Hematology (ASH) and the American Society of Clinical Oncology (ASCO), created the Curricular Milestones to provide these connections.7,8 The intent was to establish a system whereby Hematology/Oncology faculty were comfortable and confident in fellow performance on major rotations according to subspecialty standards and were able to relate this performance back to the more general format of the “IM Subspecialty Reporting Milestones.” This technique started with identifying Entrustable Professional Activities for the specific IM subspecialty and then using them to create Curricular Milestones. When these Curricular Milestones were organized into the competency domains already found in the “IM Subspecialty Reporting Milestones,” they logically linked Entrustable Professional Activities, Curricular Milestones, and Subspecialty Reporting Milestones. Most important, this permitted transition of evaluation data on fellows from specific to general. The sample evaluation templates included in this article were designed for Hematology/Oncology, but the same methodology can be applied to other subspecialties.
0002-9343/$ -see front matter Ó 2015 Alliance for Academic Internal Medicine. All rights reserved. http://dx.doi.org/10.1016/j.amjmed.2014.10.025
Collichio et al
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
201
DEVELOPING ENTRUSTABLE PROFESSIONAL ACTIVITIES AND CURRICULAR MILESTONES
competence in medical knowledge (the fellow must know the proper systemic therapy for the disease), interpersonal communication skills (the fellow must be ASH and ASCO organized teams that included program able to explain the treatment to the patient and famdirectors and subject matter experts from institutions ilies), and patient care (the fellow must know how to across North America to articulate the fundamental effectively order the treatment). By taking these activelements of the subspecialty ities together, the ASH and and to develop the Hematology/ ASCO committees thought that Oncology Curricular Milestones. PERSPECTIVES VIEWPOINTS this was best placed in subThe teams started by clarifying The Accreditation Council for the Grad- competence “Patient Care 2 the components of a milestone uate Medical Education (ACGME) Next (PC2): Develops and achieves (Figure 1) and then defined the Entrustable Professional Accreditation System (NAS) requires a comprehensive management plan for each patient.” Activities (Table 2) of the different evaluation system for resiDuring this exercise, the specialty. The ASH/ASCO dents and fellows. groups thought that most of the teams appreciated that although Entrustable Professional Activ Although the ACGME has given programs Entrustable Professional Activities ities best fit in the subthe latitude to define approaches to the are usually composed of multiple competencies within Patient NAS, it would be helpful for specialty competencies, to make this Care, Medical Knowledge, and programs to share evaluation strategies. process work for our societies, Interpersonal Communication the Entrustable Professional Hematology and medical oncology spe- Skills. If the language in a Activities in hematology and cialties have worked together to specific IM Subspecialty oncology literally were written Reporting Milestone could be develop an approach to the NAS that we to fit to the “IM Subspecialty applied to the skills required for believe can be used by other societies Reporting Milestones” as best the subspecialty, it was not and programs. as possible. As the reader will changed. Thus, in the compesee, by fitting our Entrustable tencies of Systems-Based Professional Activities to the “IM Subspecialty Practice and Professionalism and most of PracticeReporting Milestones,” we developed one document that Based Learning and Improvement, no changes were can be used to create most of our tools to assess made. When the Entrustable Professional Activities rotations. The Entrustable Professional Activity cannot were integrated into the “IM Subspecialty Reporting be the tool to fit to the “IM Subspecialty Reporting Milestones,” there were 39 separate subcompetencies, Milestones” because the Entrustable Professional 25 unique to the specialty and 14 identical to the “IM Activity does not describe a behavior. Subspecialty Reporting Milestones” (Figure 2). An example of the process used is assignment of The work groups next addressed the task of articuEntrustable Professional Activity 17: “Writes accurate lating the progressive achievement of behaviors and and safe orders in the Electronic Medical Record for skills for each Entrustable Professional Activity. This systemic therapy and supportive care.” This activity, task is called streaming, and whenever possible, the which is an integral part of the subspecialty, requires
Table 1
Key Definitions in the Next Accreditation System
ACGME: Accreditation Council for Graduate Medical Education ACGME Core Competencies: knowledge, skills, or attitudes applied to the clinical environment in the following domains: patient care, medical knowledge, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice. Subcompetence: a component of a core competence. IM Reporting Milestones: the document that lists the progressive skills required for residents in IM to achieve competence in all required domains.4 IM Subspecialty Milestones: the document that lists the progressive skills required for fellows in the subspecialties of IM to achieve competence in all required domains.5 Entrustable Professional Activity: descriptions of observable work-based clinical activities that can serve as meaningful and manageable points of assessment.6 Curricular Milestones: the observable skills and activities required for residents or fellows to progress to competence in practice of a specialty or subspecialty. Clinical Competency Committee: ACGME-required committee tasked with reviewing all evaluations of all residents and fellows semiannually, and to ensure a report is submitted to the ACGME. ACGME ¼ Accreditation Council for Graduate Medical Education; IM ¼ Internal Medicine.
202
The American Journal of Medicine, Vol 128, No 2, February 2015
Figure 1 Deconstruction of a milestone. A sample format for a milestone stream showing the format for submission of this information to the ACGME.
work groups used similar language so that the readers of the document could easily see patterns and move from one subcompetence to another. The resulting document is the Hematology/Oncology Curricular Milestones, posted on the websites for ASH and ASCO on March 10, 2014.7,8
NEXT STEPS: DEVELOPMENT OF TOOLS TO INFORM THE “IM SUBSPECIALTY REPORTING MILESTONES” The evaluation system before the NAS has been criticized for being unable to describe behaviors of residents and fellows in the many different environments supervised by many different preceptors. With the NAS, direct observation to inform our assessment is required, because the Milestones documents describe behaviors. The beauty of the Curricular Milestones document is that by aligning subspecialty Entrustable Professional Activities to Subcompetencies, the necessary skills and behaviors in subspecialty work environments are described. This provides a connection between the “IM Subspecialty Reporting Milestones” and fellowship outcomes.
The next step is to determine which Entrustable Professional Activities can be observed on fellowship rotations. These should overlap with the Curricular Guidelines of each fellowship. It is best to be as specific as possible, so the skills and behaviors referenced are regularly observed on the rotation. For example, it would be easy to include a variation of “IM Subspecialty Reporting Milestone” Patient Care 1 “Gathers and synthesizes patient and disease specific information necessary to understand the presenting hematologic or oncologic disorder” on evaluation forms for every clinical rotation, because these skills are fundamental. However, the recommendation would be to include this subcompetence only on a service for which the fellow has to use the skill frequently. Table 3 shows sample rotation evaluations for clinical rotations common to Hematology/Oncology fellowships. Because the number of domains on each form is relatively modest, it allows faculty to focus their observations without overwhelming them. Some subcompetencies of the “IM Subspecialty Reporting Milestones” do not involve direct patient care activities. For these, the method we have described for creating tools from Curricular Milestones will not apply. The most obvious of these is MK3 (Medical
Collichio et al Table 2
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
203
Entrustable Professional Activities for Hematology/Oncology
1. Gathers and synthesizes patient- and disease-specific information necessary to understand the presenting hematologic or oncologic disorder 2. Demonstrates the ability to diagnose and assign stage or severity of hematology and oncology disorders in all adult age groups 3. Formulates the overall plan for hematology and oncology disorders, including urgent/emergent conditions 4. Demonstrates the ability to analyze response to treatment and adjust therapy for hematology or oncology disorders over time using standard measurements and guidelines 5. Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies 6. Demonstrates the ability to facilitate patient participation in clinical trials 7. Demonstrates the ability to effectively manage older adult patients with hematologic and oncologic diseases 8. Demonstrates understanding and effective application of principles of transfusion medicine 9. Demonstrates appropriate understanding and management of complications of transfusion 10. Demonstrates knowledge of, principles of, indications for, and complications from stem cell transplantation and ability to effectively manage these patients 11. Demonstrates the ability to effectively manage patients with pain, anxiety, and depression 12. Demonstrates the ability to effectively manage patients requiring palliative care, hospice care, or rehabilitation 13. Demonstrates the ability to effectively recognize and promote cancer prevention and control strategies and survivorship 14. Demonstrates the ability to effectively manage patients during transitions of care 15. Demonstrates competent performance of invasive procedures required for diagnosis, treatment, and management of patients with hematology and oncology 16. Demonstrates the ability to perform and interpret peripheral blood smears 17. Writes accurate and safe orders in the Electronic Medical Record for systemic therapy and supportive care 18. Requests and provides effective consultative care for patients with hematologic and oncologic diseases 19. Demonstrates a fund of knowledge in solid tumor oncology, malignant hematology, and non-neoplastic hematology 20. Demonstrates knowledge of and indications for genetic, genomic, molecular, and laboratory tests related to hematologic and oncologic disorders 21. Delivers safe, effective, patient-centered, cost-efficient care, and advocates for system improvements 22. Demonstrates personal habits of lifelong learning and self-improvement 23. Cares for patients in a manner that supersedes self-interest 24. Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care 25. Demonstrates appropriate use and completion of health records and procedure documents 26. Works effectively within an interprofessional team Competence in each of these clinical activities expected to be demonstrated by all graduates of combined fellowships in Hematology/ Oncology. Each EPA has significant developmental points that help define the training trajectory for each fellow. EPA ¼ Entrustable Professional Activity.
Knowledge 3) (Scholarship). In addition, the subcompetencies dealing with performance audits practiced-based learning and improvement 1 (PBLI-1): monitors practice with a goal for improvement, and PBLI-1: learns and improves via performance audit, and recognition of system errors systems-based practice-2 (SBP-2): recognizes system error and advocates for improvement generally are outside the scope of direct patient care activities. ASH and ASCO have developed modules9,10 that may assist programs in collecting normalized data in these areas, and other societies are encouraged to develop similar products, so program graduates will be well positioned to actively participate in the current dynamic clinical environment.
ROLL OUT AND EDUCATION OF FACULTY AND FELLOWS A critical step in the NAS is to roll out the new system and teach it to all of the stakeholders, from fellows to faculty. Eventually, there should be a seamless
approach starting in medical school. Faculty and fellows will need to be oriented to changes from standard evaluation templates, many of which currently use Likert scales instead of developmental milestones to evaluate performance in each competence or subcompetence. Fellows will need orientation and education about the types of specific feedback they will receive and the anticipated trajectory for mastery of skills and behaviors. Likewise, there will need to be more faculty development, particularly to make faculty comfortable with the developmental scale used for the milestones and increasing direct observation in assessing fellows. It is essential that all clinical supervisors understand the progressive expectations. The evaluations are intended to focus their judgment regarding performance of fellows. Faculty must feel empowered to provide specific feedback to both fellows and program leadership. Therefore, although the strategy we have proposed for constructing evaluations is fundamental to a new structure, it is not intended to be the only programmatic change in the NAS.
204
The American Journal of Medicine, Vol 128, No 2, February 2015
PC
MK
SBP
PBLI
PROF
ICS
PC1a
PC2g
PC3
MK1a
SBP1
PBLI1
PROF1
ICS1
PC2a
PC2h
PC4a
MK1b
SBP2
PBLI2
PROF2
ICS2
PC2b
PC2i
PC4b
MK1c
SBP3
PBLI3
PROF3
ICS3
PC2c
PC2j
PC4c
MK2
SBP4
PBLI4
PROF4
PC2d
PC2k
PC5
MK3
PC2e
PC2l
PC2f
PC2m
Figure 2 Comparison of “IM Subspecialty Reporting Milestones” and Hematology/Oncology curricular milestones. The 6 core competencies at the header in patient care, medical knowledge, systems-based practice, practiced-based learning and improvement, professionalism, and interpersonal communication skills. The boxes below the header represent the subcompetencies in each area. Pink: specific to hematology oncology. Grey: same as the subspecialty reporting milestones. ICS ¼ interpersonal communication skills; MK ¼ medical knowledge; PBLI ¼ practiced-based learning and improvement; PC ¼ patient care; PROF ¼ professionalism; SBP ¼ systems-based practice.
STEPS TO FOLLOW BY THE SUBSPECIALTY TO DEVELOP CURRICULAR MILESTONES 1. Identify the Entrustable Professional Activities for your specialty. 2. Fit the Entrustable Professional Activities according to the ACGME competence and subcompetence of the “IM Subspecialty Reporting Milestones” most relevant for the activity. 3. Articulate progressive achievement of behaviors or skills in tabular form under each point in step 2. 4. Once you have completed step 3 for all of the Entrustable Professional Activities you have organized, you have completed Curricular Milestones.
STEPS TO FOLLOW BY THE PROGRAM 1. Define the major rotations for your program. 2. Determine which of the Entrustable Professional Activities would best fit with the educational experiences on each rotation. 3. Add the corresponding Curricular Milestone narrative streams for each Entrustable Professional Activity and collate the group to make the new evaluation tool for faculty. 5. Make sure to “tag” the Curricular Milestone with the relevant reporting “IM Subspecialty Reporting Milestones” to which scores should be directed (included in Table 3).
6. Schedule meetings with faculty and fellows so they are aware of the new evaluation strategy and its rationale. 7. Ask the faculty members and interdisciplinary team members to discuss the evaluation tool with the fellow before and after the rotation. 8. The Clinical Competence Committee (CCC) will study the reports and fill out the “IM Subspecialty Reporting Milestones,” and determine whether remediation or other actions are necessary. 9. Submit the “IM Subspecialty Reporting Milestones” report for each fellow using the CCC recommendations.
CONCLUSIONS A key component of the NAS is evaluation of progressive development of training milestones using the “IM Subspecialty Reporting Milestones” every 6 months. We have shown a unique process whereby fellowship programs can fit the Entrustable Professional Activities as closely as possible to the “IM Subspecialty Reporting Milestones” document. When the progressive skills and behaviors are added, the resulting document becomes the Curricular Milestones of the subspecialty. Evaluation tools using the Curricular Milestones provide assessment of progress in achieving competence and give meaningful feedback to the fellow in the clinical context of the modern health care system. The strategy used provides a blueprint for subspecialties to define the fundamental aspects that should be included in all fellowship programs. Emphasis should be placed on faculty
Collichio et al Table 3
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
Selected Rotations for Hematology/Oncology Training Programs
A
Inpatient Consultative Service Evaluation
EPA
Observable Skill or Behavior
1
Gathers and synthesizes patient- and disease-specific information necessary to understand the presenting hematologic or oncologic disorder Demonstrates the ability to diagnose and assign stage or severity of hematology and oncology disorders in all adult age groups Formulates the overall plan for hematology and oncology disorders, including urgent/emergent conditions Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates the ability to effectively manage patients requiring palliative care, hospice care, or rehabilitation Demonstrates the ability to effectively manage patients during transitions of care Writes accurate and safe orders in the Electronic Medical Record for systemic therapy and supportive care Requests and provides effective consultative care for patients with hematologic and oncologic diseases Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care
2 3 5 12 14 17 18 24 B
Medical Oncology Specialty Clinic Evaluation
EPA
Observable Skill or Behavior
4
Demonstrates the ability to analyze response to treatment and adjust therapy for hematology or oncology disorders over time using standard measurements and guideline Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates the ability to facilitate patient participation in clinical trials Demonstrates the ability to effectively manage older adult patients with hematologic and oncologic diseases Demonstrates the ability to effectively recognize and promote cancer prevention and control strategies and survivorship Requests and provides effective consultative care for patients with hematologic and oncologic diseases Demonstrates knowledge of and indications for genetic, genomic, molecular, and laboratory tests related to hematologic and oncologic disorders Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care Demonstrates appropriate use and completion of health records and procedure documents
5 6 7 13 18 20 24 25 C
Hematology Clinic Evaluation
EPA
Observable Skill or Behavior
5
Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates understanding and effective application of principles of transfusion medicine Demonstrates appropriate understanding and management of complications of transfusion medicine Demonstrates the ability to effectively manage patients with pain, anxiety, and depression Demonstrates the ability to perform and interpret peripheral blood smears Demonstrates knowledge of and indications for genetic, genomic, molecular, and laboratory tests related to hematologic and oncologic disorders
8 9 11 16 20
205
Curricular Milestone
Reporting Milestone
PC1a
PC1
PC2a
PC2
PC2b
PC2
PC2d
PC2
PC2k
PC2
PC2m PC4c
PC2, SBP4 PC4b
PC5
PC5
ICS1
ICS1
Curricular Milestone
Reporting Milestone
PC2c
PC2
PC2d
PC2
PC2e PC2f
PC2 PC2
PC2l
PC2
PC5
PC5
MK2
MK2
ICS1
ICS1
ICS3
ICS3, PROF2
Curricular Milestone
Reporting Milestone
PC2d
PC2
PC2g
PC2
PC2h
PC2
PC2j
PC2
PC4b MK2
PC4b MK2
206 Table 3
The American Journal of Medicine, Vol 128, No 2, February 2015 Continued
C
Hematology Clinic Evaluation
EPA
Observable Skill or Behavior
21
Delivers safe, effective, patient-centered, cost-efficient care and advocates for system improvements Demonstrates appropriate use and completion of health records and procedure documents
25 D
Infusion Center Evaluation
EPA
Observable Skill or Behavior
8
Demonstrates understanding and effective application of principles of transfusion medicine Demonstrates competent performance of invasive procedures required for diagnosis, treatment, and management of patients with hematology and oncology Writes accurate and safe orders in the Electronic Medical Record for systemic therapy, including appropriate supportive care Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates a fund of knowledge in solid tumor oncology Demonstrates a fund of knowledge in hematologic malignancies Demonstrates personal habits of lifelong learning and self-improvement Works effectively within an interprofessional team (eg, peers, consultants, nursing, and other health professionals)
15
17 5 19 19 22 26 E
BMT Service Evaluation
EPA
Observable Skill or Behavior
10
Demonstrates knowledge of principles of indications for and complications from stem cell transplantation and ability to effectively manage these patients Demonstrates competent performance of invasive procedures required for diagnosis, treatment, and management of patients with hematology and oncology disorders Writes accurate and safe orders in the Electronic Medical Record for systemic therapy, including appropriate supportive care Requests and provides effective consultative care for patients with hematologic and oncologic diseases Cares for patients in a manner that supersedes self-interest Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care Works effectively within an interprofessional team (eg, with peers, consultants, nursing, ancillary professionals, and other support personnel)
15
17 18 23 24 26
Curricular Milestone
Reporting Milestone
SBP3
SBP3
ICS3
ICS3, Prof2
Curricular Milestone
Reporting Milestone
PC2g
PC2
PC4a
PC4a
PC4c
PC4b
PC2d
PC2
MK1a MK1b PBLI4 SBP1
MK1 MK1 PBLI4 SBP1, ICS2, PROF1
Curricular Milestone
Reporting Milestone
PC2i
PC2
PC4a
PC4a
PC4c
PC4b
PC5
PC5
PROF3 ICS1
PROF3, PROF4 ICS1
SBP1, PROF1
SBP1, PROF1
A sample rotation evaluation for a core rotation for Hematology/Oncology fellowship. The relevant EPA is listed in column 1, and the observable clinical activity is listed in column 2. Column 3 shows the reference for the relevant Hematology/Oncology Curricular Milestone, and column 4 shows the relevant IM Subspecialty Reporting Milestone(s). Note that data from all 5 evaluations (A-E) must be taken together by the CCC to determine progress in all “IM Subspecialty Reporting Milestones” that are related to direct patient care activities. BMT ¼ blood and marrow transplantation; EPA ¼ Entrustable Professional Activity; ICS ¼ interpersonal communication skills; MK ¼ medical knowledge; PC ¼ patient care; PROF ¼ professionalism; SBP ¼ systems-based practice.
development so that training expectations are shared and embraced. Because evaluation of fellows on the basis of progressive development of specific skills is a new concept for our community, it is anticipated that there will be suggestions for improvement as we move forward. The intention is that Entrustable Professional Activities and Curricular Milestones will assist the community in achieving requirements of the NAS, and
that these documents will be shared and reviewed at regular intervals.
ACKNOWLEDGMENTS The authors thank Eric Holmboe, MD, for review of the article, and the working groups for the development of Entrustable Professional Activities and curricular
Collichio et al
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
milestones. ASCO Working Group: Frances A. Collichio, MD, Jill Gilbert, MD, Robert Wolff, MD, Helen K. Chew, MD, Thomas George, Jr, MD, FACP, Lee M. Krug, MD, Alexandra P. Wolanskyj, MD, Ronald J. Maggiore, MD, Jennifer S. Temel, MD, and Tamara N. Shenkier, MD. ASH Working Group: Elaine A. Muchmore, MD, Kevin Imrie, MD, Charles Bolan, MD, Joanne Filicko-O’Hara, MD, Jason Gotlib, MD, Ann LeCasce, MD, Robert Redner, MD, Gerald Soff, MD, and Katherine Walsh, MD.
5.
6.
7.
References 1. Nasca T, Philibert I, Brigham T, Flynn T. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051-1056. 2. Irby D. Educating physicians for the future: Carnegie’s calls for reform. Med Teach. 2011;33:547-550. 3. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5:157-158. 4. Alliance for Academic Internal Medicine. ACGME and ABIM Release 22 Internal Medicine Reporting Milestones. Available at:
8.
9.
10.
207
http://www.im.org/AcademicAffairs/milestones/Pages/ACGME andABIMRelease22InternalMedicineReportingMilestones.aspx. Accessed March 30, 2014. Accreditation Council for Graduate Medical Education. The Internal Medicine Subspecialty Milestones Project. Available at: http://acgme.org/acgmeweb/Portals/0/PDFs/Milestones/Internal MedicineSubspecialtyMilestones.pdf. Accessed March 30, 2014. Accreditation Council for Graduate Medical Education. Next Accreditation System. Available at: http://www.acgme.org/ acgmeweb/tabid/435/ProgramandInstitutionalAccreditation/Next AccreditationSystem.aspx. Accessed March 30, 2014. American Society of Hematology. ASH and ASCO Present Hematology-Oncology Curricular Milestones. Available at: http://www.hematology.org/Training/Trainees/12505.aspx. Accessed March 30, 2014. American Society of Clinical Oncology. ASCO and ASH Present Hematology-Oncology Curricular Milestones. Available at: http://www.asco.org/professional-development/hematology-oncologycurricular-milestones. Accessed March 30, 2014. American Society of Clinical Oncology. QOPI: The Quality Oncology Practice Initiative. Available at: http://qopi.asco.org. Accessed March 30, 2014. American Society of Hematology. ASH Academy. Available at: http://www.ashacademy.org. Accessed March 30, 2014.
AAIM Perspectives AAIM is the largest academically focused specialty organization representing departments of internal medicine at medical schools and teaching hospitals in the United States and Canada. As a consortium of five organizations, AAIM represents department chairs and chiefs; clerkship, residency, and fellowship program directors; division chiefs; and academic and business administrators as well as other faculty and staff in departments of internal medicine and their divisions.
The Next Accreditation System: A Strategy for Implementing New Reporting Standards Using a Hematology/Oncology Model Frances Collichio, MD,a Karen Kayoumi, BA,b Lyndsey Sierra, BA,c Charles P. Clayton, BA,b Marilyn Raymond, PT, PhD,c Elaine A. Muchmore, MDd a c
The University of North Carolina, Chapel Hill; bAmerican Society of Hematology, Washington, DC; American Society of Clinical Oncology, Alexandria, Va; dDepartment of Medicine, University of California, San Diego.
The Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME) began phased implementation in July of 2013.1 The NAS was developed, in part, to align the education of residents and fellows with 21st century medical care. In the report from the Carnegie Foundation regarding the current status of medical education, standardization of progressive learning outcomes was one of the major recommendations cited as necessary to improve medical training.2 By changing accreditation standards, this alignment is addressed. Key definitions of the NAS are included in Table 1.3-5 Starting in July of 2014, Internal Medicine (IM) Fellowship Programs and other “phase II” programs were included in the NAS, which requires new evaluation standards for fellows expanding on the original ACGME 6 competencies.6 Funding: American Society of Hematology and American Society of Clinical Oncology. Conflict of Interest: None. Authorship: All authors had access to the data and played a role in writing this manuscript. Requests for reprints should be addressed to Elaine A. Muchmore, MD, Department of Medicine, University of California, San Diego, VA Medical Center, 3350 La Jolla Village Dr, 11-E, San Diego, CA 92161. E-mail address: [email protected]
The IM subspecialty societies were asked to relate the distinctive qualities of their professions to the “IM Subspecialty Reporting Milestones.” Hematology and medical oncology work groups, sponsored by the American Society of Hematology (ASH) and the American Society of Clinical Oncology (ASCO), created the Curricular Milestones to provide these connections.7,8 The intent was to establish a system whereby Hematology/Oncology faculty were comfortable and confident in fellow performance on major rotations according to subspecialty standards and were able to relate this performance back to the more general format of the “IM Subspecialty Reporting Milestones.” This technique started with identifying Entrustable Professional Activities for the specific IM subspecialty and then using them to create Curricular Milestones. When these Curricular Milestones were organized into the competency domains already found in the “IM Subspecialty Reporting Milestones,” they logically linked Entrustable Professional Activities, Curricular Milestones, and Subspecialty Reporting Milestones. Most important, this permitted transition of evaluation data on fellows from specific to general. The sample evaluation templates included in this article were designed for Hematology/Oncology, but the same methodology can be applied to other subspecialties.
0002-9343/$ -see front matter Ó 2015 Alliance for Academic Internal Medicine. All rights reserved. http://dx.doi.org/10.1016/j.amjmed.2014.10.025
Collichio et al
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
201
DEVELOPING ENTRUSTABLE PROFESSIONAL ACTIVITIES AND CURRICULAR MILESTONES
competence in medical knowledge (the fellow must know the proper systemic therapy for the disease), interpersonal communication skills (the fellow must be ASH and ASCO organized teams that included program able to explain the treatment to the patient and famdirectors and subject matter experts from institutions ilies), and patient care (the fellow must know how to across North America to articulate the fundamental effectively order the treatment). By taking these activelements of the subspecialty ities together, the ASH and and to develop the Hematology/ ASCO committees thought that Oncology Curricular Milestones. PERSPECTIVES VIEWPOINTS this was best placed in subThe teams started by clarifying The Accreditation Council for the Grad- competence “Patient Care 2 the components of a milestone uate Medical Education (ACGME) Next (PC2): Develops and achieves (Figure 1) and then defined the Entrustable Professional Accreditation System (NAS) requires a comprehensive management plan for each patient.” Activities (Table 2) of the different evaluation system for resiDuring this exercise, the specialty. The ASH/ASCO dents and fellows. groups thought that most of the teams appreciated that although Entrustable Professional Activ Although the ACGME has given programs Entrustable Professional Activities ities best fit in the subthe latitude to define approaches to the are usually composed of multiple competencies within Patient NAS, it would be helpful for specialty competencies, to make this Care, Medical Knowledge, and programs to share evaluation strategies. process work for our societies, Interpersonal Communication the Entrustable Professional Hematology and medical oncology spe- Skills. If the language in a Activities in hematology and cialties have worked together to specific IM Subspecialty oncology literally were written Reporting Milestone could be develop an approach to the NAS that we to fit to the “IM Subspecialty applied to the skills required for believe can be used by other societies Reporting Milestones” as best the subspecialty, it was not and programs. as possible. As the reader will changed. Thus, in the compesee, by fitting our Entrustable tencies of Systems-Based Professional Activities to the “IM Subspecialty Practice and Professionalism and most of PracticeReporting Milestones,” we developed one document that Based Learning and Improvement, no changes were can be used to create most of our tools to assess made. When the Entrustable Professional Activities rotations. The Entrustable Professional Activity cannot were integrated into the “IM Subspecialty Reporting be the tool to fit to the “IM Subspecialty Reporting Milestones,” there were 39 separate subcompetencies, Milestones” because the Entrustable Professional 25 unique to the specialty and 14 identical to the “IM Activity does not describe a behavior. Subspecialty Reporting Milestones” (Figure 2). An example of the process used is assignment of The work groups next addressed the task of articuEntrustable Professional Activity 17: “Writes accurate lating the progressive achievement of behaviors and and safe orders in the Electronic Medical Record for skills for each Entrustable Professional Activity. This systemic therapy and supportive care.” This activity, task is called streaming, and whenever possible, the which is an integral part of the subspecialty, requires
Table 1
Key Definitions in the Next Accreditation System
ACGME: Accreditation Council for Graduate Medical Education ACGME Core Competencies: knowledge, skills, or attitudes applied to the clinical environment in the following domains: patient care, medical knowledge, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice. Subcompetence: a component of a core competence. IM Reporting Milestones: the document that lists the progressive skills required for residents in IM to achieve competence in all required domains.4 IM Subspecialty Milestones: the document that lists the progressive skills required for fellows in the subspecialties of IM to achieve competence in all required domains.5 Entrustable Professional Activity: descriptions of observable work-based clinical activities that can serve as meaningful and manageable points of assessment.6 Curricular Milestones: the observable skills and activities required for residents or fellows to progress to competence in practice of a specialty or subspecialty. Clinical Competency Committee: ACGME-required committee tasked with reviewing all evaluations of all residents and fellows semiannually, and to ensure a report is submitted to the ACGME. ACGME ¼ Accreditation Council for Graduate Medical Education; IM ¼ Internal Medicine.
202
The American Journal of Medicine, Vol 128, No 2, February 2015
Figure 1 Deconstruction of a milestone. A sample format for a milestone stream showing the format for submission of this information to the ACGME.
work groups used similar language so that the readers of the document could easily see patterns and move from one subcompetence to another. The resulting document is the Hematology/Oncology Curricular Milestones, posted on the websites for ASH and ASCO on March 10, 2014.7,8
NEXT STEPS: DEVELOPMENT OF TOOLS TO INFORM THE “IM SUBSPECIALTY REPORTING MILESTONES” The evaluation system before the NAS has been criticized for being unable to describe behaviors of residents and fellows in the many different environments supervised by many different preceptors. With the NAS, direct observation to inform our assessment is required, because the Milestones documents describe behaviors. The beauty of the Curricular Milestones document is that by aligning subspecialty Entrustable Professional Activities to Subcompetencies, the necessary skills and behaviors in subspecialty work environments are described. This provides a connection between the “IM Subspecialty Reporting Milestones” and fellowship outcomes.
The next step is to determine which Entrustable Professional Activities can be observed on fellowship rotations. These should overlap with the Curricular Guidelines of each fellowship. It is best to be as specific as possible, so the skills and behaviors referenced are regularly observed on the rotation. For example, it would be easy to include a variation of “IM Subspecialty Reporting Milestone” Patient Care 1 “Gathers and synthesizes patient and disease specific information necessary to understand the presenting hematologic or oncologic disorder” on evaluation forms for every clinical rotation, because these skills are fundamental. However, the recommendation would be to include this subcompetence only on a service for which the fellow has to use the skill frequently. Table 3 shows sample rotation evaluations for clinical rotations common to Hematology/Oncology fellowships. Because the number of domains on each form is relatively modest, it allows faculty to focus their observations without overwhelming them. Some subcompetencies of the “IM Subspecialty Reporting Milestones” do not involve direct patient care activities. For these, the method we have described for creating tools from Curricular Milestones will not apply. The most obvious of these is MK3 (Medical
Collichio et al Table 2
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
203
Entrustable Professional Activities for Hematology/Oncology
1. Gathers and synthesizes patient- and disease-specific information necessary to understand the presenting hematologic or oncologic disorder 2. Demonstrates the ability to diagnose and assign stage or severity of hematology and oncology disorders in all adult age groups 3. Formulates the overall plan for hematology and oncology disorders, including urgent/emergent conditions 4. Demonstrates the ability to analyze response to treatment and adjust therapy for hematology or oncology disorders over time using standard measurements and guidelines 5. Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies 6. Demonstrates the ability to facilitate patient participation in clinical trials 7. Demonstrates the ability to effectively manage older adult patients with hematologic and oncologic diseases 8. Demonstrates understanding and effective application of principles of transfusion medicine 9. Demonstrates appropriate understanding and management of complications of transfusion 10. Demonstrates knowledge of, principles of, indications for, and complications from stem cell transplantation and ability to effectively manage these patients 11. Demonstrates the ability to effectively manage patients with pain, anxiety, and depression 12. Demonstrates the ability to effectively manage patients requiring palliative care, hospice care, or rehabilitation 13. Demonstrates the ability to effectively recognize and promote cancer prevention and control strategies and survivorship 14. Demonstrates the ability to effectively manage patients during transitions of care 15. Demonstrates competent performance of invasive procedures required for diagnosis, treatment, and management of patients with hematology and oncology 16. Demonstrates the ability to perform and interpret peripheral blood smears 17. Writes accurate and safe orders in the Electronic Medical Record for systemic therapy and supportive care 18. Requests and provides effective consultative care for patients with hematologic and oncologic diseases 19. Demonstrates a fund of knowledge in solid tumor oncology, malignant hematology, and non-neoplastic hematology 20. Demonstrates knowledge of and indications for genetic, genomic, molecular, and laboratory tests related to hematologic and oncologic disorders 21. Delivers safe, effective, patient-centered, cost-efficient care, and advocates for system improvements 22. Demonstrates personal habits of lifelong learning and self-improvement 23. Cares for patients in a manner that supersedes self-interest 24. Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care 25. Demonstrates appropriate use and completion of health records and procedure documents 26. Works effectively within an interprofessional team Competence in each of these clinical activities expected to be demonstrated by all graduates of combined fellowships in Hematology/ Oncology. Each EPA has significant developmental points that help define the training trajectory for each fellow. EPA ¼ Entrustable Professional Activity.
Knowledge 3) (Scholarship). In addition, the subcompetencies dealing with performance audits practiced-based learning and improvement 1 (PBLI-1): monitors practice with a goal for improvement, and PBLI-1: learns and improves via performance audit, and recognition of system errors systems-based practice-2 (SBP-2): recognizes system error and advocates for improvement generally are outside the scope of direct patient care activities. ASH and ASCO have developed modules9,10 that may assist programs in collecting normalized data in these areas, and other societies are encouraged to develop similar products, so program graduates will be well positioned to actively participate in the current dynamic clinical environment.
ROLL OUT AND EDUCATION OF FACULTY AND FELLOWS A critical step in the NAS is to roll out the new system and teach it to all of the stakeholders, from fellows to faculty. Eventually, there should be a seamless
approach starting in medical school. Faculty and fellows will need to be oriented to changes from standard evaluation templates, many of which currently use Likert scales instead of developmental milestones to evaluate performance in each competence or subcompetence. Fellows will need orientation and education about the types of specific feedback they will receive and the anticipated trajectory for mastery of skills and behaviors. Likewise, there will need to be more faculty development, particularly to make faculty comfortable with the developmental scale used for the milestones and increasing direct observation in assessing fellows. It is essential that all clinical supervisors understand the progressive expectations. The evaluations are intended to focus their judgment regarding performance of fellows. Faculty must feel empowered to provide specific feedback to both fellows and program leadership. Therefore, although the strategy we have proposed for constructing evaluations is fundamental to a new structure, it is not intended to be the only programmatic change in the NAS.
204
The American Journal of Medicine, Vol 128, No 2, February 2015
PC
MK
SBP
PBLI
PROF
ICS
PC1a
PC2g
PC3
MK1a
SBP1
PBLI1
PROF1
ICS1
PC2a
PC2h
PC4a
MK1b
SBP2
PBLI2
PROF2
ICS2
PC2b
PC2i
PC4b
MK1c
SBP3
PBLI3
PROF3
ICS3
PC2c
PC2j
PC4c
MK2
SBP4
PBLI4
PROF4
PC2d
PC2k
PC5
MK3
PC2e
PC2l
PC2f
PC2m
Figure 2 Comparison of “IM Subspecialty Reporting Milestones” and Hematology/Oncology curricular milestones. The 6 core competencies at the header in patient care, medical knowledge, systems-based practice, practiced-based learning and improvement, professionalism, and interpersonal communication skills. The boxes below the header represent the subcompetencies in each area. Pink: specific to hematology oncology. Grey: same as the subspecialty reporting milestones. ICS ¼ interpersonal communication skills; MK ¼ medical knowledge; PBLI ¼ practiced-based learning and improvement; PC ¼ patient care; PROF ¼ professionalism; SBP ¼ systems-based practice.
STEPS TO FOLLOW BY THE SUBSPECIALTY TO DEVELOP CURRICULAR MILESTONES 1. Identify the Entrustable Professional Activities for your specialty. 2. Fit the Entrustable Professional Activities according to the ACGME competence and subcompetence of the “IM Subspecialty Reporting Milestones” most relevant for the activity. 3. Articulate progressive achievement of behaviors or skills in tabular form under each point in step 2. 4. Once you have completed step 3 for all of the Entrustable Professional Activities you have organized, you have completed Curricular Milestones.
STEPS TO FOLLOW BY THE PROGRAM 1. Define the major rotations for your program. 2. Determine which of the Entrustable Professional Activities would best fit with the educational experiences on each rotation. 3. Add the corresponding Curricular Milestone narrative streams for each Entrustable Professional Activity and collate the group to make the new evaluation tool for faculty. 5. Make sure to “tag” the Curricular Milestone with the relevant reporting “IM Subspecialty Reporting Milestones” to which scores should be directed (included in Table 3).
6. Schedule meetings with faculty and fellows so they are aware of the new evaluation strategy and its rationale. 7. Ask the faculty members and interdisciplinary team members to discuss the evaluation tool with the fellow before and after the rotation. 8. The Clinical Competence Committee (CCC) will study the reports and fill out the “IM Subspecialty Reporting Milestones,” and determine whether remediation or other actions are necessary. 9. Submit the “IM Subspecialty Reporting Milestones” report for each fellow using the CCC recommendations.
CONCLUSIONS A key component of the NAS is evaluation of progressive development of training milestones using the “IM Subspecialty Reporting Milestones” every 6 months. We have shown a unique process whereby fellowship programs can fit the Entrustable Professional Activities as closely as possible to the “IM Subspecialty Reporting Milestones” document. When the progressive skills and behaviors are added, the resulting document becomes the Curricular Milestones of the subspecialty. Evaluation tools using the Curricular Milestones provide assessment of progress in achieving competence and give meaningful feedback to the fellow in the clinical context of the modern health care system. The strategy used provides a blueprint for subspecialties to define the fundamental aspects that should be included in all fellowship programs. Emphasis should be placed on faculty
Collichio et al Table 3
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
Selected Rotations for Hematology/Oncology Training Programs
A
Inpatient Consultative Service Evaluation
EPA
Observable Skill or Behavior
1
Gathers and synthesizes patient- and disease-specific information necessary to understand the presenting hematologic or oncologic disorder Demonstrates the ability to diagnose and assign stage or severity of hematology and oncology disorders in all adult age groups Formulates the overall plan for hematology and oncology disorders, including urgent/emergent conditions Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates the ability to effectively manage patients requiring palliative care, hospice care, or rehabilitation Demonstrates the ability to effectively manage patients during transitions of care Writes accurate and safe orders in the Electronic Medical Record for systemic therapy and supportive care Requests and provides effective consultative care for patients with hematologic and oncologic diseases Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care
2 3 5 12 14 17 18 24 B
Medical Oncology Specialty Clinic Evaluation
EPA
Observable Skill or Behavior
4
Demonstrates the ability to analyze response to treatment and adjust therapy for hematology or oncology disorders over time using standard measurements and guideline Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates the ability to facilitate patient participation in clinical trials Demonstrates the ability to effectively manage older adult patients with hematologic and oncologic diseases Demonstrates the ability to effectively recognize and promote cancer prevention and control strategies and survivorship Requests and provides effective consultative care for patients with hematologic and oncologic diseases Demonstrates knowledge of and indications for genetic, genomic, molecular, and laboratory tests related to hematologic and oncologic disorders Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care Demonstrates appropriate use and completion of health records and procedure documents
5 6 7 13 18 20 24 25 C
Hematology Clinic Evaluation
EPA
Observable Skill or Behavior
5
Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates understanding and effective application of principles of transfusion medicine Demonstrates appropriate understanding and management of complications of transfusion medicine Demonstrates the ability to effectively manage patients with pain, anxiety, and depression Demonstrates the ability to perform and interpret peripheral blood smears Demonstrates knowledge of and indications for genetic, genomic, molecular, and laboratory tests related to hematologic and oncologic disorders
8 9 11 16 20
205
Curricular Milestone
Reporting Milestone
PC1a
PC1
PC2a
PC2
PC2b
PC2
PC2d
PC2
PC2k
PC2
PC2m PC4c
PC2, SBP4 PC4b
PC5
PC5
ICS1
ICS1
Curricular Milestone
Reporting Milestone
PC2c
PC2
PC2d
PC2
PC2e PC2f
PC2 PC2
PC2l
PC2
PC5
PC5
MK2
MK2
ICS1
ICS1
ICS3
ICS3, PROF2
Curricular Milestone
Reporting Milestone
PC2d
PC2
PC2g
PC2
PC2h
PC2
PC2j
PC2
PC4b MK2
PC4b MK2
206 Table 3
The American Journal of Medicine, Vol 128, No 2, February 2015 Continued
C
Hematology Clinic Evaluation
EPA
Observable Skill or Behavior
21
Delivers safe, effective, patient-centered, cost-efficient care and advocates for system improvements Demonstrates appropriate use and completion of health records and procedure documents
25 D
Infusion Center Evaluation
EPA
Observable Skill or Behavior
8
Demonstrates understanding and effective application of principles of transfusion medicine Demonstrates competent performance of invasive procedures required for diagnosis, treatment, and management of patients with hematology and oncology Writes accurate and safe orders in the Electronic Medical Record for systemic therapy, including appropriate supportive care Demonstrates the ability to anticipate, recognize, and effectively manage toxicities of systemic therapies Demonstrates a fund of knowledge in solid tumor oncology Demonstrates a fund of knowledge in hematologic malignancies Demonstrates personal habits of lifelong learning and self-improvement Works effectively within an interprofessional team (eg, peers, consultants, nursing, and other health professionals)
15
17 5 19 19 22 26 E
BMT Service Evaluation
EPA
Observable Skill or Behavior
10
Demonstrates knowledge of principles of indications for and complications from stem cell transplantation and ability to effectively manage these patients Demonstrates competent performance of invasive procedures required for diagnosis, treatment, and management of patients with hematology and oncology disorders Writes accurate and safe orders in the Electronic Medical Record for systemic therapy, including appropriate supportive care Requests and provides effective consultative care for patients with hematologic and oncologic diseases Cares for patients in a manner that supersedes self-interest Communicates effectively and compassionately with patients, caregivers, and interprofessional teams during all phases of care Works effectively within an interprofessional team (eg, with peers, consultants, nursing, ancillary professionals, and other support personnel)
15
17 18 23 24 26
Curricular Milestone
Reporting Milestone
SBP3
SBP3
ICS3
ICS3, Prof2
Curricular Milestone
Reporting Milestone
PC2g
PC2
PC4a
PC4a
PC4c
PC4b
PC2d
PC2
MK1a MK1b PBLI4 SBP1
MK1 MK1 PBLI4 SBP1, ICS2, PROF1
Curricular Milestone
Reporting Milestone
PC2i
PC2
PC4a
PC4a
PC4c
PC4b
PC5
PC5
PROF3 ICS1
PROF3, PROF4 ICS1
SBP1, PROF1
SBP1, PROF1
A sample rotation evaluation for a core rotation for Hematology/Oncology fellowship. The relevant EPA is listed in column 1, and the observable clinical activity is listed in column 2. Column 3 shows the reference for the relevant Hematology/Oncology Curricular Milestone, and column 4 shows the relevant IM Subspecialty Reporting Milestone(s). Note that data from all 5 evaluations (A-E) must be taken together by the CCC to determine progress in all “IM Subspecialty Reporting Milestones” that are related to direct patient care activities. BMT ¼ blood and marrow transplantation; EPA ¼ Entrustable Professional Activity; ICS ¼ interpersonal communication skills; MK ¼ medical knowledge; PC ¼ patient care; PROF ¼ professionalism; SBP ¼ systems-based practice.
development so that training expectations are shared and embraced. Because evaluation of fellows on the basis of progressive development of specific skills is a new concept for our community, it is anticipated that there will be suggestions for improvement as we move forward. The intention is that Entrustable Professional Activities and Curricular Milestones will assist the community in achieving requirements of the NAS, and
that these documents will be shared and reviewed at regular intervals.
ACKNOWLEDGMENTS The authors thank Eric Holmboe, MD, for review of the article, and the working groups for the development of Entrustable Professional Activities and curricular
Collichio et al
NAS: Fellowship Programs Can Implement Standards Using a Hematology/Oncology Model
milestones. ASCO Working Group: Frances A. Collichio, MD, Jill Gilbert, MD, Robert Wolff, MD, Helen K. Chew, MD, Thomas George, Jr, MD, FACP, Lee M. Krug, MD, Alexandra P. Wolanskyj, MD, Ronald J. Maggiore, MD, Jennifer S. Temel, MD, and Tamara N. Shenkier, MD. ASH Working Group: Elaine A. Muchmore, MD, Kevin Imrie, MD, Charles Bolan, MD, Joanne Filicko-O’Hara, MD, Jason Gotlib, MD, Ann LeCasce, MD, Robert Redner, MD, Gerald Soff, MD, and Katherine Walsh, MD.
5.
6.
7.
References 1. Nasca T, Philibert I, Brigham T, Flynn T. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051-1056. 2. Irby D. Educating physicians for the future: Carnegie’s calls for reform. Med Teach. 2011;33:547-550. 3. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5:157-158. 4. Alliance for Academic Internal Medicine. ACGME and ABIM Release 22 Internal Medicine Reporting Milestones. Available at:
8.
9.
10.
207
http://www.im.org/AcademicAffairs/milestones/Pages/ACGME andABIMRelease22InternalMedicineReportingMilestones.aspx. Accessed March 30, 2014. Accreditation Council for Graduate Medical Education. The Internal Medicine Subspecialty Milestones Project. Available at: http://acgme.org/acgmeweb/Portals/0/PDFs/Milestones/Internal MedicineSubspecialtyMilestones.pdf. Accessed March 30, 2014. Accreditation Council for Graduate Medical Education. Next Accreditation System. Available at: http://www.acgme.org/ acgmeweb/tabid/435/ProgramandInstitutionalAccreditation/Next AccreditationSystem.aspx. Accessed March 30, 2014. American Society of Hematology. ASH and ASCO Present Hematology-Oncology Curricular Milestones. Available at: http://www.hematology.org/Training/Trainees/12505.aspx. Accessed March 30, 2014. American Society of Clinical Oncology. ASCO and ASH Present Hematology-Oncology Curricular Milestones. Available at: http://www.asco.org/professional-development/hematology-oncologycurricular-milestones. Accessed March 30, 2014. American Society of Clinical Oncology. QOPI: The Quality Oncology Practice Initiative. Available at: http://qopi.asco.org. Accessed March 30, 2014. American Society of Hematology. ASH Academy. Available at: http://www.ashacademy.org. Accessed March 30, 2014.
