Special Reports Assessing Competencies in a Master of Science in Clinical Research Program: The Comprehensive Competency Review Georgeanna F.W.B. Robinson, Ed.D.1, Charity G. Moore, Ph.D.2, Kathleen M. McTigue, M.D., M.S.1,3, Doris M. Rubio, Ph.D.1,3, and Wishwa N. Kapoor, M.D., M.S.1,3 Abstract Competencies in Master of Science Clinical Research programs are becoming increasingly common. However, students and programs can only benefit fully from competency-based education if students’ competence is formally assessed. Prior to a summative assessment, students must have at least one formative, formal assessment to be sure they are developing competence appropriate for their stage of training. This paper describes the comprehensive competency review (CCR), a milestone for MS students in Clinical Research at the University of Pittsburgh’s Institute for Clinical Research Education. The CCR involves metacognitive reflection of the student’s learning as a whole, written evidence of each competency, a narrative explaining the choice of evidence for demonstrating competencies, and a meeting in which two faculty members review the evidence and solicit further oral evidence of competence. CCRs allow for individualized feedback at the midpoint in degree programs, providing students with confidence that they will have the means and strategies to develop competence in all areas by the summative assessment of competence at their thesis defense. CCRs have also provided programmatic insight on the need for curricular revisions and additions. These benefits outweigh the time cost on the part of students and faculty in the CCR process. Clin Trans Sci 2015; Volume 8: 770–775
Keywords: competency assessment, education, degree milestones
Introduction
Clinical research as a discipline is relatively young.1 As such, in recent years it has benefited from being more clearly defined by the Clinical and Translational Science Award (CTSA) community. A workgroup from the former CTSA Education and Career Development Key Function Committee collaborated over several years to specify 14 thematic areas of clinical research in which competency statements were written.2 The definition of these competencies allows program directors of Master of Clinical Research degree programs to be sure that their curriculum is comprehensive and that students who have these competencies will be considered well trained by their colleagues around the country. Defining competencies also allows the discipline of clinical research to evolve and solidify with a defined body of knowledge, skills, and attitudes. The clinical research competencies have been broadly adopted by the CTSA educational community, which has used them to implement competency-based education (CBE). The utility and evolution of CBE for clinical and translational science have been further described elsewhere.3 In summary, the inclusion of defined competencies in a Master of Clinical Research degree program ensures that students deemed competent upon graduation have been well trained to conduct high quality, ethical research. Therefore, it is surprising that little progress has been made in establishing formal assessment of competencies in Clinical Research Master of Science (MS) programs. How best to assess competence is not a problem singular to this discipline; scholars have proposed assessment principles4 and quality criteria,5 but overall, recommendations about assessing competencies in academic, rather than clinical, programs remain diffuse. Because competencies are developmental and involve knowledge, skills, and attitudes, they are complex to achieve and require multiple practice opportunities. Therefore, in assessing competence, students should have multiple low-risk
opportunities to develop and demonstrate competence, as well as receive feedback prior to undergoing a summative high-stakes competency assessment.6 For a master’s degree, an obvious point for a summative assessment of competence is the thesis defense. A relatively early, formal, formative assessment in the degree program would allow students who were not fully competent the opportunity to develop the additional competence necessary to graduate. To meet this need, the Institute for Clinical Research Education (ICRE) created the comprehensive competency review (CCR). This paper outlines the CCR process, outcomes, and some challenges and benefits to both the students and the program. Methods
Development of CCR The MS in Clinical Research at the University of Pittsburgh is a 30-credit degree, which usually takes 2 years to complete. All master’s degree students select one of the following four tracks: T1 translational; clinical trials; comparative effectiveness; and health services research. Each track has a set of required courses in addition to common core courses, as well as some elective courses and thesis credits. Once the CBE group within the ICRE had identified the need for a formative, mid-point assessment of students’ competence, the first step was to develop a rubric against which students’ competence would be evaluated, both at the mid-point of the degree and at the thesis defense. Figure 1 presents the rubric for two of the eleven competencies: oral and written communication. The rubric can be found in its entirety on the ICRE website.7 The CBE group views competence as a continuum from no exposure to a skill set to advanced. However, for purposes of scoring students’ competence, four discrete levels were necessary,
Institute for Clinical Research Education, University of Pittsburgh, Pittsburgh, Pennsylvania, USA; 2Carolinas HealthCare System, Charlotte, North Carolina, USA; 3Department of Medicine, Division of General Internal Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA. 1
Correspondence: Georgeanna Robinson ([email protected]) DOI: 10.1111/cts.12322
770
VOLUME 8 • ISSUE 6
WWW.CTSJOURNAL.COM
Robinson et al. Assessing Competencies in a CTS MS Program n
Figure 1. Assessment rubric for the oral communication and written communication competencies.
and thus included in the rubric. The four levels of achievement for each of the 11 competencies that we use in the ICRE3,7 are: “not demonstrated,” “developing competence,” “competent,” and “advanced.” Faculty members use the rubric to assess where students are in each competency along the developmental continuum. For example, for each of the four levels of competence in the Problem Formulation competency, the rubric for assessing the student’s competence in proposing research questions reads: (1) not demonstrated—“No clear research question is identified;” (2) developing competence—“Research topic is outlined but research question is either poorly articulated or unanswerable;” (3) competent—“Research question is readily identifiable, and is testable;” and (4) advanced—“Research question is novel and significant, and clearly articulated as such.” Constructing portfolios is a common method for students to demonstrate and be assessed on their competence.6 We combined the portfolio approach with that of metacognition, or thinking about one’s learning. Metacognition involves evaluating one’s learning methods, efforts, and the associated outcomes. It encompasses both learning strategies: “What is the best way to learn this?” and self-appraisal of knowledge: “Do I know this WWW.CTSJOURNAL.COM
fact” or “Do I know how to do this?”.8 It has been described as enhancing learning8 and a central component to developing expertise.9 CCR document For the CCR, students provide one piece of written evidence for each of the 11 competencies. Evidence of competence can come from a variety of sources, such as students’ coursework, grant applications, manuscripts, e-mail strings, presentations, feedback from others, protocols the student has developed, and so on. Students are required to select only the portion of a longer document that specifically demonstrates the competency for which it is included. Students must also write a narrative introducing the piece of evidence. In this text, they explain what the evidence is and why it is appropriate for the competency concerned. Students are presented with a list of questions (Figure 2) to help them write this narrative or think about what to include. Rather than requiring an answer to each question, this list serves as a prompt to promote students’ thinking about their learning and how their competence is developing in each particular domain. Students are also told that faculty may ask them similar questions VOLUME 8 • ISSUE 6
771
Robinson et al. Assessing Competencies in a CTS MS Program n
the questions, because students often have limited experience in this practice. The three questions are intended to engage students in higher order thinking, encouraging them to take ownership of their mastery and pursue strategies that they recognize as having helped them in the past. In addition to prompting students to be aware of their learning and their ability to drive it, the answers to these reflective questions help faculty advise students regarding coursework and research goals for the remainder of the master’s program. Figure 2. Reflective questions to prompt students’ thinking in writing narratives for each piece of evidence of competence.
Figure 3. Reflective questions for which students must compose short answers.
Figure 4. CCR time line and process.
about the evidence during their CCR meeting. To reduce the time burden on students, the narrative and piece of evidence together may be no more than one page for each competency. In addition, students compose brief (250 words maximum) answers to three reflective questions to encourage them to think metacognitively. Short instructions follow each question to help students who may have limited experience thinking about their own learning (Figure 3). These instructions are designed to serve as guidance in metacognition as the students answer 772
VOLUME 8 • ISSUE 6
Student time line and process Figure 4 summarizes the CCR time line and process. The competency rubric is distributed to students at orientation when they enter the master’s program. Students are expected to complete the CCR once they have earned 15 of the 30 credits they need to earn a master’s degree. Therefore, in the semester in which students are due to complete 15 credits, program staff and advisors remind students of the CCR milestone and provide them with instructions regarding the process. The student is responsible for scheduling the CCR meeting. Admission to the required 3-credit grant writing course is withheld until students have completed the CCR. Therefore, students have a strong motivation to complete this milestone in a timely fashion. As students are preparing their CCR document, they have the option to meet with one member of the CBE group for an informal review of their document prior to submission to faculty. One week prior to the CCR meeting, the student sends the CCR document to two faculty members: the director of their master’s program track, who also serves as their academic advisor, and any one member of the four-person CBE group. The faculty members review the document and prepare questions for the student during their meeting. If anything in the document appears below the expected level or contrary to requirements, the two faculty members may request that the student resubmit new documents and reschedule the meeting.
CCR meeting During the CCR meeting, which lasts 60–90 minutes, the faculty members explain the formative nature of the CCR, emphasizing that students are not expected to be competent in all areas at this point. Rather, the intent of the CCR is to ensure that students are on track to become competent by the time of their thesis defense, to engage students in thinking about their learning, and to provide students with feedback on their current progress toward competence. The committee then questions the student about each element WWW.CTSJOURNAL.COM
Robinson et al. Assessing Competencies in a CTS MS Program n
of the document, starting with the three reflective questions and proceeding through each of the competencies and narratives. The discussion focuses on any elements that the faculty members think are particularly strong, particularly weak, missing, or interesting and pertinent to the degree completion or career plans of the student. Faculty members are intentional about discussing students’ learning processes and the student’s work in both creating the evidence, as well as choosing it for a specific competency. This aspect of the conversation allows faculty to be confident that they understand the decision process in choosing the evidence, the student’s effort in creating the work chosen as evidence, and the nature of any collaboration in either the creation or choice of evidence. At the end of the meeting, the student remains in the room, and the committee members indicate on a CCR Evaluation Form whether the student has demonstrated competence or whether additional work is needed for each of the 11 domains. Faculty members discuss with the student other steps the student might take to develop further competence wherever necessary; this is documented on the CCR form. The dialogue then shifts to a conversation more akin to an advising session, with faculty suggesting relevant courses to help the student to develop additional competence. The advisor leads the discussion of thesis plans, asking the student to describe their planned research projects and title of the thesis. Finally, the faculty members pass or fail the student for the CCR, and all parties sign the form. Students pass the CCR if they have written thoughtful answers to the three reflective questions, written persuasive and thoughtful narratives for each piece of evidence, and provided appropriate evidence for each competency. Importantly, needing additional work in any number of competency domains is unrelated to passing the CCR milestone. Students who do not engage thoughtfully or do not display any effort in putting their document together fail their CCR. These students are required to rewrite portions of their document, providing different evidence or improved narratives, as appropriate. Whether another meeting is necessary to discuss a new document, or whether the revision will be accepted via e-mail, is at the discretion of the faculty. Completion and any necessary further steps is tracked by the Degree Program Coordinator within the ICRE. Results
Outcomes and challenges In implementing the CCR, we encountered some initial challenges. While most students and MS track directors have understood the intent and process of the CCR, a minority of students needed significant coaching to engage in metacognitive reflection, which is an unfamiliar exercise for them. Also, the purpose of the narrative for each piece of evidence was unclear to a minority of students, who subsequently had to rewrite their narrative, either after a prereview or after the CCR meeting. Table 1 shows the percentages of students who were deemed competent in the first 18 months since the CCR was introduced as a milestone in the Clinical Research MS program. These data make evident one of the programmatic benefits of introducing the CCR milestone. Only one quarter of the students who have completed the CCRs thus far were deemed competent in Management; it became clear from discussions in the CCR meetings that students felt they had received limited instruction in this area, either as part of their master’s degree or informally from mentors while conducting their research. To fill this gap in the curriculum, a WWW.CTSJOURNAL.COM
Competency Problem formulation
No. (%) N = 20 17 (85)
Methodology
14 (70)
Sampling
18 (90)
Measurement
18 (90)
Data management and biomedical informatics
15 (75)
Applied analytical techniques
14 (70)
Oral communication
16 (80)
Written communication
18 (90)
Ethics and professional norms
19 (95)
Teamwork and leadership
14 (70)
Management
5 (25)
Table 1. Students deemed competent at the comprehensive competency review in each competency.
new course, “Managing Your Career in Clinical and Translational Science,” was created and approved by the curriculum committee. The course covers management and leadership basics to facilitate careers in this field. Students who need to develop additional competencies in this domain are now referred to this course. Another area of improvement pertained to the students’ abilities to write an analysis plan. Our biostatistics courses had mainly focused on conducting analysis but have now incorporated a written plan for analysis into each homework assignment prior to the analyses being done. This helps students think about how they will analyze data to align with a research question rather than focusing solely on statistical software commands. One challenge inherent in the CCR procedure is the ability to capture in writing competencies that are more behaviorally oriented. For example, providing evidence of oral communication, teamwork and leadership, and management has necessitated some creativity on the part of some students. For oral communication, some students have provided the title slide of a talk they presented at a conference. By itself, this evidence is not demonstrative of competence. However, a carefully crafted narrative describing how the individual prepared for the talk, how they felt they performed, and any feedback they received about the talk can provide the CCR committee with sufficient insight to determine where the student may be along the competency continuum. For evidence of teamwork skills, some students have very persuasively included an e-mail string between group members and a narrative to explain the team’s goals and highlighting their role in overcoming obstacles and making progress toward these goals. In some cases, students have struggled to determine what constitutes appropriate evidence of their competence in a particular domain, frequently from a lack of exposure to situations that require that skill set and knowledge. If the evidence provided in the CCR document suggests some level of experience and/or competence but does not provide a complete picture, faculty have been able to simulate during the CCR meeting a recreation of a situation through dialogue, facilitated role play, solicitation of feedback received by the student, and description of process and task- or situation-specific behaviors. Doing so provides faculty with additional information upon which to base their judgment of competence. VOLUME 8 • ISSUE 6
773
Robinson et al. Assessing Competencies in a CTS MS Program n
Metacognitive thinking has proved difficult for some students, particularly those with clinical, rather than research, backgrounds. Some students, throughout their academic careers until this point, have been told what knowledge to learn, what skills to acquire, and how to perform particular clinical tasks. It can be a daunting and new experience for these students to be asked to ponder their learning; the skills, knowledge, and attitudes that they have or have not yet acquired; what was easy or difficult; and where they need to focus their efforts next. In some cases, guided reflection during the CCR meeting has facilitated such metacognition and the articulation of pertinent thoughts on a student’s academic course in the master’s program. Student feedback on the CCR process Seventeen (85%) of the 20 students who have completed their CCR provided voluntary, anonymous feedback on the process via an online survey in which responses were framed on a 5-point Likert scale where 1 indicated “not at all” and 5 indicated “very much.” Likert values were averaged for a summary measure. Students felt that the CCR was somewhat to fairly beneficial for their learning (mean [SD] score = 3.4 [1.0]), and it was somewhat helpful in planning the remainder of their courses (mean [SD] score = 2.9 [1.0]). Writing free-text responses about how they would improve the CCR and other comments about the process, students felt that the CCR process was time-consuming, with several noting that it would be helpful to have a template or exemplary CCR documents available. They also felt that expectations of the document could have been communicated more clearly, and that it was important to ensure that the CCR actually occurred at the midpoint of the MS program, as it was too late after that to choose additional courses. A number of students appreciated the opportunity to pause in their studies and discuss their objectives and their related achievement individually with faculty. For example, one student commented, “I appreciated the time and effort that the faculty put into the review; their interest in my education and success was clear.” Students who articulated satisfaction in their comments with their mentoring and advising throughout the program noted that the CCR was of limited benefit for course selection, because they had been receiving similar advice throughout the program. Discussion
The introduction of the CCR milestone into the MS in Clinical Research program at the University of Pittsburgh has led to a number of benefits, both to students and the program as a whole, despite the added demands of time and effort required from students and faculty. Students all receive in-depth individual feedback from faculty on their academic progress and development of competencies important for clinical researchers. By going through this process, students can approach their thesis defense with greater confidence, knowing that they will have acquired the knowledge, skills, and attitudes appropriate for master’s-level graduates in clinical research. They can be confident that they have been trained adequately and learned what they expected to learn when applying to the degree program. They also have the opportunity to brainstorm any issues with faculty other than their mentors and plan their progress to graduation in a focused meeting with faculty that is less rushed than other advising meetings. Individualized training is becoming increasingly important in the CTSA community.10 The individual attention, feedback, and focus on students’ learning in this two-faculty-to-one-student 774
VOLUME 8 • ISSUE 6
context allows the program to ensure that students who have crafted original paths to the MS degree, or who have unusual academic goals, will receive appropriate and substantial training. The program has benefited from a clear demonstration of where additional didactic work is needed for the competencies that we expect our graduates to acquire. Conversations with students about their learning in general, and specific areas in which they would appreciate additional help, has already led to the creation of one new course, with another currently in the planning stage. We anticipate additional changes to the curriculum over time as we amass more evidence and as the career goals and needs of our students evolve. One of the most common complaints about the CCR from students is the time burden. Compiling 11 pieces of evidence and writing a narrative for each competency, as well as answering the three reflective questions, takes a considerable amount of time and effort. To alleviate this, many students would have appreciated a template or examples of well-received CCR documents. While this perspective is understandable, we do not anticipate providing either of these for students because doing so would disengage the student from reflective thinking. While this may be easier for the student, we feel the metacognitive process is time well spent. Metacognition is recognized as an important component to developing expertise11 and, like most elements of higher order thinking and advanced skill development, can be challenging. Engaging in metacognitive reflection demands individual thought, attention, and effort. For those new to reflective thinking, the process may be unwelcome because the benefits of the unfamiliar process may not be immediately obvious. Therefore, we take the opportunity of students’ complaints about the need to think deeply to help them acquire reflective skills and educate them about the benefits of metacognition. Another common theme from students concerned the timing of the CCR. In the initial year, a number of students were late in taking their CCR for various reasons. As a result, their remaining courses had already been established, and thus it was less helpful for them in selecting courses to complete their master’s degree. Attention to timing has been improved in the process, which will likely prevent this experience from recurring. The revised explanation and guidelines appear to be clearer, and the CCR is generally taken at the midpoint of the MS degree, as it was intended. Thus, initial implementation problems have been resolved, and students who have completed the CCR more recently reported higher appreciation of the process. Although the CCR process is time-consuming for both faculty and students, we deem it worth the effort and time cost. Through the process, students have an opportunity to engage more with their learning and connect it to their career goals. Faculty can provide substantive feedback to students, enhance student learning, and suggest other opportunities and connections from which students may benefit. The CCR meeting is an important component of the process; we stress here that programs wishing to replicate the process for assessment of their students’ competence should include it despite the potential challenges of scheduling busy faculty and students. The contribution faculty members make in guiding students in metacognition, discussing competency development, and leading students to articulate events and actions that clarify their competence in any particular domain is a vital addition to the written CCR document. Although faculty in the clinical and translational sciences may have little formal experience with metacognition, guiding students to think about their learning is a skill with which many academic advisors are WWW.CTSJOURNAL.COM
Robinson et al. Assessing Competencies in a CTS MS Program n
likely familiar, and one that can be readily learned by other faculty involved in CCRs. Conclusion
The CCR provides a formative assessment of competence for students in the MS in Clinical Research program at the University of Pittsburgh. It benefits the students individually and the program as a whole. The main resource cost is time of program faculty (i.e., track directors and members of the CBE group) and students. It allows students to be confident that they are on course to acquire the competencies they will need in their careers as clinical researchers. It allows the program to complete the CBE loop of defining, teaching, and assessing competencies, thereby ensuring that the program provides a comprehensive educational offering to its master’s-level students. Acknowledgments
The authors wish to thank Juliana Tambellini for her help gathering the student feedback data. Funding
This work was supported by the National Institutes of Health through grant number UL1TR000005 to the University of Pittsburgh.
WWW.CTSJOURNAL.COM
References
1. Switzer GE, Robinson GF, Rubio DM, Fowler NR, Kapoor WN. Doctoral programs to train future leaders in clinical and translational science. Acad Med. Sep 2013; 88(9): 1332–1339. 2. CTSA Consortium. Core competencies for clinical and translational research. 2015. Available at: https://ctsacentral.org/wp-content/documents/CTSA%20Core%20Competencies_%20final% 202011.pdf. Accessed March 3, 2015. 3. Dilmore TC, Moore DW, Bjork Z. Developing a competency-based educational structure within clinical and translational science. Clin Transl Sci. 2013; 6(2): 98–102. 4. McMullan M, Endacott R, Gray MA, et al. Portfolios and assessment of competence: a review of the literature. J Adv Nurs. 2003; 41(3): 283–294. 5. Baartman LKJ, Bastiaens TJ, Kirschner PA, van der Vleuten CPM. The wheel of competency assessment: presenting quality criteria for competency assessment programs. Stud Edu Evalu. 2006; 32(2): 153–170. 6. Kaslow NJ, Rubin NJ, Bebeau MJ, Leigh I, Lichtenberg J, Nelson P, Portnoy S, Smith I. Guiding principles and recommendations for the assessment of competence. Prof Psychol-Res Pr. 2007; 38(5): 441. 7. Institute for Clinical Research Education. Assessment rubric for MS in clinical research. 2015. Available at: http://www.icre.pitt.edu/cbe/MSrubric.pdf. Accessed March 3, 2015. 8. Paris SG, Winograd P. How metacognition can promote academic learning and instruction. In: Jones BF, Idol L, eds. Dimensions of Thinking and Cognitive Instruction. Elmhurst, Ill. Hillsdale, N.J.: North Central Regional Educational Laboratory; L. Erlbaum; 1990: 15–52. 9. Ertmer P, Newby T. The expert learner: strategic, self-regulated, and reflective. Instr Sci. 1996; 24(1): 1–24. 10. National Center for Advancing Translational Sciences. Funding Opportunity: Clinical and Translational Science Award (U54). Bethesda, MD: National Center for Advancing Translational Sciences; 2014. RFA-TR-14-009. 11. Sternberg R. Metacognition, abilities, and developing expertise: what makes an expert student? Instr Sci. 1998; 26(1–2): 127–140.
VOLUME 8 • ISSUE 6
775
Keywords: competency assessment, education, degree milestones
Introduction
Clinical research as a discipline is relatively young.1 As such, in recent years it has benefited from being more clearly defined by the Clinical and Translational Science Award (CTSA) community. A workgroup from the former CTSA Education and Career Development Key Function Committee collaborated over several years to specify 14 thematic areas of clinical research in which competency statements were written.2 The definition of these competencies allows program directors of Master of Clinical Research degree programs to be sure that their curriculum is comprehensive and that students who have these competencies will be considered well trained by their colleagues around the country. Defining competencies also allows the discipline of clinical research to evolve and solidify with a defined body of knowledge, skills, and attitudes. The clinical research competencies have been broadly adopted by the CTSA educational community, which has used them to implement competency-based education (CBE). The utility and evolution of CBE for clinical and translational science have been further described elsewhere.3 In summary, the inclusion of defined competencies in a Master of Clinical Research degree program ensures that students deemed competent upon graduation have been well trained to conduct high quality, ethical research. Therefore, it is surprising that little progress has been made in establishing formal assessment of competencies in Clinical Research Master of Science (MS) programs. How best to assess competence is not a problem singular to this discipline; scholars have proposed assessment principles4 and quality criteria,5 but overall, recommendations about assessing competencies in academic, rather than clinical, programs remain diffuse. Because competencies are developmental and involve knowledge, skills, and attitudes, they are complex to achieve and require multiple practice opportunities. Therefore, in assessing competence, students should have multiple low-risk
opportunities to develop and demonstrate competence, as well as receive feedback prior to undergoing a summative high-stakes competency assessment.6 For a master’s degree, an obvious point for a summative assessment of competence is the thesis defense. A relatively early, formal, formative assessment in the degree program would allow students who were not fully competent the opportunity to develop the additional competence necessary to graduate. To meet this need, the Institute for Clinical Research Education (ICRE) created the comprehensive competency review (CCR). This paper outlines the CCR process, outcomes, and some challenges and benefits to both the students and the program. Methods
Development of CCR The MS in Clinical Research at the University of Pittsburgh is a 30-credit degree, which usually takes 2 years to complete. All master’s degree students select one of the following four tracks: T1 translational; clinical trials; comparative effectiveness; and health services research. Each track has a set of required courses in addition to common core courses, as well as some elective courses and thesis credits. Once the CBE group within the ICRE had identified the need for a formative, mid-point assessment of students’ competence, the first step was to develop a rubric against which students’ competence would be evaluated, both at the mid-point of the degree and at the thesis defense. Figure 1 presents the rubric for two of the eleven competencies: oral and written communication. The rubric can be found in its entirety on the ICRE website.7 The CBE group views competence as a continuum from no exposure to a skill set to advanced. However, for purposes of scoring students’ competence, four discrete levels were necessary,
Institute for Clinical Research Education, University of Pittsburgh, Pittsburgh, Pennsylvania, USA; 2Carolinas HealthCare System, Charlotte, North Carolina, USA; 3Department of Medicine, Division of General Internal Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA. 1
Correspondence: Georgeanna Robinson ([email protected]) DOI: 10.1111/cts.12322
770
VOLUME 8 • ISSUE 6
WWW.CTSJOURNAL.COM
Robinson et al. Assessing Competencies in a CTS MS Program n
Figure 1. Assessment rubric for the oral communication and written communication competencies.
and thus included in the rubric. The four levels of achievement for each of the 11 competencies that we use in the ICRE3,7 are: “not demonstrated,” “developing competence,” “competent,” and “advanced.” Faculty members use the rubric to assess where students are in each competency along the developmental continuum. For example, for each of the four levels of competence in the Problem Formulation competency, the rubric for assessing the student’s competence in proposing research questions reads: (1) not demonstrated—“No clear research question is identified;” (2) developing competence—“Research topic is outlined but research question is either poorly articulated or unanswerable;” (3) competent—“Research question is readily identifiable, and is testable;” and (4) advanced—“Research question is novel and significant, and clearly articulated as such.” Constructing portfolios is a common method for students to demonstrate and be assessed on their competence.6 We combined the portfolio approach with that of metacognition, or thinking about one’s learning. Metacognition involves evaluating one’s learning methods, efforts, and the associated outcomes. It encompasses both learning strategies: “What is the best way to learn this?” and self-appraisal of knowledge: “Do I know this WWW.CTSJOURNAL.COM
fact” or “Do I know how to do this?”.8 It has been described as enhancing learning8 and a central component to developing expertise.9 CCR document For the CCR, students provide one piece of written evidence for each of the 11 competencies. Evidence of competence can come from a variety of sources, such as students’ coursework, grant applications, manuscripts, e-mail strings, presentations, feedback from others, protocols the student has developed, and so on. Students are required to select only the portion of a longer document that specifically demonstrates the competency for which it is included. Students must also write a narrative introducing the piece of evidence. In this text, they explain what the evidence is and why it is appropriate for the competency concerned. Students are presented with a list of questions (Figure 2) to help them write this narrative or think about what to include. Rather than requiring an answer to each question, this list serves as a prompt to promote students’ thinking about their learning and how their competence is developing in each particular domain. Students are also told that faculty may ask them similar questions VOLUME 8 • ISSUE 6
771
Robinson et al. Assessing Competencies in a CTS MS Program n
the questions, because students often have limited experience in this practice. The three questions are intended to engage students in higher order thinking, encouraging them to take ownership of their mastery and pursue strategies that they recognize as having helped them in the past. In addition to prompting students to be aware of their learning and their ability to drive it, the answers to these reflective questions help faculty advise students regarding coursework and research goals for the remainder of the master’s program. Figure 2. Reflective questions to prompt students’ thinking in writing narratives for each piece of evidence of competence.
Figure 3. Reflective questions for which students must compose short answers.
Figure 4. CCR time line and process.
about the evidence during their CCR meeting. To reduce the time burden on students, the narrative and piece of evidence together may be no more than one page for each competency. In addition, students compose brief (250 words maximum) answers to three reflective questions to encourage them to think metacognitively. Short instructions follow each question to help students who may have limited experience thinking about their own learning (Figure 3). These instructions are designed to serve as guidance in metacognition as the students answer 772
VOLUME 8 • ISSUE 6
Student time line and process Figure 4 summarizes the CCR time line and process. The competency rubric is distributed to students at orientation when they enter the master’s program. Students are expected to complete the CCR once they have earned 15 of the 30 credits they need to earn a master’s degree. Therefore, in the semester in which students are due to complete 15 credits, program staff and advisors remind students of the CCR milestone and provide them with instructions regarding the process. The student is responsible for scheduling the CCR meeting. Admission to the required 3-credit grant writing course is withheld until students have completed the CCR. Therefore, students have a strong motivation to complete this milestone in a timely fashion. As students are preparing their CCR document, they have the option to meet with one member of the CBE group for an informal review of their document prior to submission to faculty. One week prior to the CCR meeting, the student sends the CCR document to two faculty members: the director of their master’s program track, who also serves as their academic advisor, and any one member of the four-person CBE group. The faculty members review the document and prepare questions for the student during their meeting. If anything in the document appears below the expected level or contrary to requirements, the two faculty members may request that the student resubmit new documents and reschedule the meeting.
CCR meeting During the CCR meeting, which lasts 60–90 minutes, the faculty members explain the formative nature of the CCR, emphasizing that students are not expected to be competent in all areas at this point. Rather, the intent of the CCR is to ensure that students are on track to become competent by the time of their thesis defense, to engage students in thinking about their learning, and to provide students with feedback on their current progress toward competence. The committee then questions the student about each element WWW.CTSJOURNAL.COM
Robinson et al. Assessing Competencies in a CTS MS Program n
of the document, starting with the three reflective questions and proceeding through each of the competencies and narratives. The discussion focuses on any elements that the faculty members think are particularly strong, particularly weak, missing, or interesting and pertinent to the degree completion or career plans of the student. Faculty members are intentional about discussing students’ learning processes and the student’s work in both creating the evidence, as well as choosing it for a specific competency. This aspect of the conversation allows faculty to be confident that they understand the decision process in choosing the evidence, the student’s effort in creating the work chosen as evidence, and the nature of any collaboration in either the creation or choice of evidence. At the end of the meeting, the student remains in the room, and the committee members indicate on a CCR Evaluation Form whether the student has demonstrated competence or whether additional work is needed for each of the 11 domains. Faculty members discuss with the student other steps the student might take to develop further competence wherever necessary; this is documented on the CCR form. The dialogue then shifts to a conversation more akin to an advising session, with faculty suggesting relevant courses to help the student to develop additional competence. The advisor leads the discussion of thesis plans, asking the student to describe their planned research projects and title of the thesis. Finally, the faculty members pass or fail the student for the CCR, and all parties sign the form. Students pass the CCR if they have written thoughtful answers to the three reflective questions, written persuasive and thoughtful narratives for each piece of evidence, and provided appropriate evidence for each competency. Importantly, needing additional work in any number of competency domains is unrelated to passing the CCR milestone. Students who do not engage thoughtfully or do not display any effort in putting their document together fail their CCR. These students are required to rewrite portions of their document, providing different evidence or improved narratives, as appropriate. Whether another meeting is necessary to discuss a new document, or whether the revision will be accepted via e-mail, is at the discretion of the faculty. Completion and any necessary further steps is tracked by the Degree Program Coordinator within the ICRE. Results
Outcomes and challenges In implementing the CCR, we encountered some initial challenges. While most students and MS track directors have understood the intent and process of the CCR, a minority of students needed significant coaching to engage in metacognitive reflection, which is an unfamiliar exercise for them. Also, the purpose of the narrative for each piece of evidence was unclear to a minority of students, who subsequently had to rewrite their narrative, either after a prereview or after the CCR meeting. Table 1 shows the percentages of students who were deemed competent in the first 18 months since the CCR was introduced as a milestone in the Clinical Research MS program. These data make evident one of the programmatic benefits of introducing the CCR milestone. Only one quarter of the students who have completed the CCRs thus far were deemed competent in Management; it became clear from discussions in the CCR meetings that students felt they had received limited instruction in this area, either as part of their master’s degree or informally from mentors while conducting their research. To fill this gap in the curriculum, a WWW.CTSJOURNAL.COM
Competency Problem formulation
No. (%) N = 20 17 (85)
Methodology
14 (70)
Sampling
18 (90)
Measurement
18 (90)
Data management and biomedical informatics
15 (75)
Applied analytical techniques
14 (70)
Oral communication
16 (80)
Written communication
18 (90)
Ethics and professional norms
19 (95)
Teamwork and leadership
14 (70)
Management
5 (25)
Table 1. Students deemed competent at the comprehensive competency review in each competency.
new course, “Managing Your Career in Clinical and Translational Science,” was created and approved by the curriculum committee. The course covers management and leadership basics to facilitate careers in this field. Students who need to develop additional competencies in this domain are now referred to this course. Another area of improvement pertained to the students’ abilities to write an analysis plan. Our biostatistics courses had mainly focused on conducting analysis but have now incorporated a written plan for analysis into each homework assignment prior to the analyses being done. This helps students think about how they will analyze data to align with a research question rather than focusing solely on statistical software commands. One challenge inherent in the CCR procedure is the ability to capture in writing competencies that are more behaviorally oriented. For example, providing evidence of oral communication, teamwork and leadership, and management has necessitated some creativity on the part of some students. For oral communication, some students have provided the title slide of a talk they presented at a conference. By itself, this evidence is not demonstrative of competence. However, a carefully crafted narrative describing how the individual prepared for the talk, how they felt they performed, and any feedback they received about the talk can provide the CCR committee with sufficient insight to determine where the student may be along the competency continuum. For evidence of teamwork skills, some students have very persuasively included an e-mail string between group members and a narrative to explain the team’s goals and highlighting their role in overcoming obstacles and making progress toward these goals. In some cases, students have struggled to determine what constitutes appropriate evidence of their competence in a particular domain, frequently from a lack of exposure to situations that require that skill set and knowledge. If the evidence provided in the CCR document suggests some level of experience and/or competence but does not provide a complete picture, faculty have been able to simulate during the CCR meeting a recreation of a situation through dialogue, facilitated role play, solicitation of feedback received by the student, and description of process and task- or situation-specific behaviors. Doing so provides faculty with additional information upon which to base their judgment of competence. VOLUME 8 • ISSUE 6
773
Robinson et al. Assessing Competencies in a CTS MS Program n
Metacognitive thinking has proved difficult for some students, particularly those with clinical, rather than research, backgrounds. Some students, throughout their academic careers until this point, have been told what knowledge to learn, what skills to acquire, and how to perform particular clinical tasks. It can be a daunting and new experience for these students to be asked to ponder their learning; the skills, knowledge, and attitudes that they have or have not yet acquired; what was easy or difficult; and where they need to focus their efforts next. In some cases, guided reflection during the CCR meeting has facilitated such metacognition and the articulation of pertinent thoughts on a student’s academic course in the master’s program. Student feedback on the CCR process Seventeen (85%) of the 20 students who have completed their CCR provided voluntary, anonymous feedback on the process via an online survey in which responses were framed on a 5-point Likert scale where 1 indicated “not at all” and 5 indicated “very much.” Likert values were averaged for a summary measure. Students felt that the CCR was somewhat to fairly beneficial for their learning (mean [SD] score = 3.4 [1.0]), and it was somewhat helpful in planning the remainder of their courses (mean [SD] score = 2.9 [1.0]). Writing free-text responses about how they would improve the CCR and other comments about the process, students felt that the CCR process was time-consuming, with several noting that it would be helpful to have a template or exemplary CCR documents available. They also felt that expectations of the document could have been communicated more clearly, and that it was important to ensure that the CCR actually occurred at the midpoint of the MS program, as it was too late after that to choose additional courses. A number of students appreciated the opportunity to pause in their studies and discuss their objectives and their related achievement individually with faculty. For example, one student commented, “I appreciated the time and effort that the faculty put into the review; their interest in my education and success was clear.” Students who articulated satisfaction in their comments with their mentoring and advising throughout the program noted that the CCR was of limited benefit for course selection, because they had been receiving similar advice throughout the program. Discussion
The introduction of the CCR milestone into the MS in Clinical Research program at the University of Pittsburgh has led to a number of benefits, both to students and the program as a whole, despite the added demands of time and effort required from students and faculty. Students all receive in-depth individual feedback from faculty on their academic progress and development of competencies important for clinical researchers. By going through this process, students can approach their thesis defense with greater confidence, knowing that they will have acquired the knowledge, skills, and attitudes appropriate for master’s-level graduates in clinical research. They can be confident that they have been trained adequately and learned what they expected to learn when applying to the degree program. They also have the opportunity to brainstorm any issues with faculty other than their mentors and plan their progress to graduation in a focused meeting with faculty that is less rushed than other advising meetings. Individualized training is becoming increasingly important in the CTSA community.10 The individual attention, feedback, and focus on students’ learning in this two-faculty-to-one-student 774
VOLUME 8 • ISSUE 6
context allows the program to ensure that students who have crafted original paths to the MS degree, or who have unusual academic goals, will receive appropriate and substantial training. The program has benefited from a clear demonstration of where additional didactic work is needed for the competencies that we expect our graduates to acquire. Conversations with students about their learning in general, and specific areas in which they would appreciate additional help, has already led to the creation of one new course, with another currently in the planning stage. We anticipate additional changes to the curriculum over time as we amass more evidence and as the career goals and needs of our students evolve. One of the most common complaints about the CCR from students is the time burden. Compiling 11 pieces of evidence and writing a narrative for each competency, as well as answering the three reflective questions, takes a considerable amount of time and effort. To alleviate this, many students would have appreciated a template or examples of well-received CCR documents. While this perspective is understandable, we do not anticipate providing either of these for students because doing so would disengage the student from reflective thinking. While this may be easier for the student, we feel the metacognitive process is time well spent. Metacognition is recognized as an important component to developing expertise11 and, like most elements of higher order thinking and advanced skill development, can be challenging. Engaging in metacognitive reflection demands individual thought, attention, and effort. For those new to reflective thinking, the process may be unwelcome because the benefits of the unfamiliar process may not be immediately obvious. Therefore, we take the opportunity of students’ complaints about the need to think deeply to help them acquire reflective skills and educate them about the benefits of metacognition. Another common theme from students concerned the timing of the CCR. In the initial year, a number of students were late in taking their CCR for various reasons. As a result, their remaining courses had already been established, and thus it was less helpful for them in selecting courses to complete their master’s degree. Attention to timing has been improved in the process, which will likely prevent this experience from recurring. The revised explanation and guidelines appear to be clearer, and the CCR is generally taken at the midpoint of the MS degree, as it was intended. Thus, initial implementation problems have been resolved, and students who have completed the CCR more recently reported higher appreciation of the process. Although the CCR process is time-consuming for both faculty and students, we deem it worth the effort and time cost. Through the process, students have an opportunity to engage more with their learning and connect it to their career goals. Faculty can provide substantive feedback to students, enhance student learning, and suggest other opportunities and connections from which students may benefit. The CCR meeting is an important component of the process; we stress here that programs wishing to replicate the process for assessment of their students’ competence should include it despite the potential challenges of scheduling busy faculty and students. The contribution faculty members make in guiding students in metacognition, discussing competency development, and leading students to articulate events and actions that clarify their competence in any particular domain is a vital addition to the written CCR document. Although faculty in the clinical and translational sciences may have little formal experience with metacognition, guiding students to think about their learning is a skill with which many academic advisors are WWW.CTSJOURNAL.COM
Robinson et al. Assessing Competencies in a CTS MS Program n
likely familiar, and one that can be readily learned by other faculty involved in CCRs. Conclusion
The CCR provides a formative assessment of competence for students in the MS in Clinical Research program at the University of Pittsburgh. It benefits the students individually and the program as a whole. The main resource cost is time of program faculty (i.e., track directors and members of the CBE group) and students. It allows students to be confident that they are on course to acquire the competencies they will need in their careers as clinical researchers. It allows the program to complete the CBE loop of defining, teaching, and assessing competencies, thereby ensuring that the program provides a comprehensive educational offering to its master’s-level students. Acknowledgments
The authors wish to thank Juliana Tambellini for her help gathering the student feedback data. Funding
This work was supported by the National Institutes of Health through grant number UL1TR000005 to the University of Pittsburgh.
WWW.CTSJOURNAL.COM
References
1. Switzer GE, Robinson GF, Rubio DM, Fowler NR, Kapoor WN. Doctoral programs to train future leaders in clinical and translational science. Acad Med. Sep 2013; 88(9): 1332–1339. 2. CTSA Consortium. Core competencies for clinical and translational research. 2015. Available at: https://ctsacentral.org/wp-content/documents/CTSA%20Core%20Competencies_%20final% 202011.pdf. Accessed March 3, 2015. 3. Dilmore TC, Moore DW, Bjork Z. Developing a competency-based educational structure within clinical and translational science. Clin Transl Sci. 2013; 6(2): 98–102. 4. McMullan M, Endacott R, Gray MA, et al. Portfolios and assessment of competence: a review of the literature. J Adv Nurs. 2003; 41(3): 283–294. 5. Baartman LKJ, Bastiaens TJ, Kirschner PA, van der Vleuten CPM. The wheel of competency assessment: presenting quality criteria for competency assessment programs. Stud Edu Evalu. 2006; 32(2): 153–170. 6. Kaslow NJ, Rubin NJ, Bebeau MJ, Leigh I, Lichtenberg J, Nelson P, Portnoy S, Smith I. Guiding principles and recommendations for the assessment of competence. Prof Psychol-Res Pr. 2007; 38(5): 441. 7. Institute for Clinical Research Education. Assessment rubric for MS in clinical research. 2015. Available at: http://www.icre.pitt.edu/cbe/MSrubric.pdf. Accessed March 3, 2015. 8. Paris SG, Winograd P. How metacognition can promote academic learning and instruction. In: Jones BF, Idol L, eds. Dimensions of Thinking and Cognitive Instruction. Elmhurst, Ill. Hillsdale, N.J.: North Central Regional Educational Laboratory; L. Erlbaum; 1990: 15–52. 9. Ertmer P, Newby T. The expert learner: strategic, self-regulated, and reflective. Instr Sci. 1996; 24(1): 1–24. 10. National Center for Advancing Translational Sciences. Funding Opportunity: Clinical and Translational Science Award (U54). Bethesda, MD: National Center for Advancing Translational Sciences; 2014. RFA-TR-14-009. 11. Sternberg R. Metacognition, abilities, and developing expertise: what makes an expert student? Instr Sci. 1998; 26(1–2): 127–140.
VOLUME 8 • ISSUE 6
775
