MILITARY MEDICINE, 180, 4:113, 2015
Assessing Curriculum Effectiveness: A Survey of Uniformed Services University Medical School Graduates Katherine Picho, PhD*; COL William R. Gilliland, MC USA (Ret.)f; CDR Anthony R. Artino Jr., MSC USNf; LTC Kent J. DeZee, MC USA*; Ting Dong, PhD*; Col John E. McManigle, USAF MC (Ret.)f, David F. Cruess, PhD§; Steven J. Durning, MD, PhD*
ABSTRACT Purpose: This study assessed alumni perceptions of their preparedness for clinical practice using the Accreditation Council for Graduate Medical Education (ACGME) competencies. We hypothesized that our alumni’s perception of preparedness would be highest for military-unique practice and professionalism and lowest for system-based practice and practice-based learning and improvement. Method: 1,189 alumni who graduated from the Uniformed Services University (USU) between 1980 and 2001 completed a survey modeled to assess the ACGME competencies on a 5-point, Likert-type scale. Specifically, self-reports of competencies related to patient care, communication and interpersonal skills, medical knowledge, professionalism, systems-based practice, practice-based learning and improvement, and militaryunique practice were evaluated. Results: Consistent with our expectations as the nation’s military medical school, our graduates were most confident in their preparedness for military-unique practice, which included items assessing military leadership (M = 4,30, SD = 0.65). USU graduates also indicated being well prepared for the challenges of residency education in the domain of professionalism (M = 4.02, SD = 0.72). Self-reports were also high for competencies related to patient care (M = 3.86, SD = 0.68), communication and interpersonal skills ( M = 3.88, SD = 0.66), and medical knowledge (M = 3.78. SD = 0.73). Consistent with expectations, systems-based practice (M = 3.50, SD = 0.70) and practice-based learning and improvement ( M = 3.57, SD = 0.62) were the lowest rated competencies, although self-reported preparedness was still quite high. Discussion: Our findings suggest that, from the perspective of our graduates, USU is providing both an effective military-unique curriculum and is preparing trainees for residency training. Further, these results support the notion that graduates are prepared to lead and to practice medicine in austere environments. Compared to other compe tencies that were assessed, self-ratings for systems-based practice and practice-based learning and improvement were the lowest, which suggests the need to continue to improve USU education in these areas.
INTRODUCTION M edical school faculty play an essential role in society— educating physicians who will com plete additional training to provide health care for our nation. This is a com plex and daunting task with several notable challenges. C om pleting m edical school is only the first step in a physician’s training w ith the eventual “product” not yet having been realized. Thus, m edical educators grapple w ith selecting appropriate outcom e m easures for this phase o f education. Second, changes continue to em erge in our educational system s; these include the establishm ent o f new evaluation dom ains and the continued grow th o f m edical know ledge. Third, it can be difficult for m edical schools to track their graduates over the long term because o f the num ber o f residencies and practice *Department of Medicine. Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. tF. Edward Hebert School of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. ^Department of Military and Emergency Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. §Department of Preventive Medicine and Biometrics, Uniformed Ser vices University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of Defense or the U.S. Government. doi: 10.7205/MILMED-D-14-00570
MILITARY MEDICINE, Vol. 180, April Supplement 2015
locations available to graduates. An additional challenge for the U niform ed Services U niversity (USU) is our unique charge to educate m ilitary physicians with dual professional identities as both physicians and m ilitary officers. D espite these challenges, it is critical that USU track these outcom es to ensure that the school is fulfilling its societal obligation in both peacetim e and wartim e. The m ost prom inent new evaluation dom ain, a set o f six com petencies for all graduating residents, was developed in 2002 by the A ccreditation Council for G raduate M edical Education (A C G M E ).1 T his com petency rubric is consistent with several international com petency fram ew orks for gradu ate m edical education such as C anM eds2 and G ood M edical Practice.3 Som e m edical schools in the U nited States now use the A CG M E fram ew ork as a foundation for preparing their students for future success in graduate m edical education. W hen m easuring m edical school “ success,” one im portant perspective is that o f the graduates. Do they believe, when reflecting on their clinical experiences, that the m edical school has prepared them for their practice needs and chal lenges? Such evaluations, as part o f a larger portfolio of outcom es, can help m edical educators identify areas in their curriculum that m ight need revision. Such evaluations can also assist in the assessm ent o f m ore com plex outcom es such as preparedness for independent practice. To that end, we surveyed two decades o f USU graduates to assess their percep tions about their preparedness for residency training, using
113
Assessing Curriculum Effectiveness: A Survey ofU SU Medical School Graduates
the ACGME competencies as a framework. We believe that such aggregate data can help medical educators formulate better policies and educational practices. We hypothesized that USU graduates’ perceptions of preparedness would be highest for military-unique practice and professionalism given that USU is the only federal medical school in the United States, and, as such, can and does provide additional focus in these areas. Conversely, we hypothesized that graduates’ perceptions of preparedness would be lowest for the two ACGME com petencies that were not a focus of the curriculum during the time frame under investigation (i.e., 1980-2001): systems-based practice and practice-based learning and improvement. METHODS This study was part of the larger Long-Term Career Outcome Study (LTCOS) conducted by the F. Edward Hebert School of Medicine, the only federal medical school in the United States. This study was approved by USU’s Institutional Review Board. Participants and Procedures The LTCOS alumni questionnaire was initially sent electron ically to graduates who had earned their MD degree between 1980 and 2001. These data were collected in March 2012 using PHPSurveyor. To increase the response rate, we sent both paper and electronic reminders, and also reached out, via social media, to alumni through USU’s Alumni Affairs Office. Data collection was completed in December 2013. Questionnaire Development The questionnaire was developed through an interdepartmen tal vetting process involving LTCOS members, the USU’s Executive Curriculum Committee, and Associate Dean of Medical Education. Reliability and validity evidence for this instrument has been previously collected and reported else where.4 Items on this survey were subsequently mapped to the ACGME competencies and, through a deliberative group process, additional items to address ACGME competencies were created; these items were not included in previous ver sions of the survey. Following these revisions, we conducted several cognitive interviews with selected faculty and gradu ates and made additional edits to the survey. The full survey is provided in the Appendix. The competency section of the survey was organized into seven sections: (1) patient care (12 items), (2) com munication and interpersonal skills (10 items), (3) medi cal knowledge (3 items), (4) professionalism (5 items), (5) systems-based practice (3 items), (6) practice-based learn ing and improvement (7 items), and (7) military-unique practice (8 items). The items were anchored on a 5-point, Likert-type response scale with the following options: (1) poor, (2) significantly below your peers, (3) on par with your peers, (4) better than your peers, and (5) consistently higher level than most of your peers. A sixth option, unable to judge, was also included in the response scale but was treated as missing data in the analysis.
114
Data Analysis Analyses were conducted using the statistical software Stata 12 (College Station, Texas). Internal consistency (reliability) of the items measuring each of the competency factors was examined by reliability analysis using Cronbach’s a. Univar iate descriptive statistics were also calculated to assess rat ings on each of the competencies. RESULTS Surveys were electronically distributed to 2,825 graduates for whom e-mail addresses were available. Of these, we estimated that 2,400 alumni had valid contact information (i.e., e-mail addresses). Of these, 1,189 alumni completed and returned the survey, giving a response rate of approximately 50%. There were no significant differences on cumulative medical school GPA between alumni who participated in the study and those who did not (t = 1.57, df = 2958, p = 0.12). These results suggest that, relative to medical school per formance, our sample was representative of the entire USU alumni population who graduated between 1980 and 2001. The internal consistency reliability scores for each of the seven scales were excellent, with Cronbach’s a ranging from 0.95 (patient care) to 0.72 (systems-based practice) (Table I). We calculated descriptive statistics for the seven subsec tions of the alumni survey. Tables I and II show subscale means and standard deviations for each competency and cor responding items, respectively. Overall, mean scores were relatively consistent across over the decades, for all compe tencies (Fig. 1). Mean scores for competencies ranged from 3.50 (systems-based practice) to 4.04 (military-unique prac tice), with professionalism and military-unique practice being the highest rated competencies. Consistent with our expectations as the nation’s military medical school, our graduates were most confident in their preparedness for military-unique practice (M = 4.04, SD = 0.65), which included items assessing military leadership (M = 4.30, SD = 0.65). USU graduates also indicated being well prepared for the challenges of residency education in the domain of professionalism (M = 4.02, SD = 0.72). Selfreports were also high for competencies related to patient care (M = 3.86, SD = 0.68), communication and interpersonal skills (M = 3.88, SD = 0.66), and medical knowledge (M = 3.78, SD = 0.73). Consistent with expectations, systemsbased practice (M = 3.50, SD = 0.70) and practice-based TABLE I.
Competencies: Reliability and Mean Scores
Factor
N
M(SD)
Reliability, a
Patient Care Com m unication and Interpersonal Skills Medical Knowledge Professionalism System s-Based Practice Practice-Based Learning M ilitary-Unique Practice
1116 1118
3.86 (0.68) 3.88 (0.66)
0.95 0.95
1108 1116 1092 1112 1069
3.78 4.02 3.50 3.57 4.04
0.91 0.93 0.72 0.86 0.92
(0.73) (0.72) (0.70) (0.62) (0.65)
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey of USU Medical School Graduates TABLE II.
Descriptive Statistics of USU Graduates’ Competency Self-Evaluation Items (Classes of 1980-2001) P atien t C are
C o m m u n ic atio n and Interp erso n al Skills M ean
SD
In te rv iew P atien ts
4 .0 0
0.82
O ral C o m m u n icatio n Skills
3.97
0.81
P erform P h y sical E x am in atio n
3.86
0.83
W ritten C om m u n icatio n Skills
3.91
0.82
M ea n
SD
P erform D aily P atien t E v alu atio n s (C lin ical S etting)
3.89
0.83
R elatio n sh ip W ith P atients
4.01
0.80
P erform D aily P atien t E v a lu a tio n s (In p atien t S etting) P erform B asic T e c h n ic al Skills
3.87
0.84
R elatio n sh ip W ith F am ilies
0.82
3.92
0.86
R elatio n sh ip W ith P eers and O th e r H ealth
3.97 3.94
0.81
C are P ersonnel P erform A d v an ced P ro ced u ral Skills
3.73
0.92
E ffectiveness as a T e a c h er
3.86
0.81
A n aly sis o f C lin ical D ata and In te rp retatio n o f T ests
3.84
0.81
S ensitiv ity to P a tie n t's A ge and G ender
3.72
0.80
M an a g e m e n t o f P atien ts W ith L ife-T h reaten in g Illness
3.78
0.84
S ensitiv ity to P a tie n t’s C ulture and D isabilities
3.74
0.80
M an a g e m e n t o f P atien ts W ith C o m p lex Illness
3.70
0.83
E ffectiv en ess as a T eam M em ber
4.06
0.79
E ffectiv en ess W ith E nd o f L ife C are Issues
3.62
0.82
K n o w le d g e and S electio n o f T rea tm e n t O ptions
3.75
0.81
C o o rd in a tio n an d C o n tin u ity o f C are
3.77
0.81
P e rc e iv e d P rep aratio n fo r P a tie n t C are in a D ep lo y m e n t/ H u m an itarian E n v iro n m en t
4 .2 4
0.79
F u n d o f B asic S cien ce K n ow ledge
3.66
0.80
Initiative and M otivation
4.12
0.81
F und o f C lin ical S cien ce K n ow ledge
3.76
0.80
C o n scien tio u sn ess
4.12
0.80
C linical lu d g m e n t
3.91
0.80
E thical C o n d u ct
4.03
0.83
A w are o f O w n L im itation
3.92
0.81
W illin g n e ss to A dm it an E rro r in Judgm ent
3.92
0.81
M edical K n o w led g e
P ro fessio n alism
S y stem s-B ased P ractice
P ractice-B ased L e arning and Im provem ent
U n d e rstan d in g o f the C o n tex ts and Sy stem s in T R IC A R E
3.74
0.88
S e lf-D irected L e a rn in g Skills
3.69
C o n sid eratio n o f C o sts in D iag n o sis and M an ag em en t
3.25
0.90
T im e M an ag em en t Skills
3.70
0.84
Q u a lity A ssu ran ce and Im p ro v em en t In itiativ es
3.55
0.80
Q u ality M edical R ecord D ocu m en tatio n
3.79
0.81
A ccessing and C ritically E v a lu a tin g C urrent
3.57
0.78
E lec tiv e In v o lv em en t in R esearch
3.14
0.94
P articip atio n in V o lunteerism or H um anitarian
3.42
0.81
3.57
0.80
0.80
M edical In form ation
C lin ical A ctivities A dap tatio n to N ew T ech n o lo g y M ilitary -U n iq u e P ractice, D ep lo y m en t, an d H u m an itarian M issio n s M ilitary L ead ersh ip
4.21
0.75
A bility to C o n d u ct P atien t C are in D ep lo y ed
4.21
0.76
M ission E n v iro n m en t U n d e rstan d in g o f P sy ch o lo g ical Im pacts o f D ep lo y m en t
4.15
0.78
K now ledge o f C o m m o n P o std ep lo y m en t M ed ical o r
3.97
0.82
K now ledge o f E lectro n ic H ealth R eco rd A p p licatio n s
3.68
0.88
K n o w ledg e o f E lectro n ic H ealth R eco rd an d T ech n o lo g y U sed in T h e a te r
3.66
0.91
A bility to C ope W ith the Stress o f M ilitary M ed ical P ractice
4.13
0.79
A d ap tatio n to U n iq u e S itu atio n and S tresso rs
4.17
0.76
P sycholo g ical C o n d itio n s
learning and improvement (M = 3.57, SD = 0.62) were the lowest rated competencies, albeit graduate self-reported pre paredness was still high. An examination of individual items for each competency also revealed that respondents reported being better than their peers on preparation for patient care in a deployment or humanitarian environment and interviewing patients (patient care), and effectiveness as a team member and relationship with patients (communication and interper sonal skills). As expected, in general, self-ratings for systemsbased practice and practice-based learning and improvement were relatively low especially with regards to the average ratings on consideration of costs in diagnosis and management
MILITARY MEDICINE, Vol. 180, April Supplement 2015
(systems-based practice), as well as quality medical record documentation and elective involvement in research (practicebased learning and improvement). D IS C U S S IO N
Our findings provide additional evidence that USU is pro viding general and military residency training prepared ness, as reflected by our alumni’s views. Consistent with our expectations as the nation’s military medical school, our grad uates felt most prepared for professionalism and militaryunique practice to include high ratings for preparedness for leadership. Our curriculum offers a number of military-unique
115
Assessing Curriculum Effectiveness: A Survey of USE Medical School Graduates
Trends in Core Competencies: 1980-2001 o
u 5.00 --------------------------------------------------G
QJ
n. 4.50 --------------------------------------------------£
c 3.00 ---------------------------------------------------------g 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Years from 1980 - 2001 Patient Care
— Communication Skills
i. i—iProfessionalism
— —Systems Based Practice
■ ....Medical Knowledge " Practice Based Learning
.... . Military Unique Practice FIGURE 1.
Trends across time— 1980-2001.
components, including leadership activities and field exer cises that may account for these findings. These results are also consistent with views expressed by our alumni who have achieved flag officer rank.5 We believe these findings are important given that USU graduates are asked to practice medicine in austere environments. Alumni preparedness ratings were also high for patient care, communication and interpersonal skills, and medical knowledge. We believe this provides additional evidence that our medical curriculum educates our students well, at least from the graduates’ perspective. The opinions expressed by our graduates are also consistent with other outcome mea surements. For example, USU graduates have higher board certification rates than the national average.6 Comparatively, USU alumni were less confident on systemsbased practice and practice-based learning and improvement. However, this result was expected given that these competen cies were not formally defined, taught, or evaluated during the time frame under investigation. We believe that this lends additional validity evidence to our findings. We were pleased to find that our graduates felt they were at least on par with their peers (and in some cases, above their peers) with respect to these competencies. We continue to make changes to improve our students’ education in these needed competen cies to include additional emphasis being placed in the cur riculum on topics such as patient safety, reducing medical errors, cost-effective management strategies, evidence-based medicine, and a team-based approach to improving the quality of patient care. We believe that such findings can help make important contributions to curriculum reform efforts to include other relatively low ratings such as medical record docu mentation and involvement in research. For example, in our new curriculum, students have additional elective time for research projects and all students receive training in the use of our electronic medical record system throughout the curriculum starting in the preclerkship period. It will be inter
116
esting to see if graduates’ confidence in these areas improve in future studies involving more recent graduates.
Limitations and Recommendations for Future Research The study had a number of limitations. First, respondents were asked to self-assess their own competencies, and there is a fair amount of research in medical education, which suggests that individuals often do a poor job self-assessing broad competencies. Second, respondents were asked to rate their perceptions on different competencies using response options anchored on a 5-point Likert-type scale. These data, however, are self-reported and so they represent perceived competencies as opposed to actual competencies as measured by more objective assessment measurements. That said, find ings from more objective measures of our alumni’s perfor mance corroborate the findings presented here.6 Another potential limitation is nonresponse bias, although considering the fact that there were no differences between respondents and nonrespondents on medical school GPA, we feel non response bias is less problematic. Also, since respondents graduated over the course of a 20-year period, it is likely that the reflections or experiences recalled by respondents who graduated more than 5 years ago, might not be as accurate as that of more recent graduates. Therefore, the possibility of recall bias, especially among earlier graduates, cannot be overlooked as a potential limitation to the study. Last, this study was completed at a single institution and so there is no opportunity to compare the preparedness perceptions of our alumni with alumni form other medical schools.
CONCLUSION In summary, USU graduates indicate being well prepared for the challenges of residency education especially in the competences of military-unique practice and professionalism.
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates
This is reassuring since leadership and service to the nation is what sets USU apart from other schools. Findings from this study provide insight into which aspects of the curricu lum might be improved to further enhance student educa tion and competencies in areas outlined by the ACGME
and other accreditation bodies. Alumni perceptions about preparedness will be important to track over time and could provide medical educators with a viable outcome to study the effects of a wide range of medical school initiatives, such as curricular reform.
APPENDIX
USU ALUMNI SURVEY C lass (G r a d u a tio n Y e a r ) _______________ N a m e
T h a n k yo u f o r ta k in g th e tim e to p ro v id e fe e d b a c k on y o u r t r a in in g a t U S U . W e w ill u se a g g r e g a te d a ta fr o m th is s u rv e y to g u id e o u r d e c is io n s a b o u t o u r c u rric u la a n d o t h e r a s p e c ts o f m e d ic a l t r a in in g . T h e in fo r m a tio n w e g a th e r w ill re m a in c o n fid e n tia l a n d w ill n o t be s h a re d w ith y o u r s u p e rv is o rs . Y o u r c o n tr ib u tio n s to m ilit a r y m e d ic in e , in c lu d in g a n s w e r in g th is s u rv e y , a r e v e ry m u c h a p p r e c ia te d .______________________________________ A ) P o s t G r a d u a t e C a r e e r E x p e r ie n c e O la : I n w h a t a r e a d id y o u c o m p le te y o u r PG Y 1 tr a in in g ? I f y o u h a v e d o n e a t r a n s itio n a l y e a r b u t w e r e p re -s e le c te d fo r s p e c ia lty tr a in in g a t th e P G Y -2 le v e l b e fo re s ta r tin g P G Y -1 , p le a s e c h o o s e th e p r e s e le c te d s p e c ia lty (e .g . tr a n s itio n a l p r e s e le c t A n e s th e s ia , c h o s e A n e s th e s ia ). Please choose o n ly o n e o f th e fo llo w in g :
Pediatrics
Dermatology
O Neurology
□ o
Emergency Medicine
O OB/GYN
Q
Preventive Medicine
Q
Ophthalmology
Q Anesthesiology
□ □
Q Family Medicine □
□
□
□
Psychiatry Surgery
Q Internal Medicine
o n
Occupational Medicine Pathology
□
Physical and Rehab Medicine
Radiology
Joint PGY1
Transitional, not preselected for PGY-2 training at the start o f PGY-1
Other (please list)----------------------------------------- _ _
[O n ly a n s w e r th is q u e s tio n i f you a n sw e re d 'S u rg e ry ' to q u e s tio n 'O la '] 0 1 b : S u r g e r y - P le a s e s p e c ify ty p e : e .g . G e n e ra l, E N T, O R T H O , e tc . Please w rite y o u r a n s w e r h ere:
[O n ly a n s w e r th is q u e s tio n if you answ e re d 'J o in t PGY1 ' to q u e s tio n 'O la '] 0 1 c : J o in t PG Y 1 - P le a s e s p e c ify ty p e : e .g . p s y c h ia t r y /in t e r n a l m e d ic in e : Please w rite v o u r a n s w e r h ere:
0 2 : D id y o u h a v e a to u r o f d u ty a s a p h y s ic ia n b e fo re c o m p le tio n o f y o u r fir s t re s id e n c y ? Please choose o n ly o n e o f th e fo llo w in g :
Yes
U
No
[O n ly a n s w e r th is q u e s tio n if you a n sw e re d 'Yes' to q u e s tio n '0 2 '] 0 2 a : W h a t k in d o f to u r o f d u ty d id yo u c o m p le te as a p h y s ic ia n b e fo re y o u r f ir s t re s id e n c y ? Please choose o n ly o n e o f th e fo llo w in g :
M IL IT A R Y M E D IC IN E , Vol. 180, April Supplement 2015
117
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates Flight Surgeon
Dive Medical Officer_____ GMO (supporting peacetime or operational)
l—' O ther--------------------------------------------------------------------------------------
03: Have you completed your first residency? r~t t—1 .. 1No
Yes
04: In w h a t area did you com plete your first residency? Choose only one Anesthesiology
Dennatology
Emergency Medicine
Preventive Medicine
Family Medicine
Psychiatry
Radiation Oncology
Physical & Rehab Med Pediatrics
n Internal Medicine Radiology
• Neurology
Nuclear Medicine
Undersea Medicine
OB/GYN
Occupational Medicine
Military Service Unique
Surgery
Ophthalmology
Joint Residencies
Pathology
Other--------------------------
[Only answer this question if you answered 'Surgery ' to question '04 '] 04a: Surgery (please specify type; e.g. General, ENT, ORTHO, etc: Answer here:
[Only answer this question if you answered 'Military Service Unique ' to question '04 '] 04b: M ilitary Service Unique - please specify type: Write your answer here:
[Only answer this question if you answered 'Joint Residencies' to question '04 '] 04c: Joint Residencies - please specify type; e.g. p sv ch iatrv/in te rn al Answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 05: Did you com plete your first residency in a m ilitary hospital or facility (P rim ary Program A ffiliation Site)? Please choose only one of the following: Yes
- No
[Only answer this question if you answered 'Yes' to question '03 '] 06: Are you currently Board Certified in your first residency trained specialty? - Yes
No
[Only answer this question if you answered 'No' to question '06 '] 06a: I f you answered "No" to question 6, please explain: Please write your answer here:
118
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates
[Only answer this question if you answered 'Yes' to question '06 '] 06b: Have you recertified in your first residency trained specialty? '-
Yes
' No
[Only answer this question if you answered 'No' to question '06b '] 06c: I f you answered "No" to question 6b, please explain: Please write vour answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 07: Have you com pleted a second residency or fellowship? Yes
—^ No
[Only answer this question if you answered 'Yes' to question '07 '] 07a: Please identify the type of your second residency or fellow ship training: Please write vour answer here:____________
[Only answer this question if you answered 'Yes' to question '07 '] 07b: Did you com plete your second residency or fellow ship in a m ilitary hospital or facility (Prim ary Program Affiliation Site)? Yes
No
[Only answer this question if you answered 'Yes' to question '07 '] 07c: Are you currently Board Certified in your second residency or fellowship specialty/subspecialty? Please choose only one of the following: Yes
' No
[Only answer this question if you answered 'No' to question '07c '] 07d: I f you answered "No" to question 7c, please explain: Please write vour answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 08: Have you com pleted post-m edical school education resulting in a Masters Degree? (Please do not include any pre-m edical school education.) Yes
1—■ No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
119
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates [Only answer this question if you answered 'Yes' to question '08 '] 08a: Please specify the type of degree and your area of study: Write your answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 09: Have you com pleted post-m edical school education resulting in a Doctoral level degree (e.g ., PhD, DSc, JD)? (Please do not include pre-m edical school education.) Please choose only one of the following: Yes
No
[Only answer this question if you answered 'Yes' to question '09 '] 09a: Please specify the type o f degree and your area of study; Write vour answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 10: Have you com pleted post-m edical school healthcare adm inistration or m anagem ent education, which resulted in some kind of form al certification (e.g. ACPE, CHE, etc.)? (Please do not include pre-m edical school education Yes
No
[Only answer this question if you answered 'Yes' to question '10 '] 10a: Please specify the type of certificate and your area of study: Write your answer here:
B) O perational Experiences l l : As a physician, have you ever deployed for 30 or more consecutive days to a th e a te r of com bat operations? Yes
No
[Only answer this question if you answered 'Yes' to question ' l l '] 11a: How many tim es have you deployed for 30 or more consecutive days to a th e a te r of com bat operations? Please choose only one of the following: once
twice
3 times
4 times
5 times
6 or more times
[Only answer this question if you answered 'Yes' to question ' l l '] Total num ber of months d eployed in the last tw o years and career: Please write vour answer:
12: As a physician, have you ever deployed for 7 or m ore consecutive days on a peacetim e hum anitarian mission? Please choose only one of the following: -
120
Yes
1— '
No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey of USU Medical School Graduates
[Only answer this question if you answered 'Yes' to question '12 '] 1 2 a : H o w m a n y tim e s h a v e y o u d e p lo y e d fo r 7 o r m o re c o n s e c u tiv e d a y s on a p e a c e tim e h u m a n ita r ia n m is s io n ? Please choose o n ly o n e of the following:
□
once
twice
□
□
3 times
4 times
5 times
□
6 or more times
[Only answer this question if you answered 'Yes' to question '12 '] 12 b : A p p r o x im a te t o ta l n u m b e r o f days^Please write your answer here:
1 3 : H a v e y o u e v e r d e p lo y e d in s u p p o rt o f a s p e c ific o p e ra tio n a l (c o m b a t) o r h u m a n ita r ia n ( p e a c e t im e ) m is s io n in t h e s e n io r m e d ic a l le a d e rs h ip ro le (a s d e s ig n a te d by o ffic ia l o rd e rs , fo r e x a m p le , D iv is io n S u rg e o n , F le e t S u rg e o n , EM ED S C o m m a n d e r, e tc .)? Please choose o n ly o n e of the following:
Yes
No
[Only answer this question if you answered 'Yes' to question '13 '] 1 3 a : P le a s e lis t t it l e ( s ) o f p o s itio n (s ) (p le a s e d o n o t u s e a c ro n y m s ): Write your answer here:
C ) C a re e r L a n d m a rk s 14: W h ic h o f th e fo llo w in g p o s itio n s h a v e y o u r h e ld o v e r th e c o u rs e o f y o u r e n t ir e c a r e e r, if an y? Please choose a ll that apply:
Full-time Clinician (seeing patients at least 70% of the time during a typical week)
□ □
Chief of Service/Clinic (within an MTF) Chief/Director o f a Clinic (which is part of a larger MTF but located elsewhere; e.g. a branch clinic) Clinical Clerkship or Medical School Course Director
Department Cliair/Chief?Head at a Medical Center or Community Hospital without a residency program in your department Department Chair/ChiefHead at a Medical Center or Community Hospital with a residency program in your department Service ChiefDirector/Head of multidisciplinary service at a Medical Center or Community Hospital (e.g. Chief of all Surgical Services) Military Treatment Facility (MTF) Chief of the Medical Staff (SGH, DCCS, Medical Director) D em i tv M TF F o m m a n H in o O f f ir p r tttQ M — F v .v i i t i ,- / , Off! cer) Deputy MTF Commanding Officer (USN —Executive
Fellow ship director
Brigade surgeon
Squadron com mander
Senior medical officer
□
Residency Program Director D ivision surgeon Research director
Other (list up to 3 roles)__
1 5 : H a v e yo u h e ld th e p o s itio n o f C o m m a n d in g O ffic e r (o n o ffic ia l o rd e rs d e s ig n a tin g C o m m a n d e r o r C o m m a n d in g O ffic e r a n d th u s p o s sessin g UCMJ le g a l a u t h o r ity )? Please choose o n ly o n e of the following:
Yes
No
1 5 a : I f y e s , w h ic h o f th e fo llo w in g p o s itio n s , if a n y , h a v e y o u held?
MILITARY MEDICINE, Vol. 180, April Supplement 2015
121
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates Please choose all that apply: Free-Standing Clinic Commanding Officer
Medical Center Commanding Officer
Bedded Community-Sized Hospital Commanding Officer Squadron or similar service-specific organizational Commanding Officer Deployed/Deployable Medical Unit Commanding Officer Other list positions____________________________________
16: Have you held a staff position at your service's Medical Headquarters SG Office level? Please choose only one of the following: Yes
No
17: Have you ever had a "jo in t" assignm ent (e.g ., DoD, USU, JtF, etc)? I—' Yes
— No
18: W hat is the highest rank you have achieved to date? I f you are currently on a "select" list, please identify the rank selected for prom otion. n n n n n 03
'*—'0 4
05
*—1 06
0 7 and above
19: Have you ever been selected for prom otion, as a physician, below the prim ary zone for eligibility (i.e ., "deep select")? Choose only one: Yes
No
[Only answer this question if you answered 'Yes' to question '19 '] 19a: To which of th e follow ing ranks does this apply? (check all th a t apply) Choose all that apply: 04
05
06
20: Did USU prepare you for your current (o r past) career leadership roles? Yes
No
20a: W hy or w hy not? Please write vour answer here:
21: Have you ever engaged in volunteer medical activities (fo r exam ple, public clinic)? Please choose only one of the following:
Yes
No
22: Have you ever published an article in a professional peer-review ed journal? Please choose only one of the following:
Yes
No
22a: Approxim ately how many published articles in the last 12 months and last 5 years? Last 12 m onths:__________Last 5 years____________ 23: Have you ever presented papers before national or international professional groups? Please choose only one of the following: Yes
No
23a: I f yes, approximately how many in the last year and the last 5 years? Last y e a r___________ Last 5 years_____________
122
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates 24: Have you ever presented papers before m ilitary professional groups? Yes
O No
24a: If yes, approximately how many in the last year and the last 5 years? Last year _________ Last 5 years________________ 25: Have you ever served as a review er for a peer-review ed journal? —1 Yes No 25a If yes, approximately how many medical journals as a peer reviewer?______________ 26: Have you ever served as a principal investigator on a research grant? Yes
No
[Only answer this question if you answered 'Yes' to question '26 '] 26a: Total num ber o f grants as a principal investigator: Please write vour answer here:
27: Do you currently hold one or m ore university academ ic appointm ents? Yes
No
[Only answer this question if you answered 'Yes' to question '27 '] 27a: Which of the following title s best describes the highest academ ic appointm ent you have ever held? Please choose only one of the following: Instructor
Assistant Professor
Associate Professor
Professor
Other (list)____
[Only answer this question if you answered 'Yes' to question '27 '] 27c: Which of the following best describes the university from which you received the above appointm ent? Please choose only one of the following: USU
Other Medical School
Other college/university (list)_____________________
28: As a physician, which of th e follow ing medals have your received, if any? Please choose all that apply: (—1 /■““N Bronze Star Purple Heart Silver Star Air Medal
1— Navy Cross
Air Force Cross
POW Medal
Legion of Merit
Defense Meritorious Service
United Nations
Distinguished Flying Cross Defense Superior Service Humanitarian Service Medal
r-i
Other - not campaign medals (list)
29: Has your Active Duty career concluded? Choose only one of the following: *- ^ Yes
^ ^ No
[Only answer this question if you answered 'Yes' to question '29 '] 30: Did you separate before retirem ent? Please choose only one of the following: Yes
No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
123
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates [Only answer this question if you answered 'Yes' to question '29 ’] 31: Did you continue to serve in an Active R eserve/G uard Unit? Please choose only one: f— \
w Yes
' ■No
[Only answer this question if you answered 'Yes' to question '29 '] 32: Which of the follow ing statem ents best describes your service? Please choose only one of the following:
I retired after 20 years or more commissioned Medical Corps service. I retired after 20 or more years of total service, but less than 20 years commissioned Medical Corps service. I separated prior to 20 years total active service as a result of a Medical Board action. D ) M e d ic a l School E x p e rie n c e 33: When considering which medical school you would attend, how im portant w e re the financial aspects of attending USU versus other institutions in the civilian setting? The IMPORTANCE of financial aspects of attending USU w ere: Please choose one not at all important
slightly
n somewhat important
- quite important
extremely important
34: W hen you began taking classes at USU, w h at was your m arital status? Please choose only one of the following: a t _______ _ Not married
Married
Married but separated
Divorced
............... _ ......
Widowed
35: W hen you began taking classes at USU, did you have dependent children? Please choose only one of the following:
Yes
.. No
36: W hen you graduated from USU, w h at was your m arital status?
n Not married
Married
Married but separated
u
Divorced
Widowed
37: W hen you graduated from USU, did you have dependent children?
L 1Yes
No
38: FOR QUESTIONS 39a - 39f, PLEASE EVALUATE YOUR PREPAREDNESS IN COMPARISON TO ALL OF YOUR PEERS IN YOUR CLASS YEAR OF RESIDENCY: 39a: Patient Care Please choose the appropriate response for each item: S ig n ific a n tly
124
B e tte r
C o n s is te n tly
w ith
th a n
h ig h e r le v e l th a n
b e lo w y o u r
your
your
m ost o f your
U n a b le
p e e rs
p e e rs
p e e rs
p e e rs
to ju d g e
Q □ □ rr
Q O □ □
□ □ □ □
□ □ □ □
O □
CD
CD
□
□
□
Poor
In te rvie w in g patients a n d /o r fam ilies fo r initial histories Perform ing a physical exam ination Perform ing daily pa tien t evaluations (clinic se ttin g ) Perform ing daily pa tien t evaluations (in p a tie n t se ttin g ) Perform ing basic technical skills (i.e. inserting IV, sutu ring , inserting a-line )
O n par
o
□
□ □
□
□
t,,ni
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates P e rfo rm in g a d v a n c e d p ro ce d u ra l s k ills (i.e . in s e rtin g c e n tra l lin e , o p e ra tin g ro o m p e rfo rm a n c e )
□
A n a ly s is o f c lin ic a l d a ta , d iffe re n tia l d ia g n o s is and s e le c tio n /in te rp re ta tio n o f te sts
□
□
Q
□
□
□
□
□
□
□
M a n a g e m e n t a n d /o r re fe rra l o f p a tie n ts w ith life th re a te n in g illn e ss
□
□
Q
□
o
o
M a n a g e m e n t o f p a tie n ts w ith c o m p le x m u lti-s y s te m illness
□
□
o
□
□
□
K n o w le d g e a nd se le c tio n o f tr e a tm e n t o p tio n s /p a tie n t m anagem ent
□
□
□
□
□
□
C o o rd in a tio n and c o n tin u ity o f ca re
Q
□
□
□
D
□
P erceived p re p a ra tio n fo r p a tie n t ca re in a d e p lo y m e n t o r h u m a n ita ria n m issio n e n v iro n m e n t
□
□
□
□
□
□
3 9 b : C o m m u n ic a tio n a n d In te r p e r s o n a l S k ills Please choose th e aDDroDriate response fo r each ite m : On par with your peers
Significantly below your peers
Poor
O ral c o m m u n ic a tio n s k ills
□
Q
W ritte n c o m m u n ic a tio n s k ills
□
R e la tio n s h ip s w ith p a tie n ts R e la tio n s h ip s w ith fa m ilie s R e la tio n s h ip s w ith p e ers, s ta ff and o th e r h e a lth ca re p e rs o n n e l
Better than your peers
Consistently higher level than most of your peers
Unable to judge
□
□
□
□
□
□
□
□
□
□
□
□
Q
□
□
□
□
□
□
□
□
□
Q
□
O
□
E ffe c tiv e n e s s as a te a c h e r
0
o
□
□
D
S e n s itiv ity to p a tie n t's age a n d g e n d e r
□
Q
□
o
□
□
S e n s itiv ity to p a tie n t's c u ltu re and d is a b ilitie s
□
□
□
□
□
0
E ffe c tiv e n e s s as a te a m m em ber
□
□
□
□
□
Q
E ffe c tiv e n e s s w ith end o f life ca re issues
□
□
Q
□
□
□
3 9 c : M e d ic a l K n o w le d g e Please choose th e a p p ro p ria te response fo r each ite m : Poor
Fund o f basic science k n o w le d g e Fund o f clin ic a l science
□ □
Significantly below y our peers
On par with your peers
Better than your peers
Consistently higher level than m ost o f your peers
Unable to judge
□ □
□ □
□ □
□ □
□ □
M I L I T A R Y M E D IC IN E , Vol. 180, April Supplement 2015
125
Assessing Curriculum Effectiveness: A Sun’ey of USE Medical School Graduates knowledge
(—\
Clinical ju d g m e n t
1 -J
□
□
□
P le a s e c h o o s e t h e a D D r o D r ia te r e s p o n s e f o r e a c h i t e m : Consistently higher Better On par level than most of Significantly with your than your Poor peers peers your peers below your peers
3 9 d : P r o fe s s io n a lis m
In itia tiv e and m otivation
LJ
Conscientiousness
O
Ethical conduct
Q
Aware o f own lim ita tio n s W illingness to a d m it an e rro r in ju d g m e n t
□ □ □ □ □
□ u
o □ □ □ □
0
□ □ □
Unable to judge Q
□ Q □
□
□ □ □
/—•» LJ
□
□
3 9 e : S y s te m s -B a s e d P ra c tic e & P ra c tic e -B a s e d L e a rn in g a n d Im p r o v e m e n t
Poor
Significantly below your peers
S elf-directed learning skills
□
□
Tim e m anagem ent skills
O
□
Medical record documentation quality Accessing and critically evaluating cu rre n t medical inform ation and scientific evidence U nderstanding o f the contexts and system s in TRICARE Elective invo lve m e n t in research Participation in volu n te e rism , social, or hum anitarian clinical activities C onsideration o f costs in diagnosis and m anagem ent Adaptation to new technology Q uality assurance and im p ro ve m e n t in itia tive s
On par with your peers
u
Better than your peers
Consistently higher level than most o f your peers
Unable to judge
.
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
LJ
LJ
□
□
O
□
i \
o
□
V____ J
□
□
□
0
□
□
□
o
□
□
□
Q
□
□
□
□
□
□
□
□
□
□
Q
□
□
3 9 e e : I f y o u s e le c te d " E le c tiv e in v o lv e m e n t in re s e a rc h " p le a s e s p e c ify ty p e :
Please write vour answer here:
3 9 e e e : I f y o u s e le c te d " P a r tic ip a tio n in v o lu n te e r is m , s o c ia l, o r h u m a n ita r ia n c lin ic a l a c tiv itie s " p le a s e s p e c ify : Please write vour answer here:
126
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates
39f: M ilitary Unique Practice, Deploym ents, and H um anitarian Missions Please choose the appropriate response for each item: O n par
Poor
B e tte r
S ig n if ic a n tly
w ith
th a n
C o n s is te n tly
b e lo w y o u r
your
your
h ig h e r le v e l th a n
U n a b le
p e e rs
p e e rs
p e e rs
m o s t o f y o u r p e e rs
to ju d g e
Military leadership
□
O
□
Ability to conduct patient care in deployed or humanitarian mission environment Understanding of psycho social impacts of deployment on service members' families Knowledge of common post deployment medical or psychological conditions (e.g. TBI, PTSD) Knowledge of electronic health record applications Knowledge of electronic health record and technology used in theater Ability to cope with the stress of military medical practice Adaptation to unique situations and stressors in military medical practice
□ □
□
Q
□ □
□ □
□
O
Q
Q
□
□ □
□
Q
□
□
Q
□
□
□
□
©
□
□
□
□
□
□
□
□
□
o
□
□
□
□
□
□
Q
□ □
41: I believe my peers would tru st me to participate in the care of one of th eir loved ones: Please choose only one of the follow ing: - ‘ Yes
No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
O 1 Not sure
127
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates W h y o rW h y n o t?
4 2 : I n g e n e r a l, h o w w e ll d id y o u r m e d ic a l sch ool e d u c a tio n p re p a re yo u fo r y o u r c a r e e r in m e d ic in e ? Please choose o n ly o n e of the follow ing:
Not well at all
Slightly well
Moderately well
Quite well
Extremely well
4 3 : F ro m y o u r c a r e e r e x p e r ie n c e , in w h a t a s p e c ts , if a n y , do y o u fe e l USU m e d ic a l e d u c a tio n p ro g ra m is u n iq u e (c o m p a re d w ith c iv ilia n m e d ic a l s c h o o ls )? P le a s e a ls o s h a re a d d itio n a l in fo rm a tio n , c o m m e n ts o r fe e d b a c k fo r USU: Write your answer here (you may use the back side of page if you like):
S u b m it Y o u r S u rv e y .
Thank you for completing this survey. Please email your completed survey to [email protected] ________________ or fax your completed survey to Steven Durning at 301-295-3557.________________
REFERENCES 1. Accreditation Council for Graduate Medical Education. The Next Accreditation System: A Resident Perspective. Available at www .acgme.org/acgmeweb/Portals/0/PDFs/Resident-Services/9NASResidents May2014.pdf; accessed September 10, 2014. 2. General Medical Council. Good Medical Practice. Available at http:// www.gmc-uk.org/guidance/good_medical_practice.asp; accessed Sep tember 9, 2014 3. Royal College of Physicians and Surgeons of Canada. The Draft CanMEDS 2015. Physician Competency Framework. Available at http://www.royalcollege.ca/porlal/page/portal/rc/common/documents/
128
canmeds/framework/framework_series_l_e.pdf; accessed September 9, 2014. 4. Durning SJ, Pangaro LN, Lawrence LL, Waechter D, McManigle J, Jackson JL: The feasibility, reliability, and validity of a program director’s (supervisor’s) evaluation form for medical school graduates. Acad Med 2005; 80: 964. 5. Dong T, Durning SJ, Gilliland W R, et al ; Leadership success and the Uniformed Services University: perspectives of flag officer alusmni. Mil Med 2012; 177(9): 61-7. 6. DeZee K, Durning SJ, Dong T, et al: Where are they now? USU School of Medicine graduates after their military obligation is complete. Mil Med 2012; 177(9): 68-71.
M IL IT A R Y M E D IC IN E , Vol. 180, A pril Supplem ent 2015
Copyright of Military Medicine is the property of AMSUS and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Assessing Curriculum Effectiveness: A Survey of Uniformed Services University Medical School Graduates Katherine Picho, PhD*; COL William R. Gilliland, MC USA (Ret.)f; CDR Anthony R. Artino Jr., MSC USNf; LTC Kent J. DeZee, MC USA*; Ting Dong, PhD*; Col John E. McManigle, USAF MC (Ret.)f, David F. Cruess, PhD§; Steven J. Durning, MD, PhD*
ABSTRACT Purpose: This study assessed alumni perceptions of their preparedness for clinical practice using the Accreditation Council for Graduate Medical Education (ACGME) competencies. We hypothesized that our alumni’s perception of preparedness would be highest for military-unique practice and professionalism and lowest for system-based practice and practice-based learning and improvement. Method: 1,189 alumni who graduated from the Uniformed Services University (USU) between 1980 and 2001 completed a survey modeled to assess the ACGME competencies on a 5-point, Likert-type scale. Specifically, self-reports of competencies related to patient care, communication and interpersonal skills, medical knowledge, professionalism, systems-based practice, practice-based learning and improvement, and militaryunique practice were evaluated. Results: Consistent with our expectations as the nation’s military medical school, our graduates were most confident in their preparedness for military-unique practice, which included items assessing military leadership (M = 4,30, SD = 0.65). USU graduates also indicated being well prepared for the challenges of residency education in the domain of professionalism (M = 4.02, SD = 0.72). Self-reports were also high for competencies related to patient care (M = 3.86, SD = 0.68), communication and interpersonal skills ( M = 3.88, SD = 0.66), and medical knowledge (M = 3.78. SD = 0.73). Consistent with expectations, systems-based practice (M = 3.50, SD = 0.70) and practice-based learning and improvement ( M = 3.57, SD = 0.62) were the lowest rated competencies, although self-reported preparedness was still quite high. Discussion: Our findings suggest that, from the perspective of our graduates, USU is providing both an effective military-unique curriculum and is preparing trainees for residency training. Further, these results support the notion that graduates are prepared to lead and to practice medicine in austere environments. Compared to other compe tencies that were assessed, self-ratings for systems-based practice and practice-based learning and improvement were the lowest, which suggests the need to continue to improve USU education in these areas.
INTRODUCTION M edical school faculty play an essential role in society— educating physicians who will com plete additional training to provide health care for our nation. This is a com plex and daunting task with several notable challenges. C om pleting m edical school is only the first step in a physician’s training w ith the eventual “product” not yet having been realized. Thus, m edical educators grapple w ith selecting appropriate outcom e m easures for this phase o f education. Second, changes continue to em erge in our educational system s; these include the establishm ent o f new evaluation dom ains and the continued grow th o f m edical know ledge. Third, it can be difficult for m edical schools to track their graduates over the long term because o f the num ber o f residencies and practice *Department of Medicine. Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. tF. Edward Hebert School of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. ^Department of Military and Emergency Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. §Department of Preventive Medicine and Biometrics, Uniformed Ser vices University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of Defense or the U.S. Government. doi: 10.7205/MILMED-D-14-00570
MILITARY MEDICINE, Vol. 180, April Supplement 2015
locations available to graduates. An additional challenge for the U niform ed Services U niversity (USU) is our unique charge to educate m ilitary physicians with dual professional identities as both physicians and m ilitary officers. D espite these challenges, it is critical that USU track these outcom es to ensure that the school is fulfilling its societal obligation in both peacetim e and wartim e. The m ost prom inent new evaluation dom ain, a set o f six com petencies for all graduating residents, was developed in 2002 by the A ccreditation Council for G raduate M edical Education (A C G M E ).1 T his com petency rubric is consistent with several international com petency fram ew orks for gradu ate m edical education such as C anM eds2 and G ood M edical Practice.3 Som e m edical schools in the U nited States now use the A CG M E fram ew ork as a foundation for preparing their students for future success in graduate m edical education. W hen m easuring m edical school “ success,” one im portant perspective is that o f the graduates. Do they believe, when reflecting on their clinical experiences, that the m edical school has prepared them for their practice needs and chal lenges? Such evaluations, as part o f a larger portfolio of outcom es, can help m edical educators identify areas in their curriculum that m ight need revision. Such evaluations can also assist in the assessm ent o f m ore com plex outcom es such as preparedness for independent practice. To that end, we surveyed two decades o f USU graduates to assess their percep tions about their preparedness for residency training, using
113
Assessing Curriculum Effectiveness: A Survey ofU SU Medical School Graduates
the ACGME competencies as a framework. We believe that such aggregate data can help medical educators formulate better policies and educational practices. We hypothesized that USU graduates’ perceptions of preparedness would be highest for military-unique practice and professionalism given that USU is the only federal medical school in the United States, and, as such, can and does provide additional focus in these areas. Conversely, we hypothesized that graduates’ perceptions of preparedness would be lowest for the two ACGME com petencies that were not a focus of the curriculum during the time frame under investigation (i.e., 1980-2001): systems-based practice and practice-based learning and improvement. METHODS This study was part of the larger Long-Term Career Outcome Study (LTCOS) conducted by the F. Edward Hebert School of Medicine, the only federal medical school in the United States. This study was approved by USU’s Institutional Review Board. Participants and Procedures The LTCOS alumni questionnaire was initially sent electron ically to graduates who had earned their MD degree between 1980 and 2001. These data were collected in March 2012 using PHPSurveyor. To increase the response rate, we sent both paper and electronic reminders, and also reached out, via social media, to alumni through USU’s Alumni Affairs Office. Data collection was completed in December 2013. Questionnaire Development The questionnaire was developed through an interdepartmen tal vetting process involving LTCOS members, the USU’s Executive Curriculum Committee, and Associate Dean of Medical Education. Reliability and validity evidence for this instrument has been previously collected and reported else where.4 Items on this survey were subsequently mapped to the ACGME competencies and, through a deliberative group process, additional items to address ACGME competencies were created; these items were not included in previous ver sions of the survey. Following these revisions, we conducted several cognitive interviews with selected faculty and gradu ates and made additional edits to the survey. The full survey is provided in the Appendix. The competency section of the survey was organized into seven sections: (1) patient care (12 items), (2) com munication and interpersonal skills (10 items), (3) medi cal knowledge (3 items), (4) professionalism (5 items), (5) systems-based practice (3 items), (6) practice-based learn ing and improvement (7 items), and (7) military-unique practice (8 items). The items were anchored on a 5-point, Likert-type response scale with the following options: (1) poor, (2) significantly below your peers, (3) on par with your peers, (4) better than your peers, and (5) consistently higher level than most of your peers. A sixth option, unable to judge, was also included in the response scale but was treated as missing data in the analysis.
114
Data Analysis Analyses were conducted using the statistical software Stata 12 (College Station, Texas). Internal consistency (reliability) of the items measuring each of the competency factors was examined by reliability analysis using Cronbach’s a. Univar iate descriptive statistics were also calculated to assess rat ings on each of the competencies. RESULTS Surveys were electronically distributed to 2,825 graduates for whom e-mail addresses were available. Of these, we estimated that 2,400 alumni had valid contact information (i.e., e-mail addresses). Of these, 1,189 alumni completed and returned the survey, giving a response rate of approximately 50%. There were no significant differences on cumulative medical school GPA between alumni who participated in the study and those who did not (t = 1.57, df = 2958, p = 0.12). These results suggest that, relative to medical school per formance, our sample was representative of the entire USU alumni population who graduated between 1980 and 2001. The internal consistency reliability scores for each of the seven scales were excellent, with Cronbach’s a ranging from 0.95 (patient care) to 0.72 (systems-based practice) (Table I). We calculated descriptive statistics for the seven subsec tions of the alumni survey. Tables I and II show subscale means and standard deviations for each competency and cor responding items, respectively. Overall, mean scores were relatively consistent across over the decades, for all compe tencies (Fig. 1). Mean scores for competencies ranged from 3.50 (systems-based practice) to 4.04 (military-unique prac tice), with professionalism and military-unique practice being the highest rated competencies. Consistent with our expectations as the nation’s military medical school, our graduates were most confident in their preparedness for military-unique practice (M = 4.04, SD = 0.65), which included items assessing military leadership (M = 4.30, SD = 0.65). USU graduates also indicated being well prepared for the challenges of residency education in the domain of professionalism (M = 4.02, SD = 0.72). Selfreports were also high for competencies related to patient care (M = 3.86, SD = 0.68), communication and interpersonal skills (M = 3.88, SD = 0.66), and medical knowledge (M = 3.78, SD = 0.73). Consistent with expectations, systemsbased practice (M = 3.50, SD = 0.70) and practice-based TABLE I.
Competencies: Reliability and Mean Scores
Factor
N
M(SD)
Reliability, a
Patient Care Com m unication and Interpersonal Skills Medical Knowledge Professionalism System s-Based Practice Practice-Based Learning M ilitary-Unique Practice
1116 1118
3.86 (0.68) 3.88 (0.66)
0.95 0.95
1108 1116 1092 1112 1069
3.78 4.02 3.50 3.57 4.04
0.91 0.93 0.72 0.86 0.92
(0.73) (0.72) (0.70) (0.62) (0.65)
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey of USU Medical School Graduates TABLE II.
Descriptive Statistics of USU Graduates’ Competency Self-Evaluation Items (Classes of 1980-2001) P atien t C are
C o m m u n ic atio n and Interp erso n al Skills M ean
SD
In te rv iew P atien ts
4 .0 0
0.82
O ral C o m m u n icatio n Skills
3.97
0.81
P erform P h y sical E x am in atio n
3.86
0.83
W ritten C om m u n icatio n Skills
3.91
0.82
M ea n
SD
P erform D aily P atien t E v alu atio n s (C lin ical S etting)
3.89
0.83
R elatio n sh ip W ith P atients
4.01
0.80
P erform D aily P atien t E v a lu a tio n s (In p atien t S etting) P erform B asic T e c h n ic al Skills
3.87
0.84
R elatio n sh ip W ith F am ilies
0.82
3.92
0.86
R elatio n sh ip W ith P eers and O th e r H ealth
3.97 3.94
0.81
C are P ersonnel P erform A d v an ced P ro ced u ral Skills
3.73
0.92
E ffectiveness as a T e a c h er
3.86
0.81
A n aly sis o f C lin ical D ata and In te rp retatio n o f T ests
3.84
0.81
S ensitiv ity to P a tie n t's A ge and G ender
3.72
0.80
M an a g e m e n t o f P atien ts W ith L ife-T h reaten in g Illness
3.78
0.84
S ensitiv ity to P a tie n t’s C ulture and D isabilities
3.74
0.80
M an a g e m e n t o f P atien ts W ith C o m p lex Illness
3.70
0.83
E ffectiv en ess as a T eam M em ber
4.06
0.79
E ffectiv en ess W ith E nd o f L ife C are Issues
3.62
0.82
K n o w le d g e and S electio n o f T rea tm e n t O ptions
3.75
0.81
C o o rd in a tio n an d C o n tin u ity o f C are
3.77
0.81
P e rc e iv e d P rep aratio n fo r P a tie n t C are in a D ep lo y m e n t/ H u m an itarian E n v iro n m en t
4 .2 4
0.79
F u n d o f B asic S cien ce K n ow ledge
3.66
0.80
Initiative and M otivation
4.12
0.81
F und o f C lin ical S cien ce K n ow ledge
3.76
0.80
C o n scien tio u sn ess
4.12
0.80
C linical lu d g m e n t
3.91
0.80
E thical C o n d u ct
4.03
0.83
A w are o f O w n L im itation
3.92
0.81
W illin g n e ss to A dm it an E rro r in Judgm ent
3.92
0.81
M edical K n o w led g e
P ro fessio n alism
S y stem s-B ased P ractice
P ractice-B ased L e arning and Im provem ent
U n d e rstan d in g o f the C o n tex ts and Sy stem s in T R IC A R E
3.74
0.88
S e lf-D irected L e a rn in g Skills
3.69
C o n sid eratio n o f C o sts in D iag n o sis and M an ag em en t
3.25
0.90
T im e M an ag em en t Skills
3.70
0.84
Q u a lity A ssu ran ce and Im p ro v em en t In itiativ es
3.55
0.80
Q u ality M edical R ecord D ocu m en tatio n
3.79
0.81
A ccessing and C ritically E v a lu a tin g C urrent
3.57
0.78
E lec tiv e In v o lv em en t in R esearch
3.14
0.94
P articip atio n in V o lunteerism or H um anitarian
3.42
0.81
3.57
0.80
0.80
M edical In form ation
C lin ical A ctivities A dap tatio n to N ew T ech n o lo g y M ilitary -U n iq u e P ractice, D ep lo y m en t, an d H u m an itarian M issio n s M ilitary L ead ersh ip
4.21
0.75
A bility to C o n d u ct P atien t C are in D ep lo y ed
4.21
0.76
M ission E n v iro n m en t U n d e rstan d in g o f P sy ch o lo g ical Im pacts o f D ep lo y m en t
4.15
0.78
K now ledge o f C o m m o n P o std ep lo y m en t M ed ical o r
3.97
0.82
K now ledge o f E lectro n ic H ealth R eco rd A p p licatio n s
3.68
0.88
K n o w ledg e o f E lectro n ic H ealth R eco rd an d T ech n o lo g y U sed in T h e a te r
3.66
0.91
A bility to C ope W ith the Stress o f M ilitary M ed ical P ractice
4.13
0.79
A d ap tatio n to U n iq u e S itu atio n and S tresso rs
4.17
0.76
P sycholo g ical C o n d itio n s
learning and improvement (M = 3.57, SD = 0.62) were the lowest rated competencies, albeit graduate self-reported pre paredness was still high. An examination of individual items for each competency also revealed that respondents reported being better than their peers on preparation for patient care in a deployment or humanitarian environment and interviewing patients (patient care), and effectiveness as a team member and relationship with patients (communication and interper sonal skills). As expected, in general, self-ratings for systemsbased practice and practice-based learning and improvement were relatively low especially with regards to the average ratings on consideration of costs in diagnosis and management
MILITARY MEDICINE, Vol. 180, April Supplement 2015
(systems-based practice), as well as quality medical record documentation and elective involvement in research (practicebased learning and improvement). D IS C U S S IO N
Our findings provide additional evidence that USU is pro viding general and military residency training prepared ness, as reflected by our alumni’s views. Consistent with our expectations as the nation’s military medical school, our grad uates felt most prepared for professionalism and militaryunique practice to include high ratings for preparedness for leadership. Our curriculum offers a number of military-unique
115
Assessing Curriculum Effectiveness: A Survey of USE Medical School Graduates
Trends in Core Competencies: 1980-2001 o
u 5.00 --------------------------------------------------G
QJ
n. 4.50 --------------------------------------------------£
c 3.00 ---------------------------------------------------------g 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Years from 1980 - 2001 Patient Care
— Communication Skills
i. i—iProfessionalism
— —Systems Based Practice
■ ....Medical Knowledge " Practice Based Learning
.... . Military Unique Practice FIGURE 1.
Trends across time— 1980-2001.
components, including leadership activities and field exer cises that may account for these findings. These results are also consistent with views expressed by our alumni who have achieved flag officer rank.5 We believe these findings are important given that USU graduates are asked to practice medicine in austere environments. Alumni preparedness ratings were also high for patient care, communication and interpersonal skills, and medical knowledge. We believe this provides additional evidence that our medical curriculum educates our students well, at least from the graduates’ perspective. The opinions expressed by our graduates are also consistent with other outcome mea surements. For example, USU graduates have higher board certification rates than the national average.6 Comparatively, USU alumni were less confident on systemsbased practice and practice-based learning and improvement. However, this result was expected given that these competen cies were not formally defined, taught, or evaluated during the time frame under investigation. We believe that this lends additional validity evidence to our findings. We were pleased to find that our graduates felt they were at least on par with their peers (and in some cases, above their peers) with respect to these competencies. We continue to make changes to improve our students’ education in these needed competen cies to include additional emphasis being placed in the cur riculum on topics such as patient safety, reducing medical errors, cost-effective management strategies, evidence-based medicine, and a team-based approach to improving the quality of patient care. We believe that such findings can help make important contributions to curriculum reform efforts to include other relatively low ratings such as medical record docu mentation and involvement in research. For example, in our new curriculum, students have additional elective time for research projects and all students receive training in the use of our electronic medical record system throughout the curriculum starting in the preclerkship period. It will be inter
116
esting to see if graduates’ confidence in these areas improve in future studies involving more recent graduates.
Limitations and Recommendations for Future Research The study had a number of limitations. First, respondents were asked to self-assess their own competencies, and there is a fair amount of research in medical education, which suggests that individuals often do a poor job self-assessing broad competencies. Second, respondents were asked to rate their perceptions on different competencies using response options anchored on a 5-point Likert-type scale. These data, however, are self-reported and so they represent perceived competencies as opposed to actual competencies as measured by more objective assessment measurements. That said, find ings from more objective measures of our alumni’s perfor mance corroborate the findings presented here.6 Another potential limitation is nonresponse bias, although considering the fact that there were no differences between respondents and nonrespondents on medical school GPA, we feel non response bias is less problematic. Also, since respondents graduated over the course of a 20-year period, it is likely that the reflections or experiences recalled by respondents who graduated more than 5 years ago, might not be as accurate as that of more recent graduates. Therefore, the possibility of recall bias, especially among earlier graduates, cannot be overlooked as a potential limitation to the study. Last, this study was completed at a single institution and so there is no opportunity to compare the preparedness perceptions of our alumni with alumni form other medical schools.
CONCLUSION In summary, USU graduates indicate being well prepared for the challenges of residency education especially in the competences of military-unique practice and professionalism.
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates
This is reassuring since leadership and service to the nation is what sets USU apart from other schools. Findings from this study provide insight into which aspects of the curricu lum might be improved to further enhance student educa tion and competencies in areas outlined by the ACGME
and other accreditation bodies. Alumni perceptions about preparedness will be important to track over time and could provide medical educators with a viable outcome to study the effects of a wide range of medical school initiatives, such as curricular reform.
APPENDIX
USU ALUMNI SURVEY C lass (G r a d u a tio n Y e a r ) _______________ N a m e
T h a n k yo u f o r ta k in g th e tim e to p ro v id e fe e d b a c k on y o u r t r a in in g a t U S U . W e w ill u se a g g r e g a te d a ta fr o m th is s u rv e y to g u id e o u r d e c is io n s a b o u t o u r c u rric u la a n d o t h e r a s p e c ts o f m e d ic a l t r a in in g . T h e in fo r m a tio n w e g a th e r w ill re m a in c o n fid e n tia l a n d w ill n o t be s h a re d w ith y o u r s u p e rv is o rs . Y o u r c o n tr ib u tio n s to m ilit a r y m e d ic in e , in c lu d in g a n s w e r in g th is s u rv e y , a r e v e ry m u c h a p p r e c ia te d .______________________________________ A ) P o s t G r a d u a t e C a r e e r E x p e r ie n c e O la : I n w h a t a r e a d id y o u c o m p le te y o u r PG Y 1 tr a in in g ? I f y o u h a v e d o n e a t r a n s itio n a l y e a r b u t w e r e p re -s e le c te d fo r s p e c ia lty tr a in in g a t th e P G Y -2 le v e l b e fo re s ta r tin g P G Y -1 , p le a s e c h o o s e th e p r e s e le c te d s p e c ia lty (e .g . tr a n s itio n a l p r e s e le c t A n e s th e s ia , c h o s e A n e s th e s ia ). Please choose o n ly o n e o f th e fo llo w in g :
Pediatrics
Dermatology
O Neurology
□ o
Emergency Medicine
O OB/GYN
Q
Preventive Medicine
Q
Ophthalmology
Q Anesthesiology
□ □
Q Family Medicine □
□
□
□
Psychiatry Surgery
Q Internal Medicine
o n
Occupational Medicine Pathology
□
Physical and Rehab Medicine
Radiology
Joint PGY1
Transitional, not preselected for PGY-2 training at the start o f PGY-1
Other (please list)----------------------------------------- _ _
[O n ly a n s w e r th is q u e s tio n i f you a n sw e re d 'S u rg e ry ' to q u e s tio n 'O la '] 0 1 b : S u r g e r y - P le a s e s p e c ify ty p e : e .g . G e n e ra l, E N T, O R T H O , e tc . Please w rite y o u r a n s w e r h ere:
[O n ly a n s w e r th is q u e s tio n if you answ e re d 'J o in t PGY1 ' to q u e s tio n 'O la '] 0 1 c : J o in t PG Y 1 - P le a s e s p e c ify ty p e : e .g . p s y c h ia t r y /in t e r n a l m e d ic in e : Please w rite v o u r a n s w e r h ere:
0 2 : D id y o u h a v e a to u r o f d u ty a s a p h y s ic ia n b e fo re c o m p le tio n o f y o u r fir s t re s id e n c y ? Please choose o n ly o n e o f th e fo llo w in g :
Yes
U
No
[O n ly a n s w e r th is q u e s tio n if you a n sw e re d 'Yes' to q u e s tio n '0 2 '] 0 2 a : W h a t k in d o f to u r o f d u ty d id yo u c o m p le te as a p h y s ic ia n b e fo re y o u r f ir s t re s id e n c y ? Please choose o n ly o n e o f th e fo llo w in g :
M IL IT A R Y M E D IC IN E , Vol. 180, April Supplement 2015
117
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates Flight Surgeon
Dive Medical Officer_____ GMO (supporting peacetime or operational)
l—' O ther--------------------------------------------------------------------------------------
03: Have you completed your first residency? r~t t—1 .. 1No
Yes
04: In w h a t area did you com plete your first residency? Choose only one Anesthesiology
Dennatology
Emergency Medicine
Preventive Medicine
Family Medicine
Psychiatry
Radiation Oncology
Physical & Rehab Med Pediatrics
n Internal Medicine Radiology
• Neurology
Nuclear Medicine
Undersea Medicine
OB/GYN
Occupational Medicine
Military Service Unique
Surgery
Ophthalmology
Joint Residencies
Pathology
Other--------------------------
[Only answer this question if you answered 'Surgery ' to question '04 '] 04a: Surgery (please specify type; e.g. General, ENT, ORTHO, etc: Answer here:
[Only answer this question if you answered 'Military Service Unique ' to question '04 '] 04b: M ilitary Service Unique - please specify type: Write your answer here:
[Only answer this question if you answered 'Joint Residencies' to question '04 '] 04c: Joint Residencies - please specify type; e.g. p sv ch iatrv/in te rn al Answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 05: Did you com plete your first residency in a m ilitary hospital or facility (P rim ary Program A ffiliation Site)? Please choose only one of the following: Yes
- No
[Only answer this question if you answered 'Yes' to question '03 '] 06: Are you currently Board Certified in your first residency trained specialty? - Yes
No
[Only answer this question if you answered 'No' to question '06 '] 06a: I f you answered "No" to question 6, please explain: Please write your answer here:
118
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates
[Only answer this question if you answered 'Yes' to question '06 '] 06b: Have you recertified in your first residency trained specialty? '-
Yes
' No
[Only answer this question if you answered 'No' to question '06b '] 06c: I f you answered "No" to question 6b, please explain: Please write vour answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 07: Have you com pleted a second residency or fellowship? Yes
—^ No
[Only answer this question if you answered 'Yes' to question '07 '] 07a: Please identify the type of your second residency or fellow ship training: Please write vour answer here:____________
[Only answer this question if you answered 'Yes' to question '07 '] 07b: Did you com plete your second residency or fellow ship in a m ilitary hospital or facility (Prim ary Program Affiliation Site)? Yes
No
[Only answer this question if you answered 'Yes' to question '07 '] 07c: Are you currently Board Certified in your second residency or fellowship specialty/subspecialty? Please choose only one of the following: Yes
' No
[Only answer this question if you answered 'No' to question '07c '] 07d: I f you answered "No" to question 7c, please explain: Please write vour answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 08: Have you com pleted post-m edical school education resulting in a Masters Degree? (Please do not include any pre-m edical school education.) Yes
1—■ No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
119
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates [Only answer this question if you answered 'Yes' to question '08 '] 08a: Please specify the type of degree and your area of study: Write your answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 09: Have you com pleted post-m edical school education resulting in a Doctoral level degree (e.g ., PhD, DSc, JD)? (Please do not include pre-m edical school education.) Please choose only one of the following: Yes
No
[Only answer this question if you answered 'Yes' to question '09 '] 09a: Please specify the type o f degree and your area of study; Write vour answer here:
[Only answer this question if you answered 'Yes' to question '03 '] 10: Have you com pleted post-m edical school healthcare adm inistration or m anagem ent education, which resulted in some kind of form al certification (e.g. ACPE, CHE, etc.)? (Please do not include pre-m edical school education Yes
No
[Only answer this question if you answered 'Yes' to question '10 '] 10a: Please specify the type of certificate and your area of study: Write your answer here:
B) O perational Experiences l l : As a physician, have you ever deployed for 30 or more consecutive days to a th e a te r of com bat operations? Yes
No
[Only answer this question if you answered 'Yes' to question ' l l '] 11a: How many tim es have you deployed for 30 or more consecutive days to a th e a te r of com bat operations? Please choose only one of the following: once
twice
3 times
4 times
5 times
6 or more times
[Only answer this question if you answered 'Yes' to question ' l l '] Total num ber of months d eployed in the last tw o years and career: Please write vour answer:
12: As a physician, have you ever deployed for 7 or m ore consecutive days on a peacetim e hum anitarian mission? Please choose only one of the following: -
120
Yes
1— '
No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey of USU Medical School Graduates
[Only answer this question if you answered 'Yes' to question '12 '] 1 2 a : H o w m a n y tim e s h a v e y o u d e p lo y e d fo r 7 o r m o re c o n s e c u tiv e d a y s on a p e a c e tim e h u m a n ita r ia n m is s io n ? Please choose o n ly o n e of the following:
□
once
twice
□
□
3 times
4 times
5 times
□
6 or more times
[Only answer this question if you answered 'Yes' to question '12 '] 12 b : A p p r o x im a te t o ta l n u m b e r o f days^Please write your answer here:
1 3 : H a v e y o u e v e r d e p lo y e d in s u p p o rt o f a s p e c ific o p e ra tio n a l (c o m b a t) o r h u m a n ita r ia n ( p e a c e t im e ) m is s io n in t h e s e n io r m e d ic a l le a d e rs h ip ro le (a s d e s ig n a te d by o ffic ia l o rd e rs , fo r e x a m p le , D iv is io n S u rg e o n , F le e t S u rg e o n , EM ED S C o m m a n d e r, e tc .)? Please choose o n ly o n e of the following:
Yes
No
[Only answer this question if you answered 'Yes' to question '13 '] 1 3 a : P le a s e lis t t it l e ( s ) o f p o s itio n (s ) (p le a s e d o n o t u s e a c ro n y m s ): Write your answer here:
C ) C a re e r L a n d m a rk s 14: W h ic h o f th e fo llo w in g p o s itio n s h a v e y o u r h e ld o v e r th e c o u rs e o f y o u r e n t ir e c a r e e r, if an y? Please choose a ll that apply:
Full-time Clinician (seeing patients at least 70% of the time during a typical week)
□ □
Chief of Service/Clinic (within an MTF) Chief/Director o f a Clinic (which is part of a larger MTF but located elsewhere; e.g. a branch clinic) Clinical Clerkship or Medical School Course Director
Department Cliair/Chief?Head at a Medical Center or Community Hospital without a residency program in your department Department Chair/ChiefHead at a Medical Center or Community Hospital with a residency program in your department Service ChiefDirector/Head of multidisciplinary service at a Medical Center or Community Hospital (e.g. Chief of all Surgical Services) Military Treatment Facility (MTF) Chief of the Medical Staff (SGH, DCCS, Medical Director) D em i tv M TF F o m m a n H in o O f f ir p r tttQ M — F v .v i i t i ,- / , Off! cer) Deputy MTF Commanding Officer (USN —Executive
Fellow ship director
Brigade surgeon
Squadron com mander
Senior medical officer
□
Residency Program Director D ivision surgeon Research director
Other (list up to 3 roles)__
1 5 : H a v e yo u h e ld th e p o s itio n o f C o m m a n d in g O ffic e r (o n o ffic ia l o rd e rs d e s ig n a tin g C o m m a n d e r o r C o m m a n d in g O ffic e r a n d th u s p o s sessin g UCMJ le g a l a u t h o r ity )? Please choose o n ly o n e of the following:
Yes
No
1 5 a : I f y e s , w h ic h o f th e fo llo w in g p o s itio n s , if a n y , h a v e y o u held?
MILITARY MEDICINE, Vol. 180, April Supplement 2015
121
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates Please choose all that apply: Free-Standing Clinic Commanding Officer
Medical Center Commanding Officer
Bedded Community-Sized Hospital Commanding Officer Squadron or similar service-specific organizational Commanding Officer Deployed/Deployable Medical Unit Commanding Officer Other list positions____________________________________
16: Have you held a staff position at your service's Medical Headquarters SG Office level? Please choose only one of the following: Yes
No
17: Have you ever had a "jo in t" assignm ent (e.g ., DoD, USU, JtF, etc)? I—' Yes
— No
18: W hat is the highest rank you have achieved to date? I f you are currently on a "select" list, please identify the rank selected for prom otion. n n n n n 03
'*—'0 4
05
*—1 06
0 7 and above
19: Have you ever been selected for prom otion, as a physician, below the prim ary zone for eligibility (i.e ., "deep select")? Choose only one: Yes
No
[Only answer this question if you answered 'Yes' to question '19 '] 19a: To which of th e follow ing ranks does this apply? (check all th a t apply) Choose all that apply: 04
05
06
20: Did USU prepare you for your current (o r past) career leadership roles? Yes
No
20a: W hy or w hy not? Please write vour answer here:
21: Have you ever engaged in volunteer medical activities (fo r exam ple, public clinic)? Please choose only one of the following:
Yes
No
22: Have you ever published an article in a professional peer-review ed journal? Please choose only one of the following:
Yes
No
22a: Approxim ately how many published articles in the last 12 months and last 5 years? Last 12 m onths:__________Last 5 years____________ 23: Have you ever presented papers before national or international professional groups? Please choose only one of the following: Yes
No
23a: I f yes, approximately how many in the last year and the last 5 years? Last y e a r___________ Last 5 years_____________
122
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates 24: Have you ever presented papers before m ilitary professional groups? Yes
O No
24a: If yes, approximately how many in the last year and the last 5 years? Last year _________ Last 5 years________________ 25: Have you ever served as a review er for a peer-review ed journal? —1 Yes No 25a If yes, approximately how many medical journals as a peer reviewer?______________ 26: Have you ever served as a principal investigator on a research grant? Yes
No
[Only answer this question if you answered 'Yes' to question '26 '] 26a: Total num ber o f grants as a principal investigator: Please write vour answer here:
27: Do you currently hold one or m ore university academ ic appointm ents? Yes
No
[Only answer this question if you answered 'Yes' to question '27 '] 27a: Which of the following title s best describes the highest academ ic appointm ent you have ever held? Please choose only one of the following: Instructor
Assistant Professor
Associate Professor
Professor
Other (list)____
[Only answer this question if you answered 'Yes' to question '27 '] 27c: Which of the following best describes the university from which you received the above appointm ent? Please choose only one of the following: USU
Other Medical School
Other college/university (list)_____________________
28: As a physician, which of th e follow ing medals have your received, if any? Please choose all that apply: (—1 /■““N Bronze Star Purple Heart Silver Star Air Medal
1— Navy Cross
Air Force Cross
POW Medal
Legion of Merit
Defense Meritorious Service
United Nations
Distinguished Flying Cross Defense Superior Service Humanitarian Service Medal
r-i
Other - not campaign medals (list)
29: Has your Active Duty career concluded? Choose only one of the following: *- ^ Yes
^ ^ No
[Only answer this question if you answered 'Yes' to question '29 '] 30: Did you separate before retirem ent? Please choose only one of the following: Yes
No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
123
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates [Only answer this question if you answered 'Yes' to question '29 ’] 31: Did you continue to serve in an Active R eserve/G uard Unit? Please choose only one: f— \
w Yes
' ■No
[Only answer this question if you answered 'Yes' to question '29 '] 32: Which of the follow ing statem ents best describes your service? Please choose only one of the following:
I retired after 20 years or more commissioned Medical Corps service. I retired after 20 or more years of total service, but less than 20 years commissioned Medical Corps service. I separated prior to 20 years total active service as a result of a Medical Board action. D ) M e d ic a l School E x p e rie n c e 33: When considering which medical school you would attend, how im portant w e re the financial aspects of attending USU versus other institutions in the civilian setting? The IMPORTANCE of financial aspects of attending USU w ere: Please choose one not at all important
slightly
n somewhat important
- quite important
extremely important
34: W hen you began taking classes at USU, w h at was your m arital status? Please choose only one of the following: a t _______ _ Not married
Married
Married but separated
Divorced
............... _ ......
Widowed
35: W hen you began taking classes at USU, did you have dependent children? Please choose only one of the following:
Yes
.. No
36: W hen you graduated from USU, w h at was your m arital status?
n Not married
Married
Married but separated
u
Divorced
Widowed
37: W hen you graduated from USU, did you have dependent children?
L 1Yes
No
38: FOR QUESTIONS 39a - 39f, PLEASE EVALUATE YOUR PREPAREDNESS IN COMPARISON TO ALL OF YOUR PEERS IN YOUR CLASS YEAR OF RESIDENCY: 39a: Patient Care Please choose the appropriate response for each item: S ig n ific a n tly
124
B e tte r
C o n s is te n tly
w ith
th a n
h ig h e r le v e l th a n
b e lo w y o u r
your
your
m ost o f your
U n a b le
p e e rs
p e e rs
p e e rs
p e e rs
to ju d g e
Q □ □ rr
Q O □ □
□ □ □ □
□ □ □ □
O □
CD
CD
□
□
□
Poor
In te rvie w in g patients a n d /o r fam ilies fo r initial histories Perform ing a physical exam ination Perform ing daily pa tien t evaluations (clinic se ttin g ) Perform ing daily pa tien t evaluations (in p a tie n t se ttin g ) Perform ing basic technical skills (i.e. inserting IV, sutu ring , inserting a-line )
O n par
o
□
□ □
□
□
t,,ni
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates P e rfo rm in g a d v a n c e d p ro ce d u ra l s k ills (i.e . in s e rtin g c e n tra l lin e , o p e ra tin g ro o m p e rfo rm a n c e )
□
A n a ly s is o f c lin ic a l d a ta , d iffe re n tia l d ia g n o s is and s e le c tio n /in te rp re ta tio n o f te sts
□
□
Q
□
□
□
□
□
□
□
M a n a g e m e n t a n d /o r re fe rra l o f p a tie n ts w ith life th re a te n in g illn e ss
□
□
Q
□
o
o
M a n a g e m e n t o f p a tie n ts w ith c o m p le x m u lti-s y s te m illness
□
□
o
□
□
□
K n o w le d g e a nd se le c tio n o f tr e a tm e n t o p tio n s /p a tie n t m anagem ent
□
□
□
□
□
□
C o o rd in a tio n and c o n tin u ity o f ca re
Q
□
□
□
D
□
P erceived p re p a ra tio n fo r p a tie n t ca re in a d e p lo y m e n t o r h u m a n ita ria n m issio n e n v iro n m e n t
□
□
□
□
□
□
3 9 b : C o m m u n ic a tio n a n d In te r p e r s o n a l S k ills Please choose th e aDDroDriate response fo r each ite m : On par with your peers
Significantly below your peers
Poor
O ral c o m m u n ic a tio n s k ills
□
Q
W ritte n c o m m u n ic a tio n s k ills
□
R e la tio n s h ip s w ith p a tie n ts R e la tio n s h ip s w ith fa m ilie s R e la tio n s h ip s w ith p e ers, s ta ff and o th e r h e a lth ca re p e rs o n n e l
Better than your peers
Consistently higher level than most of your peers
Unable to judge
□
□
□
□
□
□
□
□
□
□
□
□
Q
□
□
□
□
□
□
□
□
□
Q
□
O
□
E ffe c tiv e n e s s as a te a c h e r
0
o
□
□
D
S e n s itiv ity to p a tie n t's age a n d g e n d e r
□
Q
□
o
□
□
S e n s itiv ity to p a tie n t's c u ltu re and d is a b ilitie s
□
□
□
□
□
0
E ffe c tiv e n e s s as a te a m m em ber
□
□
□
□
□
Q
E ffe c tiv e n e s s w ith end o f life ca re issues
□
□
Q
□
□
□
3 9 c : M e d ic a l K n o w le d g e Please choose th e a p p ro p ria te response fo r each ite m : Poor
Fund o f basic science k n o w le d g e Fund o f clin ic a l science
□ □
Significantly below y our peers
On par with your peers
Better than your peers
Consistently higher level than m ost o f your peers
Unable to judge
□ □
□ □
□ □
□ □
□ □
M I L I T A R Y M E D IC IN E , Vol. 180, April Supplement 2015
125
Assessing Curriculum Effectiveness: A Sun’ey of USE Medical School Graduates knowledge
(—\
Clinical ju d g m e n t
1 -J
□
□
□
P le a s e c h o o s e t h e a D D r o D r ia te r e s p o n s e f o r e a c h i t e m : Consistently higher Better On par level than most of Significantly with your than your Poor peers peers your peers below your peers
3 9 d : P r o fe s s io n a lis m
In itia tiv e and m otivation
LJ
Conscientiousness
O
Ethical conduct
Q
Aware o f own lim ita tio n s W illingness to a d m it an e rro r in ju d g m e n t
□ □ □ □ □
□ u
o □ □ □ □
0
□ □ □
Unable to judge Q
□ Q □
□
□ □ □
/—•» LJ
□
□
3 9 e : S y s te m s -B a s e d P ra c tic e & P ra c tic e -B a s e d L e a rn in g a n d Im p r o v e m e n t
Poor
Significantly below your peers
S elf-directed learning skills
□
□
Tim e m anagem ent skills
O
□
Medical record documentation quality Accessing and critically evaluating cu rre n t medical inform ation and scientific evidence U nderstanding o f the contexts and system s in TRICARE Elective invo lve m e n t in research Participation in volu n te e rism , social, or hum anitarian clinical activities C onsideration o f costs in diagnosis and m anagem ent Adaptation to new technology Q uality assurance and im p ro ve m e n t in itia tive s
On par with your peers
u
Better than your peers
Consistently higher level than most o f your peers
Unable to judge
.
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
LJ
LJ
□
□
O
□
i \
o
□
V____ J
□
□
□
0
□
□
□
o
□
□
□
Q
□
□
□
□
□
□
□
□
□
□
Q
□
□
3 9 e e : I f y o u s e le c te d " E le c tiv e in v o lv e m e n t in re s e a rc h " p le a s e s p e c ify ty p e :
Please write vour answer here:
3 9 e e e : I f y o u s e le c te d " P a r tic ip a tio n in v o lu n te e r is m , s o c ia l, o r h u m a n ita r ia n c lin ic a l a c tiv itie s " p le a s e s p e c ify : Please write vour answer here:
126
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates
39f: M ilitary Unique Practice, Deploym ents, and H um anitarian Missions Please choose the appropriate response for each item: O n par
Poor
B e tte r
S ig n if ic a n tly
w ith
th a n
C o n s is te n tly
b e lo w y o u r
your
your
h ig h e r le v e l th a n
U n a b le
p e e rs
p e e rs
p e e rs
m o s t o f y o u r p e e rs
to ju d g e
Military leadership
□
O
□
Ability to conduct patient care in deployed or humanitarian mission environment Understanding of psycho social impacts of deployment on service members' families Knowledge of common post deployment medical or psychological conditions (e.g. TBI, PTSD) Knowledge of electronic health record applications Knowledge of electronic health record and technology used in theater Ability to cope with the stress of military medical practice Adaptation to unique situations and stressors in military medical practice
□ □
□
Q
□ □
□ □
□
O
Q
Q
□
□ □
□
Q
□
□
Q
□
□
□
□
©
□
□
□
□
□
□
□
□
□
o
□
□
□
□
□
□
Q
□ □
41: I believe my peers would tru st me to participate in the care of one of th eir loved ones: Please choose only one of the follow ing: - ‘ Yes
No
MILITARY MEDICINE, Vol. 180, April Supplement 2015
O 1 Not sure
127
Assessing Curriculum Effectiveness: A Survey ofUSU Medical School Graduates W h y o rW h y n o t?
4 2 : I n g e n e r a l, h o w w e ll d id y o u r m e d ic a l sch ool e d u c a tio n p re p a re yo u fo r y o u r c a r e e r in m e d ic in e ? Please choose o n ly o n e of the follow ing:
Not well at all
Slightly well
Moderately well
Quite well
Extremely well
4 3 : F ro m y o u r c a r e e r e x p e r ie n c e , in w h a t a s p e c ts , if a n y , do y o u fe e l USU m e d ic a l e d u c a tio n p ro g ra m is u n iq u e (c o m p a re d w ith c iv ilia n m e d ic a l s c h o o ls )? P le a s e a ls o s h a re a d d itio n a l in fo rm a tio n , c o m m e n ts o r fe e d b a c k fo r USU: Write your answer here (you may use the back side of page if you like):
S u b m it Y o u r S u rv e y .
Thank you for completing this survey. Please email your completed survey to [email protected] ________________ or fax your completed survey to Steven Durning at 301-295-3557.________________
REFERENCES 1. Accreditation Council for Graduate Medical Education. The Next Accreditation System: A Resident Perspective. Available at www .acgme.org/acgmeweb/Portals/0/PDFs/Resident-Services/9NASResidents May2014.pdf; accessed September 10, 2014. 2. General Medical Council. Good Medical Practice. Available at http:// www.gmc-uk.org/guidance/good_medical_practice.asp; accessed Sep tember 9, 2014 3. Royal College of Physicians and Surgeons of Canada. The Draft CanMEDS 2015. Physician Competency Framework. Available at http://www.royalcollege.ca/porlal/page/portal/rc/common/documents/
128
canmeds/framework/framework_series_l_e.pdf; accessed September 9, 2014. 4. Durning SJ, Pangaro LN, Lawrence LL, Waechter D, McManigle J, Jackson JL: The feasibility, reliability, and validity of a program director’s (supervisor’s) evaluation form for medical school graduates. Acad Med 2005; 80: 964. 5. Dong T, Durning SJ, Gilliland W R, et al ; Leadership success and the Uniformed Services University: perspectives of flag officer alusmni. Mil Med 2012; 177(9): 61-7. 6. DeZee K, Durning SJ, Dong T, et al: Where are they now? USU School of Medicine graduates after their military obligation is complete. Mil Med 2012; 177(9): 68-71.
M IL IT A R Y M E D IC IN E , Vol. 180, A pril Supplem ent 2015
Copyright of Military Medicine is the property of AMSUS and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
