STUDENT ASSESSMENT
Correlation of Pre-Veterinary Admissions Criteria, Intra-Professional Curriculum Measures, AVMA-COE Professional Competency Scores, and the NAVLE James K. Roush n Bonnie R. Rush n Brad J. White n Melinda J. Wilkerson
Journal of Veterinary Medical Education 2014.41:19-26.
ABSTRACT Data consisting of preadmission criteria scores, annual and final cumulative grade point averages (GPAs), grades from individual professional courses, American Veterinary Medical Association Council on Education (AVMA-COE) Competency scores, annual class rank, and North American Veterinary Licensing Exam (NAVLE) scores were collected on all graduating DVM students at Kansas State University in 2009 and 2010. Associations among the collected data were compared by Pearson correlation. Pre-veterinary admissions criteria infrequently correlated with annual GPAs of Years 1–3, rarely correlated with the AVMA-COE Competencies, and never correlated with the annual GPA of Year 4. Low positive correlations occurred between the NAVLE and the Verbal Graduate Record Examination (GRE) (r ¼ .214), Total GRE (r ¼ .171), and the mean GPA of pre-professional science courses (SGPA) (r ¼ .236). Annual GPAs strongly correlated with didactic course scores. Annual GPAs and final class rank strongly correlated (mean r ¼ .849), and both strongly correlated with the NAVLE score (NAVLE: GPAs mean r ¼ .628, NAVLE: final class rank r ¼ .714). Annual GPAs at the end of Years 1–4 weakly correlated or did not correlate with the AVMA-COE Competencies. The AVMA-COE Competencies weakly correlated with scores earned in didactic courses of Years 1–3. AVMA-COE Competencies were internally consistent (mean r ¼ .796) but only moderately correlated with performance on the NAVLE (mean r ¼ .319). Low correlations between admissions criteria and outcomes indicate a need to reevaluate admission criteria as predictors of school success. If the NAVLE remains the primary discriminator for veterinary licensure (and the gateway to professional activity), then the AVMA-COE Competencies should be refined to better improve and reflect the NAVLE, or the NAVLE examination should change to reflect AVMA-COE Competencies. Key words: Council on Education, COE, AVMA-COE, professional competencies, NAVLE, admissions criteria, predictors of veterinary school performance
INTRODUCTION Valid predictive criteria for academic success during veterinary training or professional success after graduation are constantly sought, evaluated, and reevaluated by veterinary colleges, state veterinary licensing boards, and other veterinary professional organizations. The definition of veterinary school success is debatable, with vocal proponents for measurements such as graduating class rank, North American Veterinary Licensing Exam (NAVLE) scores, and state exam board scores. The current trend of most states to place more weight on NAVLE scores for licensing requirements suggests that the NAVLE has become the de facto gateway to professional activity for veterinary students and, by extension, is an important measure of the success of a veterinary college since NAVLE passing rates affect American Veterinary Medical Association (AVMA) accreditation. The NAVLE, however,
doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
may not be appropriate as a measure of success since the intention of the NAVLE is to provide a minimum standard of competency for state licensing boards. More refined measures to determine veterinary school success have yet to be developed. Preadmission criteria for professional veterinary curricula are established independently by each of the 28 veterinary medical colleges currently accredited in the United States. Admissions policies in most veterinary medical colleges have been loosely based on scientific data related to the predictive validity of admission requirements and are more often based on conjecture, historical precedence, and habit. The most commonly used criteria for admission to veterinary school are calculations of the pre-veterinary grade point average (GPA), standardized tests, the personal interview, veterinaryrelated experience, personal recommendations or evaluations, and an essay or narrative.1 Several previous studies
19
Box 1: Outcome assessment competencies of the AVMA-COE * OAC1 OAC2 OAC3 OAC4 OAC5 OAC6 OAC7 OAC8 OAC9
Comprehensive patient diagnosis (problem-solving skills), appropriate use of clinical laboratory testing, and record management Comprehensive treatment planning including patient referral when indicated Anesthesia and pain management, patient welfare Basic surgery skills, experience, and case management Basic medicine skills, experience, and case management Emergency and intensive care case management Health promotion, disease prevention/biosecurity, zoonosis, and food safety Client communications and ethical conduct Critical analysis of new information and research findings relevant to veterinary medicine
Journal of Veterinary Medical Education 2014.41:19-26.
AVMA-COE ¼ American Veterinary Medical Association Council on Education; OAC ¼ outcome assessment competency * Copied from the Accreditation Policies and Procedures of the AVMA Council on Education, April 201220
have demonstrated correlation between undergraduate GPA, Graduate Record Examination (GRE) scores, performance in veterinary school, and even performance on the NAVLE, but these conclusions are often limited to the venue and time period of the study.2–19 Weight allocated to each of these admissions components varies among veterinary schools and often varies from year to year within each veterinary school. In 2007, the AVMA Council on Education (AVMA-COE) identified nine professional competencies that each graduating veterinary student must attain and required each accredited College of Veterinary Medicine to develop and record measures of each professional competency for student outcome assessment. The nine AVMA-COE Competencies were developed to fulfill the AVMA-COE missions of ‘‘assisting the schools/colleges to improve veterinary medical education, and assuring the public that accredited programs provide a quality education,’’20(p.6) and are listed in the AVMA-COE Accreditation Policies and Procedures Manual, section 7.11, Standard 11.20 Those competencies, in short summary, are diagnostic skills, treatment planning, anesthesia and pain management, surgical skills, medical skills, emergency and critical care management, zoonosis and biosecurity, communication and ethics, and life-long learning skills (Box 1). At the Kansas State University College of Veterinary Medicine (KSUCVM), assessment of the clinical competencies mandated by AVMA-COE was instituted in 2008 at the start of the senior year of the Class of 2009. Subjective assessment of the nine competencies on a 5-point scale by the faculty member in charge of the rotation took place at the end of each of the 17 clinical rotations, and results were recorded, averaged for each student, and summarized on a commercial software system.a Although the majority of competencies could be assessed on all rotations, there were exceptions on individual rotations where a competency was not applicable to that rotation and thus was not assessed (e.g., there was no assessment of surgical skills on a medicine rotation). The subjective assessment scale for each competency was specifically (1) far below expected/unacceptable, (2) below expected/area of weakness, (3) expected/proficiency, (4) next 25%/area of strength, and (5) top 5%/mastery. A list of required and optional clinical skills was also maintained
20
for each student and reviewed by the attending clinician. In addition, the clinical rotations were graded by assignment of a standard percentage grade by each instructor as per University grading policy. There have been no published reports of collected assessments under the AVMA-COE Professional Competencies or of their correlation with pre-veterinary predictive measures, more traditional veterinary school outcomes, or the NAVLE. The goal of this study was to collect individual admissions data, veterinary school grades, AVMA-COE Competency scores, and NAVLE scores of the students and to determine correlations between these measures. We hypothesized that the AVMA-COE Professional Competencies could be reasonably predicted by common pre-veterinary measures and that the AVMA-COE Professional Competencies would be predictive of performance on the NAVLE.
MATERIALS AND METHODS Data consisting of preadmission criteria scores, intraprofessional program data, and self-reported raw NAVLE scores were collected on all students graduating from KSUCVM in 2009 (108 students) and 2010 (107 students). Collected data were ruled exempt from consideration by the University Human Subjects Institutional Review Board. The preadmission criteria that were selected were those used in the admissions process at KSUCVM and included scores of the verbal, quantitative, and analytical portions of the GRE, the total GRE score, and the mean GPA of the 35 hours of required pre-veterinary science courses (SGPA). Intra-professional program data selected included the annual GPA of the student at the end of each of the four years of the professional curriculum and the final class rank of the individual. Numerical grades were collected from several courses in the professional veterinary curriculum, with selection of the individual course determined by willingness of the course instructor to release the scores of the entire course to the investigators. Recorded first-year scores included Immunology, Physiology I, Physiology II, and Gross Anatomy I. Recorded second-year scores included Microbiology, Radiology, Systemic Pathology, and Pharmacology. Recorded third-year individual course scores were Medicine I,
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Table 1: Mean, range, and median of the primary outcome variables of the NAVLE score, final class rank, final cumulative GPA (4.0 scale), and final cumulative mean competencies * Mean e SD
Median
Minimum
Maximum
522.24 e 62.75 – 3.28 e 0.37 3.51 e 0.24
516 54 3.30 3.50
345 1 2.40 2.9
665 108 3.96 4.3
Outcome NAVLE score Final class rank Final GPA Final mean competency score
NAVLE ¼ North American Veterinary Licensing Exam; GPA ¼ grade point average * All outcomes except final class rank are normally distributed. Mean competencies are subjectively assessed on a scale of 1 (far below expectations/unacceptable) to 5 (top 5%/mastery).
Journal of Veterinary Medical Education 2014.41:19-26.
Table 2: Correlation of pre-veterinary criteria with selected outcome measures *
Verbal GRE Quantitative GRE Analytical GRE Total GRE SGPA
Year 1 GPA
Year 2 GPA
Year 3 GPA
Year 4 GPA
Final rank
Mean AVMA-COE Competency score
NAVLE score
†
†
†
†
†
†
.214
.173 .172 .232 .318
†
†
†
†
†
†
.161 .214 .38
†
†
†
†
†
†
.33
†
.186 .197 .346
†
.171 .236
†
GPA ¼ grade point average; AVMA-COE ¼ American Veterinary Medical Association Council on Education; NAVLE ¼ North American Veterinary Licensing Exam; GRE ¼ Graduate Record Examination; SGPA ¼ mean GPA of pre-professional science courses * Numbers represent a significant r value for the correlation. Negative correlations with final class rank reflect the inverse relationship between lower class rank and GPA. † Indicates the variables were not significantly correlated at p a .05.
Medicine II, Equine Medicine III, Surgery I, Surgery II, and Food Animal Medicine. The cumulative mean scores of each student for the nine individual clinical competencies for the senior year (Box 1) were recorded along with the overall mean of all clinical competency scores. The NAVLE score of each student as self-reported to the KSUCVM was also collected. All categories of collected data were compared by Pearson correlation to determine the degree of correlation between individual categories. Negative correlations that are reported with final class rank reflect the inverse relationship between class rank and other data such as grades, GRE, and NAVLE scores. Positive and negative correlations were considered significant if p a .05. For purposes of description, a comparison was considered to be a weak correlation if the association was significant and the r value was between .29 and .29. The comparison was considered to be a moderate correlation if the association was significant and the r value was between .30 and .59 or between .30 and .59. The comparison was considered to be a strong correlation if the association was significant and the r value was between 1 and .60 or greater than .60. Only statistically significant correlation values at p a .05 are reported.
RESULTS The 2009 and 2010 classes were composed collectively of 215 graduates. Only five students from the 2009 class and six students from the 2010 class attended the professional curriculum for more than 4 years; these numbers were doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
too insignificant to allow separate analysis of the small group of students with recognized academic difficulty. For those students, individual course scores used in the analysis were the passing grade for the course. The NAVLE, final cumulative GPA, final class rank, and final cumulative mean competency scores were chosen as measures of outcome success for comparison purposes, and particular attention was given to significant correlations to those measures (see Table 1). These outcomes were normally distributed except for class rank.
Pre-Veterinary Criteria The mean raw total GRE score of all students was 1797.7 e 165.3 (maximum 2400). The mean pre-veterinary SGPA was 3.47 e 0.33 (maximum 4.0). Neither the total GRE score nor its components were significantly associated with the SGPA. The verbal GRE score did not correlate with the annual GPA of any year or with the final class rank. Other pre-veterinary criteria used for admission were only sporadically significantly weakly correlated with the annual GPA in curriculum Years 1–3 (see Table 2). No pre-veterinary criteria used for admission were significantly correlated with the fourth-year GPA. Significant, but weak, negative correlations existed between the final class rank and the analytical portion of the GRE (r ¼ .186), the total GRE (r ¼ .197), and the SGPA (r ¼ .346). Low positive correlations occurred between the NAVLE and the verbal GRE (r ¼ .214), the total GRE (r ¼ .171), and the SGPA (r ¼ .236) (Table 2). Pre-veterinary criteria correlated at low positive levels 21
Table 3: Correlation of didactic course grades with annual GPA, final class rank, and the NAVLE score *
Journal of Veterinary Medical Education 2014.41:19-26.
Anatomy Equine Medicine III Food Animal Medicine Immunology Medicine I Medicine II Microbiology Pharmacology Physiology I Physiology II Radiology Surgery I Surgery II Systemic Pathology
Year 1 GPA
Year 2 GPA
Year 3 GPA
Year 4 GPA
Final rank
NAVLE score
.766 .574 .573 .697 .639 .647 .693 .592 .639 .635 .701 .62 .518 .561
.661 .701 .71 .724 .883 .736 .676 .696 .688 .653 .817 .679 .567 .507
.597 .77 .725 .608 .749 .767 .565 .715 .532 .541 .645 .749 .724 .484
.641
.724 .774 .751 .693 .793 .803 .691 .791 .641 .62 .75 .779 .682 .544
.524 .668 .534 .423 .636 .605 .541 .578 .39 .444 .603 .604 .565 .423
† † † † † † †
.587 .584 † † †
.479
GPA ¼ grade point average; NAVLE ¼ North American Veterinary Licensing Exam * Numbers represent a significant r value for the correlation. † Indicates the variables were not significantly correlated at p a .05.
with grades in some individual professional courses, with the SGPA being the most consistent significantly correlated factor. The average correlation of the total GRE with a professional course was r ¼ .233 (range .039 to .396). The average correlation of the SGPA with a professional course was r ¼ .309 (range .221 to .398). Preveterinary criteria did not significantly correlate with any of the AVMA-COE Professional Competencies with the exception that the SGPA was negatively correlated with surgical skills (r ¼ .168).
Intra-Professional Curricula Measures Annual GPA was strongly correlated with individual course scores (see Table 3). The average correlation of annual GPA with an individual course was r ¼ .653 (range .132 to .883). Final cumulative GPA at the end of the fourth year was also strongly negatively correlated with the final class rank (mean r ¼ .849). Annual GPA at the end of Years 1–3 was weakly correlated with the AVMA Competencies (mean r ¼ .270, range .188 to .414) (see Table 4). Final cumulative GPA at the end of the fourth year was not significantly associated with the AVMA Competencies. Annual GPA at the end of the fourth year was strongly correlated with the NAVLE score (mean r ¼ .628). Final class rank was strongly negatively correlated with individual course scores. The average correlation of final class rank with individual course scores was r ¼ .717. Final class rank was moderately negatively correlated with the AVMA-COE Competencies (r ¼ .426, range .549 to .286) (Table 4). Final class rank was strongly negatively correlated with the NAVLE score (r ¼ .714). Student grades in individual courses were usually strongly and significantly correlated with other course grades (mean r ¼ .583, median r ¼ .599), ranging from a low correlation of r ¼ .275 between Sys-
22
temic Pathology and Food Animal Medicine to a high of r ¼ .772 between the Medicine I and Medicine II courses.
AVMA-COE Competencies The AVMA-COE Competencies were weakly correlated with scores earned in the didactic courses of professional curricula Years 1–3 (mean r ¼ .290, range ¼ no significance to .516), but there was strong correlation among individual AVMA-COE Competencies (mean r ¼ .796, range .524 to .937) (see Table 5). The AVMA-COE Competencies were moderately correlated with performance on the NAVLE (mean r ¼ .319, range .257 to .418).
DISCUSSION The results of the current study underscore the limited value of pre-veterinary data for prediction of performance in veterinary school, the strong correlations between individual professional-course grades and overall performance in veterinary school as measured by class rank (r ¼ .717), and the strong correlation between veterinary school performance as measured by class rank and performance on the NAVLE (r ¼ .714). The derived correlation r values of the latter comparisons are negative because as the number for class rank improves (decreases), the GPA and NAVLE scores increase as expected, but the association between the variables is strong. Contrary to our hypothesis, pre-veterinary criteria measures have little value in prediction of the AVMA-COE Professional Competencies scores at Kansas State University, and the AVMA Competencies have weak correlation with grades from didactic professional courses and only moderate correlation with performance on the NAVLE (r ¼ .319). AVMA-COE scores are internally consistent, with an average correlation between the AVMA-COE Competencies of r ¼ .796.
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Table 4: Correlation of AVMA-COE Competency scores with annual cumulative GPA, final class rank, and the NAVLE score *
OAC1—Diagnostic skills OAC2—Treatment planning OAC3—Anesthesia & pain OAC4—Surgical skills OAC5—Medical skills OAC6—Emergency/critical care OAC7—Zoonosis & biosecurity OAC8—Communication & ethics OAC9—Life-long learning skills
Year 1 GPA
Year 2 GPA
Year 3 GPA
Year 4 GPA
Final rank
NAVLE score
.365 .29 .267 .239 .332 .221 .168 .257 .31
.414 .353 .236 .188 .366 .241 .189 .195 .315
.441 .387 .312 .282 .412 .31 .249 .325 .386
†
.549 .465 .418 .343 .512 .385 .286 .39 .489
.418 .336 .326 .257 .374 .266
† † † † † † † †
†
.297 .331
Journal of Veterinary Medical Education 2014.41:19-26.
AVMA-COE ¼ American Veterinary Medical Association Council on Education; GPA ¼ grade point average; NAVLE ¼ North American Veterinary Licensing Exam; OAC ¼ outcome assessment competency * Numbers represent a significant r value for the correlation. The abbreviations for the Competencies (OAC1–OAC9) refer to the designations in Box 1. † Indicates the variables were not significantly correlated at p a .05.
Table 5: Intra-correlation of AVMA-COE Competency scores *
OAC1—Diagnostic skills OAC2—Treatment planning OAC3—Anesthesia & pain OAC4—Surgical skills OAC5—Medical skills OAC6—Emergency/critical care OAC7—Zoonosis & biosecurity OAC8—Communication & ethics OAC9—Life-long learning skills
OAC1
OAC2
OAC3
OAC4
OAC5
OAC6
OAC7
OAC8
OAC9
1
0.886 1
0.851 0.805 1
0.748 0.642 0.736 1
0.937 0.895 0.853 0.706 1
0.778 0.799 0.759 0.672 0.787 1
0.697 0.646 0.669 0.524 0.715 0.633 1
0.777 0.772 0.839 0.703 0.787 0.714 0.582 1
0.878 0.827 0.826 0.662 0.87 0.74 0.686 0.807 1
AVMA-COE ¼ American Veterinary Medical Association Council on Education; OAC ¼ outcome assessment competency * All variables were significantly correlated at p b .05. The abbreviations for the Competencies (OAC1–OAC9) refer to the designations in Box 1.
Preadmission GPAs are among the most widely-used and time-honored criteria for admission selection to veterinary college. Various GPA calculations have been reported as selection criteria, including overall undergraduate GPA, the GPA of required pre-veterinary courses, the GPA of science courses, and the GPA of the last 45 undergraduate credit hours taken. Numerous studies have found an association between the pre-veterinary GPA and success in veterinary school with correlation r values ranging from r ¼ .58 to r ¼ .32.3–11 The correlation of SGPA to the final class rank found in this study (r ¼ .346) is consistent with the associations from previous studies and the most recent report available.2 The GPA of required pre-veterinary school courses was a moderate predictor (r ¼ .20) of the forerunner to the NAVLE, the National Board Examination, in a previous report.12 The correlation of SGPA to the NAVLE in the current study (r ¼ .236) is consistent with and falls in the middle of the results of a similar two-institution com-
doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
parison of undergraduate GPA to the NAVLE from 2009 students (r ¼ .37) and 2010 students (r ¼ .16).2 In the current study, correlation of the SGPA with the individual AVMA-COE Professional Competencies scores was not statistically significant except for a moderate correlation with the individual communication and ethics score. Standardized tests that have been used in the selection of candidates for admission to veterinary medical colleges include the GRE, the Veterinary College Admission Test, and the Medical College Admission Test.5,7–15 The GRE is the most widely used with 25 US Veterinary Schools reporting scores.2 In the current study, the total GRE was only weakly associated with the annual GPA in the first (r ¼ .232) and second (r ¼ .214) years of the curriculum and with the final class rank (r ¼ .197). GRE was also weakly associated with the NAVLE (r ¼ .171) in the current study. Both of the latter findings are consistent with a recent report that found correlations of r ¼ .22 and r ¼ .14 between GRE and the cumulative veterinary GPA
23
Journal of Veterinary Medical Education 2014.41:19-26.
for the classes of 2009 and 2010, respectively, and found a weak correlation of GRE with the NAVLE (r ¼ .23 and r ¼ .30, respectively).2 In the current report, GRE did not significantly correlate with any AVMA-COE Professional Competency except for a negative correlation of the verbal portion of the GRE with the basic surgical skills competency score. Based on the data of this report and others, the traditional key objective criteria for veterinary school admissions (GRE or pre-veterinary GPA) are not reliable predictors of success during veterinary school. More reliable objective predictors of success should be investigated. One goal of the current study was to find predictive metrics for success in veterinary school. Variables such as the total GRE score and the mean AVMA-COE Competency score were chosen to evaluate their predictive ability, not to serve as valid evaluation measures. Summation of the diverse components of the GRE into a total GRE score is not a valid evaluation point because each component measures a separate skill. Subsequently, there are multiple score combinations of individual components that could result in the same total GRE score. In this study, individual GRE values were combined for the purpose of determining if the combined figure, or total GRE, could be used as a predictor of other measured variables. Thus, although the actual total GRE score cannot be used to help explain a significant correlation with another measure, the fact remains that it might be a useful predictor of another measure and was useful in some comparisons. Similarly, the average of the AVMA-COE Competency scores in this study into a mean AVMA-COE value does not necessarily establish the validity of the value itself since the AVMA-COE Competencies are diverse and each assesses a different skill set, but it was the predictive value of the mean score that was assessed. The results of the current study, as expected, confirm that individual course scores are often strongly correlated with overall annual GPA and with the final class rank (Table 3) and that one individual course score often strongly correlates with another. Both these results ultimately support a hypothesis that student performances are consistent across a wide variation in courses. Individual course scores were weakly correlated with AVMA-COE Competencies, with a mean correlation of r ¼ .290, as was the annual GPA for Years 1–3 of the professional curriculum (mean r ¼ .270). The AVMA-COE scores are also moderately negatively correlated with the final class rank (mean r ¼ .426). These results indicate either that the evaluation method used for AVMA-COE Competency assessments at the KSUCVM are less consistent than didactic testing or else that the AVMA-COE Competencies are measuring different skills than the didactic testing. Despite this, AVMA-COE Competencies still moderately reflect overall student performance in the curriculum. The complete lack of significant correlation between the fourth-year GPA and the AVMA-COE scores, however, indicates that the subjective assignments of grades in the fourth year is not significantly related to the scoring system used to assess the AVMA-COE Competencies. This discrepancy suggests that the assessments of AVMA-COE Competencies at the KSUCVM do not reflect overall faculty impression of fourth-year clinical 24
performance by the student and/or that the rating of AVMA-COE Competencies is not being routinely considered as a factor in assigning grades. The assessment methodology at the KSUCVM is not unique and is used at other veterinary, dental, and medical institutions.16 Further refinement of grading procedures and assessments of AVMA-COE Competencies should be carried out to resolve this discrepancy. The AVMA-COE Competency assessments at the KSUCVM are internally consistent, as demonstrated by a mean r ¼ .796 correlation among each of the core competencies (Table 5), but are only moderately correlated with performance on the NAVLE (mean r ¼ .319). Performance on the NAVLE is better predicted by the fourth-year clinical grades (r ¼ .643) than the AVMA-COE Competencies. The predictive advantage of fourth-year grades over the AVMA-COE Competency scores indicates either that the current AVMA-COE Competency scoring system is less effective than traditional grading schema regarding prediction of NAVLE performance and thus an improved scoring system for the Competencies is needed, that the Competencies themselves need to be reassessed, or that both the scoring system and the competencies need to be improved. Standard deviation of the assigned raw COE scores ranged from .204 to .326 (median ¼ .264, mean ¼ .240), which was less variable, and therefore less discriminatory, than the standard deviation of the fourthyear GPA (.366). This study does not address the question of whether the primary purpose of the AVMA-COE Competencies assessment is to serve as a mechanism for curriculum adjustment and better preparation of students for the NAVLE assessment, or if the AVMA-COE skills themselves are important education outcomes that need separate assessment. If the reason for mandating the AVMA-COE Competencies assessments was originally as a guide to revising the curriculum for better NAVLE preparation, then the AVMA-COE assessments as currently implemented are not ideal. Certainly the AVMA-COE Competencies are subjective clinical assessments, and we have shown that they likely do not measure the same skills as the more objectively oriented NAVLE. Alternatively, if the AVMA-COE Competencies are important outcomes in and of themselves as they were intended to be by the AVMA-COE,20 then the current NAVLE is not assessing them well. A determination of which outcome is more important (AVMA-COE Competencies or NAVLE) to complete the AVMA-COE missions to improve veterinary medical education and ensure public interests, will ultimately help determine whether the Competencies or the NAVLE needs revision. Unfortunately, the published evidence for designing reliable and valid assessment methods to assess competence is scant17 although detailed suggestions for assessment have been suggested.18,19 National emphasis on competency assessment to ensure public safety as a responsibility of institutions is only increasing, as evidenced by the most recent report on veterinary education, the ‘‘Roadmap for Veterinary Medical Education in the 21st Century,’’ which listed ‘‘Ensure that admissions, curricula, accreditation, and testing/licensure are competency driven,’’ as its second goal.21(p.322)
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Journal of Veterinary Medical Education 2014.41:19-26.
Many previous reports, as well as the current study, have biases inherent in the methodology. Students who completely failed to progress for either academic or personal reasons were not included in the analysis of the majority of these studies, possibly allowing a form of nonresponse bias to insinuate itself into the results. Exclusion of this data raises the mean and decreases the standard deviation of the resulting population (eliminating the ‘‘deficient’’ outliers), but most published reports do not address or speculate on this bias. The current study did include 11 students who took longer to progress than the normal four-year professional program but did not include students who did not ultimately graduate from veterinary school. The low number of these students in the present study is unlikely to have had significant influence on the overall outcomes and associations among variables. Academic difficulty during veterinary school has been associated with a low prerequisite GPA, a low GRE score, and older age,22 but any relationship between students who have academic difficulty and individual performances on AVMA-COE Competencies has yet to be determined. The ‘‘gold standard’’ suggested by these authors to define success and assess the predictive value of the various measures is NAVLE performance. The danger of relying solely on the NAVLE as a measure of success is one of validity, of whether the full range of NAVLE scores are valid predictors of success as a graduate veterinarian. Increasingly, the NAVLE score is the primary discriminator affecting veterinary licensure, and the NAVLE pass rate is subsequently used to assess veterinary programs for accreditation purposes. However, in actual fact, only the NAVLE pass point is relevant to licensure and thus is a gateway to continued professional activity. AVMA-COE Competencies are internally consistent but poorly related to performance measures within veterinary school and only moderately related to NAVLE scores. The AVMA-COE Competencies may, in fact, be more valid as a measure of post-DVM success as they were intended; however, there is little current evidence to support that hypothesis, and the conflict of whether the AVMA-COE Competencies or the NAVLE are better predictors of graduate success may exist until either the licensing examination is changed to better reflect AVMA-COE Competencies or the AVMA-COE Competencies are better refined as an aid to improve NAVLE performance. If professional success is indeed related to NAVLE performance, and NAVLE scores are strongly related to overall academic success as demonstrated here, then class rank and cumulative grades are also good indicators of professional success and are demonstratively better predictors of future professional success than the professional competencies currently mandated for assessment.
2
3 4
5
6
7
8 9
10
11
12
13
14
15
16
NOTE a
one45 Medical School Management Software. Vancouver, BC: one45.
17
REFERENCES 1
Fuentealba C, Hecker KG, Nelson PD, et al. Relationships between admissions requirements and pre-clinical and clinical performance in a distributed
doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
18
veterinary curriculum. J Vet Med Educ. 2011;38(1):52–9. http://dx.doi.org/10.3138/jvme.38.1.52. Medline:21805935 Danielson JA, Wu TF, Molgaard LK, et al. Relationships among common measures of student performance and scores on the North American Veterinary Licensing Examination. J Am Vet Med Assoc. 2011;238(4):454–61. http://dx.doi.org/10.2460/javma.238.4.454. Medline:21320014 Noeth RJ. Predicting success in the study of veterinary science and medicine. J Educ Res. 1974;67:213–5. Niedzwiedz ER, Friedman BA. A comparative analysis of the validity of pre-admissions information at four colleges of veterinary medicine. J Vet Med Educ. 1976;3:32–7. Render GR, Jackson HD. Preveterinary performance, admissions criteria and personality variables as predictors of success in veterinary school. J Vet Med Educ. 1975;2:3–6. Julius MF, Kaiser HE. Pre-veterinary medical grade point averages as predictors of academic success in veterinary college. J Vet Med Educ. 1978;5:162–4. Cawunder P, Tasker JB. Use of the new medical college admissions test (MCAT) in veterinary school admissions. J Vet Med Educ. 1982;9:1–3. Kelman EG. Predicting success in veterinary medical college. J Vet Med Educ. 1982;8:92–4. Kearney MT, Shane SM, Tasker JB. The value of the medical college admissions test for selection of applicants to a veterinary school. J Vet Med Educ. 1985;11:54–62. Confer AW. Preadmission GRE scores and GPAs as predictors of academic performance in a college of veterinary medicine. J Vet Med Educ. 1990;17:56–62. Zachary JF, Schaeffer DJ. Correlations between preveterinary admissions variables and academic success in core courses during the first two years of the veterinary curriculum. J Vet Med Educ. 1994;21:56–60. Shane SM, Kearney MT, Thayer RW. Analysis of national board examination scores with reference to achievement during the professional veterinary curriculum. J Vet Med Educ. 1985;11:50–4. Lafi SQ, Kaneene JB, Tasker JB. Predicting Medical College Admission Test (MCAT) scores using Graduate Record Examination (GRE) scores. J Vet Med Educ. 1990;17:38–43. Shane SM, Harrington KS. Survey of standard tests as criteria for admission to North American veterinary colleges. J Vet Med Educ. 1989;16:45–6. Latshaw WK. The use of multivariate analysis to analyze the admissions process. J Vet Med Educ. 1982;9:6–9. Hardie EM. Current methods in use for assessing clinical competencies: what works? J Vet Med Educ. 2008;35(3):359–68. http://dx.doi.org/10.3138/ jvme.35.3.359. Medline:19066352 Rhind SM, Baillie S, Brown F, et al. Assessing competence in veterinary medical education: where’s the evidence? J Vet Med Educ. 2008;35(3):407–11. http:// dx.doi.org/10.3138/jvme.35.3.407. Medline:19066358 Turnwald G, Stone E, Bristol D, et al. Assessing clinical competency: reports from discussion groups. J Vet Med
25
19
20
21
Journal of Veterinary Medical Education 2014.41:19-26.
22
26
Educ. 2008;35(3):343–53. http://dx.doi.org/10.3138/ jvme.35.3.343. Medline:19066350 Lloyd JW, King LJ, Klausner JS, et al. National workshop on core competencies for success in the veterinary profession. J Vet Med Educ. 2003;30(3):280–4. http://dx.doi.org/10.3138/jvme.30.3.280. Medline:14648504 AVMA. Accreditation policies and procedures of the AVMA Council on Education. Schaumburg, IL: American Veterinary Medical Association; April 2012 [updated 2012 Oct; cited 2013 Jun 19]. Available from: https://www.avma.org/ProfessionalDevelopment/ Education/Accreditation/Colleges/Pages/coe-pp.aspx. Schurig G, Osburn BI. The North American Veterinary Medical Education Consortium (NAVMEC) looks to veterinary medical education for the future: ‘‘Roadmap for Veterinary Medical Education in the 21st Century: Responsive, Collaborative, Flexible’’. J Vet Med Educ. 2011;38(4):320–6. http://dx.doi.org/10.3138/ jvme.38.4.320. Rush BR, Sanderson MW, Elmore RG. Pre-matriculation indicators of academic difficulty during veterinary school. J Vet Med Educ. 2005;32(4):517–22. http:// dx.doi.org/10.3138/jvme.32.4.517. Medline:16421838
AUTHOR INFORMATION James K. Roush, DVM, MS, Dipl ACVS, is the Doughman Professor of Small Animal Surgery, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected]. Bonnie R. Rush, DVM, MS, Dipl ACVIM, is Head of the Department of Clinical Sciences and Professor of Equine Internal Medicine, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected]. Brad J. White, DVM, MS, is Associate Professor of Beef Production and Management, Department of Clinical Sciences and Department of Anatomy and Physiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected]. Melinda J. Wilkerson, MS, DVM, PhD, Dipl ACVP, is Professor of Immunology, Department of Diagnostic Medicine and Pathobiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected].
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Correlation of Pre-Veterinary Admissions Criteria, Intra-Professional Curriculum Measures, AVMA-COE Professional Competency Scores, and the NAVLE James K. Roush n Bonnie R. Rush n Brad J. White n Melinda J. Wilkerson
Journal of Veterinary Medical Education 2014.41:19-26.
ABSTRACT Data consisting of preadmission criteria scores, annual and final cumulative grade point averages (GPAs), grades from individual professional courses, American Veterinary Medical Association Council on Education (AVMA-COE) Competency scores, annual class rank, and North American Veterinary Licensing Exam (NAVLE) scores were collected on all graduating DVM students at Kansas State University in 2009 and 2010. Associations among the collected data were compared by Pearson correlation. Pre-veterinary admissions criteria infrequently correlated with annual GPAs of Years 1–3, rarely correlated with the AVMA-COE Competencies, and never correlated with the annual GPA of Year 4. Low positive correlations occurred between the NAVLE and the Verbal Graduate Record Examination (GRE) (r ¼ .214), Total GRE (r ¼ .171), and the mean GPA of pre-professional science courses (SGPA) (r ¼ .236). Annual GPAs strongly correlated with didactic course scores. Annual GPAs and final class rank strongly correlated (mean r ¼ .849), and both strongly correlated with the NAVLE score (NAVLE: GPAs mean r ¼ .628, NAVLE: final class rank r ¼ .714). Annual GPAs at the end of Years 1–4 weakly correlated or did not correlate with the AVMA-COE Competencies. The AVMA-COE Competencies weakly correlated with scores earned in didactic courses of Years 1–3. AVMA-COE Competencies were internally consistent (mean r ¼ .796) but only moderately correlated with performance on the NAVLE (mean r ¼ .319). Low correlations between admissions criteria and outcomes indicate a need to reevaluate admission criteria as predictors of school success. If the NAVLE remains the primary discriminator for veterinary licensure (and the gateway to professional activity), then the AVMA-COE Competencies should be refined to better improve and reflect the NAVLE, or the NAVLE examination should change to reflect AVMA-COE Competencies. Key words: Council on Education, COE, AVMA-COE, professional competencies, NAVLE, admissions criteria, predictors of veterinary school performance
INTRODUCTION Valid predictive criteria for academic success during veterinary training or professional success after graduation are constantly sought, evaluated, and reevaluated by veterinary colleges, state veterinary licensing boards, and other veterinary professional organizations. The definition of veterinary school success is debatable, with vocal proponents for measurements such as graduating class rank, North American Veterinary Licensing Exam (NAVLE) scores, and state exam board scores. The current trend of most states to place more weight on NAVLE scores for licensing requirements suggests that the NAVLE has become the de facto gateway to professional activity for veterinary students and, by extension, is an important measure of the success of a veterinary college since NAVLE passing rates affect American Veterinary Medical Association (AVMA) accreditation. The NAVLE, however,
doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
may not be appropriate as a measure of success since the intention of the NAVLE is to provide a minimum standard of competency for state licensing boards. More refined measures to determine veterinary school success have yet to be developed. Preadmission criteria for professional veterinary curricula are established independently by each of the 28 veterinary medical colleges currently accredited in the United States. Admissions policies in most veterinary medical colleges have been loosely based on scientific data related to the predictive validity of admission requirements and are more often based on conjecture, historical precedence, and habit. The most commonly used criteria for admission to veterinary school are calculations of the pre-veterinary grade point average (GPA), standardized tests, the personal interview, veterinaryrelated experience, personal recommendations or evaluations, and an essay or narrative.1 Several previous studies
19
Box 1: Outcome assessment competencies of the AVMA-COE * OAC1 OAC2 OAC3 OAC4 OAC5 OAC6 OAC7 OAC8 OAC9
Comprehensive patient diagnosis (problem-solving skills), appropriate use of clinical laboratory testing, and record management Comprehensive treatment planning including patient referral when indicated Anesthesia and pain management, patient welfare Basic surgery skills, experience, and case management Basic medicine skills, experience, and case management Emergency and intensive care case management Health promotion, disease prevention/biosecurity, zoonosis, and food safety Client communications and ethical conduct Critical analysis of new information and research findings relevant to veterinary medicine
Journal of Veterinary Medical Education 2014.41:19-26.
AVMA-COE ¼ American Veterinary Medical Association Council on Education; OAC ¼ outcome assessment competency * Copied from the Accreditation Policies and Procedures of the AVMA Council on Education, April 201220
have demonstrated correlation between undergraduate GPA, Graduate Record Examination (GRE) scores, performance in veterinary school, and even performance on the NAVLE, but these conclusions are often limited to the venue and time period of the study.2–19 Weight allocated to each of these admissions components varies among veterinary schools and often varies from year to year within each veterinary school. In 2007, the AVMA Council on Education (AVMA-COE) identified nine professional competencies that each graduating veterinary student must attain and required each accredited College of Veterinary Medicine to develop and record measures of each professional competency for student outcome assessment. The nine AVMA-COE Competencies were developed to fulfill the AVMA-COE missions of ‘‘assisting the schools/colleges to improve veterinary medical education, and assuring the public that accredited programs provide a quality education,’’20(p.6) and are listed in the AVMA-COE Accreditation Policies and Procedures Manual, section 7.11, Standard 11.20 Those competencies, in short summary, are diagnostic skills, treatment planning, anesthesia and pain management, surgical skills, medical skills, emergency and critical care management, zoonosis and biosecurity, communication and ethics, and life-long learning skills (Box 1). At the Kansas State University College of Veterinary Medicine (KSUCVM), assessment of the clinical competencies mandated by AVMA-COE was instituted in 2008 at the start of the senior year of the Class of 2009. Subjective assessment of the nine competencies on a 5-point scale by the faculty member in charge of the rotation took place at the end of each of the 17 clinical rotations, and results were recorded, averaged for each student, and summarized on a commercial software system.a Although the majority of competencies could be assessed on all rotations, there were exceptions on individual rotations where a competency was not applicable to that rotation and thus was not assessed (e.g., there was no assessment of surgical skills on a medicine rotation). The subjective assessment scale for each competency was specifically (1) far below expected/unacceptable, (2) below expected/area of weakness, (3) expected/proficiency, (4) next 25%/area of strength, and (5) top 5%/mastery. A list of required and optional clinical skills was also maintained
20
for each student and reviewed by the attending clinician. In addition, the clinical rotations were graded by assignment of a standard percentage grade by each instructor as per University grading policy. There have been no published reports of collected assessments under the AVMA-COE Professional Competencies or of their correlation with pre-veterinary predictive measures, more traditional veterinary school outcomes, or the NAVLE. The goal of this study was to collect individual admissions data, veterinary school grades, AVMA-COE Competency scores, and NAVLE scores of the students and to determine correlations between these measures. We hypothesized that the AVMA-COE Professional Competencies could be reasonably predicted by common pre-veterinary measures and that the AVMA-COE Professional Competencies would be predictive of performance on the NAVLE.
MATERIALS AND METHODS Data consisting of preadmission criteria scores, intraprofessional program data, and self-reported raw NAVLE scores were collected on all students graduating from KSUCVM in 2009 (108 students) and 2010 (107 students). Collected data were ruled exempt from consideration by the University Human Subjects Institutional Review Board. The preadmission criteria that were selected were those used in the admissions process at KSUCVM and included scores of the verbal, quantitative, and analytical portions of the GRE, the total GRE score, and the mean GPA of the 35 hours of required pre-veterinary science courses (SGPA). Intra-professional program data selected included the annual GPA of the student at the end of each of the four years of the professional curriculum and the final class rank of the individual. Numerical grades were collected from several courses in the professional veterinary curriculum, with selection of the individual course determined by willingness of the course instructor to release the scores of the entire course to the investigators. Recorded first-year scores included Immunology, Physiology I, Physiology II, and Gross Anatomy I. Recorded second-year scores included Microbiology, Radiology, Systemic Pathology, and Pharmacology. Recorded third-year individual course scores were Medicine I,
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Table 1: Mean, range, and median of the primary outcome variables of the NAVLE score, final class rank, final cumulative GPA (4.0 scale), and final cumulative mean competencies * Mean e SD
Median
Minimum
Maximum
522.24 e 62.75 – 3.28 e 0.37 3.51 e 0.24
516 54 3.30 3.50
345 1 2.40 2.9
665 108 3.96 4.3
Outcome NAVLE score Final class rank Final GPA Final mean competency score
NAVLE ¼ North American Veterinary Licensing Exam; GPA ¼ grade point average * All outcomes except final class rank are normally distributed. Mean competencies are subjectively assessed on a scale of 1 (far below expectations/unacceptable) to 5 (top 5%/mastery).
Journal of Veterinary Medical Education 2014.41:19-26.
Table 2: Correlation of pre-veterinary criteria with selected outcome measures *
Verbal GRE Quantitative GRE Analytical GRE Total GRE SGPA
Year 1 GPA
Year 2 GPA
Year 3 GPA
Year 4 GPA
Final rank
Mean AVMA-COE Competency score
NAVLE score
†
†
†
†
†
†
.214
.173 .172 .232 .318
†
†
†
†
†
†
.161 .214 .38
†
†
†
†
†
†
.33
†
.186 .197 .346
†
.171 .236
†
GPA ¼ grade point average; AVMA-COE ¼ American Veterinary Medical Association Council on Education; NAVLE ¼ North American Veterinary Licensing Exam; GRE ¼ Graduate Record Examination; SGPA ¼ mean GPA of pre-professional science courses * Numbers represent a significant r value for the correlation. Negative correlations with final class rank reflect the inverse relationship between lower class rank and GPA. † Indicates the variables were not significantly correlated at p a .05.
Medicine II, Equine Medicine III, Surgery I, Surgery II, and Food Animal Medicine. The cumulative mean scores of each student for the nine individual clinical competencies for the senior year (Box 1) were recorded along with the overall mean of all clinical competency scores. The NAVLE score of each student as self-reported to the KSUCVM was also collected. All categories of collected data were compared by Pearson correlation to determine the degree of correlation between individual categories. Negative correlations that are reported with final class rank reflect the inverse relationship between class rank and other data such as grades, GRE, and NAVLE scores. Positive and negative correlations were considered significant if p a .05. For purposes of description, a comparison was considered to be a weak correlation if the association was significant and the r value was between .29 and .29. The comparison was considered to be a moderate correlation if the association was significant and the r value was between .30 and .59 or between .30 and .59. The comparison was considered to be a strong correlation if the association was significant and the r value was between 1 and .60 or greater than .60. Only statistically significant correlation values at p a .05 are reported.
RESULTS The 2009 and 2010 classes were composed collectively of 215 graduates. Only five students from the 2009 class and six students from the 2010 class attended the professional curriculum for more than 4 years; these numbers were doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
too insignificant to allow separate analysis of the small group of students with recognized academic difficulty. For those students, individual course scores used in the analysis were the passing grade for the course. The NAVLE, final cumulative GPA, final class rank, and final cumulative mean competency scores were chosen as measures of outcome success for comparison purposes, and particular attention was given to significant correlations to those measures (see Table 1). These outcomes were normally distributed except for class rank.
Pre-Veterinary Criteria The mean raw total GRE score of all students was 1797.7 e 165.3 (maximum 2400). The mean pre-veterinary SGPA was 3.47 e 0.33 (maximum 4.0). Neither the total GRE score nor its components were significantly associated with the SGPA. The verbal GRE score did not correlate with the annual GPA of any year or with the final class rank. Other pre-veterinary criteria used for admission were only sporadically significantly weakly correlated with the annual GPA in curriculum Years 1–3 (see Table 2). No pre-veterinary criteria used for admission were significantly correlated with the fourth-year GPA. Significant, but weak, negative correlations existed between the final class rank and the analytical portion of the GRE (r ¼ .186), the total GRE (r ¼ .197), and the SGPA (r ¼ .346). Low positive correlations occurred between the NAVLE and the verbal GRE (r ¼ .214), the total GRE (r ¼ .171), and the SGPA (r ¼ .236) (Table 2). Pre-veterinary criteria correlated at low positive levels 21
Table 3: Correlation of didactic course grades with annual GPA, final class rank, and the NAVLE score *
Journal of Veterinary Medical Education 2014.41:19-26.
Anatomy Equine Medicine III Food Animal Medicine Immunology Medicine I Medicine II Microbiology Pharmacology Physiology I Physiology II Radiology Surgery I Surgery II Systemic Pathology
Year 1 GPA
Year 2 GPA
Year 3 GPA
Year 4 GPA
Final rank
NAVLE score
.766 .574 .573 .697 .639 .647 .693 .592 .639 .635 .701 .62 .518 .561
.661 .701 .71 .724 .883 .736 .676 .696 .688 .653 .817 .679 .567 .507
.597 .77 .725 .608 .749 .767 .565 .715 .532 .541 .645 .749 .724 .484
.641
.724 .774 .751 .693 .793 .803 .691 .791 .641 .62 .75 .779 .682 .544
.524 .668 .534 .423 .636 .605 .541 .578 .39 .444 .603 .604 .565 .423
† † † † † † †
.587 .584 † † †
.479
GPA ¼ grade point average; NAVLE ¼ North American Veterinary Licensing Exam * Numbers represent a significant r value for the correlation. † Indicates the variables were not significantly correlated at p a .05.
with grades in some individual professional courses, with the SGPA being the most consistent significantly correlated factor. The average correlation of the total GRE with a professional course was r ¼ .233 (range .039 to .396). The average correlation of the SGPA with a professional course was r ¼ .309 (range .221 to .398). Preveterinary criteria did not significantly correlate with any of the AVMA-COE Professional Competencies with the exception that the SGPA was negatively correlated with surgical skills (r ¼ .168).
Intra-Professional Curricula Measures Annual GPA was strongly correlated with individual course scores (see Table 3). The average correlation of annual GPA with an individual course was r ¼ .653 (range .132 to .883). Final cumulative GPA at the end of the fourth year was also strongly negatively correlated with the final class rank (mean r ¼ .849). Annual GPA at the end of Years 1–3 was weakly correlated with the AVMA Competencies (mean r ¼ .270, range .188 to .414) (see Table 4). Final cumulative GPA at the end of the fourth year was not significantly associated with the AVMA Competencies. Annual GPA at the end of the fourth year was strongly correlated with the NAVLE score (mean r ¼ .628). Final class rank was strongly negatively correlated with individual course scores. The average correlation of final class rank with individual course scores was r ¼ .717. Final class rank was moderately negatively correlated with the AVMA-COE Competencies (r ¼ .426, range .549 to .286) (Table 4). Final class rank was strongly negatively correlated with the NAVLE score (r ¼ .714). Student grades in individual courses were usually strongly and significantly correlated with other course grades (mean r ¼ .583, median r ¼ .599), ranging from a low correlation of r ¼ .275 between Sys-
22
temic Pathology and Food Animal Medicine to a high of r ¼ .772 between the Medicine I and Medicine II courses.
AVMA-COE Competencies The AVMA-COE Competencies were weakly correlated with scores earned in the didactic courses of professional curricula Years 1–3 (mean r ¼ .290, range ¼ no significance to .516), but there was strong correlation among individual AVMA-COE Competencies (mean r ¼ .796, range .524 to .937) (see Table 5). The AVMA-COE Competencies were moderately correlated with performance on the NAVLE (mean r ¼ .319, range .257 to .418).
DISCUSSION The results of the current study underscore the limited value of pre-veterinary data for prediction of performance in veterinary school, the strong correlations between individual professional-course grades and overall performance in veterinary school as measured by class rank (r ¼ .717), and the strong correlation between veterinary school performance as measured by class rank and performance on the NAVLE (r ¼ .714). The derived correlation r values of the latter comparisons are negative because as the number for class rank improves (decreases), the GPA and NAVLE scores increase as expected, but the association between the variables is strong. Contrary to our hypothesis, pre-veterinary criteria measures have little value in prediction of the AVMA-COE Professional Competencies scores at Kansas State University, and the AVMA Competencies have weak correlation with grades from didactic professional courses and only moderate correlation with performance on the NAVLE (r ¼ .319). AVMA-COE scores are internally consistent, with an average correlation between the AVMA-COE Competencies of r ¼ .796.
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Table 4: Correlation of AVMA-COE Competency scores with annual cumulative GPA, final class rank, and the NAVLE score *
OAC1—Diagnostic skills OAC2—Treatment planning OAC3—Anesthesia & pain OAC4—Surgical skills OAC5—Medical skills OAC6—Emergency/critical care OAC7—Zoonosis & biosecurity OAC8—Communication & ethics OAC9—Life-long learning skills
Year 1 GPA
Year 2 GPA
Year 3 GPA
Year 4 GPA
Final rank
NAVLE score
.365 .29 .267 .239 .332 .221 .168 .257 .31
.414 .353 .236 .188 .366 .241 .189 .195 .315
.441 .387 .312 .282 .412 .31 .249 .325 .386
†
.549 .465 .418 .343 .512 .385 .286 .39 .489
.418 .336 .326 .257 .374 .266
† † † † † † † †
†
.297 .331
Journal of Veterinary Medical Education 2014.41:19-26.
AVMA-COE ¼ American Veterinary Medical Association Council on Education; GPA ¼ grade point average; NAVLE ¼ North American Veterinary Licensing Exam; OAC ¼ outcome assessment competency * Numbers represent a significant r value for the correlation. The abbreviations for the Competencies (OAC1–OAC9) refer to the designations in Box 1. † Indicates the variables were not significantly correlated at p a .05.
Table 5: Intra-correlation of AVMA-COE Competency scores *
OAC1—Diagnostic skills OAC2—Treatment planning OAC3—Anesthesia & pain OAC4—Surgical skills OAC5—Medical skills OAC6—Emergency/critical care OAC7—Zoonosis & biosecurity OAC8—Communication & ethics OAC9—Life-long learning skills
OAC1
OAC2
OAC3
OAC4
OAC5
OAC6
OAC7
OAC8
OAC9
1
0.886 1
0.851 0.805 1
0.748 0.642 0.736 1
0.937 0.895 0.853 0.706 1
0.778 0.799 0.759 0.672 0.787 1
0.697 0.646 0.669 0.524 0.715 0.633 1
0.777 0.772 0.839 0.703 0.787 0.714 0.582 1
0.878 0.827 0.826 0.662 0.87 0.74 0.686 0.807 1
AVMA-COE ¼ American Veterinary Medical Association Council on Education; OAC ¼ outcome assessment competency * All variables were significantly correlated at p b .05. The abbreviations for the Competencies (OAC1–OAC9) refer to the designations in Box 1.
Preadmission GPAs are among the most widely-used and time-honored criteria for admission selection to veterinary college. Various GPA calculations have been reported as selection criteria, including overall undergraduate GPA, the GPA of required pre-veterinary courses, the GPA of science courses, and the GPA of the last 45 undergraduate credit hours taken. Numerous studies have found an association between the pre-veterinary GPA and success in veterinary school with correlation r values ranging from r ¼ .58 to r ¼ .32.3–11 The correlation of SGPA to the final class rank found in this study (r ¼ .346) is consistent with the associations from previous studies and the most recent report available.2 The GPA of required pre-veterinary school courses was a moderate predictor (r ¼ .20) of the forerunner to the NAVLE, the National Board Examination, in a previous report.12 The correlation of SGPA to the NAVLE in the current study (r ¼ .236) is consistent with and falls in the middle of the results of a similar two-institution com-
doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
parison of undergraduate GPA to the NAVLE from 2009 students (r ¼ .37) and 2010 students (r ¼ .16).2 In the current study, correlation of the SGPA with the individual AVMA-COE Professional Competencies scores was not statistically significant except for a moderate correlation with the individual communication and ethics score. Standardized tests that have been used in the selection of candidates for admission to veterinary medical colleges include the GRE, the Veterinary College Admission Test, and the Medical College Admission Test.5,7–15 The GRE is the most widely used with 25 US Veterinary Schools reporting scores.2 In the current study, the total GRE was only weakly associated with the annual GPA in the first (r ¼ .232) and second (r ¼ .214) years of the curriculum and with the final class rank (r ¼ .197). GRE was also weakly associated with the NAVLE (r ¼ .171) in the current study. Both of the latter findings are consistent with a recent report that found correlations of r ¼ .22 and r ¼ .14 between GRE and the cumulative veterinary GPA
23
Journal of Veterinary Medical Education 2014.41:19-26.
for the classes of 2009 and 2010, respectively, and found a weak correlation of GRE with the NAVLE (r ¼ .23 and r ¼ .30, respectively).2 In the current report, GRE did not significantly correlate with any AVMA-COE Professional Competency except for a negative correlation of the verbal portion of the GRE with the basic surgical skills competency score. Based on the data of this report and others, the traditional key objective criteria for veterinary school admissions (GRE or pre-veterinary GPA) are not reliable predictors of success during veterinary school. More reliable objective predictors of success should be investigated. One goal of the current study was to find predictive metrics for success in veterinary school. Variables such as the total GRE score and the mean AVMA-COE Competency score were chosen to evaluate their predictive ability, not to serve as valid evaluation measures. Summation of the diverse components of the GRE into a total GRE score is not a valid evaluation point because each component measures a separate skill. Subsequently, there are multiple score combinations of individual components that could result in the same total GRE score. In this study, individual GRE values were combined for the purpose of determining if the combined figure, or total GRE, could be used as a predictor of other measured variables. Thus, although the actual total GRE score cannot be used to help explain a significant correlation with another measure, the fact remains that it might be a useful predictor of another measure and was useful in some comparisons. Similarly, the average of the AVMA-COE Competency scores in this study into a mean AVMA-COE value does not necessarily establish the validity of the value itself since the AVMA-COE Competencies are diverse and each assesses a different skill set, but it was the predictive value of the mean score that was assessed. The results of the current study, as expected, confirm that individual course scores are often strongly correlated with overall annual GPA and with the final class rank (Table 3) and that one individual course score often strongly correlates with another. Both these results ultimately support a hypothesis that student performances are consistent across a wide variation in courses. Individual course scores were weakly correlated with AVMA-COE Competencies, with a mean correlation of r ¼ .290, as was the annual GPA for Years 1–3 of the professional curriculum (mean r ¼ .270). The AVMA-COE scores are also moderately negatively correlated with the final class rank (mean r ¼ .426). These results indicate either that the evaluation method used for AVMA-COE Competency assessments at the KSUCVM are less consistent than didactic testing or else that the AVMA-COE Competencies are measuring different skills than the didactic testing. Despite this, AVMA-COE Competencies still moderately reflect overall student performance in the curriculum. The complete lack of significant correlation between the fourth-year GPA and the AVMA-COE scores, however, indicates that the subjective assignments of grades in the fourth year is not significantly related to the scoring system used to assess the AVMA-COE Competencies. This discrepancy suggests that the assessments of AVMA-COE Competencies at the KSUCVM do not reflect overall faculty impression of fourth-year clinical 24
performance by the student and/or that the rating of AVMA-COE Competencies is not being routinely considered as a factor in assigning grades. The assessment methodology at the KSUCVM is not unique and is used at other veterinary, dental, and medical institutions.16 Further refinement of grading procedures and assessments of AVMA-COE Competencies should be carried out to resolve this discrepancy. The AVMA-COE Competency assessments at the KSUCVM are internally consistent, as demonstrated by a mean r ¼ .796 correlation among each of the core competencies (Table 5), but are only moderately correlated with performance on the NAVLE (mean r ¼ .319). Performance on the NAVLE is better predicted by the fourth-year clinical grades (r ¼ .643) than the AVMA-COE Competencies. The predictive advantage of fourth-year grades over the AVMA-COE Competency scores indicates either that the current AVMA-COE Competency scoring system is less effective than traditional grading schema regarding prediction of NAVLE performance and thus an improved scoring system for the Competencies is needed, that the Competencies themselves need to be reassessed, or that both the scoring system and the competencies need to be improved. Standard deviation of the assigned raw COE scores ranged from .204 to .326 (median ¼ .264, mean ¼ .240), which was less variable, and therefore less discriminatory, than the standard deviation of the fourthyear GPA (.366). This study does not address the question of whether the primary purpose of the AVMA-COE Competencies assessment is to serve as a mechanism for curriculum adjustment and better preparation of students for the NAVLE assessment, or if the AVMA-COE skills themselves are important education outcomes that need separate assessment. If the reason for mandating the AVMA-COE Competencies assessments was originally as a guide to revising the curriculum for better NAVLE preparation, then the AVMA-COE assessments as currently implemented are not ideal. Certainly the AVMA-COE Competencies are subjective clinical assessments, and we have shown that they likely do not measure the same skills as the more objectively oriented NAVLE. Alternatively, if the AVMA-COE Competencies are important outcomes in and of themselves as they were intended to be by the AVMA-COE,20 then the current NAVLE is not assessing them well. A determination of which outcome is more important (AVMA-COE Competencies or NAVLE) to complete the AVMA-COE missions to improve veterinary medical education and ensure public interests, will ultimately help determine whether the Competencies or the NAVLE needs revision. Unfortunately, the published evidence for designing reliable and valid assessment methods to assess competence is scant17 although detailed suggestions for assessment have been suggested.18,19 National emphasis on competency assessment to ensure public safety as a responsibility of institutions is only increasing, as evidenced by the most recent report on veterinary education, the ‘‘Roadmap for Veterinary Medical Education in the 21st Century,’’ which listed ‘‘Ensure that admissions, curricula, accreditation, and testing/licensure are competency driven,’’ as its second goal.21(p.322)
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
Journal of Veterinary Medical Education 2014.41:19-26.
Many previous reports, as well as the current study, have biases inherent in the methodology. Students who completely failed to progress for either academic or personal reasons were not included in the analysis of the majority of these studies, possibly allowing a form of nonresponse bias to insinuate itself into the results. Exclusion of this data raises the mean and decreases the standard deviation of the resulting population (eliminating the ‘‘deficient’’ outliers), but most published reports do not address or speculate on this bias. The current study did include 11 students who took longer to progress than the normal four-year professional program but did not include students who did not ultimately graduate from veterinary school. The low number of these students in the present study is unlikely to have had significant influence on the overall outcomes and associations among variables. Academic difficulty during veterinary school has been associated with a low prerequisite GPA, a low GRE score, and older age,22 but any relationship between students who have academic difficulty and individual performances on AVMA-COE Competencies has yet to be determined. The ‘‘gold standard’’ suggested by these authors to define success and assess the predictive value of the various measures is NAVLE performance. The danger of relying solely on the NAVLE as a measure of success is one of validity, of whether the full range of NAVLE scores are valid predictors of success as a graduate veterinarian. Increasingly, the NAVLE score is the primary discriminator affecting veterinary licensure, and the NAVLE pass rate is subsequently used to assess veterinary programs for accreditation purposes. However, in actual fact, only the NAVLE pass point is relevant to licensure and thus is a gateway to continued professional activity. AVMA-COE Competencies are internally consistent but poorly related to performance measures within veterinary school and only moderately related to NAVLE scores. The AVMA-COE Competencies may, in fact, be more valid as a measure of post-DVM success as they were intended; however, there is little current evidence to support that hypothesis, and the conflict of whether the AVMA-COE Competencies or the NAVLE are better predictors of graduate success may exist until either the licensing examination is changed to better reflect AVMA-COE Competencies or the AVMA-COE Competencies are better refined as an aid to improve NAVLE performance. If professional success is indeed related to NAVLE performance, and NAVLE scores are strongly related to overall academic success as demonstrated here, then class rank and cumulative grades are also good indicators of professional success and are demonstratively better predictors of future professional success than the professional competencies currently mandated for assessment.
2
3 4
5
6
7
8 9
10
11
12
13
14
15
16
NOTE a
one45 Medical School Management Software. Vancouver, BC: one45.
17
REFERENCES 1
Fuentealba C, Hecker KG, Nelson PD, et al. Relationships between admissions requirements and pre-clinical and clinical performance in a distributed
doi: 10.3138/jvme.0613-087R1
JVME 41(1) 8 2014 AAVMC
18
veterinary curriculum. J Vet Med Educ. 2011;38(1):52–9. http://dx.doi.org/10.3138/jvme.38.1.52. Medline:21805935 Danielson JA, Wu TF, Molgaard LK, et al. Relationships among common measures of student performance and scores on the North American Veterinary Licensing Examination. J Am Vet Med Assoc. 2011;238(4):454–61. http://dx.doi.org/10.2460/javma.238.4.454. Medline:21320014 Noeth RJ. Predicting success in the study of veterinary science and medicine. J Educ Res. 1974;67:213–5. Niedzwiedz ER, Friedman BA. A comparative analysis of the validity of pre-admissions information at four colleges of veterinary medicine. J Vet Med Educ. 1976;3:32–7. Render GR, Jackson HD. Preveterinary performance, admissions criteria and personality variables as predictors of success in veterinary school. J Vet Med Educ. 1975;2:3–6. Julius MF, Kaiser HE. Pre-veterinary medical grade point averages as predictors of academic success in veterinary college. J Vet Med Educ. 1978;5:162–4. Cawunder P, Tasker JB. Use of the new medical college admissions test (MCAT) in veterinary school admissions. J Vet Med Educ. 1982;9:1–3. Kelman EG. Predicting success in veterinary medical college. J Vet Med Educ. 1982;8:92–4. Kearney MT, Shane SM, Tasker JB. The value of the medical college admissions test for selection of applicants to a veterinary school. J Vet Med Educ. 1985;11:54–62. Confer AW. Preadmission GRE scores and GPAs as predictors of academic performance in a college of veterinary medicine. J Vet Med Educ. 1990;17:56–62. Zachary JF, Schaeffer DJ. Correlations between preveterinary admissions variables and academic success in core courses during the first two years of the veterinary curriculum. J Vet Med Educ. 1994;21:56–60. Shane SM, Kearney MT, Thayer RW. Analysis of national board examination scores with reference to achievement during the professional veterinary curriculum. J Vet Med Educ. 1985;11:50–4. Lafi SQ, Kaneene JB, Tasker JB. Predicting Medical College Admission Test (MCAT) scores using Graduate Record Examination (GRE) scores. J Vet Med Educ. 1990;17:38–43. Shane SM, Harrington KS. Survey of standard tests as criteria for admission to North American veterinary colleges. J Vet Med Educ. 1989;16:45–6. Latshaw WK. The use of multivariate analysis to analyze the admissions process. J Vet Med Educ. 1982;9:6–9. Hardie EM. Current methods in use for assessing clinical competencies: what works? J Vet Med Educ. 2008;35(3):359–68. http://dx.doi.org/10.3138/ jvme.35.3.359. Medline:19066352 Rhind SM, Baillie S, Brown F, et al. Assessing competence in veterinary medical education: where’s the evidence? J Vet Med Educ. 2008;35(3):407–11. http:// dx.doi.org/10.3138/jvme.35.3.407. Medline:19066358 Turnwald G, Stone E, Bristol D, et al. Assessing clinical competency: reports from discussion groups. J Vet Med
25
19
20
21
Journal of Veterinary Medical Education 2014.41:19-26.
22
26
Educ. 2008;35(3):343–53. http://dx.doi.org/10.3138/ jvme.35.3.343. Medline:19066350 Lloyd JW, King LJ, Klausner JS, et al. National workshop on core competencies for success in the veterinary profession. J Vet Med Educ. 2003;30(3):280–4. http://dx.doi.org/10.3138/jvme.30.3.280. Medline:14648504 AVMA. Accreditation policies and procedures of the AVMA Council on Education. Schaumburg, IL: American Veterinary Medical Association; April 2012 [updated 2012 Oct; cited 2013 Jun 19]. Available from: https://www.avma.org/ProfessionalDevelopment/ Education/Accreditation/Colleges/Pages/coe-pp.aspx. Schurig G, Osburn BI. The North American Veterinary Medical Education Consortium (NAVMEC) looks to veterinary medical education for the future: ‘‘Roadmap for Veterinary Medical Education in the 21st Century: Responsive, Collaborative, Flexible’’. J Vet Med Educ. 2011;38(4):320–6. http://dx.doi.org/10.3138/ jvme.38.4.320. Rush BR, Sanderson MW, Elmore RG. Pre-matriculation indicators of academic difficulty during veterinary school. J Vet Med Educ. 2005;32(4):517–22. http:// dx.doi.org/10.3138/jvme.32.4.517. Medline:16421838
AUTHOR INFORMATION James K. Roush, DVM, MS, Dipl ACVS, is the Doughman Professor of Small Animal Surgery, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected]. Bonnie R. Rush, DVM, MS, Dipl ACVIM, is Head of the Department of Clinical Sciences and Professor of Equine Internal Medicine, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected]. Brad J. White, DVM, MS, is Associate Professor of Beef Production and Management, Department of Clinical Sciences and Department of Anatomy and Physiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected]. Melinda J. Wilkerson, MS, DVM, PhD, Dipl ACVP, is Professor of Immunology, Department of Diagnostic Medicine and Pathobiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506 USA. E-mail: [email protected].
JVME 41(1) 8 2014 AAVMC
doi: 10.3138/jvme.0613-087R1
