EDITORIAL
Reliability of the Oral Certifying Examination for Neurologists: Comparison of Test Results Using Live Patient Interviews and Audiovisual Techniques Patrick F. Bray, M D
Each of the twenty-two specialty boards approved by the American Board of Medical Specialties and the American Medical Association evaluates the qualifications of candidates in its field who appear voluntarily for examination, and certifies as diplomates those who are considered competent. To accomplish this function, specialty boards determine whether candidates have received adequate preparation in accordance with established educational standards. They provide comprehensive examinations to evaluate the ability of candidates, and they certify as physicians qualified to practice within a specialty those who have satisfied the requirements. Prior to October, 1976, the examination of the American Board of Psychiatry and Neurology included two hour-long sessions with patients. For reasons to be cited in the Discussion, the Board began at that time to substitute one hour of television-taped patient examination for one of the two Part I1 oral examination hours in the candidate’s major field. To determine the degree of concordance or nonconcordance between the different formats of oral examination and to ascertain whether one of the two methods of testing in clinical neurology is a better discriminator in identifying a candidate’s competence o r lack of competence, the pass-fail-conditional grades achieved with these two test methods were compared for 816 candidates at seven consecutive sessions over a two-year period.
Method of Study The oral examination in neurology lasts four hours. Starting in October, 1976, each of the four hours was divided into 30-minute patient evaluations followed by 30-minute interviews with two or three examiners. The 30-minute interview was divided into a 10o r 15-minute discussion of the patient just examined
or viewed on tape, followed by a 15-minute discussion of common, standard clinical problems posed by the examiner, in which emphasis was placed on the clinical approach to diagnosis, treatment, management, and counseling. In the major specialty (adult o r child neurology), the candidate interviewed and examined a patient live and viewed an audiovisual taped interview and examination; two live patient hours in the minor field (psychiatry and adult or child neurology) followed. Concordance in the major specialty (adult or child neurology) was defined as a passing grade in both hours, failing grade in both hours, conditional grade in both hours, or failing in one hour and conditional grade in one hour. Nonconcordance was defined as passing one hour and failing one hour of the major specialty or a passing grade one hour and conditional grade the other hour. The last group deserves special mention. The Board concedes that oral examinations are not infallible, and by convention a special review of examiners’ grading cards is done and the candidate’s performance in other sections is considered. After careful review by all directors, the conditional grade is either raised to pass or held at conditional. The nonconcordant grades might be particularly pertinent in deciding whether the live or the TV-taped patient hour is a better discriminator of the performance of a candidate who might be judged less than competent in his major specialty. Because the two experiences differ in their nature, careful inspection of the evaluation forms and the examiners’ grading cards was carried out by three directors of the Board to determine whether a consistent trend appeared in different phases of the candidate’s evaluations, particularly among candidates who failed either hour. Attention was also directed to whether any correlation could be detected in the inci-
From the Department of Neurology, University of Utah College of Medicine, Salt Lake City, UT.
Address reprint requests to Dr Bray, University of Utah Medical Center, 50 N Medical Dr, Salt Lake City, UT 84132.
0~64-513~/79/080091-03$01.25 @ 1978 by Patrick L. Bray 91
dence of failures recorded by examining teams headed by particular directors.
Results The frequency distribution of pass-fail-conditional grades is shown in the Figure. A high degree of concordance was demonstrated in the examination of 541 candidates who passed or failed in seven consecutive examinations over a two-year examination cycle. The degree of concordance for passing or failing was 88.3%. The correlation coefficient (phi) for concordance between the two different formats was highly significant (phi = 0.636;p < 0.001) [l]. The actual numbers of nonconcordant grades for those who passed one hour and failed the other are also indicated in the Figure. One can see that of the 275 who had nonconcordant grades, 30 candidates received a failing grade for the TV-taped patient hour and passed the live patient hour, whereas 42 candidates received a failing grade on the live patient test and passed the taped patient hour. A chi-square test showed that this difference in pass-fail grades was not significant k2= 2.0; p > 0.10). When a comparison is made between the number of candidates who passed the live patient examination and received a conditional grade on the TV-taped examination with those who passed the T V examination and had a conditional result on the live examination, the results are similar: 105 in the former group and 98 in the latter. As indicated in the Figure, if one considers the two nonconcordant groups cumulatively, the numbers are even closer: 135 compared to 140. Inspection of the evaluation forms for the two hours of examination has revealed no consistently discernible reason for failure in either the live or taped patient hours. Inspection of pass-fail rates among different directors and their teams of examiners showed no consistent trends. Discussion The decision made in 1975 by the American Board of Psychiatry and Neurology to change the oral examination format was arbitrary but grew out of several practical and idealistic concerns. N o t the least pressing were the increasing cost and cumbersome nature of an oral examination using patients. A growing number of medical institutions were unable or unwilling to assume responsibility for an undertaking that had come t o involve large amounts of money and energy-money and energy that had to come from candidates, institutions, and examiners. In addition, the directors of the Board continually ask themselves about the reliability and validity of the certification process and are heatedly questioned on these points by candidates and professionals in the field. Cognizance was taken of the fact that ours is the
92
Annals of Neurology
Vol 6 No 2
August 1979
Pass (TV)
Fail
Conditional
(TV)
(TV)
Distribution of pass-fail-conditional grades achieved by 816 candidates examined for one hour with a live patient interview and for one hour with a TV-tapedpatient interview. The grades for the groups of candidates indicated by an asterisk were concordant on the two types of examination, and the value was highly significant (p < 0.001). The grades for the groups marked with a dagger were nonconcordant. Note that the total number of candidates passing the live portion and receiving failing or conditionalgrades on the T V examination was 135, compared with 140 candidates who passed the televised portion and received failing or conditional grades on the live interview. The difference between these two groups is obviously insignif cant.
only specialty board continuing to use patient interviews in the certification process. In the absence of previous documentation of the examination's reliability, it comes as no surprise that both successful and unsuccessful candidates, as well as the professional organizations which sponsor the specialty board, question whether the examination is comprehensive or capricious, reliable or inconsistent. Validity of c o n t e n t a n even knottier question that is continually raised about all examinations, including those which attest to professional medical competence-is not dealt with in this study. Individual neurologists have generally viewed the wisdom of eliminating half of the live patient time with reluctance, doubt, and disdain. These attitudes are easy to understand at first glance if one compares the two types of test experience. When the candidate is told to interview and examine a randomly assigned patient, he is observed closely by two or three examiners who are attempting to measure his ability to extract the essentials of the history and to carry out a formal neurological examination under close scrutiny in a short period of time, sometimes under difficult conditions. Contrast this active, emotionally charged experience with a passive one in which a small group of candidates views a similar performance by an ex-
perienced neurologist on a television screen over the same period. Following the half-hour of patient contact, the 30-minute interview portion of the oral examination is identical for both types of patient exposure. Despite these prevalent doubts, the data speak for themselves. Using the two oral examination techniques, examiners essentially agree 90% of the time about whether the candidate should pass or fail. In 10% of cases they do not concur, and this small difference proves to be statistically insignificant. What is most reassuring about the results is the almost 50-50 split among the numbers of candidates for whom the two different techniques of examination led to nonconcordant views of pass, fail, or conditional grades (see Figure legend). What can one conclude about such an oral certification process? That whether one uses the live or the taped interview technique makes no significant difference in the outcome, whether pass or fail, in a large sample of candidates. One can also conclude that regardless of which examination technique is used, experienced examiners agree 9 out of 10 times about whether a candidate should pass or fail. The results of this study support the reliability of the oral certifying examination in neurology regardless of the nature of the patient encounter. At a time when demands for training in both undergraduate and graduate medical education have greatly increased, those institutions which are properly equipped should be encouraged by the experience reported here to make television cassettes of important case material. The tapes can be pooled in a library where trainees can view them at their leisure. Cases of different
disorders-live examples of which may not be available at any given t i m e - c a n be used alone by the trainee, as the focus of group discussion with a faculty member, or as a technique to evaluate the trainee. Whether either technique is a valid measure of clinical competence is an issue not covered in this study and certainly not answered by it. Different techniques to assess the validity of the oral examination are used by other certifying specialties to get at this question, including detailed reports from training programs and their directors, or chart audits. O n the basis of this study one could and should raise the question whether the decision to pass or fail a candidate has much to do with the candidate’s 30minute exposure to a patient, whether live or taped. It may be that the pass-fail decision is made in the 30 minutes of discussion following patient exposure. If this is the case, as seems possible, one must question whether the oral examination is measuring an element of competence already covered in the contentand information-oriented Part I written examination. Studies are being planned to answer some of these questions. ~
~
~~
The data used in this study were generously provided by the staff of the American Board of Psychiatry and Neurology, and helpful criticisms were offered by all Directors of the Board. We thank Robert E. Dustman, PhD, of the Department of Neurology, Division of Neuropsychology, University of Utah College of Medicine, for help with the statistical analysis.
Reference 1. Guilford JP: Fundamental Statistics in Psychology and Education. Second edition. New York, McGraw-Hill, 1950
Editorial: Bray: Oral Examination for Neurologists
93
Reliability of the Oral Certifying Examination for Neurologists: Comparison of Test Results Using Live Patient Interviews and Audiovisual Techniques Patrick F. Bray, M D
Each of the twenty-two specialty boards approved by the American Board of Medical Specialties and the American Medical Association evaluates the qualifications of candidates in its field who appear voluntarily for examination, and certifies as diplomates those who are considered competent. To accomplish this function, specialty boards determine whether candidates have received adequate preparation in accordance with established educational standards. They provide comprehensive examinations to evaluate the ability of candidates, and they certify as physicians qualified to practice within a specialty those who have satisfied the requirements. Prior to October, 1976, the examination of the American Board of Psychiatry and Neurology included two hour-long sessions with patients. For reasons to be cited in the Discussion, the Board began at that time to substitute one hour of television-taped patient examination for one of the two Part I1 oral examination hours in the candidate’s major field. To determine the degree of concordance or nonconcordance between the different formats of oral examination and to ascertain whether one of the two methods of testing in clinical neurology is a better discriminator in identifying a candidate’s competence o r lack of competence, the pass-fail-conditional grades achieved with these two test methods were compared for 816 candidates at seven consecutive sessions over a two-year period.
Method of Study The oral examination in neurology lasts four hours. Starting in October, 1976, each of the four hours was divided into 30-minute patient evaluations followed by 30-minute interviews with two or three examiners. The 30-minute interview was divided into a 10o r 15-minute discussion of the patient just examined
or viewed on tape, followed by a 15-minute discussion of common, standard clinical problems posed by the examiner, in which emphasis was placed on the clinical approach to diagnosis, treatment, management, and counseling. In the major specialty (adult o r child neurology), the candidate interviewed and examined a patient live and viewed an audiovisual taped interview and examination; two live patient hours in the minor field (psychiatry and adult or child neurology) followed. Concordance in the major specialty (adult or child neurology) was defined as a passing grade in both hours, failing grade in both hours, conditional grade in both hours, or failing in one hour and conditional grade in one hour. Nonconcordance was defined as passing one hour and failing one hour of the major specialty or a passing grade one hour and conditional grade the other hour. The last group deserves special mention. The Board concedes that oral examinations are not infallible, and by convention a special review of examiners’ grading cards is done and the candidate’s performance in other sections is considered. After careful review by all directors, the conditional grade is either raised to pass or held at conditional. The nonconcordant grades might be particularly pertinent in deciding whether the live or the TV-taped patient hour is a better discriminator of the performance of a candidate who might be judged less than competent in his major specialty. Because the two experiences differ in their nature, careful inspection of the evaluation forms and the examiners’ grading cards was carried out by three directors of the Board to determine whether a consistent trend appeared in different phases of the candidate’s evaluations, particularly among candidates who failed either hour. Attention was also directed to whether any correlation could be detected in the inci-
From the Department of Neurology, University of Utah College of Medicine, Salt Lake City, UT.
Address reprint requests to Dr Bray, University of Utah Medical Center, 50 N Medical Dr, Salt Lake City, UT 84132.
0~64-513~/79/080091-03$01.25 @ 1978 by Patrick L. Bray 91
dence of failures recorded by examining teams headed by particular directors.
Results The frequency distribution of pass-fail-conditional grades is shown in the Figure. A high degree of concordance was demonstrated in the examination of 541 candidates who passed or failed in seven consecutive examinations over a two-year examination cycle. The degree of concordance for passing or failing was 88.3%. The correlation coefficient (phi) for concordance between the two different formats was highly significant (phi = 0.636;p < 0.001) [l]. The actual numbers of nonconcordant grades for those who passed one hour and failed the other are also indicated in the Figure. One can see that of the 275 who had nonconcordant grades, 30 candidates received a failing grade for the TV-taped patient hour and passed the live patient hour, whereas 42 candidates received a failing grade on the live patient test and passed the taped patient hour. A chi-square test showed that this difference in pass-fail grades was not significant k2= 2.0; p > 0.10). When a comparison is made between the number of candidates who passed the live patient examination and received a conditional grade on the TV-taped examination with those who passed the T V examination and had a conditional result on the live examination, the results are similar: 105 in the former group and 98 in the latter. As indicated in the Figure, if one considers the two nonconcordant groups cumulatively, the numbers are even closer: 135 compared to 140. Inspection of the evaluation forms for the two hours of examination has revealed no consistently discernible reason for failure in either the live or taped patient hours. Inspection of pass-fail rates among different directors and their teams of examiners showed no consistent trends. Discussion The decision made in 1975 by the American Board of Psychiatry and Neurology to change the oral examination format was arbitrary but grew out of several practical and idealistic concerns. N o t the least pressing were the increasing cost and cumbersome nature of an oral examination using patients. A growing number of medical institutions were unable or unwilling to assume responsibility for an undertaking that had come t o involve large amounts of money and energy-money and energy that had to come from candidates, institutions, and examiners. In addition, the directors of the Board continually ask themselves about the reliability and validity of the certification process and are heatedly questioned on these points by candidates and professionals in the field. Cognizance was taken of the fact that ours is the
92
Annals of Neurology
Vol 6 No 2
August 1979
Pass (TV)
Fail
Conditional
(TV)
(TV)
Distribution of pass-fail-conditional grades achieved by 816 candidates examined for one hour with a live patient interview and for one hour with a TV-tapedpatient interview. The grades for the groups of candidates indicated by an asterisk were concordant on the two types of examination, and the value was highly significant (p < 0.001). The grades for the groups marked with a dagger were nonconcordant. Note that the total number of candidates passing the live portion and receiving failing or conditionalgrades on the T V examination was 135, compared with 140 candidates who passed the televised portion and received failing or conditional grades on the live interview. The difference between these two groups is obviously insignif cant.
only specialty board continuing to use patient interviews in the certification process. In the absence of previous documentation of the examination's reliability, it comes as no surprise that both successful and unsuccessful candidates, as well as the professional organizations which sponsor the specialty board, question whether the examination is comprehensive or capricious, reliable or inconsistent. Validity of c o n t e n t a n even knottier question that is continually raised about all examinations, including those which attest to professional medical competence-is not dealt with in this study. Individual neurologists have generally viewed the wisdom of eliminating half of the live patient time with reluctance, doubt, and disdain. These attitudes are easy to understand at first glance if one compares the two types of test experience. When the candidate is told to interview and examine a randomly assigned patient, he is observed closely by two or three examiners who are attempting to measure his ability to extract the essentials of the history and to carry out a formal neurological examination under close scrutiny in a short period of time, sometimes under difficult conditions. Contrast this active, emotionally charged experience with a passive one in which a small group of candidates views a similar performance by an ex-
perienced neurologist on a television screen over the same period. Following the half-hour of patient contact, the 30-minute interview portion of the oral examination is identical for both types of patient exposure. Despite these prevalent doubts, the data speak for themselves. Using the two oral examination techniques, examiners essentially agree 90% of the time about whether the candidate should pass or fail. In 10% of cases they do not concur, and this small difference proves to be statistically insignificant. What is most reassuring about the results is the almost 50-50 split among the numbers of candidates for whom the two different techniques of examination led to nonconcordant views of pass, fail, or conditional grades (see Figure legend). What can one conclude about such an oral certification process? That whether one uses the live or the taped interview technique makes no significant difference in the outcome, whether pass or fail, in a large sample of candidates. One can also conclude that regardless of which examination technique is used, experienced examiners agree 9 out of 10 times about whether a candidate should pass or fail. The results of this study support the reliability of the oral certifying examination in neurology regardless of the nature of the patient encounter. At a time when demands for training in both undergraduate and graduate medical education have greatly increased, those institutions which are properly equipped should be encouraged by the experience reported here to make television cassettes of important case material. The tapes can be pooled in a library where trainees can view them at their leisure. Cases of different
disorders-live examples of which may not be available at any given t i m e - c a n be used alone by the trainee, as the focus of group discussion with a faculty member, or as a technique to evaluate the trainee. Whether either technique is a valid measure of clinical competence is an issue not covered in this study and certainly not answered by it. Different techniques to assess the validity of the oral examination are used by other certifying specialties to get at this question, including detailed reports from training programs and their directors, or chart audits. O n the basis of this study one could and should raise the question whether the decision to pass or fail a candidate has much to do with the candidate’s 30minute exposure to a patient, whether live or taped. It may be that the pass-fail decision is made in the 30 minutes of discussion following patient exposure. If this is the case, as seems possible, one must question whether the oral examination is measuring an element of competence already covered in the contentand information-oriented Part I written examination. Studies are being planned to answer some of these questions. ~
~
~~
The data used in this study were generously provided by the staff of the American Board of Psychiatry and Neurology, and helpful criticisms were offered by all Directors of the Board. We thank Robert E. Dustman, PhD, of the Department of Neurology, Division of Neuropsychology, University of Utah College of Medicine, for help with the statistical analysis.
Reference 1. Guilford JP: Fundamental Statistics in Psychology and Education. Second edition. New York, McGraw-Hill, 1950
Editorial: Bray: Oral Examination for Neurologists
93
