In response to inquiries about the development and statistical characteristic8 of the 1974 Certifying Examination of The American Bard of Allergy and Immunology (a Conjoint Board of the American Board of Internal bone and the American Board of Pediatrics), the following material is presented by the Board.
In selecting the methods the Board would use for determining qualifications for certification, The American Board of Allergy and Immunology reviewed how other medical specialty boards were presently evaluating individual8 within their disciplines. In general, it was found that two method8 were used to evaluate candidates for certification. Knowledge, including problem-solving skills, was bat&&~ assessed through the use of standardized written examinations wd through the use of oral examinations. The adequacy of a candidate’s day-to-day performance was generally appraised through the use of letter8 of recommendation or standardized report forms submitted by residency program directors. All specialty Boards employ consultants expert in the construction of examinations to assist in the preparation of their written examinations. Most of the major medical specialty groups tiork with the’ assistance of the National Board of Medical Examiners. In general, examinations are of the multiple-choice question variety. These examination8 are regarded by the National Board of Medical Examiners as being appropriate for assessing the knowledge and certain problem-solving abilities of the examinees. Thus the Board decided to use a written, multiple-choice examination in its evaluation program. The Board did not attempt to develop any standardized performance assessment instrument8 for use by program directors because The American Board of Allergy and Immunology ( ABAI) , in its first examination, expeoted that many *A Conjoint Board of the American Board of Internal Medicine and the American Board of Pediatrics. **Correspondent: Herbert C. Mansmann, Jr., M.D., Executive Secretary, 3930 Chestnut St., Philadelphia, Pa. 19104.
Vol. 55, No..%', pp. 140-144
Report
VOLUME 55 NUMBER 2
TABLE I. Number
of months
of formal
allergy
on 1974 Certifying
and
immunology
No. of candidates
O-12 months of formal training 13-23 months of formal training More than 24 months of formal Total
TABLE
II. Content
outline
by participants % of total
488 147 293 928
training
and approximate
training
emphasis
141
Examination
group
52 16 32
on topics Weight in % of questions
I.
II.
Basic sciences relevant to allergy and immunology A. Immunology, e.g., T and B cell functions, development of the immune response, immunoglobulin structure and function, antigens, mechanisms of hypersensitivity reactions, tumor and transplant immunology B. Pharmacophysiology, e.g., mediators, complement, lymphokinens, autonomic nervous system, pulmonary physiology C. Aerobiology, e.g., pollens, fungi, pollutants Clinical information relevant to allergy and immunology A. Disease states 1. Asthma and differential diagnosis from other pulmonary diseases; rhinitis and sinusitis; urticaria and angioedema ; anaphylaxis; atopic eczema and its differential diagnosis from other skin diseases; adverse reactions to drugs, serum, insulin, or insect stings 2. Other disorders characterized by immunologic features a. Immune deficiency states and graft-versus-host disease b. Granulomatous disorders, e.g., sarcoid, Hodgkin’s disease c. Hypersensitivity pneumonitis and related disorders d. Collagen vascular disease e. Immunohematologic disease f. Autoallergic diseases affecting specific organ systems g. Infectious diseases B. Diagnosis/Therapy 1. Diagnostic procedures a. Used in differential diagnosis of allergic disease, e.g., skin tests, provocative challenge tests, x-rays, pulmonary function studies, etc. b. Immunologic procedures! e.g., quantitative immunoglobulin and antibody measurement, in vitro tests of cell-mediated immunity, hormonal and other radioimmunoassays, precepitin tests, etc. 2. Therapeutic principles a. Immunotherapy b. Avoidance c. Drugs
30% 15% 10% 5% 70% 40%
15%
15%
of those who applied would not have any formal allergy residency training (Table I) and because, at present, the state of the art for evaluation of physician performance is relatively new.l In the future, the Board expects to develop such performance assessment methods for use by program directors. DEVELOPMENT OF THE EXAMINATION The ABA1 appointed a lo-person examination panel to construct the first examination. This panel met in March of 1972 with the professional staff of the National Board of Medical Examiners to plan .the first examination. The panel’s first task was to develop a content outline for the examination that validly represented the subject matter of the fields of allergy and immu-
141
The American
Board
TAME
III. Description
of Allergy
and distribution
and
of item types
Dosdpt4on
1
X types (multiple true-false) A types (one best-response multiple choice) R types (matching) Total No of items in examination X type
J. ALLERGY CLIN. IhbMuMo~, Fi!s&lAw 1975
immunology
MsPttbgllon 116 72 79 267
Sample answer-sheet
Example 1. Serum electrolytes (A) Sodium (B ) Potassium (C) Albumin (D) Chloride (E) Globulin
include :
Alternatives A, B, and D are correct, and the corresponding circles in the Fe8 column are filled rn with a heavy black pencil mark. Alternatrves C and E are incorrect, and the corresponding circles in the No column are marked. A type
R type
Example 2. To which of the following does the heart belong? (A) Digestive (B) Circulatory (C) Central nervous ( D) Endocrine (E) Musculoskeletal
Sample answer systems
of the body
Example 3-6. For each physical finding Rated below, select the most likely associated malfunctioning organ systern : (A) Cardiovascular (B) Central nervous (C) Hepatic (D) Musculoskeletal (E) Respiratory 3. 4. 5. 6.
sheet
2@,cQ@O
Sample answer
sheet
Pulmonary wheezes Varicose veins Jaundice Hot, painful joints
nology. An outline was prepared that was broken down into three baaie &W (I, Basic Science ; II, Disease States ; III, Die/Therapy). tih 0% major areas was sppropria&ely aubdivi&d into mare spwi& sub-arm, E&h of these sub-areas was then weighted in terms of the. pereentwe of q~~s,~ %& area that would be included in the examination (Table II). Using the outline as a guide, a 267-quest&% eza&nation ww ev~~a~~.~ veloped. The questions created WBIX of three tm (Table III). Appr&m per cent of the questions required the candidate to salve prob&z+s ~6% material (e.g., x-rays, physical signs). Once developed, the examtition was admi&%red to a grow sf
VOLUME 55 NUMBER 2
Report
TABLE IV. DiFiculty,
discrimination,
Subtort
Section I (X type) Section II (A and B types) Composite standard score (Section I + Section II)
and
on 1974 Certifying
Examination
143
reliability
Numbor of scomblo udts
Avera$o P
AVWogO r bh
540 140
0.83 0.79
0.25 0.36
0.93 0.87
-
-
0.95
Reliability
allergists to check for ambiguous items and to be certain that the &hour period allocated for the examination was sufficient for the examination. ADMINISTRATION
AND
ANALYSIS
OF THE EXAMINATION
On March 1, 1974, the examination was administered to 928 candidates for certification. Following the administration of the examination, the psychometrics staff of the National Board of Medical Examiners analyzed the performance of both the candidates and the examination. Three characteristics of the examination were carefully assessed(Table IV). A. DitTiculty
(P value)
This statistic (P) shows the percentage of a reference group* that answered an individual question correctly. The more difficult the question, the lower its P value will be. For a group of questions (or for the entire examination) an average of these P values also may be interpreted as the percentage of questions that were answered correctly by the average individual in the reference group. Thus the average P value serves as an index of difficulty for an examination as a whole and also as a ,reflection of the performance of a typical member of the reference group in that examination. B. Discrimination
(’ bid
For an individual question this statistic (r bis) indicates the extent to which the question is able to distinguish between the best candidates as identified by performance on the entire examination and the poorest candidates as identified by performance on the entire examination. A perfect r bis is 1.00, which would occur if a question was answered correctly by all of the best candidates and incorrectly by all of the poorest candidates. A question with an r bis greater than 0.20 is usually considered acceptable by the National Board of Medical Examiners, and an average * bis greater than 0.25 for an examination at the specialty board level is considered acceptable by the National Board of Medical Examiners. C. Reliability
IKR-201
The reliability coefficient is an index of the reproducibility of scores from a given examination. It is an estimate of the degree to which a group of candidates “The reference group for the ABA1 examination consisted of individuals certified by the American Board of Internal Medicine or the American Board of Pediatrics who had cornpleted two years of fellowship in Allergy and Immunology.
144
The American
Board
of Allergy
and Immunology
J. ALLERGY CLIN. IMMJNOL. FEBRWA-RY1975
would have the same rank order if they were to take a similar examination a second time without further preparation. The National Board of Medical Examiners holds that a reliability coefficient of greater than 0.90 for a total examination is necessary to feel confident in making pass-fail decisions about individual examinees.2 coefficient for Table IV shows the average I’, average r bis, and reliability the Certifying Examination and for the two major sections of that test. In the judgment of the National Board, the t.est was highly reliable, discriminated well, and was pitched at an appropriate level of difficulty.
In deciding what standard to set for passing the examination, the Board considered the past experience of the antecedent sub-boards and the experiences and policies of other specialty boards. The ABA1 sought advice from two sources, its two parent boards (the American Board of Internal Medicine and the American Board of Pediatrics) and the National Board of Medical Examiners. Before the examination was administered, a specially appointed Standards Committee* agreed upon the procedure by which the standard for passing the test would be set. Having first defined the reference group, this committee felt that the mean performance of this group would be an acceptable criterion for certification. However, the Committee also recognized that the testing instrument would not provide perfectly reliable measurements and, therefore, if the mean score of the reference group were chosen as the standard for passing the test, some examinees whose “true scores” (scores that would be obtained from a perfectly reliable instrument) were at or above the criterion might fail. Thus, to be virtually certain that no examinee who failed the test had a true score at or above the criterion, the committee recommended that the standard for passing the test be set 1.5 standard deviation units below the mean of the reference group, and this recommendation was adopted by the Board. When this standard was applied, approximately 90 per cent of the reference group and 68 per cent of all examinees passed the examination. *The Standards Committee of the ABA1 consisted of two members from ABAI, two members from the American Board of Internal Medicine, and two wmxbers from the American Bo5d of Pediatrics. 1 Barro, A. R.: Survey and evaluation of approaches to physician performance measurement, a supplement to the Journal of Medical EdRe@iog, vol. 48, November 19?3, p. 1047. 2 Hubbard, J. P.: Measuring medical education, Philadelphia, 1971, Lea $ Febiger, p. 57.
In selecting the methods the Board would use for determining qualifications for certification, The American Board of Allergy and Immunology reviewed how other medical specialty boards were presently evaluating individual8 within their disciplines. In general, it was found that two method8 were used to evaluate candidates for certification. Knowledge, including problem-solving skills, was bat&&~ assessed through the use of standardized written examinations wd through the use of oral examinations. The adequacy of a candidate’s day-to-day performance was generally appraised through the use of letter8 of recommendation or standardized report forms submitted by residency program directors. All specialty Boards employ consultants expert in the construction of examinations to assist in the preparation of their written examinations. Most of the major medical specialty groups tiork with the’ assistance of the National Board of Medical Examiners. In general, examinations are of the multiple-choice question variety. These examination8 are regarded by the National Board of Medical Examiners as being appropriate for assessing the knowledge and certain problem-solving abilities of the examinees. Thus the Board decided to use a written, multiple-choice examination in its evaluation program. The Board did not attempt to develop any standardized performance assessment instrument8 for use by program directors because The American Board of Allergy and Immunology ( ABAI) , in its first examination, expeoted that many *A Conjoint Board of the American Board of Internal Medicine and the American Board of Pediatrics. **Correspondent: Herbert C. Mansmann, Jr., M.D., Executive Secretary, 3930 Chestnut St., Philadelphia, Pa. 19104.
Vol. 55, No..%', pp. 140-144
Report
VOLUME 55 NUMBER 2
TABLE I. Number
of months
of formal
allergy
on 1974 Certifying
and
immunology
No. of candidates
O-12 months of formal training 13-23 months of formal training More than 24 months of formal Total
TABLE
II. Content
outline
by participants % of total
488 147 293 928
training
and approximate
training
emphasis
141
Examination
group
52 16 32
on topics Weight in % of questions
I.
II.
Basic sciences relevant to allergy and immunology A. Immunology, e.g., T and B cell functions, development of the immune response, immunoglobulin structure and function, antigens, mechanisms of hypersensitivity reactions, tumor and transplant immunology B. Pharmacophysiology, e.g., mediators, complement, lymphokinens, autonomic nervous system, pulmonary physiology C. Aerobiology, e.g., pollens, fungi, pollutants Clinical information relevant to allergy and immunology A. Disease states 1. Asthma and differential diagnosis from other pulmonary diseases; rhinitis and sinusitis; urticaria and angioedema ; anaphylaxis; atopic eczema and its differential diagnosis from other skin diseases; adverse reactions to drugs, serum, insulin, or insect stings 2. Other disorders characterized by immunologic features a. Immune deficiency states and graft-versus-host disease b. Granulomatous disorders, e.g., sarcoid, Hodgkin’s disease c. Hypersensitivity pneumonitis and related disorders d. Collagen vascular disease e. Immunohematologic disease f. Autoallergic diseases affecting specific organ systems g. Infectious diseases B. Diagnosis/Therapy 1. Diagnostic procedures a. Used in differential diagnosis of allergic disease, e.g., skin tests, provocative challenge tests, x-rays, pulmonary function studies, etc. b. Immunologic procedures! e.g., quantitative immunoglobulin and antibody measurement, in vitro tests of cell-mediated immunity, hormonal and other radioimmunoassays, precepitin tests, etc. 2. Therapeutic principles a. Immunotherapy b. Avoidance c. Drugs
30% 15% 10% 5% 70% 40%
15%
15%
of those who applied would not have any formal allergy residency training (Table I) and because, at present, the state of the art for evaluation of physician performance is relatively new.l In the future, the Board expects to develop such performance assessment methods for use by program directors. DEVELOPMENT OF THE EXAMINATION The ABA1 appointed a lo-person examination panel to construct the first examination. This panel met in March of 1972 with the professional staff of the National Board of Medical Examiners to plan .the first examination. The panel’s first task was to develop a content outline for the examination that validly represented the subject matter of the fields of allergy and immu-
141
The American
Board
TAME
III. Description
of Allergy
and distribution
and
of item types
Dosdpt4on
1
X types (multiple true-false) A types (one best-response multiple choice) R types (matching) Total No of items in examination X type
J. ALLERGY CLIN. IhbMuMo~, Fi!s&lAw 1975
immunology
MsPttbgllon 116 72 79 267
Sample answer-sheet
Example 1. Serum electrolytes (A) Sodium (B ) Potassium (C) Albumin (D) Chloride (E) Globulin
include :
Alternatives A, B, and D are correct, and the corresponding circles in the Fe8 column are filled rn with a heavy black pencil mark. Alternatrves C and E are incorrect, and the corresponding circles in the No column are marked. A type
R type
Example 2. To which of the following does the heart belong? (A) Digestive (B) Circulatory (C) Central nervous ( D) Endocrine (E) Musculoskeletal
Sample answer systems
of the body
Example 3-6. For each physical finding Rated below, select the most likely associated malfunctioning organ systern : (A) Cardiovascular (B) Central nervous (C) Hepatic (D) Musculoskeletal (E) Respiratory 3. 4. 5. 6.
sheet
2@,cQ@O
Sample answer
sheet
Pulmonary wheezes Varicose veins Jaundice Hot, painful joints
nology. An outline was prepared that was broken down into three baaie &W (I, Basic Science ; II, Disease States ; III, Die/Therapy). tih 0% major areas was sppropria&ely aubdivi&d into mare spwi& sub-arm, E&h of these sub-areas was then weighted in terms of the. pereentwe of q~~s,~ %& area that would be included in the examination (Table II). Using the outline as a guide, a 267-quest&% eza&nation ww ev~~a~~.~ veloped. The questions created WBIX of three tm (Table III). Appr&m per cent of the questions required the candidate to salve prob&z+s ~6% material (e.g., x-rays, physical signs). Once developed, the examtition was admi&%red to a grow sf
VOLUME 55 NUMBER 2
Report
TABLE IV. DiFiculty,
discrimination,
Subtort
Section I (X type) Section II (A and B types) Composite standard score (Section I + Section II)
and
on 1974 Certifying
Examination
143
reliability
Numbor of scomblo udts
Avera$o P
AVWogO r bh
540 140
0.83 0.79
0.25 0.36
0.93 0.87
-
-
0.95
Reliability
allergists to check for ambiguous items and to be certain that the &hour period allocated for the examination was sufficient for the examination. ADMINISTRATION
AND
ANALYSIS
OF THE EXAMINATION
On March 1, 1974, the examination was administered to 928 candidates for certification. Following the administration of the examination, the psychometrics staff of the National Board of Medical Examiners analyzed the performance of both the candidates and the examination. Three characteristics of the examination were carefully assessed(Table IV). A. DitTiculty
(P value)
This statistic (P) shows the percentage of a reference group* that answered an individual question correctly. The more difficult the question, the lower its P value will be. For a group of questions (or for the entire examination) an average of these P values also may be interpreted as the percentage of questions that were answered correctly by the average individual in the reference group. Thus the average P value serves as an index of difficulty for an examination as a whole and also as a ,reflection of the performance of a typical member of the reference group in that examination. B. Discrimination
(’ bid
For an individual question this statistic (r bis) indicates the extent to which the question is able to distinguish between the best candidates as identified by performance on the entire examination and the poorest candidates as identified by performance on the entire examination. A perfect r bis is 1.00, which would occur if a question was answered correctly by all of the best candidates and incorrectly by all of the poorest candidates. A question with an r bis greater than 0.20 is usually considered acceptable by the National Board of Medical Examiners, and an average * bis greater than 0.25 for an examination at the specialty board level is considered acceptable by the National Board of Medical Examiners. C. Reliability
IKR-201
The reliability coefficient is an index of the reproducibility of scores from a given examination. It is an estimate of the degree to which a group of candidates “The reference group for the ABA1 examination consisted of individuals certified by the American Board of Internal Medicine or the American Board of Pediatrics who had cornpleted two years of fellowship in Allergy and Immunology.
144
The American
Board
of Allergy
and Immunology
J. ALLERGY CLIN. IMMJNOL. FEBRWA-RY1975
would have the same rank order if they were to take a similar examination a second time without further preparation. The National Board of Medical Examiners holds that a reliability coefficient of greater than 0.90 for a total examination is necessary to feel confident in making pass-fail decisions about individual examinees.2 coefficient for Table IV shows the average I’, average r bis, and reliability the Certifying Examination and for the two major sections of that test. In the judgment of the National Board, the t.est was highly reliable, discriminated well, and was pitched at an appropriate level of difficulty.
In deciding what standard to set for passing the examination, the Board considered the past experience of the antecedent sub-boards and the experiences and policies of other specialty boards. The ABA1 sought advice from two sources, its two parent boards (the American Board of Internal Medicine and the American Board of Pediatrics) and the National Board of Medical Examiners. Before the examination was administered, a specially appointed Standards Committee* agreed upon the procedure by which the standard for passing the test would be set. Having first defined the reference group, this committee felt that the mean performance of this group would be an acceptable criterion for certification. However, the Committee also recognized that the testing instrument would not provide perfectly reliable measurements and, therefore, if the mean score of the reference group were chosen as the standard for passing the test, some examinees whose “true scores” (scores that would be obtained from a perfectly reliable instrument) were at or above the criterion might fail. Thus, to be virtually certain that no examinee who failed the test had a true score at or above the criterion, the committee recommended that the standard for passing the test be set 1.5 standard deviation units below the mean of the reference group, and this recommendation was adopted by the Board. When this standard was applied, approximately 90 per cent of the reference group and 68 per cent of all examinees passed the examination. *The Standards Committee of the ABA1 consisted of two members from ABAI, two members from the American Board of Internal Medicine, and two wmxbers from the American Bo5d of Pediatrics. 1 Barro, A. R.: Survey and evaluation of approaches to physician performance measurement, a supplement to the Journal of Medical EdRe@iog, vol. 48, November 19?3, p. 1047. 2 Hubbard, J. P.: Measuring medical education, Philadelphia, 1971, Lea $ Febiger, p. 57.
