A Study of Board Certification in Child Psychiatry as a Valid Indicator of Clinical Competence
John F. McDermott, Jr., M.D., Christine McGuire, M.A., and Eta S. Berner, Ed.D.
Abstract. This report describes a research projeCl carried out by the Child Psychiatry Board to improve the validity and reliability of its examination. Suggestions are made for changing a certifit'ation "event" to a certification "process." which would begin in training. where certain competencies determined by the study of practice are best evaluated. and continue on throughout a child psvchiatrist's career via the evaluation of his or her continuing education experience.
In 1973, the Committee on Certification in Child Psychiatry of the American Board of Psychiatry and Neurology, Inc. initiated a joint study with the Center for Educational Development of the University of Illinois College of Medicine for the purposes of: (1) developing more valid and reliable procedures for certifying competence in the field by relating them to the actual requirements of practice to be determined empirically; (2) generating data that would be of assistance to training directors in increasing the efficiency and effectiveness of residency programs; and (3) serving as a pilot for a larger subsequent study in Psychiatry and Neurology (which is now in progress). In order to accomplish these objectives, the study was designed to be carried out in three phases: (1) a determination of the essential or critical components of competence for effective perforDr. McDennott is Professor and Chairmnn, Department of Psychiatry, John A. Burns School of Medicine, University of Hawaii. Ms. McGuire is Associate Director and Chief, Research and Evaluation Section, Center for EducatioTUlI Development, University of Illinois Coll£ge of Medicine. Dr. Berner is Project Director and on the staff of the Center for Educational Development, University of Illinois College of Medicine. This project was funded by the Grant Foundation, the Ittwson Family Foundation, and the Maurice Falk Medical Fund. The authors wish to express their appreciation to Lester Rudy, H. Donald Dunton, Stuart Finch, Barbara Fish, Joseph Green, Irving Philips, Jeanne Spurlock, Robert Stubblefield, and Richard Ward without whose participation the study could not have been done. Reprints may be requested from Dr. McDermott at the Department of Psychiatry, University of Hawaii, 1356 Lusitana Street, Honolulu, HI 96813.
517
518
John F. M cDrrmolt, Jr.
pt
al.
mance in the practice of the specialty; (2) an investigation of the extent to which the existing certification techniques were, in fact, adequately measuring these critical performance requirements; and (3) the development of instruments that would yield a more relevant, valid, reliable, and comprehensive assessment, partictllariy of those components of competence not pre\·iouslv assessed in the certification examination. Background. Since the 1920s, medical specialties in the United States have developed their own "Boards" with dual functions: maintaining standards of training through participating in the accreditation of training programs, and measuring the competence of individuals practicing that specialty through certification examinations. In recent years, there has been increasing criticism within the psychiatric profession about the way these functions, especially the principal one of certification, are carried out. Some (Aring, 1976; Lipp, 1976; McGuire, 1969; Porkorny and Frazier, 1966; Taylor and Torrey, 1972) have commented on the limited nature of professional competence that the Board examinations sample, and it has been noted that at best these examinations measure predominantly the candidate's ability to recall fragments of information solicited by the examiner. Traditi()nal examination procedures that emphasize the recall of information may bear little relation to such complex clinical tasks as interpreting data and solving problems, and even less to such skills as examining patients, formulating a differential diagnosis, or planning and carrying out management programs. The Current Problem. Government and third-party payment (Aring, 1976) requiring Board Certification have resulted in larger and larger numbers of physicians applying for examination. The reasons are obvious: the level of third-party payments will be increasingly determined by special competence as measured by Board Certification. Clinics and groups requiring the services of physicians, and even professional societies, are utilizing the results of Board examinations for recruitment, promotion, and salary. Yet little has been done to reverse the longstanding uncertainty in the medical profession about the validity of its own method of certifying competence. There is no firm evidence which demonstrates that Board Certification separates the more competent from less competent physicians. Thus the current national scene provides us with an urgent need as well as opportunity to improve certifying examinations, to make them more valid and reliable, and to relate them more directly to the requirements of practice of the specialty as determined by data obtained from a broad sample of practitioners themselves. An experiment toward this goal by the
A Study of Board Certification in Child Psychiatry
519
subspecialty of child psychiatry is detailed in this report. We hope that the data and experience it provides will insure greater reliability, validity, and confidence in the decisions made about an individual's competence in the field. Method. A determination of the essential components of competence is often made a priori on the basis of expert judgment about the kinds and levels of knowledge and skills that should be demonstrated by a given specialist. However, in order to establish a firm, empirical basis for determining the kinds of competencies which child psychiatrists need to have in order to function adequately in their current professional roles, primary reliance in this study was placed on systematic data collection from a combined task analysis and critical incident survey of practitioners, supplemented by expert opinion. (It is recognized that certain more subjective qualities, e.g., empathy, cannot be measured as well as others.) The critical incident technique has been used in numerous studies (McGuire, 1966; Levine and McGuire, 1971) of medical practice in the past and has been generally accepted as a valid and reliable research tool. We surveyed a stratified random sample of 506 practicing child psychiatrists, both certified and noncertified, who represented approximately 50 percent of the membership of the American Academy of Child Psychiatry in 1974. Questionnaires requesting detailed information about their professional activities in a typical work week-the types of patients they see, the types of other professionals with whom they consult, their current participation in professional organizations, etc.-were mailed to the selected individuals. Approximately 60 percent of those surveyed responded. The respondents were determined to represent a random sample of the child psychiatric community. The responses of this group were analyzed for the group as a whole and for various subgroups classified according to age, certification status, geographic locale, and practice setting of the respondent. In addition, all psychiatrists surveyed were asked to provide two descriptions of critical incidents (Flanagan, 1954): one in which they had recently done, or had seen a colleague do, something which they considered especially effective; the other, an incident in which they had recently done, or had seen a colleague do, something which they considered especially ineffective. The physician was asked to describe the setting in which the incident occurred, exactly what was done, and why the respondent considered it especially effective or especially ineffective. A total of 487 incidents were reported by the 298 respondents. As a final check on the critical performance requirements of
john F. McDermott,jr. et al.
520
child psychiatrists, two consumer groups who frequently request consultations from specialists in this field were surveyed. The 227 pediatricians and 100 judges included in this survey were selected to reflect the geographic distribution of child psychiatrists. This sampling technique was used in order to assure that respondents had adequate access to the services of child psychiatry. Questionnaires sent to these consumer groups requested information about the types of assistance they had sought and had received from child psychiatrists, and the areas in which they hoped to obtain assistance but had not received satisfactory support. In addition, all respondents were asked to report two specific critical incidents: one in which a child psychiatrist had been particularly helpful to them, and one in which he had been especially ineffective in assisting them. For each incident the consumer was asked to describe the situation, exactly what the child psychiatrist did, and why the consumer considered it especially effective or especially ineffective. A total of 98 responses (approximately 40%) was obtained from the pediatricians, and a total of 22 responses (22%) was obtained from the judges. This low response rate, in contrast to that of the child psychiatrists, most likely represents primarily a lower motivation for participation as well as a lack of significant experience with identified child psychiatrists. Critical incidents submitted by the child psychiatrists were used to generate an empirical classification of the essential components of competency for the practice of the specialty. Incidents from the consumer groups were then considered to determine whether new categories of performance were required in addition to those already derived from the classification of incidents reported by physicians themselves. The critical performance requirements emerging from this empirical classification were then checked against the data obtained from the task analysis and against expert opinion, in order to formulate a comprehensive statement of the essential components of competency which should be developed in the training programs, and assessed in the certification process for this field (table 1).
DISCUSSION
Inspection of the definition of professional competence which emerged from this process reveals, as would be expected, that much of the requisite knowledge and many of the skills are common to all physicians, and are not unique to the practice of child
A Study of Board Certification in Child Psychiatry
521
psychiatry. This observation applies not only to skills required for effective performance in clinical roles, but also to those entailed in nonclinical responsibilities of consulting, administration, teaching, and research evaluation. Table I Competencies of Child Psychiatrists Skill Category Skill in gathering information Skill in formulating diagnoses Skill in therapeutic planning Skill in crisis and emergenc\' management Skill in management of acute and ch ronic disorders Skill in recording information Skill in consultation Skill in administration Skill in evaluating research Skill in teaching Ethics
Number of Incidents in Sample from: Psychiatrists Pediatricians Judges E*
1*
E*
1*
E*
1*
:!6 :!I
:!O
4
3
~O
4
:! 0
~8
4~
I~
9
II 3
0 II
:! I 4 0
94
96
I ;i
7
,I 48
,I
0 8
()
()
:!;!
(i
:!
:>
(i
()
I
.~
:1
:l
:1 II
()
()
0
0
0 0 0 0
:!
:!
~
0
0 2 0 0 0 0
* E = dTecti\'t~ incidents: 1= inefTecti\'e incidents.
The definition of essential skills, based on a combination of: (a) the critical incident study, (b) the task analysis, and (c) expert review, has been fully detailed in a monograph entitled, Roles and Functions of Child Psychiatrists (McDermott et al., 1976). In the monograph each Skill is elaborated in a separate chapter which includes the data base from which the Indicator:; of Competence were derived, a statement of the prerequisite Knowledge Base for the development of this Skill, suggested criteria and techniques for Evaluation of performance in each Skill area, and suggestions as to where and how this Evaluation might best be done, e.g., written or oral examination, in training or afterward. The report is divided into three sections: 1. Competence in patient care roles: information gathering, differential diagnosis, therapeutic planning, therapeutic management, and recording information. 2. Competence in nonpatient care roles: consultation, administration, evaluation of research. 3. Professional habits and attitudes: ethical behavior; this is elaborated with illustrations of the application of ethical standards relating to each of the component Skills involved in patient care and noncare activities.
522
John F. McDermott, Jr. et ai.
MODIFICATION OF THE CERTIFICATION PROCESS
In effect, the Board Certification Examination in Child Psychiatry has gradually been modified to measure the prime requisites for the model of good clinical practice elicited from the study in an attempt to improve its validity. Instead of evaluating the candidate in previously agreed-upon content areas alone, the same basic techniques are being used to provide a profile of scores in the basic,. Skill areas as determined by the competency study. However, it is also clear that each Skill score may be determined by data from several different sources (table 2). For example, it is clear that if training directors can agree and establish consensus with the Board as to a common core training experience, then training directors can become part of the certification process itself. Training directors could use standardized forms to provide evidence of competency in certain Skill areas which would supplement evaluation of Table
~
Suggested Scorc Profile till' Certification Examination Asscssed during Board Exam Techniquc
Skill Area Data gatheringcom prehensi veness
Data gathering techniquc
Asscssed bv Data from Training i)irectors or Candidates ~Iaior
Written exam: multiple choice. and patielll management problems: videotape and/or oral exam Observation of live patient
Su pplemcnta r}
x
exanlinatic)ll
Diffcrential diagnosis
\\'ritten exam: multiple choice and patient management problems: videotape and/or
Therapeutic planning
Wrillen exam: multiple choice and patient managcment problems: videotape and/or oral exam Not assessed "ot assessed Case vignettes and role plaving \; ot assessed Written exam on a Journal article Not assessed :-.I ot assessed
oral
Therapeutic management Record keeping Consultation Administratillll Ev'aluation of research Teaching Ethics
x
eXatll
x
X X X X X X X
A Study of Board Certification in Child Psychiatry
523
some of the same competencies in the examination. The use of these multiple sources of data allows a broader sampling of skills than is currently possible in a single certification examination. This would insure greater reliability, validity, and confidence in the decisions made about an individual's competence in the field. It is obvious that general agreement on core competencies or Skills for a specialist could lead to more uniform training objectives and the involvement of training programs in the preparation for Board Certification, without detracting from the uniqueness of each program, or the variety of "routes" utilized to achieve these standard basic clinical skills among trainees. Furthermore, all candidates for the Examination can now know more precisely what skills will be assessed and how this will be done. Perhaps of most significance, however, is that it is clear from table 2 that the Board Certification Examination can tap only some of the skills required for the practice of the specialty. Therapeutic management, record keeping, administration, teaching, and ethical behaviors have been established (at least in child psychiatry) as essential skills for competence in the field. Yet, they are not being assessed in the Board Certification examination. It would appear logical that linking the Examination with the training process in which these skills are established and gradually developed, is a natural future direction. Training directors could fill our assessment forms documenting a candidate's performance, and candidates themselves might provide documentation (e.g., patient records and/or video tapes of diagnostic or therapeutic interviews) to be reviewed. This process is urgently in need of development if child psychiatrists are to have more appropriate input into their own review process. CONCLUSIONS AND FUTURE DIRECTIONS
Perhaps the most important consideration in binding (1) the process of training in a specialty with (2) the practice of that specialty, and (3) the measurement of competence via a certification process is the design which this "package" will take, the identification of its parts, and their most appropriate delegation. Specifically, it is likely that certain aspects of competence in a specialty should be measured during the training experience itself; e.g., treatment skills and attitudes toward patients are best assessed longitudinally. This evaluation could consist of Part I of the certification process (local). Part II might then consist of a written examination (regional) during or following the training experience in which essential "knowl-
john F. McDermott,jr. et ai.
524
edge and information" are measured. Part II would test for generally agreed-upon, important, basic knowledge and new information in the field which were gained during the training process itself. Part III (national) of the certification process would then consist of a smaller and less dramatic Board Certification examination, a live assessment of clinical judgment and skills which have been determined to be measurable in such a simulated setting. Part IV might consist of an ongoing recertification process based on continuing self and group education, the continued process of upgrading knowledge and skills. This would of course require a system for identification of significant new developments in a field and might very well be worked out with our constituent organization, the American Academy of Child Psychiatry. The assessment of need, design, and delivery of continuing education could be the responsibility of the Academy; and evaluation of the impact, a proper responsibility of the Board or a joint responsibility. The Board could provide immediate evaluation of knowledge and information gained by participants in continuing education programs. However, the true test of maintenance or development of new clinical skills and integration of new knowledge is in actual practice. How continuing education has affected an individual physician's actual clinical practice could be a joint responsibility of the Board and the Academy through periodic chart audit, utilization review, or other methods designed for this purpose. The important consideration is that through the relationship of these four hypothetical parts of a possible future certification process, the physician can enter it early and continue in it for the life of his or her career, one that is less dramatic but more valid and effective. It is toward this goal that we all look, to make child psychiatric practice a more meaningful contribution to the health needs of our nation and our young patients. REFERENCES ARING, C. (1976), Whither the specialty boards?]. A mer. Med. Assn., 235: 1849-1850. FLANAGAN, J. C. (1954), The critical incident technique. Psychol. Bull., 51 :327-358. LEVINE, H. & MCGUIRE, C. (1971), Use of profile system for scoring and reporting certifying examinations in orthopedic surgery.]. Med. Educ., 46:78-85. LIPP, M. R. (1976), Experiences of Psychiatry Board exam casualties. Amer. ]. Psychiat., 133:279-283. McDERMOTT, J. F., MCGUIRE, C., & BERNER, E. (1976), Roles and Functions of Child Psychiatrists: Report of the Project on Certification in Child Psychiatry. Evanston, Ill.: American Board of Psychiatry and Neurology, Inc. MCGUIRE, C. (1966), The oral examination as a measure of professional competence.]. Med. Educ., 41:267-274.
A Study of Board Certification in Child Psychiatry
525
- - - (1969), Evaluation of certification examination. Presented at the 13th Annual Scientific Assembly of the College of Family Physicians of Canada. PORKORNY, A. D. & FRAZIER, F. H., JR. (1966), An evaluation of oral examinations.]. Med. Educ., 4:28-40. TAYLOR, R. L. & TORREY, E. F. (1972), The pseudoregulation of American psychiatry. Amer. ]. Psychiat., 129:658-683.
John F. McDermott, Jr., M.D., Christine McGuire, M.A., and Eta S. Berner, Ed.D.
Abstract. This report describes a research projeCl carried out by the Child Psychiatry Board to improve the validity and reliability of its examination. Suggestions are made for changing a certifit'ation "event" to a certification "process." which would begin in training. where certain competencies determined by the study of practice are best evaluated. and continue on throughout a child psvchiatrist's career via the evaluation of his or her continuing education experience.
In 1973, the Committee on Certification in Child Psychiatry of the American Board of Psychiatry and Neurology, Inc. initiated a joint study with the Center for Educational Development of the University of Illinois College of Medicine for the purposes of: (1) developing more valid and reliable procedures for certifying competence in the field by relating them to the actual requirements of practice to be determined empirically; (2) generating data that would be of assistance to training directors in increasing the efficiency and effectiveness of residency programs; and (3) serving as a pilot for a larger subsequent study in Psychiatry and Neurology (which is now in progress). In order to accomplish these objectives, the study was designed to be carried out in three phases: (1) a determination of the essential or critical components of competence for effective perforDr. McDennott is Professor and Chairmnn, Department of Psychiatry, John A. Burns School of Medicine, University of Hawaii. Ms. McGuire is Associate Director and Chief, Research and Evaluation Section, Center for EducatioTUlI Development, University of Illinois Coll£ge of Medicine. Dr. Berner is Project Director and on the staff of the Center for Educational Development, University of Illinois College of Medicine. This project was funded by the Grant Foundation, the Ittwson Family Foundation, and the Maurice Falk Medical Fund. The authors wish to express their appreciation to Lester Rudy, H. Donald Dunton, Stuart Finch, Barbara Fish, Joseph Green, Irving Philips, Jeanne Spurlock, Robert Stubblefield, and Richard Ward without whose participation the study could not have been done. Reprints may be requested from Dr. McDermott at the Department of Psychiatry, University of Hawaii, 1356 Lusitana Street, Honolulu, HI 96813.
517
518
John F. M cDrrmolt, Jr.
pt
al.
mance in the practice of the specialty; (2) an investigation of the extent to which the existing certification techniques were, in fact, adequately measuring these critical performance requirements; and (3) the development of instruments that would yield a more relevant, valid, reliable, and comprehensive assessment, partictllariy of those components of competence not pre\·iouslv assessed in the certification examination. Background. Since the 1920s, medical specialties in the United States have developed their own "Boards" with dual functions: maintaining standards of training through participating in the accreditation of training programs, and measuring the competence of individuals practicing that specialty through certification examinations. In recent years, there has been increasing criticism within the psychiatric profession about the way these functions, especially the principal one of certification, are carried out. Some (Aring, 1976; Lipp, 1976; McGuire, 1969; Porkorny and Frazier, 1966; Taylor and Torrey, 1972) have commented on the limited nature of professional competence that the Board examinations sample, and it has been noted that at best these examinations measure predominantly the candidate's ability to recall fragments of information solicited by the examiner. Traditi()nal examination procedures that emphasize the recall of information may bear little relation to such complex clinical tasks as interpreting data and solving problems, and even less to such skills as examining patients, formulating a differential diagnosis, or planning and carrying out management programs. The Current Problem. Government and third-party payment (Aring, 1976) requiring Board Certification have resulted in larger and larger numbers of physicians applying for examination. The reasons are obvious: the level of third-party payments will be increasingly determined by special competence as measured by Board Certification. Clinics and groups requiring the services of physicians, and even professional societies, are utilizing the results of Board examinations for recruitment, promotion, and salary. Yet little has been done to reverse the longstanding uncertainty in the medical profession about the validity of its own method of certifying competence. There is no firm evidence which demonstrates that Board Certification separates the more competent from less competent physicians. Thus the current national scene provides us with an urgent need as well as opportunity to improve certifying examinations, to make them more valid and reliable, and to relate them more directly to the requirements of practice of the specialty as determined by data obtained from a broad sample of practitioners themselves. An experiment toward this goal by the
A Study of Board Certification in Child Psychiatry
519
subspecialty of child psychiatry is detailed in this report. We hope that the data and experience it provides will insure greater reliability, validity, and confidence in the decisions made about an individual's competence in the field. Method. A determination of the essential components of competence is often made a priori on the basis of expert judgment about the kinds and levels of knowledge and skills that should be demonstrated by a given specialist. However, in order to establish a firm, empirical basis for determining the kinds of competencies which child psychiatrists need to have in order to function adequately in their current professional roles, primary reliance in this study was placed on systematic data collection from a combined task analysis and critical incident survey of practitioners, supplemented by expert opinion. (It is recognized that certain more subjective qualities, e.g., empathy, cannot be measured as well as others.) The critical incident technique has been used in numerous studies (McGuire, 1966; Levine and McGuire, 1971) of medical practice in the past and has been generally accepted as a valid and reliable research tool. We surveyed a stratified random sample of 506 practicing child psychiatrists, both certified and noncertified, who represented approximately 50 percent of the membership of the American Academy of Child Psychiatry in 1974. Questionnaires requesting detailed information about their professional activities in a typical work week-the types of patients they see, the types of other professionals with whom they consult, their current participation in professional organizations, etc.-were mailed to the selected individuals. Approximately 60 percent of those surveyed responded. The respondents were determined to represent a random sample of the child psychiatric community. The responses of this group were analyzed for the group as a whole and for various subgroups classified according to age, certification status, geographic locale, and practice setting of the respondent. In addition, all psychiatrists surveyed were asked to provide two descriptions of critical incidents (Flanagan, 1954): one in which they had recently done, or had seen a colleague do, something which they considered especially effective; the other, an incident in which they had recently done, or had seen a colleague do, something which they considered especially ineffective. The physician was asked to describe the setting in which the incident occurred, exactly what was done, and why the respondent considered it especially effective or especially ineffective. A total of 487 incidents were reported by the 298 respondents. As a final check on the critical performance requirements of
john F. McDermott,jr. et al.
520
child psychiatrists, two consumer groups who frequently request consultations from specialists in this field were surveyed. The 227 pediatricians and 100 judges included in this survey were selected to reflect the geographic distribution of child psychiatrists. This sampling technique was used in order to assure that respondents had adequate access to the services of child psychiatry. Questionnaires sent to these consumer groups requested information about the types of assistance they had sought and had received from child psychiatrists, and the areas in which they hoped to obtain assistance but had not received satisfactory support. In addition, all respondents were asked to report two specific critical incidents: one in which a child psychiatrist had been particularly helpful to them, and one in which he had been especially ineffective in assisting them. For each incident the consumer was asked to describe the situation, exactly what the child psychiatrist did, and why the consumer considered it especially effective or especially ineffective. A total of 98 responses (approximately 40%) was obtained from the pediatricians, and a total of 22 responses (22%) was obtained from the judges. This low response rate, in contrast to that of the child psychiatrists, most likely represents primarily a lower motivation for participation as well as a lack of significant experience with identified child psychiatrists. Critical incidents submitted by the child psychiatrists were used to generate an empirical classification of the essential components of competency for the practice of the specialty. Incidents from the consumer groups were then considered to determine whether new categories of performance were required in addition to those already derived from the classification of incidents reported by physicians themselves. The critical performance requirements emerging from this empirical classification were then checked against the data obtained from the task analysis and against expert opinion, in order to formulate a comprehensive statement of the essential components of competency which should be developed in the training programs, and assessed in the certification process for this field (table 1).
DISCUSSION
Inspection of the definition of professional competence which emerged from this process reveals, as would be expected, that much of the requisite knowledge and many of the skills are common to all physicians, and are not unique to the practice of child
A Study of Board Certification in Child Psychiatry
521
psychiatry. This observation applies not only to skills required for effective performance in clinical roles, but also to those entailed in nonclinical responsibilities of consulting, administration, teaching, and research evaluation. Table I Competencies of Child Psychiatrists Skill Category Skill in gathering information Skill in formulating diagnoses Skill in therapeutic planning Skill in crisis and emergenc\' management Skill in management of acute and ch ronic disorders Skill in recording information Skill in consultation Skill in administration Skill in evaluating research Skill in teaching Ethics
Number of Incidents in Sample from: Psychiatrists Pediatricians Judges E*
1*
E*
1*
E*
1*
:!6 :!I
:!O
4
3
~O
4
:! 0
~8
4~
I~
9
II 3
0 II
:! I 4 0
94
96
I ;i
7
,I 48
,I
0 8
()
()
:!;!
(i
:!
:>
(i
()
I
.~
:1
:l
:1 II
()
()
0
0
0 0 0 0
:!
:!
~
0
0 2 0 0 0 0
* E = dTecti\'t~ incidents: 1= inefTecti\'e incidents.
The definition of essential skills, based on a combination of: (a) the critical incident study, (b) the task analysis, and (c) expert review, has been fully detailed in a monograph entitled, Roles and Functions of Child Psychiatrists (McDermott et al., 1976). In the monograph each Skill is elaborated in a separate chapter which includes the data base from which the Indicator:; of Competence were derived, a statement of the prerequisite Knowledge Base for the development of this Skill, suggested criteria and techniques for Evaluation of performance in each Skill area, and suggestions as to where and how this Evaluation might best be done, e.g., written or oral examination, in training or afterward. The report is divided into three sections: 1. Competence in patient care roles: information gathering, differential diagnosis, therapeutic planning, therapeutic management, and recording information. 2. Competence in nonpatient care roles: consultation, administration, evaluation of research. 3. Professional habits and attitudes: ethical behavior; this is elaborated with illustrations of the application of ethical standards relating to each of the component Skills involved in patient care and noncare activities.
522
John F. McDermott, Jr. et ai.
MODIFICATION OF THE CERTIFICATION PROCESS
In effect, the Board Certification Examination in Child Psychiatry has gradually been modified to measure the prime requisites for the model of good clinical practice elicited from the study in an attempt to improve its validity. Instead of evaluating the candidate in previously agreed-upon content areas alone, the same basic techniques are being used to provide a profile of scores in the basic,. Skill areas as determined by the competency study. However, it is also clear that each Skill score may be determined by data from several different sources (table 2). For example, it is clear that if training directors can agree and establish consensus with the Board as to a common core training experience, then training directors can become part of the certification process itself. Training directors could use standardized forms to provide evidence of competency in certain Skill areas which would supplement evaluation of Table
~
Suggested Scorc Profile till' Certification Examination Asscssed during Board Exam Techniquc
Skill Area Data gatheringcom prehensi veness
Data gathering techniquc
Asscssed bv Data from Training i)irectors or Candidates ~Iaior
Written exam: multiple choice. and patielll management problems: videotape and/or oral exam Observation of live patient
Su pplemcnta r}
x
exanlinatic)ll
Diffcrential diagnosis
\\'ritten exam: multiple choice and patient management problems: videotape and/or
Therapeutic planning
Wrillen exam: multiple choice and patient managcment problems: videotape and/or oral exam Not assessed "ot assessed Case vignettes and role plaving \; ot assessed Written exam on a Journal article Not assessed :-.I ot assessed
oral
Therapeutic management Record keeping Consultation Administratillll Ev'aluation of research Teaching Ethics
x
eXatll
x
X X X X X X X
A Study of Board Certification in Child Psychiatry
523
some of the same competencies in the examination. The use of these multiple sources of data allows a broader sampling of skills than is currently possible in a single certification examination. This would insure greater reliability, validity, and confidence in the decisions made about an individual's competence in the field. It is obvious that general agreement on core competencies or Skills for a specialist could lead to more uniform training objectives and the involvement of training programs in the preparation for Board Certification, without detracting from the uniqueness of each program, or the variety of "routes" utilized to achieve these standard basic clinical skills among trainees. Furthermore, all candidates for the Examination can now know more precisely what skills will be assessed and how this will be done. Perhaps of most significance, however, is that it is clear from table 2 that the Board Certification Examination can tap only some of the skills required for the practice of the specialty. Therapeutic management, record keeping, administration, teaching, and ethical behaviors have been established (at least in child psychiatry) as essential skills for competence in the field. Yet, they are not being assessed in the Board Certification examination. It would appear logical that linking the Examination with the training process in which these skills are established and gradually developed, is a natural future direction. Training directors could fill our assessment forms documenting a candidate's performance, and candidates themselves might provide documentation (e.g., patient records and/or video tapes of diagnostic or therapeutic interviews) to be reviewed. This process is urgently in need of development if child psychiatrists are to have more appropriate input into their own review process. CONCLUSIONS AND FUTURE DIRECTIONS
Perhaps the most important consideration in binding (1) the process of training in a specialty with (2) the practice of that specialty, and (3) the measurement of competence via a certification process is the design which this "package" will take, the identification of its parts, and their most appropriate delegation. Specifically, it is likely that certain aspects of competence in a specialty should be measured during the training experience itself; e.g., treatment skills and attitudes toward patients are best assessed longitudinally. This evaluation could consist of Part I of the certification process (local). Part II might then consist of a written examination (regional) during or following the training experience in which essential "knowl-
john F. McDermott,jr. et ai.
524
edge and information" are measured. Part II would test for generally agreed-upon, important, basic knowledge and new information in the field which were gained during the training process itself. Part III (national) of the certification process would then consist of a smaller and less dramatic Board Certification examination, a live assessment of clinical judgment and skills which have been determined to be measurable in such a simulated setting. Part IV might consist of an ongoing recertification process based on continuing self and group education, the continued process of upgrading knowledge and skills. This would of course require a system for identification of significant new developments in a field and might very well be worked out with our constituent organization, the American Academy of Child Psychiatry. The assessment of need, design, and delivery of continuing education could be the responsibility of the Academy; and evaluation of the impact, a proper responsibility of the Board or a joint responsibility. The Board could provide immediate evaluation of knowledge and information gained by participants in continuing education programs. However, the true test of maintenance or development of new clinical skills and integration of new knowledge is in actual practice. How continuing education has affected an individual physician's actual clinical practice could be a joint responsibility of the Board and the Academy through periodic chart audit, utilization review, or other methods designed for this purpose. The important consideration is that through the relationship of these four hypothetical parts of a possible future certification process, the physician can enter it early and continue in it for the life of his or her career, one that is less dramatic but more valid and effective. It is toward this goal that we all look, to make child psychiatric practice a more meaningful contribution to the health needs of our nation and our young patients. REFERENCES ARING, C. (1976), Whither the specialty boards?]. A mer. Med. Assn., 235: 1849-1850. FLANAGAN, J. C. (1954), The critical incident technique. Psychol. Bull., 51 :327-358. LEVINE, H. & MCGUIRE, C. (1971), Use of profile system for scoring and reporting certifying examinations in orthopedic surgery.]. Med. Educ., 46:78-85. LIPP, M. R. (1976), Experiences of Psychiatry Board exam casualties. Amer. ]. Psychiat., 133:279-283. McDERMOTT, J. F., MCGUIRE, C., & BERNER, E. (1976), Roles and Functions of Child Psychiatrists: Report of the Project on Certification in Child Psychiatry. Evanston, Ill.: American Board of Psychiatry and Neurology, Inc. MCGUIRE, C. (1966), The oral examination as a measure of professional competence.]. Med. Educ., 41:267-274.
A Study of Board Certification in Child Psychiatry
525
- - - (1969), Evaluation of certification examination. Presented at the 13th Annual Scientific Assembly of the College of Family Physicians of Canada. PORKORNY, A. D. & FRAZIER, F. H., JR. (1966), An evaluation of oral examinations.]. Med. Educ., 4:28-40. TAYLOR, R. L. & TORREY, E. F. (1972), The pseudoregulation of American psychiatry. Amer. ]. Psychiat., 129:658-683.
