Original Research
Workplace-Based Assessment for Vocational Registration of International Medical Graduates
STEVEN LILLIS, MBCHB, FRNZCGP, MGP, PHD; VALENCIA VAN DYK, BTECH, PGDIPBUSADMIN Introduction: Medical regulatory authorities need efficient and effective methods of ensuring the competence of immigrating international medical graduates (IMGs). Not all IMGs who apply for specialist vocational registration will have directly comparable qualifications to those usually accepted. As general licensure examinations are inappropriate for these doctors, workplace-based assessment (WBA) techniques would appear to provide a solution. However, there is little published data on such outcomes. Methods: All cases of WBA (n = 81) used for vocational registration of IMGs in New Zealand between 2008 and 2013 were collated and analyzed. Results: The successful completion rate of IMGs through the pathway was 87%. The majority (64%) undertook the year of supervised practice and the final assessment in a provincial center. For those unsuccessful in the pathway, inadequate clinical knowledge was the most common deficit found, followed by poor clinical reasoning. Discussion: A WBA approach for assessing readiness of IMGs for vocational registration is feasible. The constructivist theoretical perspective of WBA has particular advantages in assessing the standard of practice for experienced practitioners working in narrow scopes than traditional methods of assessment. The majority of IMGs undertook both the clinical year and the assessment in provincial hospitals, thus providing a workforce for underserved areas. Key Words: physician assessment/remediation, workforce development/issues, workplace learning, workplacebased assessment
Introduction Workplace-based assessment (WBA) is an established method of assessment and learning in medical training with a described philosophical approach.1 In general, a number of tools are used from a suite that includes Multisource Feedback (MSF), Direct Observation of Procedural Skills (DOPS), Mini Clinical Examination (Mini-CEx) and casebased oral examination (CBO).2 The key advantage of WBA over alternative assessment strategies is the ability to assess integrated performance rather than more traditional testing methods that target maximal competence in deconstructed tests.3,4 Consequently, well-constructed WBA can facilitate Disclosures: The authors report none. Dr. Lillis: Medical Adviser, Medical Council of New Zealand; Ms. Van Dyk: Registration Manager, Medical Council of New Zealand. Correspondence: Steven Lillis, Medical Council of New Zealand, PO Box 11-649, Wellington, New Zealand; e-mail: [email protected]. © 2014 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education • Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/chp.21251
deep rather than superficial learning.5 WBA has been extensively used in undergraduate contexts.6,7 In postgraduate training environments, WBA has been studied for use in such diverse practices as psychiatry,8 surgery,9 general practice10 radiology,11 obstetrics,12 and anaesthesia.13 Although WBA has mostly been utilized for formative purposes, there is increasing interest and experience in using the methodology for summative purposes in vocational training.14 More recently, the methodology has been used for general registration of international medical graduates (IMGs) in Australia.15,16 There are insufficient data as yet to be certain of better educational outcomes of WBA in comparison to more traditional forms of assessment.17 In New Zealand, medical workforce shortages are more accentuated in provincial and rural centers.18 This reflects similar international trends.19–21 Workplace-based assessment provides potential solutions to some workforce problems in circumstances where it is inappropriate to expect an experienced physician working in a limited scope of practice to sit a broad registration assessment that is pitched at an earlier stage of professional life. IMGs who wish to work in New Zealand with vocational registration in a particular branch of medicine may be
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS, 34(4):260–264, 2014
Workplace-Based Assessment for Evaluating IMGs
required by the regulator, the Medical Council of New Zealand (MCNZ), to follow a provisional vocational registration pathway if they do not hold an approved New Zealand or Australasian postgraduate qualification.22 Entry to the pathway requires an interview with the appropriate specialist college to ascertain the equivalence of specialist qualifications, training, and experience. The pathway requires 12 to 18 months of supervised practice, 3 satisfactory monthly supervision reports during this time, participation in a recognized continuing professional development program, and an assessment at the end of the supervision period. The reports are used for formative purposes to assist the IMG in professional development but are also sent to the MCNZ for monitoring. If supervisor reports are satisfactory, the IMG undergoes a Vocational Practice Assessment (VPA).23 The purpose of the VPA is to determine if the IMG is practicing at the level of a doctor holding the prescribed fellowship, diploma, or certificate qualification and practicing at the level of a doctor registered in the same vocational scope, and competent to practice independently and unsupervised across the broad vocational scope of practice. The timing of the VPA is at the end or very soon after the supervisory period. The VPA typically includes the tools of opening interview, observations of interactions with patients in an outpatient setting, review of 20 patient records from their caseload, the CBO on clinical material uncovered during the assessment, interviews with peers and other health professionals, peer ratings from medical and nonmedical colleagues using the peer ratings tool (completed prior to the day of assessment), and closing interview. DOPS and Mini-CEx tools are used where appropriate. A suite of modified tools is available for many specialties that are based on published assessment methods. The VPA is held at the IMG’s place of employment with 2 assessors who are vocationally registered in the same scope of practice for which the IMG is applying. The assessors must be in good standing with the MCNZ and are encouraged to attend a training day each year. The training day is used to discuss changes in tools being used, share experiences, and discuss mock reports for level of detail and appropriateness of recommendation. Prior to the VPA, a teleconference is held between the assessors, Council’s medical adviser, and Council’s registration coordinator to discuss the logistics of the assessment day and case selection for those doctors who undertake procedures. The VPA is undertaken in a single day, although some interviews with peers may be undertaken by phone prior to the formal day of assessment. A report is prepared by the assessors that provides the MCNZ with a detailed description of the assessment day and a recommendation as to whether the doctor under assessment should be registered within the relevant vocational scope of practice and reasons as to the basis
of the recommendation. The report is sent to the IMG regardless of the outcome. Regulatory authorities face the difficult task of assessing the competence of doctors for vocational registration in circumstances where there is no clear equivalence of postgraduate training. Exit examinations from vocational training institutions may not be feasible and may also be inappropriate for doctors who have developed a subspecialty interest or are later in their careers. Workplace-based assessment offers a potential but largely untested solution to this problem. To date, there has been no published research on the use of workplace-based assessment for registering IMGs in a vocational scope of practice. This article presents the results of 81 IMGs applying for specialist registration on the New Zealand medical register and for whom a workplace-based assessment was ordered. Method From its inception in 2008 to August 2013, 81 IMGs progressed through the provisional vocational pathway to the final part of the process, a VPA. An additional 24 doctors who had applied for the provisional vocational pathway voluntarily withdrew from the process and were not included in the data set. All data used were part of existing databases held by the MCNZ for audit purposes or was extracted from reports by assessors. For all candidates, data were collected on location of visit (provincial or urban), date of assessment, scope of practice, and outcome of assessment by the authors. All reports were obtained for unsuccessful candidates and examined for areas of concern that were identified by the assessors. For these unsuccessful candidates, data on the MCNZ’s postassessment determination were collected. Feasibility issues in this project were assessed from the perspectives of success rate for doctors undergoing the pathway, numbers of doctors applying, and overall cost per assessment. Results Demographics The place of primary medical qualification (MD, MBChB, MBBS, or equivalent) shows 59% to be European, 22% from the United States, 10% from Indian Subcontinent, and 7% from Africa. The place where secondary (postgraduate) qualification was gained shows 58% to be from Europe, 30% from the United States, and 6% both from Indian Subcontinent and Africa. Scope Numbers of applicants by scope of practice are given in TABLE 1. Of 80 doctors, 64% had undertaken the year of
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
261
Lillis and Van Dyk
from internal medicine, obstetrics and gynecology, surgery, and pediatrics.
TABLE 1. Applicants by Scope of Practice
Scope of Practice
% of Total (n = 81)
Discussion Anesthesia
14
Cardiology
1
Colorectal surgery
1
Dermatology
1
Radiology
5
General surgery
4
General medicine
42
Medical oncology
1
Obstetrics and gynecology
4
Orthopedics
2
Pediatrics
7
Psychiatry
17
provisional vocational registration and a VPA in a provincial center, and 36% in a major urban center. Assessment Outcome One doctor (out of 81) had significant concerns about competence raised during the supervisory process and underwent a formal performance assessment, rather than proceeding to the VPA. Of 80 IMGs who underwent a VPA, 70 succeeded on their first assessment, giving an 87% success rate. For those unsuccessful in the assessment, knowledge deficits were found in all but one of the doctors. Problematic clinical reasoning (ability to combine disparate but relevant clinical information into a coherent picture) and deficient documentation was found in one-third of the doctors. Occasional problems were seen in clinical management, leadership, verbal communication, procedural skills, and basic skills such as history taking and physical examination. One doctor was assessed as having poor insight (inaccurate perception of one’s own knowledge and abilities). All but 1 of the doctors had multiple concerns. For those 10 IMGs who were unsuccessful in the VPA, 3 undertook further supervision and were successful at a second VPA attempt. Two left the country and were lost to follow-up. One ceased practice. One had further concerns notified during a period of extended supervision, and a performance assessment was ordered. One failed a second VPA and had conditions placed on their scope of practice. One was required to undertake a specific program of study and was given vocational registration once this had been completed. One is awaiting a second VPA. The scope of practice of those unsuccessful in the VPA shows 3 cases from the specialty of anesthetics; 2 were in the scope of radiology, and 1 each 262
The supervisory period of between 12 and 18 months provides an important stepping-stone for IMGs. The supervisor reports are discussed between the IMG and the supervisor, thus providing formal review at regular intervals. There is ample evidence of the importance of feedback for learning in WBA and the need for such formal regular review.24 A second benefit of the supervisory process is to provide opportunity for the IMG to acclimatize to a New Zealand culture. Previous research has highlighted difficulties of IMGs in joining the New Zealand workforce.25,26 There are substantial external factors for these doctors to contend with, such as uncertain employment and lifestyle as well as differences in level of documentation required, importance of communication skills in New Zealand medical practice, medicolegal issues unique to this country, and knowledge of hospital policy and process. This supervisory period is of benefit in overcoming such difficulties. Reports of the supervisor are discussed with the IMG for formative purposes as well as summative MCNZ requirements. Similarly, the final report of the VPA, while summative in purpose, is also released to the IMG for reasons of improving future performance. There is a tension implicit in using the same assessment for both formative and summative purposes. Formative assessment should encourage the learner to discuss self-acknowledged areas of weakness, but summative assessment may have adverse consequences if such weaknesses are discovered.27 A more contemporary view of this tension is expressed in describing the use of an assessment for both purposes: “ . . . this is acceptable provided it is entirely transparent to trainer and trainee in what circumstances they are meeting on a particular occasion.”28 All assessment processes are a balance between competing imperatives. A traditional approach to structuring an assessment considers issues of reliability, validity, acceptability, educational value, and cost.29 This psychometric approach fits poorly with WBA, particularly with regard to reliability and validity. Acknowledged problems with a psychometric analysis of WBA include the unpredictable and unstandardized nature of the tasks being assessed as well as interassessor variability.30 Further, the context in which the assessment takes place will influence the outcome of WBA, whereas in more traditional assessments, the outcome should be the same irrespective of location and context.31 In the assessment process described in this article, the intent is not to achieve indices of reliability with a particular tool, but to gather wide-ranging information from a variety of sources that produce thick descriptions of both performance and the context of the performance. Information from various
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
Workplace-Based Assessment for Evaluating IMGs
sources is triangulated where possible. The outcome of assessment is decided by discussion between assessors and, if necessary, an experienced medical adviser to Council rather than achieving a particular score or scores during the assessment. These 2 different stances to assessment can be conceptualized as a traditional psychometric-based, reductionist approach versus a socially situated, interpretivist approach. The interpretivist approach sits comfortably within sociocultural educational theory and social constructivism.31 Both seek to make judgments on professional competence that are defensible, credible, and based on robust evidence. The assessments reported in this article are clearly of a socially situated, interpretivist nature. For many scopes of practice, it would be logistically difficult to undertake such an assessment using traditional psychometrically based methods. For example, major surgical procedures may take many hours, yet composite reliability data would suggest that between 8 and 9 samples using a Direct Observation of Procedural Skills tool would be required to achieve adequate levels of reliability.32 Similarly, observation of the work of a psychiatrist with 1-hour appointments using a Mini-CEX tool would take between 7 and 8 samples to achieve adequate reliability. Currently, the cost of undertaking a VPA is approximately $US5000 (GST inclusive). The considerable majority of assessments are completed within 1 working day with 2 hours’ preparation time and 4 hours of report writing per assessor. The success rate of 87% was considered acceptable. Numbers of doctors applying for vocational registration through this pathway was also considered acceptable. A reductionist assessment philosophy would likely add substantially to the cost in terms of time required to undertake the assessment.
Conclusion
Limitations
References
As yet, there are no data on long-term outcomes (MiniCEx), of the assessment process described here. Although all practicing doctors are required to participate in formal recertification programs (including undertaking continuing professional development), this by itself does not imply that professional competence is assured or will be maintained. Since this group of doctors represents a very different cohort to doctors with a New Zealand undergraduate degree and New Zealand or Australasian recognized postgraduate qualifications, it would be of interest to observe how this group performs over a number of years. The outcomes of this approach to vocational registration are difficult to compare with other countries, as there are no published comparative data. The philosophical approach to the assessment design, while having a robust theoretical stance, is a departure from standard approaches to assessment, and comparison between the 2 stances would be valuable.
WBA is a feasible methodology of assessment for vocational registration of IMGs. The method described in this article yielded an 87% success rate on first assessment. The strengths of the process described in this article are the constructivist framework of the assessment, the preceding 12 to 18 months of experience with supervisor meetings and formal reports, and training of assessors. In assessing complex, specialized, and integrated professional competence, a constructivist perspective offers significant advantages to a psychometric approach. Tools used in such assessments need constant updating, and assessors need both formal training and regular update sessions. The preponderance of doctors applying for registration via this pathway who worked and undertook the assessment in provincial towns is encouraging in potentially providing a partial solution to provincial workforce issues.
Lessons for Practice ●
●
●
Workplace-based assessment is a feasible method of assessment for vocational registration of IMGs. A period of supervised practice should precede summative assessment. Deficient clinical knowledge is the most common reason for failing the assessment.
1. Postgraduate Medical Education and Training Board. Principles for an Assessment System for Postgraduate Medical Training. London, England: PMETB Press; 2004. 2. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29(9):855–871. 3. Rethans JJ, Sturmans F, Drop R, van der Vleuten C, Hobus P. Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. BMJ. 1991;303(6814):1377–1380. 4. Kopelow ML, Schnabl GK, Hassard TH, et al. Assessing practicing physicians in two settings using standardized patients. Acad Med. 199; 67(10 suppl):19–21. 5. Al-Kadri HM, Al-Kadi MT, van der Vleuten CP. Workplace-based assessment and students’ approaches to learning: a qualitative inquiry. Med Teach. 2013; 35(suppl 1):31–38. 6. Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3):226–231. 7. Mitchell C, Bhat S, Herbert A, Baker P. Workplace-based assessments in Foundation Programme training: do trainees in difficulty use them differently? Med Educ. 2013;47(3):292–300.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
263
Lillis and Van Dyk 8. Brittlebank A, Archer J, Longson D, Malik A, Bhugra DK. Workplacebased assessments in psychiatry: evaluation of a whole assessment system. Acad Psychiatry. 2013;37(5):301–307. 9. Ali JM. Getting lost in translation? Workplace based assessments in surgical training. Surgeon. 2013;11(5):286–289. 10. Mamelok J. Workplace-based assessment (WPBA) portfolios in licensing for general practice specialty training. Educ Prim Care. 2009;20(3):139–142. 11. Wallis A, Edey A, Prothero D, McCoubrie P. The Bristol Radiology Report Assessment Tool (BRRAT): developing a workplacebased assessment tool for radiology reporting skills. Clin Radiol. 2013;68(11):1146–1154. 12. Homer M, Setna Z, Jha V, Higham J, Roberts T, Boursicot K. Estimating and comparing the reliability of a suite of workplacebased assessments: an obstetrics and gynaecology setting. Med Teach. 2013;35(8):684–691. 13. Crossingham GV, Sice PJ, Roberts MJ, Lam WH, Gale TC. Development of workplace-based assessments of non-technical skills in anaesthesia. Anaesthesia. 2012;67(2):158–164. 14. Ali JM. Getting lost in translation? Workplace based assessments in surgical training. Surgeon. 2013;11(5):286–289. 15. Nair BR, Hensley MJ, Parvathy MS, et al. A systematic approach to workplace-based assessment for international medical graduates. Med J Aust. 2012;196(6):399–402. 16. Barnett AP, Terry DR, Mulligan B, West D. An evaluation of the Workplace Based Assessment at the Launceston General Hospital, University of Tasmania, Launceston; 2013. Available at: http://ecite.utas.edu.au/86181. Accessed August 19, 2014. 17. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010;341:c5064. 18. Gorman D. The disposition and mobility of medical practitioners in New Zealand. N Z Med J. 2011;124(1330):11–13. 19. MacDowell M, Glasser M, Fitts M, Nielsen K, Hunsaker M. A national view of rural health workforce issues in the USA. Rural and Remote Health. 2010;10(3);1531. 20. Lehmann U, Dieleman M, Martineau T. Staffing remote rural areas in middle- and low-income countries: a literature review of attraction and retention. BMC Health Serv Res. 2008;8:19.
264
21. Dussault G, Franceschini MC. Not enough there, too many here: understanding geographical imbalances in the distribution of the health workforce. Hum Resour Health. 2006 May;4:12. 22. Medical Council of New Zealand. Available at: http://www.mcnz .org.nz/assets/Policies/Registration-within-a-vocational-scope-ofpractice-Doctors-who-do-not-hold-approved-postgraduatequalification.pdf. Accessed October 24, 2013. 23. Medical Council of New Zealand. Available at: http://www.mcnz .org.nz/assets/Policies/Vocational-practice-assessments.pdf. Accessed October 24, 2013. 24. Pelgrim EA, Kramer AW, Mokkink HG, van der Vleuten CP. The process of feedback in workplace-based assessment: organisation, delivery, continuity. Med Educ. 2012;46(6):604–612. 25. Lillis S, St George I, Upsdell R. Perceptions of migrant doctors joining the New Zealand medical workforce. N Z Med J. 2006; 119(1229):U1844. 26. Narasimhan S, Ranchord A, Weatherall M. International medical graduates’ training needs: perceptions of New Zealand hospital staff. N Z Med J. 2006;119(1236):U2027. 27. Taras M. Assessment—summative and formative—some theoretical reflections. Br J Educ Stud. 2005;53(4):466–478. 28. Postgraduate Medical Education and Training Board. Developing and Maintaining an Assessment System. London, England: PMETB Press; 2007:21. 29. Van der Vleuten CP. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1(1):41–67. 30. Govaerts M, van der Vleuten CP. Validity in work-based assessment: expanding our horizons. Med Educ. 2013;47(12):1164–1174. 31. Durning SJ, Artino AR, Boulet JR, Dorrance K, van der Vleuten C, Schuwirth L. The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Adv Health Sci Educ Theory Pract. 2012;17(1):65–79. 32. Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 2013;18(5):1087–1102.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
Copyright of Journal of Continuing Education in the Health Professions is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Workplace-Based Assessment for Vocational Registration of International Medical Graduates
STEVEN LILLIS, MBCHB, FRNZCGP, MGP, PHD; VALENCIA VAN DYK, BTECH, PGDIPBUSADMIN Introduction: Medical regulatory authorities need efficient and effective methods of ensuring the competence of immigrating international medical graduates (IMGs). Not all IMGs who apply for specialist vocational registration will have directly comparable qualifications to those usually accepted. As general licensure examinations are inappropriate for these doctors, workplace-based assessment (WBA) techniques would appear to provide a solution. However, there is little published data on such outcomes. Methods: All cases of WBA (n = 81) used for vocational registration of IMGs in New Zealand between 2008 and 2013 were collated and analyzed. Results: The successful completion rate of IMGs through the pathway was 87%. The majority (64%) undertook the year of supervised practice and the final assessment in a provincial center. For those unsuccessful in the pathway, inadequate clinical knowledge was the most common deficit found, followed by poor clinical reasoning. Discussion: A WBA approach for assessing readiness of IMGs for vocational registration is feasible. The constructivist theoretical perspective of WBA has particular advantages in assessing the standard of practice for experienced practitioners working in narrow scopes than traditional methods of assessment. The majority of IMGs undertook both the clinical year and the assessment in provincial hospitals, thus providing a workforce for underserved areas. Key Words: physician assessment/remediation, workforce development/issues, workplace learning, workplacebased assessment
Introduction Workplace-based assessment (WBA) is an established method of assessment and learning in medical training with a described philosophical approach.1 In general, a number of tools are used from a suite that includes Multisource Feedback (MSF), Direct Observation of Procedural Skills (DOPS), Mini Clinical Examination (Mini-CEx) and casebased oral examination (CBO).2 The key advantage of WBA over alternative assessment strategies is the ability to assess integrated performance rather than more traditional testing methods that target maximal competence in deconstructed tests.3,4 Consequently, well-constructed WBA can facilitate Disclosures: The authors report none. Dr. Lillis: Medical Adviser, Medical Council of New Zealand; Ms. Van Dyk: Registration Manager, Medical Council of New Zealand. Correspondence: Steven Lillis, Medical Council of New Zealand, PO Box 11-649, Wellington, New Zealand; e-mail: [email protected]. © 2014 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education • Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/chp.21251
deep rather than superficial learning.5 WBA has been extensively used in undergraduate contexts.6,7 In postgraduate training environments, WBA has been studied for use in such diverse practices as psychiatry,8 surgery,9 general practice10 radiology,11 obstetrics,12 and anaesthesia.13 Although WBA has mostly been utilized for formative purposes, there is increasing interest and experience in using the methodology for summative purposes in vocational training.14 More recently, the methodology has been used for general registration of international medical graduates (IMGs) in Australia.15,16 There are insufficient data as yet to be certain of better educational outcomes of WBA in comparison to more traditional forms of assessment.17 In New Zealand, medical workforce shortages are more accentuated in provincial and rural centers.18 This reflects similar international trends.19–21 Workplace-based assessment provides potential solutions to some workforce problems in circumstances where it is inappropriate to expect an experienced physician working in a limited scope of practice to sit a broad registration assessment that is pitched at an earlier stage of professional life. IMGs who wish to work in New Zealand with vocational registration in a particular branch of medicine may be
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS, 34(4):260–264, 2014
Workplace-Based Assessment for Evaluating IMGs
required by the regulator, the Medical Council of New Zealand (MCNZ), to follow a provisional vocational registration pathway if they do not hold an approved New Zealand or Australasian postgraduate qualification.22 Entry to the pathway requires an interview with the appropriate specialist college to ascertain the equivalence of specialist qualifications, training, and experience. The pathway requires 12 to 18 months of supervised practice, 3 satisfactory monthly supervision reports during this time, participation in a recognized continuing professional development program, and an assessment at the end of the supervision period. The reports are used for formative purposes to assist the IMG in professional development but are also sent to the MCNZ for monitoring. If supervisor reports are satisfactory, the IMG undergoes a Vocational Practice Assessment (VPA).23 The purpose of the VPA is to determine if the IMG is practicing at the level of a doctor holding the prescribed fellowship, diploma, or certificate qualification and practicing at the level of a doctor registered in the same vocational scope, and competent to practice independently and unsupervised across the broad vocational scope of practice. The timing of the VPA is at the end or very soon after the supervisory period. The VPA typically includes the tools of opening interview, observations of interactions with patients in an outpatient setting, review of 20 patient records from their caseload, the CBO on clinical material uncovered during the assessment, interviews with peers and other health professionals, peer ratings from medical and nonmedical colleagues using the peer ratings tool (completed prior to the day of assessment), and closing interview. DOPS and Mini-CEx tools are used where appropriate. A suite of modified tools is available for many specialties that are based on published assessment methods. The VPA is held at the IMG’s place of employment with 2 assessors who are vocationally registered in the same scope of practice for which the IMG is applying. The assessors must be in good standing with the MCNZ and are encouraged to attend a training day each year. The training day is used to discuss changes in tools being used, share experiences, and discuss mock reports for level of detail and appropriateness of recommendation. Prior to the VPA, a teleconference is held between the assessors, Council’s medical adviser, and Council’s registration coordinator to discuss the logistics of the assessment day and case selection for those doctors who undertake procedures. The VPA is undertaken in a single day, although some interviews with peers may be undertaken by phone prior to the formal day of assessment. A report is prepared by the assessors that provides the MCNZ with a detailed description of the assessment day and a recommendation as to whether the doctor under assessment should be registered within the relevant vocational scope of practice and reasons as to the basis
of the recommendation. The report is sent to the IMG regardless of the outcome. Regulatory authorities face the difficult task of assessing the competence of doctors for vocational registration in circumstances where there is no clear equivalence of postgraduate training. Exit examinations from vocational training institutions may not be feasible and may also be inappropriate for doctors who have developed a subspecialty interest or are later in their careers. Workplace-based assessment offers a potential but largely untested solution to this problem. To date, there has been no published research on the use of workplace-based assessment for registering IMGs in a vocational scope of practice. This article presents the results of 81 IMGs applying for specialist registration on the New Zealand medical register and for whom a workplace-based assessment was ordered. Method From its inception in 2008 to August 2013, 81 IMGs progressed through the provisional vocational pathway to the final part of the process, a VPA. An additional 24 doctors who had applied for the provisional vocational pathway voluntarily withdrew from the process and were not included in the data set. All data used were part of existing databases held by the MCNZ for audit purposes or was extracted from reports by assessors. For all candidates, data were collected on location of visit (provincial or urban), date of assessment, scope of practice, and outcome of assessment by the authors. All reports were obtained for unsuccessful candidates and examined for areas of concern that were identified by the assessors. For these unsuccessful candidates, data on the MCNZ’s postassessment determination were collected. Feasibility issues in this project were assessed from the perspectives of success rate for doctors undergoing the pathway, numbers of doctors applying, and overall cost per assessment. Results Demographics The place of primary medical qualification (MD, MBChB, MBBS, or equivalent) shows 59% to be European, 22% from the United States, 10% from Indian Subcontinent, and 7% from Africa. The place where secondary (postgraduate) qualification was gained shows 58% to be from Europe, 30% from the United States, and 6% both from Indian Subcontinent and Africa. Scope Numbers of applicants by scope of practice are given in TABLE 1. Of 80 doctors, 64% had undertaken the year of
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
261
Lillis and Van Dyk
from internal medicine, obstetrics and gynecology, surgery, and pediatrics.
TABLE 1. Applicants by Scope of Practice
Scope of Practice
% of Total (n = 81)
Discussion Anesthesia
14
Cardiology
1
Colorectal surgery
1
Dermatology
1
Radiology
5
General surgery
4
General medicine
42
Medical oncology
1
Obstetrics and gynecology
4
Orthopedics
2
Pediatrics
7
Psychiatry
17
provisional vocational registration and a VPA in a provincial center, and 36% in a major urban center. Assessment Outcome One doctor (out of 81) had significant concerns about competence raised during the supervisory process and underwent a formal performance assessment, rather than proceeding to the VPA. Of 80 IMGs who underwent a VPA, 70 succeeded on their first assessment, giving an 87% success rate. For those unsuccessful in the assessment, knowledge deficits were found in all but one of the doctors. Problematic clinical reasoning (ability to combine disparate but relevant clinical information into a coherent picture) and deficient documentation was found in one-third of the doctors. Occasional problems were seen in clinical management, leadership, verbal communication, procedural skills, and basic skills such as history taking and physical examination. One doctor was assessed as having poor insight (inaccurate perception of one’s own knowledge and abilities). All but 1 of the doctors had multiple concerns. For those 10 IMGs who were unsuccessful in the VPA, 3 undertook further supervision and were successful at a second VPA attempt. Two left the country and were lost to follow-up. One ceased practice. One had further concerns notified during a period of extended supervision, and a performance assessment was ordered. One failed a second VPA and had conditions placed on their scope of practice. One was required to undertake a specific program of study and was given vocational registration once this had been completed. One is awaiting a second VPA. The scope of practice of those unsuccessful in the VPA shows 3 cases from the specialty of anesthetics; 2 were in the scope of radiology, and 1 each 262
The supervisory period of between 12 and 18 months provides an important stepping-stone for IMGs. The supervisor reports are discussed between the IMG and the supervisor, thus providing formal review at regular intervals. There is ample evidence of the importance of feedback for learning in WBA and the need for such formal regular review.24 A second benefit of the supervisory process is to provide opportunity for the IMG to acclimatize to a New Zealand culture. Previous research has highlighted difficulties of IMGs in joining the New Zealand workforce.25,26 There are substantial external factors for these doctors to contend with, such as uncertain employment and lifestyle as well as differences in level of documentation required, importance of communication skills in New Zealand medical practice, medicolegal issues unique to this country, and knowledge of hospital policy and process. This supervisory period is of benefit in overcoming such difficulties. Reports of the supervisor are discussed with the IMG for formative purposes as well as summative MCNZ requirements. Similarly, the final report of the VPA, while summative in purpose, is also released to the IMG for reasons of improving future performance. There is a tension implicit in using the same assessment for both formative and summative purposes. Formative assessment should encourage the learner to discuss self-acknowledged areas of weakness, but summative assessment may have adverse consequences if such weaknesses are discovered.27 A more contemporary view of this tension is expressed in describing the use of an assessment for both purposes: “ . . . this is acceptable provided it is entirely transparent to trainer and trainee in what circumstances they are meeting on a particular occasion.”28 All assessment processes are a balance between competing imperatives. A traditional approach to structuring an assessment considers issues of reliability, validity, acceptability, educational value, and cost.29 This psychometric approach fits poorly with WBA, particularly with regard to reliability and validity. Acknowledged problems with a psychometric analysis of WBA include the unpredictable and unstandardized nature of the tasks being assessed as well as interassessor variability.30 Further, the context in which the assessment takes place will influence the outcome of WBA, whereas in more traditional assessments, the outcome should be the same irrespective of location and context.31 In the assessment process described in this article, the intent is not to achieve indices of reliability with a particular tool, but to gather wide-ranging information from a variety of sources that produce thick descriptions of both performance and the context of the performance. Information from various
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
Workplace-Based Assessment for Evaluating IMGs
sources is triangulated where possible. The outcome of assessment is decided by discussion between assessors and, if necessary, an experienced medical adviser to Council rather than achieving a particular score or scores during the assessment. These 2 different stances to assessment can be conceptualized as a traditional psychometric-based, reductionist approach versus a socially situated, interpretivist approach. The interpretivist approach sits comfortably within sociocultural educational theory and social constructivism.31 Both seek to make judgments on professional competence that are defensible, credible, and based on robust evidence. The assessments reported in this article are clearly of a socially situated, interpretivist nature. For many scopes of practice, it would be logistically difficult to undertake such an assessment using traditional psychometrically based methods. For example, major surgical procedures may take many hours, yet composite reliability data would suggest that between 8 and 9 samples using a Direct Observation of Procedural Skills tool would be required to achieve adequate levels of reliability.32 Similarly, observation of the work of a psychiatrist with 1-hour appointments using a Mini-CEX tool would take between 7 and 8 samples to achieve adequate reliability. Currently, the cost of undertaking a VPA is approximately $US5000 (GST inclusive). The considerable majority of assessments are completed within 1 working day with 2 hours’ preparation time and 4 hours of report writing per assessor. The success rate of 87% was considered acceptable. Numbers of doctors applying for vocational registration through this pathway was also considered acceptable. A reductionist assessment philosophy would likely add substantially to the cost in terms of time required to undertake the assessment.
Conclusion
Limitations
References
As yet, there are no data on long-term outcomes (MiniCEx), of the assessment process described here. Although all practicing doctors are required to participate in formal recertification programs (including undertaking continuing professional development), this by itself does not imply that professional competence is assured or will be maintained. Since this group of doctors represents a very different cohort to doctors with a New Zealand undergraduate degree and New Zealand or Australasian recognized postgraduate qualifications, it would be of interest to observe how this group performs over a number of years. The outcomes of this approach to vocational registration are difficult to compare with other countries, as there are no published comparative data. The philosophical approach to the assessment design, while having a robust theoretical stance, is a departure from standard approaches to assessment, and comparison between the 2 stances would be valuable.
WBA is a feasible methodology of assessment for vocational registration of IMGs. The method described in this article yielded an 87% success rate on first assessment. The strengths of the process described in this article are the constructivist framework of the assessment, the preceding 12 to 18 months of experience with supervisor meetings and formal reports, and training of assessors. In assessing complex, specialized, and integrated professional competence, a constructivist perspective offers significant advantages to a psychometric approach. Tools used in such assessments need constant updating, and assessors need both formal training and regular update sessions. The preponderance of doctors applying for registration via this pathway who worked and undertook the assessment in provincial towns is encouraging in potentially providing a partial solution to provincial workforce issues.
Lessons for Practice ●
●
●
Workplace-based assessment is a feasible method of assessment for vocational registration of IMGs. A period of supervised practice should precede summative assessment. Deficient clinical knowledge is the most common reason for failing the assessment.
1. Postgraduate Medical Education and Training Board. Principles for an Assessment System for Postgraduate Medical Training. London, England: PMETB Press; 2004. 2. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29(9):855–871. 3. Rethans JJ, Sturmans F, Drop R, van der Vleuten C, Hobus P. Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. BMJ. 1991;303(6814):1377–1380. 4. Kopelow ML, Schnabl GK, Hassard TH, et al. Assessing practicing physicians in two settings using standardized patients. Acad Med. 199; 67(10 suppl):19–21. 5. Al-Kadri HM, Al-Kadi MT, van der Vleuten CP. Workplace-based assessment and students’ approaches to learning: a qualitative inquiry. Med Teach. 2013; 35(suppl 1):31–38. 6. Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3):226–231. 7. Mitchell C, Bhat S, Herbert A, Baker P. Workplace-based assessments in Foundation Programme training: do trainees in difficulty use them differently? Med Educ. 2013;47(3):292–300.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
263
Lillis and Van Dyk 8. Brittlebank A, Archer J, Longson D, Malik A, Bhugra DK. Workplacebased assessments in psychiatry: evaluation of a whole assessment system. Acad Psychiatry. 2013;37(5):301–307. 9. Ali JM. Getting lost in translation? Workplace based assessments in surgical training. Surgeon. 2013;11(5):286–289. 10. Mamelok J. Workplace-based assessment (WPBA) portfolios in licensing for general practice specialty training. Educ Prim Care. 2009;20(3):139–142. 11. Wallis A, Edey A, Prothero D, McCoubrie P. The Bristol Radiology Report Assessment Tool (BRRAT): developing a workplacebased assessment tool for radiology reporting skills. Clin Radiol. 2013;68(11):1146–1154. 12. Homer M, Setna Z, Jha V, Higham J, Roberts T, Boursicot K. Estimating and comparing the reliability of a suite of workplacebased assessments: an obstetrics and gynaecology setting. Med Teach. 2013;35(8):684–691. 13. Crossingham GV, Sice PJ, Roberts MJ, Lam WH, Gale TC. Development of workplace-based assessments of non-technical skills in anaesthesia. Anaesthesia. 2012;67(2):158–164. 14. Ali JM. Getting lost in translation? Workplace based assessments in surgical training. Surgeon. 2013;11(5):286–289. 15. Nair BR, Hensley MJ, Parvathy MS, et al. A systematic approach to workplace-based assessment for international medical graduates. Med J Aust. 2012;196(6):399–402. 16. Barnett AP, Terry DR, Mulligan B, West D. An evaluation of the Workplace Based Assessment at the Launceston General Hospital, University of Tasmania, Launceston; 2013. Available at: http://ecite.utas.edu.au/86181. Accessed August 19, 2014. 17. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010;341:c5064. 18. Gorman D. The disposition and mobility of medical practitioners in New Zealand. N Z Med J. 2011;124(1330):11–13. 19. MacDowell M, Glasser M, Fitts M, Nielsen K, Hunsaker M. A national view of rural health workforce issues in the USA. Rural and Remote Health. 2010;10(3);1531. 20. Lehmann U, Dieleman M, Martineau T. Staffing remote rural areas in middle- and low-income countries: a literature review of attraction and retention. BMC Health Serv Res. 2008;8:19.
264
21. Dussault G, Franceschini MC. Not enough there, too many here: understanding geographical imbalances in the distribution of the health workforce. Hum Resour Health. 2006 May;4:12. 22. Medical Council of New Zealand. Available at: http://www.mcnz .org.nz/assets/Policies/Registration-within-a-vocational-scope-ofpractice-Doctors-who-do-not-hold-approved-postgraduatequalification.pdf. Accessed October 24, 2013. 23. Medical Council of New Zealand. Available at: http://www.mcnz .org.nz/assets/Policies/Vocational-practice-assessments.pdf. Accessed October 24, 2013. 24. Pelgrim EA, Kramer AW, Mokkink HG, van der Vleuten CP. The process of feedback in workplace-based assessment: organisation, delivery, continuity. Med Educ. 2012;46(6):604–612. 25. Lillis S, St George I, Upsdell R. Perceptions of migrant doctors joining the New Zealand medical workforce. N Z Med J. 2006; 119(1229):U1844. 26. Narasimhan S, Ranchord A, Weatherall M. International medical graduates’ training needs: perceptions of New Zealand hospital staff. N Z Med J. 2006;119(1236):U2027. 27. Taras M. Assessment—summative and formative—some theoretical reflections. Br J Educ Stud. 2005;53(4):466–478. 28. Postgraduate Medical Education and Training Board. Developing and Maintaining an Assessment System. London, England: PMETB Press; 2007:21. 29. Van der Vleuten CP. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1(1):41–67. 30. Govaerts M, van der Vleuten CP. Validity in work-based assessment: expanding our horizons. Med Educ. 2013;47(12):1164–1174. 31. Durning SJ, Artino AR, Boulet JR, Dorrance K, van der Vleuten C, Schuwirth L. The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Adv Health Sci Educ Theory Pract. 2012;17(1):65–79. 32. Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 2013;18(5):1087–1102.
JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS—34(4), 2014 DOI: 10.1002/chp
Copyright of Journal of Continuing Education in the Health Professions is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
