COURSE REPORT
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
Helping Those Who Fail the MRCGP Examination A short residential course for a group of general practitioners who had failed the MRCGP examination is described. Evaluation of the course includes a comparison of subsequent examination performance by participants with that of a control group. Overall, and in relation to four of the five parts of the examination, the course participants performed better than the control group. For two of the five parts of the examination the differences between course participants and controls were statistically significant. The impact of examination failure on this group of doctors, and their diverse educational needs are emphasized. The idea of the course for those who had failed the examination for membership of the Royal College of General Practitioners developed from the followingobservations: 0 The significant proportion of candidates failing the examination.' 0 The lack of information on why candidates fail and hence what might be done to help them.
Moreover, it appeared to us desirable that the College should recognize a responsibility to those who had tried unsuccessfully to become Members. Therefore a course was proposed at College headquarters to provide assessment and educational opportunities for candidates who had failed a recent examination for Membership. The course was provided in October 1980, with the support of the Experimental Courses Study Group of the College. It was residential and lasted five days. Aims 0 T o identify the characteristics of a group of candidates who had failed the MRCGP examination. 0 To involve them in a variety of educational activities appropriate to their needs.
D.Adshead, BA. MBBS. General Practitioner, Tordmorden; W.
Allen,
MB. CH.R. MRCGP. General Practitioner, Leeds; J . Bahrami, MB, CH.B. D.OBST.HCOC;. MRCOC, MRCGP, General Practitioner, Bradford; A. Belton, B S : . MB. CH.B, DRCOG. FRCGP, General Practitioner, Keighley; P. Heywood, MB. CH.B. D.CH, D.OBST.RCPC. FRCCP. General Practitioner, keds; S. Jenkinson, MB,B.CHIR. D.CH. MRCGP. General Practitioner, Baildon; A. Lewis, MA. MB. CH.B, D.CH, D.OBST.RCOG. MRCGP. General Practitioner. Hayle; I. Stanley, MB. CH.B, MRCCP. FRCCP, Lecturer in General Practice, University of Leeds; C . Varnavides, B.SC, MB, CH.B. D.OBST.H(:OX. MRCGP. General Practitioner, Leeds. Reprints requests to
I. Stanley, Dept. of Community Medicine and General Practice, Clinical Sciences Building, St James's Hospital, Leeds LS9 7TF.
Medical Teacher Vol 6 No 3 1984
0 To encourage patterns of learning appropriate to future success in the MRCGP examination and to continuing professional education.
Method Course design
In planning a course specifically for individuals who had failed this examination we were conscious that failure was likely to represent the outcome of a wide variety of factors. Moreover, as such individuals discover, feedback on examination performance is limited in its usefulness, consisting simply of individual and average scores for each of the five parts. A low individual score may be interpreted in a number of ways; in the Multiple Choice Question (MCQ) paper, for example, it may be seen as evidence of limited factual knowledge or difficulty with the conventions of this type of question. Even greater difficulty arises in the interpretation by candidates or their would-be advisors of scores achieved on essay questions or in orals. The first problem for such a course, then, is one of diagnosis; what are the reasons underlying failure by an individual in the examination? With this in mind a range of assessment methods was chosen in three broad areas: cognitive abilities, professional skills in the context of practice, and attitudes to learning. Within these areas both formal and informal techniques were used to identify problems for individuals in their approach to learning and in their organization and presentation of knowledge. Some of these techniques were chosen because of their ability to provide a detailed analysis within a general area, others because they formed part of the examination and were likely to reveal technical shortcomings. In consideringthe balance to be struck by such a course between examination content and technique, and bearing in mind the stated aims, the former was favoured. Nevertheless, it was recognized that technique might be a significant factor in the failure of individuals and, for the majority, an expected part of such a course. Both content and technique were to be provided within a programme of learning options, entrance to most of the options being linked to one or more assessment techniques. A two-tier structure was chosen. Each participant was to be allocated a tutor and given the opportunity to meet with him daily in the same small group. At other times participants were to be regrouped on the basis of 101
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
assessment and involved in appropriate small group learning. A staff participant ratio of 1:3 was considered necessary to provide adequate time with tutors, teachers and organizers within this structure. A five-day residential format was considered to be acceptable to working general practitioners and to provide a time-scale within which the aims of the course might be realized. The headquarters of the Royal College of General Practitioners was the venue chosen for the course. This choice was felt to be psychologically important to the participants, and consistent with the concern showed by the College for the group.
0 A programme of opportunities for learning about examination content and technique (Figure 1). 0 Tutorials. On the first day participants were allocated at random to a course tutor in groups of about five. Thereafter, these groups met the same tutor for at least one hour per day. 0 An oral examination. All but one participant (who declined the opportunity) were involved in a 30-minute oral examination closely simulating those of the MRCGP examination, and provided by a number of College Examiners. The practice log formed the basis of this exercise.
Recruitment
Assessment
Those candidates who failed the MRCGP examination in July 1980 and who subsequently sought details of their marks were sent a letter about the course (180 letters sent). Of these, 35 expressed interest and 27 attended the course. Of the 27 participants, 23 were principals, two were trainees and two were working outside general practice.
The Modified Essay Question paper: There were nine pages; each page had been constructed to assess the participant in one area of knowledge or skills provided for in the programme (Figure 1). The course teacher who was responsible for preparing a topic marked the appropriate page from each participant.
The Course Pre-course Work About 2 weeks before the course, participants were asked to complete the following: 0 A Modified Essay Question (MEQ) paper and a Traditional Essay Question (TEQ) paper under ‘examination conditions’. 0 A number of practice-related tasks. A short clinical history was provided, and the participant was asked to respond with an appropriate entry in the medical records and with a referral letter to an orthopaedic consultant. In addition a short account of patient follow-up policy in the participant’s practice was requested. 0 A practice proforma seeking information on the participant’s professional experience; practice organization; opportunities for continuing professional education, and so on. 0 A practice log listing 50 consecutive recent consultations and 10 patients under continuing care during the preceding three months. The M E Q and TEQ were returned by participants for marking before the course. The remaining items were brought with them to the course along with details of their marks in previous attempts at the MRCGP examination.
In-course Work 0 The initial assessment of clinical skills (IACS)’ comprising three written papers, each lasting 50 minutes, was taken by the participants on the first morning of the course. 0 Manchester rating scales (MRS)’ were completed by the participants on the first day and the last day.
102
The Traditional Essay Question paper: Answers to essay questions from all participants were marked by one course teacher responsible for teaching this aspect of examination technique. Practice-related tasks: These were assessed on the first morning of the course by six tutors working together. By comparing the material submitted, the tutors identified those participants with performance below average for the group. Prior pct-formance in the M R C G P examination: Marks were available from each participant showing his performance in the Multiple Choice Question (MCQ) paper, the MEQ, the T E Q and in the two Orals. The Initial Assessment of Clinical Skills: This generates marks in 11 areas of cognitive ability related to clinical practice. From these marks profiles were drawn showing individual and group performance; these were given to the participants at the end of the first day. Tutorial assessment: In an extended tutorial on the first day the practice p r o f o m was used informally by the tutor to assess individuals; attitudes in particular were explored. Thereafter, tutorials provided opportunities for the continuous assessment of participants by their respective tutor. Course Pathways and Individual Attention At the end of the first day an individual pathway through the programme was ‘prescribed’ for each participant. This pathway was derived from the results of assessment; participants wishing to attend other parts of the programme negotiated this through their tutor. O n average the participants chose one topic in addition to those prescribed. In addition to the topics covered by the programme, a number of participants required individual help with the following: Medical T e a c h Vol 6 No 3 1984
Day 1
Day 2
II
Day 3
Day 5
Day 4
p.m.
a.m.
p.m.
Briefing U-J
.-?I
. I -
::
I 5
Pm
-sm
.-C c
e
0)
fj
I m
.-m
Self-
learning
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
I-
Postcourse tasks
Course
dinner Briefing 9 o.m. self-completed Mendester
Rating sfale
1. TheRactiw
2. Epidemiology
4. Hypothesistesting 5. Management
3. Hypothesis formulation
6. Therapeutics
7. comrmnication
8. Revention 9. Medicine and S o c i i
Figure 1 . Outline programme of 5-day residmtial course showing the assessment and learning opportunities provided.
0 specific problems of learning or examination technique; 0 hostile attitudes towards the College; 0 minor psychological problems. The tutorial groups appeared to provide an important supportive role for these individuals.
MRCGP examination at re-sit. The proportion failing the written papers (15 per cent) and the orals (17.6 per cent) was lower than that for controls (24 per cent and 31.8 per cent respectively). However, the differences are not statistically significant.
Evaluation
Tables 2 and 3 show the mean scores and standard deviations of controls and of course participants in July and December 1980 for each part of the examination, and the percentage change. Table 4 compares the percentage changes of the two groups between examinations. Apart from the M C Q where the percentage change is similar,' the course participants improved their scores more than the controls in all parts of the examination. For the M E Q and Oral 1 these changes are statistically significant (P = C0.05).
SubJequnzt MRCGP Examination Performance We have compared the performance of the 20 course participants who were committed to re-sit the MRCGP examination immediately after the course, with the performance of a control group of 58 candidates. The control group comprises all candidates for the MRCGP examination who did not attend the course, who had failed at the same time (July 1980) as the course participants and who chose to re-sit at the earliest opportunity (December 1980). The only known difference between the two groups, apart from participation in the course, relates to the immediate re-sit rate. The control group represent 25 per cent of those failing the examination in July 1980; the study group 74 per cent of those attending the course. Overall performance Table 1 shows that a higher proportion of course participants (70per cent) than controls (52 per cent) passed the Medical Teacher Vol 6 No 3 1984
Change in Average Performance
Table 1. Comparison of the performance of course participants and a control group in the subsequent MRCGP examination (December 1980). Controls Passed MRCGP overall Failed written papers Failed oral exam (percentage of those reaching orals) Total
n = 30 (52%) n = 14 (24%)
Participants
n = 14 (70%) n = 3 (15%)
n - 14 (31.8%) n - 3 (17.6%) n320 n=58
103
Table 2. Comparison of the average marks in each part of the MRCGP examination of 58 controls in July 1980 with their marks in December 1980. The change in average marks for each part is also shown. Standard deviations are in parentheses.
July 1980: marks (per cent)
(s.d.) Dec. 1980: marks (per cent) (s.d.) Percentage change in marks (s.d.) Candidates
MCQ
MEQ
TEQ
Average
Oral 1
Oral 2
Total
41.50 (8.4) 46.40 (8.7) + 4.9 (6.5) 58
35.25 (5.1) 34.27 (4.3) - 0.98 (5.0) 58
44.37 (5.8) 40.17 (9.1) - 4.20 (8.6) 58
40.36 (4.14) 40.45 (5.5) + 0.09
50.52 (8.9) 56.37 (8.8) + 5.86 (11.1) 29
49.48 (8.3) 57.58 (9.3) +8.1 (13.3) 29
46.00 (1.9) 49.27 (3.5) + 3.27
58
29
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
Table 3. Comparison of the average marks in each part of the MRCGP examination of 20 course participants in July 1980 with their marks in December 1980. The change in average marks for each part is also shown. Standard deviations are in parentheses.
July 1980: marks (per cent)
(s.d.) Dec. 1980: marks (per cent) (s.d.) Percentage change in marks (s.d.)
Candidates
MCQ
MEQ
TEQ
Average
Oral 1
Oral 2
Total
42.20 (12.3) 47.00
35.10 (5.3) 37.13 (5.8) + 2.03 (6.4) 20
46.43 (6.8) 44.03 (9.2) - 2.40 (11.2) 20
41.25 (6.4) 42.72 (6.6) + 1.47
46.15 (7.9) 60.76 (9.3) + 14.6 (13.8) 13'
45.77 (6.1) 57.31 (7.8) + 11.5 (8.5) 13'
45.58 (3.1) 50.76 (4.3) +5.18
(10.3)
+ 4.8 (8.1) 20
20
13'
'Four candidates reaching the orals in December had failed to do so in July. Thus only 13 comparisons are possible.
Table 4. Comparison between controls and course participants of the average change in marks (per cent) gained in July and December 1980 for each part of the MRCGP examination.
Control group Participants
MCQ
MEQ
TEQ
Average
Oral 1
+ 4.9
- 0.98
- 4.20
+ 0.09
+ 5.86
+ 4.8
+ 2.03
- 2.40
+ 1.47
+ 14.6
Oral 2
Total
+8.1
+3.27 +5.18
+ 11.5
(PS0.05)
(Ps0.05)
The Predictive Abilib of Course Assessment Methods
Orals
The results of in-course assessment were scrutinized to see if they predicted subsequent performance in the M R C G P examination. The following methods were examined.
T h e in-course oral examination score was compared with the average score of two orals from the subsequent M R C G P examination. T h e correlation just failed to reach significance at the 95 per cent level.
The Initial Assessment of Clinical Skills
Manchester Rating Scales
The rank-order generated on the first day of the course was compared with a rank-order derived from the M R C G P written paper scores which followed the course. A highly significant correlation was demonstrated by Spearman's formula (P = < 0.001).
These self-assessment ratings showed no significant change between the beginning and end of the course, and did not correlate with subsequent examination performance.
Impressions of the Course Teachers The Modified Essay Question and Traditional Essay Question Papers
A similar rank-order correlation showed no significant correlation between pre-course M E Q or T E Q and subsequent examination performance.
I04
Failure in a professional examination has a substantial impact on a n established doctor. Therefore, loss of selfesteem is added to any educational difficulties. W e have failed to identify other characteristics typical of the group as a whole. Within it, individuals were
Medical Teacher Vol 6 No 3 1984
encountered with language/cultural problems; difficulties in the organization and presentation of knowledge; and educational passivity - a desire to be given ‘understanding’ rather than to seek it for themselves. The consequences of failure (and its variety of possible causes) appear to constitute a special need. This is unlikely to be met by courses designed to prepare firsttime candidates for MRCGP. The particular features of this course which we would defend are:
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
0 a high teacher to participant ratio; 0 the residential nature of the course; 0 its setting within the College building; 0 the support for individuals provided by small tutorial subgroups. Other groups of doctors who have failed this examination may exhibit different needs; it is likely that their needs too will be diverse. Without careful formal and informal assessment, these needs may remain unidentified. Providing appropriate help within a course is only possible with a broadly based and flexible programme, and a willingness by participant and teacher to return to basic issues. Thus examination content needs to be addressed as ‘how do I learn?’, and examination technique as ‘how do I present what I know?’
Discussion
We have published an account of this course for two reasons: to draw attention to the problems of doctors who fail a higher professional examination and to report a method of helping them which appears useful. The declared aim of the MRCGP examination is “to assess the ability of a candidate to carry unsupervised responsibility for the care of patients in general practice”.‘ Failure in the examination carries with it the implication for the doctor concerned that he is unfit for this role. Evidence from the group involved in this course suggests that failure is often perceived as a substantial professional set-back. Moreover it occurs for some at a critical time of transition from trainee to principal. At this time the doctor may lack the support needed to see failure in the examination from a perspective that is educationally constructive. As a group of general practice teachers we are concerned about the effects of failure in the MRCGP examination on professional colleagues and suggest that the College continues to recognize a responsibility to those affected. The course was well received by the participants - one of whom documented the experience;’ should it be repeated we hope that they will become involved as teachers. Implications Our results suggest that participation in this residential course was associated with improvement in examination performance greater than that shown by a comparable group who did not undergo the experience. We cannot, of course, know whether this improved performance reMedical Teacher Vol 6 No 3 1984
sulted from the course or from the type of participant recruited. Because of self-selection our course participants are likely to have been more highly motivated than the control group. However, they do not appear distinct in terms of their examination performance in July 1980 (Tables 2 and 3). Acknowledgements We thank the following at the Royal College of General Practitioners: Experimental Courses Study Group; Members of the Panel of Examiners; Professor Walker and Pauline Dallmeyer of the Membership Division; Dr Jack Norell, formerly Dean of Studies; Elizabeth Monk, administrator, Education Division.
References ‘Walker JH. The MRCGP Examination. Twenty-seumth Annual Report. Edinburgh: Royal College of General Practitioners, 1979. ’Wright HJ, Stanley IM, Webster J . The assessment of cognitive abilities in clinical medicine. McdEd 1983; 17: 31-38. ’Byrne PS, Freeman J. Postgraduate training for general practice: an assessment of aptitudes and abilities of trainee entrants. Br J Mcd Ed 1971; 5: 292-304. ‘Council RCGP. Obtaining and maintaining Membership: a Council discussion paper. J R CONGen Rut 1981; 31: 521-524. ’Tegner H. You can’t beat them, so join ’em - if you can. Pulse 1981; 41: 22.
Talking About Video-disks Individuals interested in the use of video-disks in health sciences education are invited to form an informal group to meet once or twice a year in order to exchange information and experience. The group could discuss matters such as the problems of production, the use of available authoring languages, the equipment available for playback and computing, and interfacing between the two. There could be discussion as to whether disks should be designed as resource material or with specific objectives in mind. The place of evaluation of the use of this new teaching and training medium could also be considered. Those interested might include medical teachers at undergraduate and postgraduate level, nursing and other paramedical teachers, and representatives of the pharmaceutical industry. The group could meet at different centres. Anyone interested in forming a group along the lines suggested should contact Dr D. G. Jameson, Department of Medical Physics and the Institute of Nuclear Medicine, Middlesex Hospital Medical School, Cleveland Street, London W1P 6DB.
105
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
Helping Those Who Fail the MRCGP Examination A short residential course for a group of general practitioners who had failed the MRCGP examination is described. Evaluation of the course includes a comparison of subsequent examination performance by participants with that of a control group. Overall, and in relation to four of the five parts of the examination, the course participants performed better than the control group. For two of the five parts of the examination the differences between course participants and controls were statistically significant. The impact of examination failure on this group of doctors, and their diverse educational needs are emphasized. The idea of the course for those who had failed the examination for membership of the Royal College of General Practitioners developed from the followingobservations: 0 The significant proportion of candidates failing the examination.' 0 The lack of information on why candidates fail and hence what might be done to help them.
Moreover, it appeared to us desirable that the College should recognize a responsibility to those who had tried unsuccessfully to become Members. Therefore a course was proposed at College headquarters to provide assessment and educational opportunities for candidates who had failed a recent examination for Membership. The course was provided in October 1980, with the support of the Experimental Courses Study Group of the College. It was residential and lasted five days. Aims 0 T o identify the characteristics of a group of candidates who had failed the MRCGP examination. 0 To involve them in a variety of educational activities appropriate to their needs.
D.Adshead, BA. MBBS. General Practitioner, Tordmorden; W.
Allen,
MB. CH.R. MRCGP. General Practitioner, Leeds; J . Bahrami, MB, CH.B. D.OBST.HCOC;. MRCOC, MRCGP, General Practitioner, Bradford; A. Belton, B S : . MB. CH.B, DRCOG. FRCGP, General Practitioner, Keighley; P. Heywood, MB. CH.B. D.CH, D.OBST.RCPC. FRCCP. General Practitioner, keds; S. Jenkinson, MB,B.CHIR. D.CH. MRCGP. General Practitioner, Baildon; A. Lewis, MA. MB. CH.B, D.CH, D.OBST.RCOG. MRCGP. General Practitioner. Hayle; I. Stanley, MB. CH.B, MRCCP. FRCCP, Lecturer in General Practice, University of Leeds; C . Varnavides, B.SC, MB, CH.B. D.OBST.H(:OX. MRCGP. General Practitioner, Leeds. Reprints requests to
I. Stanley, Dept. of Community Medicine and General Practice, Clinical Sciences Building, St James's Hospital, Leeds LS9 7TF.
Medical Teacher Vol 6 No 3 1984
0 To encourage patterns of learning appropriate to future success in the MRCGP examination and to continuing professional education.
Method Course design
In planning a course specifically for individuals who had failed this examination we were conscious that failure was likely to represent the outcome of a wide variety of factors. Moreover, as such individuals discover, feedback on examination performance is limited in its usefulness, consisting simply of individual and average scores for each of the five parts. A low individual score may be interpreted in a number of ways; in the Multiple Choice Question (MCQ) paper, for example, it may be seen as evidence of limited factual knowledge or difficulty with the conventions of this type of question. Even greater difficulty arises in the interpretation by candidates or their would-be advisors of scores achieved on essay questions or in orals. The first problem for such a course, then, is one of diagnosis; what are the reasons underlying failure by an individual in the examination? With this in mind a range of assessment methods was chosen in three broad areas: cognitive abilities, professional skills in the context of practice, and attitudes to learning. Within these areas both formal and informal techniques were used to identify problems for individuals in their approach to learning and in their organization and presentation of knowledge. Some of these techniques were chosen because of their ability to provide a detailed analysis within a general area, others because they formed part of the examination and were likely to reveal technical shortcomings. In consideringthe balance to be struck by such a course between examination content and technique, and bearing in mind the stated aims, the former was favoured. Nevertheless, it was recognized that technique might be a significant factor in the failure of individuals and, for the majority, an expected part of such a course. Both content and technique were to be provided within a programme of learning options, entrance to most of the options being linked to one or more assessment techniques. A two-tier structure was chosen. Each participant was to be allocated a tutor and given the opportunity to meet with him daily in the same small group. At other times participants were to be regrouped on the basis of 101
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
assessment and involved in appropriate small group learning. A staff participant ratio of 1:3 was considered necessary to provide adequate time with tutors, teachers and organizers within this structure. A five-day residential format was considered to be acceptable to working general practitioners and to provide a time-scale within which the aims of the course might be realized. The headquarters of the Royal College of General Practitioners was the venue chosen for the course. This choice was felt to be psychologically important to the participants, and consistent with the concern showed by the College for the group.
0 A programme of opportunities for learning about examination content and technique (Figure 1). 0 Tutorials. On the first day participants were allocated at random to a course tutor in groups of about five. Thereafter, these groups met the same tutor for at least one hour per day. 0 An oral examination. All but one participant (who declined the opportunity) were involved in a 30-minute oral examination closely simulating those of the MRCGP examination, and provided by a number of College Examiners. The practice log formed the basis of this exercise.
Recruitment
Assessment
Those candidates who failed the MRCGP examination in July 1980 and who subsequently sought details of their marks were sent a letter about the course (180 letters sent). Of these, 35 expressed interest and 27 attended the course. Of the 27 participants, 23 were principals, two were trainees and two were working outside general practice.
The Modified Essay Question paper: There were nine pages; each page had been constructed to assess the participant in one area of knowledge or skills provided for in the programme (Figure 1). The course teacher who was responsible for preparing a topic marked the appropriate page from each participant.
The Course Pre-course Work About 2 weeks before the course, participants were asked to complete the following: 0 A Modified Essay Question (MEQ) paper and a Traditional Essay Question (TEQ) paper under ‘examination conditions’. 0 A number of practice-related tasks. A short clinical history was provided, and the participant was asked to respond with an appropriate entry in the medical records and with a referral letter to an orthopaedic consultant. In addition a short account of patient follow-up policy in the participant’s practice was requested. 0 A practice proforma seeking information on the participant’s professional experience; practice organization; opportunities for continuing professional education, and so on. 0 A practice log listing 50 consecutive recent consultations and 10 patients under continuing care during the preceding three months. The M E Q and TEQ were returned by participants for marking before the course. The remaining items were brought with them to the course along with details of their marks in previous attempts at the MRCGP examination.
In-course Work 0 The initial assessment of clinical skills (IACS)’ comprising three written papers, each lasting 50 minutes, was taken by the participants on the first morning of the course. 0 Manchester rating scales (MRS)’ were completed by the participants on the first day and the last day.
102
The Traditional Essay Question paper: Answers to essay questions from all participants were marked by one course teacher responsible for teaching this aspect of examination technique. Practice-related tasks: These were assessed on the first morning of the course by six tutors working together. By comparing the material submitted, the tutors identified those participants with performance below average for the group. Prior pct-formance in the M R C G P examination: Marks were available from each participant showing his performance in the Multiple Choice Question (MCQ) paper, the MEQ, the T E Q and in the two Orals. The Initial Assessment of Clinical Skills: This generates marks in 11 areas of cognitive ability related to clinical practice. From these marks profiles were drawn showing individual and group performance; these were given to the participants at the end of the first day. Tutorial assessment: In an extended tutorial on the first day the practice p r o f o m was used informally by the tutor to assess individuals; attitudes in particular were explored. Thereafter, tutorials provided opportunities for the continuous assessment of participants by their respective tutor. Course Pathways and Individual Attention At the end of the first day an individual pathway through the programme was ‘prescribed’ for each participant. This pathway was derived from the results of assessment; participants wishing to attend other parts of the programme negotiated this through their tutor. O n average the participants chose one topic in addition to those prescribed. In addition to the topics covered by the programme, a number of participants required individual help with the following: Medical T e a c h Vol 6 No 3 1984
Day 1
Day 2
II
Day 3
Day 5
Day 4
p.m.
a.m.
p.m.
Briefing U-J
.-?I
. I -
::
I 5
Pm
-sm
.-C c
e
0)
fj
I m
.-m
Self-
learning
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
I-
Postcourse tasks
Course
dinner Briefing 9 o.m. self-completed Mendester
Rating sfale
1. TheRactiw
2. Epidemiology
4. Hypothesistesting 5. Management
3. Hypothesis formulation
6. Therapeutics
7. comrmnication
8. Revention 9. Medicine and S o c i i
Figure 1 . Outline programme of 5-day residmtial course showing the assessment and learning opportunities provided.
0 specific problems of learning or examination technique; 0 hostile attitudes towards the College; 0 minor psychological problems. The tutorial groups appeared to provide an important supportive role for these individuals.
MRCGP examination at re-sit. The proportion failing the written papers (15 per cent) and the orals (17.6 per cent) was lower than that for controls (24 per cent and 31.8 per cent respectively). However, the differences are not statistically significant.
Evaluation
Tables 2 and 3 show the mean scores and standard deviations of controls and of course participants in July and December 1980 for each part of the examination, and the percentage change. Table 4 compares the percentage changes of the two groups between examinations. Apart from the M C Q where the percentage change is similar,' the course participants improved their scores more than the controls in all parts of the examination. For the M E Q and Oral 1 these changes are statistically significant (P = C0.05).
SubJequnzt MRCGP Examination Performance We have compared the performance of the 20 course participants who were committed to re-sit the MRCGP examination immediately after the course, with the performance of a control group of 58 candidates. The control group comprises all candidates for the MRCGP examination who did not attend the course, who had failed at the same time (July 1980) as the course participants and who chose to re-sit at the earliest opportunity (December 1980). The only known difference between the two groups, apart from participation in the course, relates to the immediate re-sit rate. The control group represent 25 per cent of those failing the examination in July 1980; the study group 74 per cent of those attending the course. Overall performance Table 1 shows that a higher proportion of course participants (70per cent) than controls (52 per cent) passed the Medical Teacher Vol 6 No 3 1984
Change in Average Performance
Table 1. Comparison of the performance of course participants and a control group in the subsequent MRCGP examination (December 1980). Controls Passed MRCGP overall Failed written papers Failed oral exam (percentage of those reaching orals) Total
n = 30 (52%) n = 14 (24%)
Participants
n = 14 (70%) n = 3 (15%)
n - 14 (31.8%) n - 3 (17.6%) n320 n=58
103
Table 2. Comparison of the average marks in each part of the MRCGP examination of 58 controls in July 1980 with their marks in December 1980. The change in average marks for each part is also shown. Standard deviations are in parentheses.
July 1980: marks (per cent)
(s.d.) Dec. 1980: marks (per cent) (s.d.) Percentage change in marks (s.d.) Candidates
MCQ
MEQ
TEQ
Average
Oral 1
Oral 2
Total
41.50 (8.4) 46.40 (8.7) + 4.9 (6.5) 58
35.25 (5.1) 34.27 (4.3) - 0.98 (5.0) 58
44.37 (5.8) 40.17 (9.1) - 4.20 (8.6) 58
40.36 (4.14) 40.45 (5.5) + 0.09
50.52 (8.9) 56.37 (8.8) + 5.86 (11.1) 29
49.48 (8.3) 57.58 (9.3) +8.1 (13.3) 29
46.00 (1.9) 49.27 (3.5) + 3.27
58
29
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
Table 3. Comparison of the average marks in each part of the MRCGP examination of 20 course participants in July 1980 with their marks in December 1980. The change in average marks for each part is also shown. Standard deviations are in parentheses.
July 1980: marks (per cent)
(s.d.) Dec. 1980: marks (per cent) (s.d.) Percentage change in marks (s.d.)
Candidates
MCQ
MEQ
TEQ
Average
Oral 1
Oral 2
Total
42.20 (12.3) 47.00
35.10 (5.3) 37.13 (5.8) + 2.03 (6.4) 20
46.43 (6.8) 44.03 (9.2) - 2.40 (11.2) 20
41.25 (6.4) 42.72 (6.6) + 1.47
46.15 (7.9) 60.76 (9.3) + 14.6 (13.8) 13'
45.77 (6.1) 57.31 (7.8) + 11.5 (8.5) 13'
45.58 (3.1) 50.76 (4.3) +5.18
(10.3)
+ 4.8 (8.1) 20
20
13'
'Four candidates reaching the orals in December had failed to do so in July. Thus only 13 comparisons are possible.
Table 4. Comparison between controls and course participants of the average change in marks (per cent) gained in July and December 1980 for each part of the MRCGP examination.
Control group Participants
MCQ
MEQ
TEQ
Average
Oral 1
+ 4.9
- 0.98
- 4.20
+ 0.09
+ 5.86
+ 4.8
+ 2.03
- 2.40
+ 1.47
+ 14.6
Oral 2
Total
+8.1
+3.27 +5.18
+ 11.5
(PS0.05)
(Ps0.05)
The Predictive Abilib of Course Assessment Methods
Orals
The results of in-course assessment were scrutinized to see if they predicted subsequent performance in the M R C G P examination. The following methods were examined.
T h e in-course oral examination score was compared with the average score of two orals from the subsequent M R C G P examination. T h e correlation just failed to reach significance at the 95 per cent level.
The Initial Assessment of Clinical Skills
Manchester Rating Scales
The rank-order generated on the first day of the course was compared with a rank-order derived from the M R C G P written paper scores which followed the course. A highly significant correlation was demonstrated by Spearman's formula (P = < 0.001).
These self-assessment ratings showed no significant change between the beginning and end of the course, and did not correlate with subsequent examination performance.
Impressions of the Course Teachers The Modified Essay Question and Traditional Essay Question Papers
A similar rank-order correlation showed no significant correlation between pre-course M E Q or T E Q and subsequent examination performance.
I04
Failure in a professional examination has a substantial impact on a n established doctor. Therefore, loss of selfesteem is added to any educational difficulties. W e have failed to identify other characteristics typical of the group as a whole. Within it, individuals were
Medical Teacher Vol 6 No 3 1984
encountered with language/cultural problems; difficulties in the organization and presentation of knowledge; and educational passivity - a desire to be given ‘understanding’ rather than to seek it for themselves. The consequences of failure (and its variety of possible causes) appear to constitute a special need. This is unlikely to be met by courses designed to prepare firsttime candidates for MRCGP. The particular features of this course which we would defend are:
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/02/15 For personal use only.
0 a high teacher to participant ratio; 0 the residential nature of the course; 0 its setting within the College building; 0 the support for individuals provided by small tutorial subgroups. Other groups of doctors who have failed this examination may exhibit different needs; it is likely that their needs too will be diverse. Without careful formal and informal assessment, these needs may remain unidentified. Providing appropriate help within a course is only possible with a broadly based and flexible programme, and a willingness by participant and teacher to return to basic issues. Thus examination content needs to be addressed as ‘how do I learn?’, and examination technique as ‘how do I present what I know?’
Discussion
We have published an account of this course for two reasons: to draw attention to the problems of doctors who fail a higher professional examination and to report a method of helping them which appears useful. The declared aim of the MRCGP examination is “to assess the ability of a candidate to carry unsupervised responsibility for the care of patients in general practice”.‘ Failure in the examination carries with it the implication for the doctor concerned that he is unfit for this role. Evidence from the group involved in this course suggests that failure is often perceived as a substantial professional set-back. Moreover it occurs for some at a critical time of transition from trainee to principal. At this time the doctor may lack the support needed to see failure in the examination from a perspective that is educationally constructive. As a group of general practice teachers we are concerned about the effects of failure in the MRCGP examination on professional colleagues and suggest that the College continues to recognize a responsibility to those affected. The course was well received by the participants - one of whom documented the experience;’ should it be repeated we hope that they will become involved as teachers. Implications Our results suggest that participation in this residential course was associated with improvement in examination performance greater than that shown by a comparable group who did not undergo the experience. We cannot, of course, know whether this improved performance reMedical Teacher Vol 6 No 3 1984
sulted from the course or from the type of participant recruited. Because of self-selection our course participants are likely to have been more highly motivated than the control group. However, they do not appear distinct in terms of their examination performance in July 1980 (Tables 2 and 3). Acknowledgements We thank the following at the Royal College of General Practitioners: Experimental Courses Study Group; Members of the Panel of Examiners; Professor Walker and Pauline Dallmeyer of the Membership Division; Dr Jack Norell, formerly Dean of Studies; Elizabeth Monk, administrator, Education Division.
References ‘Walker JH. The MRCGP Examination. Twenty-seumth Annual Report. Edinburgh: Royal College of General Practitioners, 1979. ’Wright HJ, Stanley IM, Webster J . The assessment of cognitive abilities in clinical medicine. McdEd 1983; 17: 31-38. ’Byrne PS, Freeman J. Postgraduate training for general practice: an assessment of aptitudes and abilities of trainee entrants. Br J Mcd Ed 1971; 5: 292-304. ‘Council RCGP. Obtaining and maintaining Membership: a Council discussion paper. J R CONGen Rut 1981; 31: 521-524. ’Tegner H. You can’t beat them, so join ’em - if you can. Pulse 1981; 41: 22.
Talking About Video-disks Individuals interested in the use of video-disks in health sciences education are invited to form an informal group to meet once or twice a year in order to exchange information and experience. The group could discuss matters such as the problems of production, the use of available authoring languages, the equipment available for playback and computing, and interfacing between the two. There could be discussion as to whether disks should be designed as resource material or with specific objectives in mind. The place of evaluation of the use of this new teaching and training medium could also be considered. Those interested might include medical teachers at undergraduate and postgraduate level, nursing and other paramedical teachers, and representatives of the pharmaceutical industry. The group could meet at different centres. Anyone interested in forming a group along the lines suggested should contact Dr D. G. Jameson, Department of Medical Physics and the Institute of Nuclear Medicine, Middlesex Hospital Medical School, Cleveland Street, London W1P 6DB.
105
