1
SELF ASSESSMENT
I
This series provides teachers with the opportunity of assessing their knowledge and understanding of some important aspects of medical education. Only after you have answered the questions below should you read the comments on page 268.
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/01/15 For personal use only.
F. R . A b batt, B .SC,ED.M, is Senior Lecturer, and Joyce A b batt, BA, B Course Administrator, Centref o r Medical Education, The University of Dundee, Ninewells Hospital and Medical School, Dundee, DDl SSY, UK.
Many teaching departments in medical schools use a variety of methods for assessing their students. This diversity is often desirable because, for example, multiple choice questions and clinical examinations assess different types of skill. However, the use of the different methods does present some problems which are especially important because they are only rarely recognized. Take as an example a department which uses continuous assessment throughout the course, plus a final MCQexamination and a clinical assessment. Each of these parts is marked out of 100, the overall pass mark is 50 per cent and it has been agreed that each of the three assessments ‘should have equal weight’ in determining the overall result. Given this information and the following marks of five students (A, B, C, D, E) selected from the 120 on the course, try answering the following questions. The answers appear on page 268. Total
A
B
C
D ~
Multiple choice questions Clinical assessment Continuous assessment
100 100 100
31 60 58
60 48 46
55 52 56
E
Mean*
SD*
84 54 56
58 61 62
20 4 8
~~~
60 63 65 ~~
*The mean and standard deviations are given for the complete group of 120 students.
Question 1 To give equal weight to the three parts, the overall percentage can be calculated by simply adding together the three marks out of 100 and dividing by 3.
True/False
Question 2 To give equal weight to the three parts you must take into account the spread of the scores in each part (the standard deviations).
True / False
Question 3 To give equal weight to the three parts you must take into account the mean mark for each of the three parts.
True/False
Question 4 When each part has been given equal weight, student E should have a higher mark than student D.
True/False
Question 5 When each part has been given equal weight, student A should have a higher mark than student B.
True/False
Medical Teacher V o l l No 5 1979
267
Answers 1. False
2. True
3. False
4. False
5.True
Comments Question 1 False. If you simply add the scores, the MCQexamination results will dominate the others because the spread of marks is so much greater. In other words, an average performance on the MCQ will more than compensate for a very bad performance on the other two parts. Question 2 True. This is essential.
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/01/15 For personal use only.
Question 3 False. The mean score does not matter at all when giving weights. Questions 4 and 5 False and true, respectively. The overall marks should be A 56.0 not A 49.7 B 48.8 B 51.3
c
D
E
53.5 62.9 58.1
c
D
E
54.3 62.3 64.7
The second set of scores would be obtained if you followed the wrong procedure of simply adding the scores and dividing by 3.
Discussion The first point must be that obtaining an overall mark from examinations consisting of several parts is a realistic and frequent problem. Within departments different methods of assessment are used, and they often do have widely differing spreads of marks (standard deviation). This is also true within an examination, where different parts are marked by different examiners. The effect is that parts of examinations or subjects which are important from educational or medical points of view may have virtually no influence on the overall scores of a student. What is the Remedy? To give equal weight to different parts of an assessment, the scores for the different parts should first be multiplied by a factor which is in inverse ratio to the standard deviation for the set of scores. In this example the standard deviations are in the ratio 5:1:2. Therefore the MCQscores should be multiplied by 1, the clinical scores by 5 and the continuous assessment by 2.5. This will make the marks look like: Total
A
B
C
D
E
Mean
SD
55 260 140 455
60 315 160 535
84 270 140 494
58 350 155
20 20 20
53.5
62.9
5 k1
Multiple choice questions Clinical assessment Continuous assessment Total
100 500 250 850
145 476
60 240 115 415
%
100
56.0
48.8
268
31
300
Medical Teacher Vol 1 No 5 1979
The result of this multiplication is that each set of scores now has the same standard deviation, so when they are added the different parts wiZZ have equal weight. This gives a total out of 850, so the total scores for each student must be multiplied by 100/850 or 0.1 176 to give a percentage score. What has happened to the pass mark? Nothing at all. If the overall pass mark for each part was 50 per cent the overall pass mark will remain at 50. Summary To give equal weight to different sets of scores:
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/01/15 For personal use only.
1. Multiply each set of scores by a factor in inverse proportion to its standard deviation. 2. Add together the amended scores. 3. Multiply the resulting total by 100, divided by the new maximum possible score.
Must you wait until all scores have been obtained before deciding on the multiplying factors? Probably not. If you look at the scores from previous years on the different parts of the exams, you will gain a good idea of the likely standard deviations. Use these figures as a rough guide. Even if they are not precisely accurate, they will help you to give a much better overall mark.
M E D I C A L & SCIENTIFIC
BOOKSHOP H. K. LEWIS hold a most comprehensive stock on all branches of MEDICINE and SCIENCE Catalogues will be sent on request.
Please state particular interests
MEDICAL & SCIENTIFIC LIBRARY Annual Subscription from f7.50 (Reduced rates for multiple subscription) Prospectus post free on request (Available in U.K. only)
H. K. LEWIS & Co Ltd., 136 GOWER STREET, LONDON WClE 6Bs Telephone: 01-387 4282
Medical Teacher Vol 1 No 5 1979
269
SELF ASSESSMENT
I
This series provides teachers with the opportunity of assessing their knowledge and understanding of some important aspects of medical education. Only after you have answered the questions below should you read the comments on page 268.
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/01/15 For personal use only.
F. R . A b batt, B .SC,ED.M, is Senior Lecturer, and Joyce A b batt, BA, B Course Administrator, Centref o r Medical Education, The University of Dundee, Ninewells Hospital and Medical School, Dundee, DDl SSY, UK.
Many teaching departments in medical schools use a variety of methods for assessing their students. This diversity is often desirable because, for example, multiple choice questions and clinical examinations assess different types of skill. However, the use of the different methods does present some problems which are especially important because they are only rarely recognized. Take as an example a department which uses continuous assessment throughout the course, plus a final MCQexamination and a clinical assessment. Each of these parts is marked out of 100, the overall pass mark is 50 per cent and it has been agreed that each of the three assessments ‘should have equal weight’ in determining the overall result. Given this information and the following marks of five students (A, B, C, D, E) selected from the 120 on the course, try answering the following questions. The answers appear on page 268. Total
A
B
C
D ~
Multiple choice questions Clinical assessment Continuous assessment
100 100 100
31 60 58
60 48 46
55 52 56
E
Mean*
SD*
84 54 56
58 61 62
20 4 8
~~~
60 63 65 ~~
*The mean and standard deviations are given for the complete group of 120 students.
Question 1 To give equal weight to the three parts, the overall percentage can be calculated by simply adding together the three marks out of 100 and dividing by 3.
True/False
Question 2 To give equal weight to the three parts you must take into account the spread of the scores in each part (the standard deviations).
True / False
Question 3 To give equal weight to the three parts you must take into account the mean mark for each of the three parts.
True/False
Question 4 When each part has been given equal weight, student E should have a higher mark than student D.
True/False
Question 5 When each part has been given equal weight, student A should have a higher mark than student B.
True/False
Medical Teacher V o l l No 5 1979
267
Answers 1. False
2. True
3. False
4. False
5.True
Comments Question 1 False. If you simply add the scores, the MCQexamination results will dominate the others because the spread of marks is so much greater. In other words, an average performance on the MCQ will more than compensate for a very bad performance on the other two parts. Question 2 True. This is essential.
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/01/15 For personal use only.
Question 3 False. The mean score does not matter at all when giving weights. Questions 4 and 5 False and true, respectively. The overall marks should be A 56.0 not A 49.7 B 48.8 B 51.3
c
D
E
53.5 62.9 58.1
c
D
E
54.3 62.3 64.7
The second set of scores would be obtained if you followed the wrong procedure of simply adding the scores and dividing by 3.
Discussion The first point must be that obtaining an overall mark from examinations consisting of several parts is a realistic and frequent problem. Within departments different methods of assessment are used, and they often do have widely differing spreads of marks (standard deviation). This is also true within an examination, where different parts are marked by different examiners. The effect is that parts of examinations or subjects which are important from educational or medical points of view may have virtually no influence on the overall scores of a student. What is the Remedy? To give equal weight to different parts of an assessment, the scores for the different parts should first be multiplied by a factor which is in inverse ratio to the standard deviation for the set of scores. In this example the standard deviations are in the ratio 5:1:2. Therefore the MCQscores should be multiplied by 1, the clinical scores by 5 and the continuous assessment by 2.5. This will make the marks look like: Total
A
B
C
D
E
Mean
SD
55 260 140 455
60 315 160 535
84 270 140 494
58 350 155
20 20 20
53.5
62.9
5 k1
Multiple choice questions Clinical assessment Continuous assessment Total
100 500 250 850
145 476
60 240 115 415
%
100
56.0
48.8
268
31
300
Medical Teacher Vol 1 No 5 1979
The result of this multiplication is that each set of scores now has the same standard deviation, so when they are added the different parts wiZZ have equal weight. This gives a total out of 850, so the total scores for each student must be multiplied by 100/850 or 0.1 176 to give a percentage score. What has happened to the pass mark? Nothing at all. If the overall pass mark for each part was 50 per cent the overall pass mark will remain at 50. Summary To give equal weight to different sets of scores:
Med Teach Downloaded from informahealthcare.com by University of Otago on 01/01/15 For personal use only.
1. Multiply each set of scores by a factor in inverse proportion to its standard deviation. 2. Add together the amended scores. 3. Multiply the resulting total by 100, divided by the new maximum possible score.
Must you wait until all scores have been obtained before deciding on the multiplying factors? Probably not. If you look at the scores from previous years on the different parts of the exams, you will gain a good idea of the likely standard deviations. Use these figures as a rough guide. Even if they are not precisely accurate, they will help you to give a much better overall mark.
M E D I C A L & SCIENTIFIC
BOOKSHOP H. K. LEWIS hold a most comprehensive stock on all branches of MEDICINE and SCIENCE Catalogues will be sent on request.
Please state particular interests
MEDICAL & SCIENTIFIC LIBRARY Annual Subscription from f7.50 (Reduced rates for multiple subscription) Prospectus post free on request (Available in U.K. only)
H. K. LEWIS & Co Ltd., 136 GOWER STREET, LONDON WClE 6Bs Telephone: 01-387 4282
Medical Teacher Vol 1 No 5 1979
269
