Midrciii kdmrrotr lc)c)2, 26, 58-02
Structured short-answer questions: an alternative examination method R H WEBBEH Lotidoti
School of Hy'qierie arid Tropical ,Lledicirie, Iitrrvrr\ity o f Lorrdorr
the considerable value of MCQs, a structured short-answer assessnietit method has been developed.
Summary. A structured short answer (SSA) examination has been developed for the MSc. C o m m u n i t y Health in Developing Countries, t o test competence in international public health. Questions can test knowledge o r aim to assess problem-solving ability, but have the advantage that the candidates construct their o w n answer (rather than choosing from p r e f o r m e d answers in an MCQ), yet encourage sufficient precision for answers to be easily marked. T h e SSA more closely reflects participatory teaching and selfdirected learning. Illustrated material is widely used.
MCQ method I n the examination process a certain amount of time is required by the examiners in setting and
marking papers. In the conventional essay written answer, setting the question takes very little time, but marking a considerable amount of time. T h c MCQ has become increasingly popular because it shifts the time allocation to the preparation o f questions and minimizes the time for marking. A large number ofcandidates can be tested at one time and the markina is so easy that automatic readers can be used. T h e MCQ format classically asks a question and supplies five alternative answers, generally one, and occasionally t w o or more being correct. Sometimes the question is asked in the negative, so that the incorrect answer is the one to score. Double negatives should be avoided, but they often do creep in causing particular problems in understanding for the person whose first language is not English. With five altcrnativc answers there is an opportunity for guessing which has been studied o n several occasions scc Cox (1976),Jolly (1976) and Fleming (1988). There has been a move in teaching practice away from the didactic lecture to more participatory teaching and self-directed learning. Students are encouraged to work i n groups or alone, formulating their o w n solutions to problems. While the acquisition ofknowledge is essential, it is the interpretation and use of this knowledge that is the more valuable skill. T h c examination process needs to test this ability, rather thanjust knowledge. T h e M C Q can be modified for
Key words: educational measurement/ *methods; 'cduc, med, grad; public health; community medicinc/educ; London
Introduction T h e multiple choice question (MCQ) method of examining has become popular due to its economy o f s a l e , ease o f marking and consistcncy. I t allows a large number ofquestions to be answered, testing a wide range of factual material. B y providing a choice o f alternatives it reniovcs the necessity to construct a reply and answers can be obtained by luck without the candidate really knowing the correct solution. Candidates whose first language is not English often have difficulty in intcrpreting the question, particulary when negatives are used. M C Q does not lend itself so well to problem-solving, which is a far more real-life situation than having t o choose between a number of prc-formed alternatives. With these disadvantages, but recognizing Corrrhpondcncc: Dr K . H. Wrbbcr, London School of Hygiene and Tropical Medicine, University of London, Kcppel Street, London W C l E 7HT. U K .
58
Structured short-answer questions problem-solving, but alternative solutions are still supplied rather than encouraging the real-life situation of solving the problem in the individual’s own way. Ferland et a / . (1987) found no difference in M C Q s which set out to measure higher cognitive levels and suggested finding other instruments. The structurcd short-answer question endeavours to do this. Structured short answer (SSA)
Short answers have been found to be more reliable than essay type questions (Mowbray 81 Davies 1967). They are often used in written examinations to break down an essay question to ensure that the candidate attempts different aspects ofthe problem. Bull (1956) found that the shorter the questions, the greater the correlation and marking efficiency between examiners. Wakcford & Roberts (1979) found the interexaminer reliability of short essay questions to be good. This process can be extended so that the question asked requires only a one- or two-line reply. A simple example to test knowledge could be: q: What species of mosquito is illustrated? a: Aedes aegypti. q: Where does it commonly breed? a. I n water containers, water tanks, old tyres and other collections of water, close to human habitation. q: What are the main diseases transmitted by this mosquito? a: Dengue and yellow fever. q: Briefly describe (no more than 4 lines each) two methods of controlling this mosquito. a: (1) I n a dengue epidemic, using a knock-down insecticide by ULV method f r o m a knap-sack sprayer, mounted on a vehicle or by aircraft. a: (2) Burying tin cans, cutting holes in tyres and making sure there are no unwanted collections of water. Covering water tanks with netting using a mammalian-safe insecticide such as pyrimethos methyl (Abate) or biological method (dragon j y larvae or larvivorousfish) in the water container. (The question paper would also include a line drawing of the mosquito). Specimen answers are prepared a t the same time the question is constructed and made avail-
59
able to the examiners. Formulating the question in this way is easier than thc MCQ (where several alternative false answers have to be invented) and having a prepared answer makes marking a rapid process. The same question could be set as an M C Q but it would need four to five separate questions to elicit the same amount of factual material. Using a fill-in diagram can elicit very precise answers, as shown in Fig. 1. Illustrated questions can be very valuable in tcsting skills ofinterpretation, such as reading graphs. The SSA is particularly effective for problem-solving questions such as the epidemiology example illustrated in Fig. 2 and Table 1. q: The graph in Fig. 2 is constructed from the data in Table 1. The doctor presenting these data claimed that mass chemoprophylaxis had controlled the cholera epidemic, do you agree? a: No.
q: Why does the graph have two peaks? a: T h e first peak represents the epidemic that occurred in Limbo (peak in M a y ) and the second the epidemic in Birin (peak inJune). q: After each peak why do the number of cases decline? a. T h e pool of susceptibles has been used up (the downward slope ofthe epidemic curve). The SSA lends itselfmore easily to the physical and biological sciences (including medicine), but can also be used in the social sciences. Discussion The examination system is often considered a necessary but onerous task that is best overcome by the easiest possible method. Seen from the examincrs’ point of vicw, such methods as the M C Q reduce the load by a considerable extent. On thc other hand, examinees often strongly dislike the M C Q . The M C Q asks for provision of a very limited piece of knowledge and although candidates may have reviewed the subject material, if they are not able to answer this precise part, they are judged as not knowing the subject (Cox 1982). In an examination format where they can construct their own answer, they at least have the opportunity to use the knowledge they have of the subject to get close to the answer. The gambling nature of the M C Q is
K . H . Wchber
fill i n t h e b l a n k s w i t h t h e f i v e E P I v a c c i n e s
The figures b e l o w show t h e b e h a v i o u r of d t o x o i d i n a c t i v a t e d by f r e e z i n g , o f an a c t i v e one, n e v e r f r o z e n . Which onescan be u s e d ?
dnd
Immediately a l t e r shakmg
B
A smooth
and cloudy
-
Not smooth -You
can see granular
panicles
30 minutes a! l e r shaking
from WHO ( 1 9 8 9 )
Figure 1. A knowledge question constructed as a fill-in diagram. T h e candidate needs to know the five vaccines used in the expanded programme of immunization (EPI) and whether they arc stored frozen or at 0°C to + 8 T . In the srcond part, A and C are the correct answers.
often resented and problems with understanding the question have already been referred to. Assessment is a valuable and powerful learning process and more thought needs to be given to designing examinations that will encourage this process. The SSA sets out to test candidates with the kind of problems likely to be faced in their career o r profession. In this way the candidate can see the reality o f the situation rather than having to learn vast amounts of factual material just to get through the examinations.
The design of SSA questions IS to elicit a progression of information. The first question asks basic matcrial which is built on step by step. This means that the first part has only one correct answer, but as it bccomcs more searching it allows greater latitude for a partially correct answer, even if the first part(s) have been answered incorrectly. For example, in the first question, the wrong mosquito may be identified, but by knowing methods of vector control there is a good likelihood that part of the last section of
Structured short-answer questions
61
No. of cases 60 r
50 .
40
30
20!
15 20 25 30 5
10 15 20 25 30 4
April
9
14 19 24 29 4
9
14 19 24 29
July
June
May
Figure 2. Cholera cases, 15 April - 2 August.
Table 1. Occurrence o f cholera cases between 15 April and 2 August in the villages of Limbo, Simb, Pambe and Birin Infected villages Data
Limbo
Simb
Pambe
15- 19 20 - 24 25 - 29 30- 4
1 5 16 20
1 2 3 4
0 0 0 1
0 0 0 0
19 25
May
5- 9 10- 14 IS- 19 20 - 24 25 - 29 30- 3
27 38 41 39 33 18
2 3 5 3 1 1
2 2 3 1 0 1
0 0 0 1 4 12
31 43 49 44 38 32
June
4- 8 9 - 13 14- 18 19 - 23 24 - 28 29- 3
16 7 2 0 1 2
0 1 0 1 0
1 0 2 1 1 0
7 21 30 29 17 12
24 29 35 30 20 14
July
4- 8 9 - 13 14- 18 19 - 23 24 - 28
1 0 0 0 0
0 0 0 0 0
1 0 0 0 0
10 6 4 2 1
12 6 4 2 1
August
29-
0
0
0
1
1
267
28
16
157
468
April
Totals
-
2
1
Birin
Total 2 7
f2
I?. H .
the question c x i b e attempted. When the qucst i o t i I \ scorcd, the nuniber o f niJrks for c ~ c h p o i n t c m bc q r e e d before marking b y the cxaiiiincrs, s o presenting consistent)-. Illu,tratcti d a t d art' commonly ~ised.as i n the csaniplcs givcn. which is ;i valuable addition to the .~ssessincntproccss. Di'igranis, photogrciplis (although less \citisfactory bccduse they may not photocopy \veil), graphs, hi5tograms. pic cii
Structured short-answer questions: an alternative examination method R H WEBBEH Lotidoti
School of Hy'qierie arid Tropical ,Lledicirie, Iitrrvrr\ity o f Lorrdorr
the considerable value of MCQs, a structured short-answer assessnietit method has been developed.
Summary. A structured short answer (SSA) examination has been developed for the MSc. C o m m u n i t y Health in Developing Countries, t o test competence in international public health. Questions can test knowledge o r aim to assess problem-solving ability, but have the advantage that the candidates construct their o w n answer (rather than choosing from p r e f o r m e d answers in an MCQ), yet encourage sufficient precision for answers to be easily marked. T h e SSA more closely reflects participatory teaching and selfdirected learning. Illustrated material is widely used.
MCQ method I n the examination process a certain amount of time is required by the examiners in setting and
marking papers. In the conventional essay written answer, setting the question takes very little time, but marking a considerable amount of time. T h c MCQ has become increasingly popular because it shifts the time allocation to the preparation o f questions and minimizes the time for marking. A large number ofcandidates can be tested at one time and the markina is so easy that automatic readers can be used. T h e MCQ format classically asks a question and supplies five alternative answers, generally one, and occasionally t w o or more being correct. Sometimes the question is asked in the negative, so that the incorrect answer is the one to score. Double negatives should be avoided, but they often do creep in causing particular problems in understanding for the person whose first language is not English. With five altcrnativc answers there is an opportunity for guessing which has been studied o n several occasions scc Cox (1976),Jolly (1976) and Fleming (1988). There has been a move in teaching practice away from the didactic lecture to more participatory teaching and self-directed learning. Students are encouraged to work i n groups or alone, formulating their o w n solutions to problems. While the acquisition ofknowledge is essential, it is the interpretation and use of this knowledge that is the more valuable skill. T h c examination process needs to test this ability, rather thanjust knowledge. T h e M C Q can be modified for
Key words: educational measurement/ *methods; 'cduc, med, grad; public health; community medicinc/educ; London
Introduction T h e multiple choice question (MCQ) method of examining has become popular due to its economy o f s a l e , ease o f marking and consistcncy. I t allows a large number ofquestions to be answered, testing a wide range of factual material. B y providing a choice o f alternatives it reniovcs the necessity to construct a reply and answers can be obtained by luck without the candidate really knowing the correct solution. Candidates whose first language is not English often have difficulty in intcrpreting the question, particulary when negatives are used. M C Q does not lend itself so well to problem-solving, which is a far more real-life situation than having t o choose between a number of prc-formed alternatives. With these disadvantages, but recognizing Corrrhpondcncc: Dr K . H. Wrbbcr, London School of Hygiene and Tropical Medicine, University of London, Kcppel Street, London W C l E 7HT. U K .
58
Structured short-answer questions problem-solving, but alternative solutions are still supplied rather than encouraging the real-life situation of solving the problem in the individual’s own way. Ferland et a / . (1987) found no difference in M C Q s which set out to measure higher cognitive levels and suggested finding other instruments. The structurcd short-answer question endeavours to do this. Structured short answer (SSA)
Short answers have been found to be more reliable than essay type questions (Mowbray 81 Davies 1967). They are often used in written examinations to break down an essay question to ensure that the candidate attempts different aspects ofthe problem. Bull (1956) found that the shorter the questions, the greater the correlation and marking efficiency between examiners. Wakcford & Roberts (1979) found the interexaminer reliability of short essay questions to be good. This process can be extended so that the question asked requires only a one- or two-line reply. A simple example to test knowledge could be: q: What species of mosquito is illustrated? a: Aedes aegypti. q: Where does it commonly breed? a. I n water containers, water tanks, old tyres and other collections of water, close to human habitation. q: What are the main diseases transmitted by this mosquito? a: Dengue and yellow fever. q: Briefly describe (no more than 4 lines each) two methods of controlling this mosquito. a: (1) I n a dengue epidemic, using a knock-down insecticide by ULV method f r o m a knap-sack sprayer, mounted on a vehicle or by aircraft. a: (2) Burying tin cans, cutting holes in tyres and making sure there are no unwanted collections of water. Covering water tanks with netting using a mammalian-safe insecticide such as pyrimethos methyl (Abate) or biological method (dragon j y larvae or larvivorousfish) in the water container. (The question paper would also include a line drawing of the mosquito). Specimen answers are prepared a t the same time the question is constructed and made avail-
59
able to the examiners. Formulating the question in this way is easier than thc MCQ (where several alternative false answers have to be invented) and having a prepared answer makes marking a rapid process. The same question could be set as an M C Q but it would need four to five separate questions to elicit the same amount of factual material. Using a fill-in diagram can elicit very precise answers, as shown in Fig. 1. Illustrated questions can be very valuable in tcsting skills ofinterpretation, such as reading graphs. The SSA is particularly effective for problem-solving questions such as the epidemiology example illustrated in Fig. 2 and Table 1. q: The graph in Fig. 2 is constructed from the data in Table 1. The doctor presenting these data claimed that mass chemoprophylaxis had controlled the cholera epidemic, do you agree? a: No.
q: Why does the graph have two peaks? a: T h e first peak represents the epidemic that occurred in Limbo (peak in M a y ) and the second the epidemic in Birin (peak inJune). q: After each peak why do the number of cases decline? a. T h e pool of susceptibles has been used up (the downward slope ofthe epidemic curve). The SSA lends itselfmore easily to the physical and biological sciences (including medicine), but can also be used in the social sciences. Discussion The examination system is often considered a necessary but onerous task that is best overcome by the easiest possible method. Seen from the examincrs’ point of vicw, such methods as the M C Q reduce the load by a considerable extent. On thc other hand, examinees often strongly dislike the M C Q . The M C Q asks for provision of a very limited piece of knowledge and although candidates may have reviewed the subject material, if they are not able to answer this precise part, they are judged as not knowing the subject (Cox 1982). In an examination format where they can construct their own answer, they at least have the opportunity to use the knowledge they have of the subject to get close to the answer. The gambling nature of the M C Q is
K . H . Wchber
fill i n t h e b l a n k s w i t h t h e f i v e E P I v a c c i n e s
The figures b e l o w show t h e b e h a v i o u r of d t o x o i d i n a c t i v a t e d by f r e e z i n g , o f an a c t i v e one, n e v e r f r o z e n . Which onescan be u s e d ?
dnd
Immediately a l t e r shakmg
B
A smooth
and cloudy
-
Not smooth -You
can see granular
panicles
30 minutes a! l e r shaking
from WHO ( 1 9 8 9 )
Figure 1. A knowledge question constructed as a fill-in diagram. T h e candidate needs to know the five vaccines used in the expanded programme of immunization (EPI) and whether they arc stored frozen or at 0°C to + 8 T . In the srcond part, A and C are the correct answers.
often resented and problems with understanding the question have already been referred to. Assessment is a valuable and powerful learning process and more thought needs to be given to designing examinations that will encourage this process. The SSA sets out to test candidates with the kind of problems likely to be faced in their career o r profession. In this way the candidate can see the reality o f the situation rather than having to learn vast amounts of factual material just to get through the examinations.
The design of SSA questions IS to elicit a progression of information. The first question asks basic matcrial which is built on step by step. This means that the first part has only one correct answer, but as it bccomcs more searching it allows greater latitude for a partially correct answer, even if the first part(s) have been answered incorrectly. For example, in the first question, the wrong mosquito may be identified, but by knowing methods of vector control there is a good likelihood that part of the last section of
Structured short-answer questions
61
No. of cases 60 r
50 .
40
30
20!
15 20 25 30 5
10 15 20 25 30 4
April
9
14 19 24 29 4
9
14 19 24 29
July
June
May
Figure 2. Cholera cases, 15 April - 2 August.
Table 1. Occurrence o f cholera cases between 15 April and 2 August in the villages of Limbo, Simb, Pambe and Birin Infected villages Data
Limbo
Simb
Pambe
15- 19 20 - 24 25 - 29 30- 4
1 5 16 20
1 2 3 4
0 0 0 1
0 0 0 0
19 25
May
5- 9 10- 14 IS- 19 20 - 24 25 - 29 30- 3
27 38 41 39 33 18
2 3 5 3 1 1
2 2 3 1 0 1
0 0 0 1 4 12
31 43 49 44 38 32
June
4- 8 9 - 13 14- 18 19 - 23 24 - 28 29- 3
16 7 2 0 1 2
0 1 0 1 0
1 0 2 1 1 0
7 21 30 29 17 12
24 29 35 30 20 14
July
4- 8 9 - 13 14- 18 19 - 23 24 - 28
1 0 0 0 0
0 0 0 0 0
1 0 0 0 0
10 6 4 2 1
12 6 4 2 1
August
29-
0
0
0
1
1
267
28
16
157
468
April
Totals
-
2
1
Birin
Total 2 7
f2
I?. H .
the question c x i b e attempted. When the qucst i o t i I \ scorcd, the nuniber o f niJrks for c ~ c h p o i n t c m bc q r e e d before marking b y the cxaiiiincrs, s o presenting consistent)-. Illu,tratcti d a t d art' commonly ~ised.as i n the csaniplcs givcn. which is ;i valuable addition to the .~ssessincntproccss. Di'igranis, photogrciplis (although less \citisfactory bccduse they may not photocopy \veil), graphs, hi5tograms. pic cii
