Brain (1992), 115, 783-794
VALIDATION OF A SCREENING QUESTIONNAIRE FOR THE DETECTION OF EPILEPTIC SEIZURES IN EPIDEMIOLOGICAL STUDIES by M. PLACENCIA, 1 J. W. A. S. SANDER, 4 S. D. SHORVON, 4 R. H. ELLISON 3 and S. M. CASCANTE 2 {From the ^Community Management of Epilepsy Project, the 2Ciga Geigy Ecuatoriana, Quito, Ecuador, the ^Diseases Control Programme, Ciba Geigy, Basel, Switerland and the ^Epilepsy Research Group, Institute of Neurology, Queen Square, London, and Chalfont Centre for Epilepsy, Gerrards Cross, Bucks, UK) SUMMARY
INTRODUCTION
It has been common practice during the last three decades, for epidemiologists to use a two-staged approach for the identification of cases of a given condition in the community (Mausner and Bahn, 1977). The first step is the application of simplified interviews or screening techniques to identify suspected cases; the procedure can be a simple laboratory test (e.g. blood sugar estimations for diabetes) or a screening questionnaire. This step is designed to be simple, inexpensive and require no sophisticated facilities. The efficacy of this screening procedure depends on the sensitivity and specificity of the screening instrument. The second step involves a specialist examination of all the suspected cases identified by the previous screening. Cases are then categorized into true positives or false positives. Here sophisticated diagnostic resources may be needed. In the majority of epidemiological studies, however, the second step is a simple clinical examination; a perfectly appropriate procedure in epilepsy, which is defined by clinical criteria, and is an entirely clinical diagnosis. Correspondence to: Dr M. Placencia, PO BOX 17-01-693, Quito, Ecuador. © Oxford University Press 1992
Downloaded from by guest on July 12, 2015
A large-scale clinico-epidemiological study of epileptic seizures has been carried out in a highland area in northern Ecuador, South America. This paper describes the design and the two-staged validation procedures undertaken to authenticate the screening questionnaire, which was used to detect epileptic seizures in the community during this project. An initial questionnaire consisting of 20 questions was devised and then validated in a group of healthy subjects and a group of patients with known epileptic seizures. This questionnaire was found to have a sensitivity of 100% and a specificity of 50.8%. A cluster analysis of the responses rates was undertaken and a set of nine questions which presented a specificity of 92% and sensitivity of 98% were chosen to be used during the survey. The validation of this nine-question instrument was then undertaken by direct application to a general population of 72 121 persons. Positive, negative, false positive and false negative rates were derived. The screening instrument was found to have a sensitivity of 79.3%, a specificity of 92.9%, a positive predictive value of 18.3%, a negative predictive value of 99.6% and a Youden's index of 0.79. The methodology of the study and the instruments developed are recommended for future neuro-epidemiological work in epilepsy. The problems of validation are discussed, and previous epidemiological studies of epilepsy reviewed with special emphasis on the handling of questions of validation.
784
M. PLACENCIA AND OTHERS
Downloaded from by guest on July 12, 2015
Methodological problems of epidemiological studies of epilepsy have been extensively reviewed (Alter et al., 1972; Robb, 1972; Rose et al., 1973; Zielinski, 1974; Hauser andKurland, 1975; Baumann, 1982; Schoenberg, \9%2a,b; Haerer etal., 1986; Sander and Shorvon, 1987; Shorvon and Fanner, 1988). Interest has generally centred on case ascertainment, classification and definitions but to date the issues regarding the validity of the screening instruments have not been fully discussed. In response to a screening questionnaire, populations under study are grouped into possible cases {positives) and into apparently healthy persons (negatives). If the sensitivity of the screening instrument is low, many false negatives will be missed in the community. On the other hand, if the screening questionnaire has a low specificity, a large number offalse positives will be included as possible cases, generating serious logistical difficulties for stage two. Thus, the effectiveness of the screening questionnaire is of critical importance to the design of any epidemiological study. Validation is the process of establishing the degree of accuracy of an instrument. MacMahon and Pugh (1981) state that validation is the procedure to assess the ability of an instrument to recognize observations which apply to real situations, established simultaneously by more accurate and precise methods. Two measures of importance are usually quoted; the sensitivity and specificity (Mausner and Bahn, 1977; Lilienfeld and Lilienfeld, 1980; MacMahon and Pugh, 1981; Rumeau-Rouquette et al., 1981; Schoenberg, 1982c). The sensitivity of an instrument is a measure of its ability to distinguish as positive those subjects who have the condition under investigation (i.e. low false negative rates). Specificity is a measure of the ability to correctly identify those subjects who do not have the condition (i.e. low false positive rates). The sensitivity of a test is defined as the number of true positives correctly identified by the test, divided by the sum of the true positives and false negatives ( x 100). The specificity is defined as the number of true negatives correctly identified by the test, divided by the sum of the false positive and true negatives (X100). In addition, the predictive value originally described by Vecchio (1966) is a useful measure in a validation process, which accounts for the prevalence of the condition as well as the specificity and sensitivity of the instrument. The positive predictive value of an instrument is the proportion of subjects who are positive and have the condition under investigation (the number of true positive cases divided by the sum of the numbers of true and false positives x 100). The negative predictive value is the proportion of subjects who are negative and do not have the condition (the number of true negatives divided by the sum of the numbers of true and false negatives x 100). Also useful is an Youden's index (Armitage and Berry, 1987) which is an estimate of the diagnostic value of the instrument; the diagnostic value is greater, the nearer to unity is the index (derived by the subtraction of the probabilities of false positives and false negatives from unity). Validation procedures can be undertaken in two different settings. In the first, two selected samples of subjects who are known to have the condition (cases) and who are healthy (non-cases) are chosen, to whom the instrument under study is applied. The positive and negative rates on screening are then compared with the previously known diagnostic status. This method is effective as data for both healthy persons and cases can be gathered. However, generalization of results is limited by the selection criteria and by the fact that it is carried out in an artificial setting. The power of the method depends on the size of the sample and the level of diagnostic sophistication; cost and
VALIDATION OF AN EPILEPSY SCREEN
785
logistical difficulties may limit these. We shall call this method the clinic-based validation. The second method to validate an instrument is to derive rates directly from its application to a general population. Generalization of findings from this approach are more powerful as they are obtained in a less artificial setting. The main limitation of the method, however, is that it is not usually possible to apply to an accurate diagnostic examination to the whole population being studied due to the high cost of such an exercise. A sample of the population must be examined for validation purposes, and the results extrapolated to the whole population. Data therefore are partly indirect. We have called this method the field validation. Between 1986 and 1987, a large-scale clinico-epidemiological study of epileptic seizures was carried out in a highland area in northern Ecuador, South America, the methodology of which is described in detail elsewhere (Placencia et al., 1991) and is summarized in the following paper. In order to undertake this study, it was imperative to have a screening questionnaire capable of being used in large communities which had a better sensitivity and a higher specificity than existing instruments. In this paper details of the design and of the validation procedures undertaken to authenticate this screening questionnaire are presented. Both clinic-based and field validation were carried out.
CLINIC BASED VALIDATION
Results Table 1 shows the sex and age distribution by years of the healthy controls (no seizures) and patients (cases with seizures). Fifty-five of the 87 patients had partial seizures (six simple partial, 16 complex partial alone, and 33 partial with secondarily generalization). Thirty of the patients had generalized seizures (17 tonic clonic, five other generalized and eight mixed generalized seizures). Of the 63 controls, 32 were screened as negative and 31 positive, and of the 87 patients, all were positive on screening; a sensitivity of 100% and a specificity of 50.8%. In order to improve specificity (and maintain the high sensitivity), a cluster analysis of the responses rates was undertaken, and a set of nine questions
Downloaded from by guest on July 12, 2015
Material and methods A screening instrument of 20 questions was designed, based on the authors' clinical experience (Appendix 1). The screening questionnaire was designed to identify patients with a history of epileptic seizures. An affirmative answer to any of the questions would identify the subject as a positive case. In developed countries where the majority of the population is registered under a health system it is, theoretically at least, relatively easy to choose a homogeneous and comparable group of patients with the same diagnosis and simultaneously select a group of people without the condition under study and apply a screening instrument. In developing countries, such as Ecuador, where medical records are less comprehensive, healthy subjects usually will have to be identified in the community. We employed a strategy utilized previously in another survey carried out locally (Placencia et al., 1984), in which the validation sample was formed from two different sources. First, healthy subjects were identified from a survey in a village. Secondly, known patients with epilepsy, with a matched socio-cultural background, were selected from three hospital clinics in Quito (Hospital Andrade Marin run by the Ecuadorian Institute of Social Security was a source for adult patients, Hospital Eugenio Espejo, a Ministry of Public Health establishment was a source for adult and paediatric patients, and Hospital Baca Ortiz, also run by the Ministry of Public Health was a source for children). Screening was carried out by six high school-leavers, none of whom was a health worker or professional surveyor. They underwent a short training course to accomplish a high level of uniformity. The reference diagnosticians were three consultant neurologists, all of whom had previously attended the training course. A specially prepared diagnostic protocol (algorithm) was used. Diagnosis was always undertaken on clinical grounds, according to stringent clinical criteria, and in the hospital cases with hospital notes.
786
M. PLACENCIA AND OTHERS TABLE 1. AGE AND SEX DISTRIBUTION OF THE CLINIC-BASED VALIDATION GROUP Males
Age in years 0- 9
Patients
>70
18 5 4 8 3 0 2 3
Total
43
10-19 20-29 30-39 40-49 50-59 60-69
Controls 12 5 3 2 1 2 1 1 27
Females Patients 8 12 7 8 6 2 1 0 44
Total
Controls 11 9 6 5 2 0 2 1 36
Patients 26 17 11 16 9 2 3 3 87
Controls 23 14 9 7 3 2 3 2 63
Total
Percentage
49 31 20 23 12 4 6 5 150
32.7 20.7 13.3 15.3 8.0 2.7 4.0 3.3
100.0
were chosen which had a combination of specificity (92%) and sensitivity (98%) (on screening, of the 63 controls, five were positive and 58 negative; and of the 87 patients, 85 were positive and two were negative). No questions that would readily identify absences or myoclonic seizures were included in the final nine questions, but this combination was felt acceptable for general screening purposes. The questionnaire is shown in Appendix 2. FIELD VALIDATION
In addition to this, three 'quality control' steps were undertaken, of which two (steps 5 and 6) allowed field validation of the screening instruments: Step 5. Rural doctor examination of some screening negative cases. Step 6. Neurologists examination of some rural doctor negative cases. Step 7. To assesss inter-observer reliability of the neurological diagnosis, a sample of 349 (33.9%) cases was re-examined with strict clinical criteria. The findings from the seven steps were as follows: Step 1. Of the 72 121 persons surveyed, 6330 (8.8%) people were positive. Step 2. One of the 11 rural doctors examined 6155 (97%) and of these 1235 were considered true positives or uncertain. Step 3. Twelve consultant neurologists then examined 1199 (97%), of whom 958 were confirmed as patients with afebrile seizures and 34 as cases of febrile seizures (these 34 cases slipped through the screening
Downloaded from by guest on July 12, 2015
Material and methods The revised questionnaire (nine questions) was then applied to 72 121 people in a door-to-door population survey, undertaken in the highlands of northern Ecuador. This was combined with a demographic instrument. The questionnaire was designed to identify all cases who had suffered epileptic seizures. All seizure types were included, as were single seizures, acute symptomatic seizures. In stage 2, febrile seizures were designed to be excluded by a simple clinical algorithm, although a small number of cases slip through (see below). The methodology and definitions used and other details of this study are described elsewhere (Placencia et al., 1992a,fc). The study plan is summarized in Fig. 1. A cascade system of diagnostic confirmation was used for each case, as follows: Step 1. Door-to-door survey involving 72 121 people, using the screening medical questionnaire (Appendix 2) and demographic questionnaire. Step 2. Positive cases were examined by rural doctors, specially trained for the use of the clinicoepidemiological definitions of epileptic seizures (see Placencia et al., 1992). Step 3. Cases considered positive or uncertain by the rural doctors were then examined by specialist neurologists, from the Ecuadorian Society of Neurology. Step 4. The research case records from the positive cases from step 3 were subjected to an international panel review.
787
VALIDATION OF AN EPILEPSY SCREEN
Downloaded from by guest on July 12, 2015
o
788
M. PLACENCIA AND OTHERS
algorithm). Two hundred and seven cases were rejected: 184 as false positives and 23 as uncertain cases. In addition, the neurologists also confirmed a history of afebrile seizures in 71 patients who had incomplete documentation from the previous stages. Thirty-eight were positive at the screening but were not seen by the rural doctor, and in the remaining 33 the screening procedure documentation data were incomplete. Therefore, the neurologists diagnosed 1029 (958+71) cases of afebrile seizures, 996 of whom were screen positive. Step 4. A research case record review of the 1029 cases (958+71) was carried out, and 881 (85.62%) were considered to be certain (definite) cases, and 148 probable cases. Step 5. To ascertain false negative rates at screening (defined as people who although having a history of epileptic seizures responded to the screening negatively), a random sample of 904 people (1.37%) out of 65 791 negatives were interviewed by a specially trained rural doctor. Of these four (0.44%) were found to be positive, and these cases were confirmed by a neurologist. If this proportion is projected to the total number of negatives (65 791), a total of 291 false negatives would be expected. Step 6. Two hundred and twenty-seven (4.61 %) people out of the 4920 deemed as false positives (defined as those without epilepsy who responded positively to screening) by the rural doctors were examined by the neurologists. Four (1.76%) were found to have a history of epileptic seizures, and if this proportion is projected to the total (4930), a further 87 cases should be expected. Step 7. In 11.5% of the 349 cases selected for the inter-observer reliability assessment, the diagnosis of epileptic seizures was questioned.
Results A sensitivity of 79.3%, a specificity of 92.9%, a positive predictive value of 18.8%, a negative predictive value of 99.6% and a Youden's index of 0.79 were found (Table 2). Age and sex specific values are shown in Table 3 (based on the crude not adjusted values). Predicted value, crude and adjusted. The calculation of this rate offers some other methodological difficulties. For instance, the overall positive predictive value of the clinic-based validation is 94.4%, but this may vary with varying relative proportions of numerator/denominator, which depend on the way the sample is selected, and this may introduce bias. In an attempt to adapt the sample utilized for the clinicbased validation as much as possible to reality, we adjusted the numbers to the corresponding proportions in the large community, as it was estimated in a previous local study (Placencia et al., 1984). In the field
Downloaded from by guest on July 12, 2015
The numbers for true and false positives, true and false negatives in this field validation are derived in the following way: True positives. As this is a validation of the field procedure, we include all the cases who were screen positives (step 1) and confirmed by the neurologist as (febrile or afebrile) seizures (1030 = 958 + 38 + 34). These cases were not missed by the screening and must therefore be included in the total true positive number. Also, included are the 87 cases who were estimated to exist amongst the 4920 people who the rural doctor deemed as false positives. This estimated (adjusted) total is therefore 1117 (1030+87). This number will be used for calculating validity. The number of cases actually identified (crude number, 1030) will be used for calculating age and sex specific rates. False positives. 6330 were screen positive cases, of whom 1117 were considered to be true positives and 234 were not taken into consideration (175 cases who were not seen by rural doctor +36 cases who were examined by the neurologist +23 cases with uncertain diagnosis). Therefore, the number of false positives is estimated to be 4979 (6330-1117-234). Accordingly, 6096 (4979+1117) screen positives will be used for the raw validity estimations. False negatives and true negatives. The total number of negatives in the screening was 65 791. The number of false negatives within this group, i.e. patients with a history of epileptic seizures, but who were negative in the survey, was estimated, based on a review of a random sample of 904, to be 291 cases out of the total of 65 791 negatives. Therefore the number of true negatives was 65 500 (65 791 —291). To determine age and sex-specific validity rates, the crude values will be used. The age and sex specific predictive value will be calculated as the percentage of sex and age distribution of these 1030 cases, the denominator as the distribution of each group in the total of positives (6330). The rate of positive answers or suspects in the general population will be calculated using the distribution of these 6330 cases and the denominator the distribution of each group in the total population (72 121).
VALIDATION OF AN EPILEPSY SCREEN
789
TABLE 2. FINDINGS FOR THE CLINIC-BASED AND FIELD VALIDATION Clinic-based (n = 150) Sensitivity Specificity Predictive value Positive Negative Total Younden's index
Crude (%) 97.7 92.1
Adjusted (%) 97.7 92.1
Field (n = 72. 121) (%)
94.4 96.7 95.3 0.9
19.7 99.9 92.2
18.3 99.6 92.7 0.79
79.3 92.9
TABLE 3. AGE AND SEX SPECIFIC RATES (PERCENTAGE) FOR POSITIVE ANSWER AND POSITIVE PREDICTIVE VALUE OF THE SCREENING AND THE CONFIRMED CASE RATE (PER 1000) Males Age in years 0-9
PAR
Females 2AA1
10-19
4.39
14.31
20-29
5.89
30-39
Total PPV
3.48
CCR 8.54
CCR 8.58
PPV
22.81
PAR 3.52
32.60
5.41
13.25
23.24
4.88
13.80
28.25
12.32
20.92
9.13
17.19
18.83
7.54
14.79
19.63
10.29
15.18
14.76
13.64
22.17
16.26
12.09
18.93
15.67
40-49
12.17
14.45
11.87
18.35
29.01
15.81
15.37
21.98
14.30
50-59
14.14
15.39
10.88
21.92
25.63
11.69
18.17
20.69
11.39
60-69
16.95
10.53
6.21
24.07
21.62
8.98
20.57
16.16
7.86
>70
20.39
10.85
5.32
25.79
17.43
6.76
23.06
14.10
6.11
Total
7.39
12.32
16.66
10.17
16.23
15.63
8.79
14.28
16.25
PAR
24.35
PAR •= positive answer rate (per 100); CCR = confirmed cases rate (per 1000); PPV = positive predictive value (per 100).
validation, the total number of healthy subjects was estimated as 70 479 (4979 false positives+65 500 true negatives) and the total number of cases estimated as 1408 (1117 true positives +291 false negatives). Thus, the ratio healthy/cases is 50.06 (70 479/1408). Using this value in the respective ratio of the clinicbased validation sample we obtained the adjusted ratio: 4355/87 (instead of the raw 63/87). By applying the raw proportion of five false positives and 58 true negatives to the 4355, the adjusted values of 346 and 4009 are obtained, respectively. With these estimated numbers, the adjusted values of the predictive value of the clinic-based validation were calculated and can be seen in Table 2, comparing them with the crude rates and with the rates obtained in the field survey. DISCUSSION
Screening questionnaires have been used in a number of epidemiological studies of epilepsy published in the past decade. The earliest study using a two-tiered system of diagnosis was that of Rose et al. (1973), who included electroencephalogram (EEG)
Downloaded from by guest on July 12, 2015
PPV
3.57
CCR 8.62
790
M. PLACENCIA AND OTHERS
Downloaded from by guest on July 12, 2015
in the diagnostic definitions. Others since have used variations of this methodology. An influential World Health Organization (WHO) group (Osuntokun et al., 1982; Schoenberg, 1982a,b,c) introduced a standardized neurological instrument for screening for a variety of neurological diseases, without recourse to E E C This instrument had three questions relevant to epilepsy, which are for subjects of 7 yrs or older: (i) have you ever lost consciousness? (ii) have you ever had an episode where you lost contact with your surroundings? (iii) have you ever had shaking of your arms and legs which you could not control? In the last decade there have been a proliferation of neuroepidemiological studies using this protocol (Osuntokun et al., 1982; Cruz et al., 1984; Placencia et al., 1984; Pradilla et al., 1984; Li et al., 1985; Ponce et al., 1985; Osuntokun et al., 1987; Bharucha et al., 1988; Chiofalo et al., 1989). Most investigations have screened using a house-to-house survey, postal or telephone survey, or have used a clinic population. Twenty of the published surveys provide some information about the validation of the screening procedures used. In 12, validation data are actually given {see Table 4 for references) and in the remaining eight validation is mentioned only briefly. In only two are sufficient details given to establish how the validation process was actually carried out (Zielinski, 1974; Placencia et al., 1984). In Table 4 the findings from these 12 studies are shown. They report a sensitivity of the screening questionnaire between 77% and 100%, a specificity between 48% and 99%, and a predictive value between 16.3% and 90.8%. Problems exist for all screening questionnaires in three particular areas. First, is the question of diagnostic confirmation. A particular issue in epilepsy is the use of the EEC As epilepsy is essentially a clinical diagnosis and as many patients with epilepsy have normal EEGs, and many have an EEG which is inconclusive, the EEG is very limited as a screening tool (in contrast to its vital role in clinical practice). Its omission from screening procedures is therefore acceptable, and furthermore, its impracticality in a large-scale field study is clear. However, the clinical criteria on which the diagnosis is based must be clearly articulated, but only in a few studies has this issue been addressed. In the current survey, diagnostic criteria were specified on a proforma, made according to clinical features and were strictly adhered to (for details, see Placencia et al., I992a,b). We emphasize this as we consider it an important design aspect of the study. The definitions of epilepsy used are also important: particularly important are methods to differentiate febrile seizures, and also to account for active and inactive epilepsy, acute symptomatic and single seizures. Demographic details are needed to differentiate febrile seizures, and the clinical algorithms should specifically specify whether single seizures and acute symptomatic seizures are to be surveyed. The second problem is that of sensitivity. In epilepsy, most screening questionnaires have concentrated on the detection of tonic clonic seizures. This is a relatively easy task, and there is usually high sensitivity. The detection of partial seizures is a more difficult problem, as their clinical manifestations are protean. Some screening instruments, used in the past, have not included questions likely to be sensitive to these cases. Often partial seizures coexist with tonic clonic attacks (i.e. secondarily generalized seizures), which may lessen but not abolish this under-reporting. The present questionnaire was piloted specifically among clinic patients with partial epilepsy, and a satisfactory sensitivity was recorded. The WHO questionnaire is an instrument for the detection of neurological morbidity which includes epilepsy as part of a wider screening process. Although sensi-
VALIDATION OF AN EPILEPSY SCREEN
791
TABLE 4. PUBLISHED STUDIES OF INCIDENCE AND PREVALENCE OF EPILEPSY WHICH GAVE DATA ON THE VALIDATION OF SCREENING INSTRUMENTS
Author Bhanicha et al. (1988)
Location Bombay, India
Carpio et al. (1986)
Cumbe, Ecuador
Chiofalo et al. (1989)
Santiago, Chile
Cruz et al. (1984)
Quiroga, Ecuador
Garcfo-Pedroza et al. (1983)
Sensitivity 100.0
Validation rates Specificity Positive predictive value -
-
-
99.9
90.8
90.0
-
51.7
Tlalpan, Mexico
79.0
10.0
19.5
Gutierrez et al. (1980)
Ajusco, Mexico
86.0
80.0
-
Osuntokun et al. (1982)
Aiyete, Nigeria
95.0
80.0
57.0
Osuntokun et al. (1987)
Igbo-Ora, Nigeria
95.0
80.0
-
Placencia et al. (1984)
Cangahua, Ecuador
96.3
48.0
27.1
Pradilla et al. (1984)
Giron, Colombia
90.0
68.0
86.8
Rowan and Hyman (1976)
Bronx, USA
77.0
-
-
Zielinslu (1974)
Warsaw, Poland
85.5
84.0
_
93.0 -
tivities, specificities and predictive values have been calculated for all neurological disease, these have not been given for epilepsy alone. The third problem is the contrasting one of specificity. The difficulty in designing seizure specific instruments is shown clearly, for instance, by the application of the WHO protocol. Apart from the atypical result reported by Chiofalo et al. (1989) of an astounding specificity of 99.9%, reported specificities have varied between 48% and 80% (Table 4). In these studies, therefore, between 20% and 52% of healthy individuals in the community will have been identified as possible cases, requiring specialist examination. The low specificity results in a large logistical effort to sieve out true cases and increasing potential for error. It is because of the problems of obtaining satisfactory sensitivity and specificity that we feel that composite neuro-epidemiological screening questionnaires should be avoided; and that for epilepsy at least, a specific screening instrument for partial and generalized seizures is necessary. Although of great use in general neurological screening, we believe that composite screening instruments such as the WHO questionnaire have only a limited role in epilepsy. We have designed a screening questionnaire specifically for epilepsy, initially of 20 questions, which, after clinic-based validation, was refined to nine questions. These were chosen to provide a good compromise between high specificity and sensitivity, both for partial and generalized seizures. These questions, however, were non-specific for both absence and myoclonic seizures; the inclusion of specific questions would have rendered the field survey impractical and produced a high rate of false positives. Patients
Downloaded from by guest on July 12, 2015
-
792
M. PLACENCIA AND OTHERS
with absence and myoclonic seizures are probably a very small proportion of cases, and many of these patients have coexistent seizure types (generalized convulsions) which would be detected by the questionnaire, it was felt that this would not jeopardize the exercise. Furthermore, the proportion of absence and myoclonic seizures detected (about 3%) is very similar to the proportion found in Western populations (Juul-Jensen and Foldspang, 1983; Sander et al., 1990), suggesting to us that not many cases were missed. The sensitivity, specificity and predictive values of this nine question instrument are presented here (sensitivity of 79.3%, specificity of 92.9%, positive predictive value of 18.3%, negative predictive value of 99.6% and a Youden's index of 0.79). The questionnaire was validated in this very large-scale trial, using quality control steps (steps 5 and 6) to detect false positive and false negative rates in the field. We were gratified to find specificity and sensitivity rates in the field validation to be similar to those of the clinic validation; and both satisfactory sensitivity and high specificity were achieved for the detection of epileptic attacks. The results of this exercise encourage us in the belief that this epidemiological survey instrument is both effective and resilient, and will prove to be a useful addition to the neuro-epidemiologist's armamentarium. ACKNOWLEDGEMENTS
REFERENCES ALTER M, MASLAND RL, KURTZKE JF, REED DM (1972) Proposed definitions and classifications of epilepsy
for epidemiological purposes. In: The Epidemiology of Epilepsy: a Workshop. N1NDS Monograph No. 14. Edited by M. Alter and W. A. Hauser. Washington, DC: US Department of Health, Education, and Welfare, pp. 147-148. ARMITAGE P, BERRY G (1987) Statistical Methods in Medical Research. Second edition. Oxford: Blackwell Scientific. BAUMANN RJ (1982) Classification and population studies of epilepsy. In: Genetic Basis of the Epilepsies. Edited by V. E. Anderson, W. A. Hauser, J. K. Penry and C. F. Singh. New York: Raven Press, pp. 11-20. BHARUCHA NE, BHARUCHA EP, BHARUCHA AE, BHISE AV, SCHOENBERG BS (1988) Prevalence of epilepsy
in the Parsi community of Bombay. Epilepsia, 29, 111 — 115. CARPIO A, MORALES J, CALLE H, TINOCO L, SANTILLAN F (1986) Prevalencia de epilepsia en la parroquia
Cumbe, Azuay, Ecuador. Revista del Instituto de Gencias de Salud Universidade de Cuenca, Ecuador, 1, 10-31. CHIOFALO N, SCHOENBERG BS, KIRSCHBAUM A, OUVARES O, ALVAREZ G, VALENZUELA B (1989) Estudios
epidemiologicos de las enfermedades neurologicas en Santiago Metropolitano, Chile. Abstracts of the IV Pan American Congress of Neuroepidemiology, Cartagena, Columbia.
Downloaded from by guest on July 12, 2015
The authors wish to thank R. Montenegro, F. Alarcon and J. Elsitdie for allowing us to use their hospital patients for the validation (Hospitals: Andrade Marin, Eugenio Espejo and Baca Ortiz); M. Roman, E. Arizaga, A. Madera, V. Paredes and C. Bimos for their neurological examination of the validation patients; IDEA (Instituto de Estudios Avanzados, Quito) for support; J. Suarez for his valuable comments on the manuscript; and G. Buendia and S. Berendsohn for logistic support. The study was supported by a grant from Diseases Control Programme, Ciba Geigy, Basel, Switzerland. The work is reported on behalf of the CME (Community Management of Epilepsy Project, through an agreement between the Ecuadorian Ministry of Public Health, the Ecuadorian League against Epilepsy, the Ecuadorian Society of Neurology and Ciba Geigy) and ICBERG (International Community-based Epilepsy Research Group).
VALIDATION OF AN EPILEPSY SCREEN
793
CRUZ M, RUALES J, BOSSANO F, PROANO J, BARBERIS P, COX L (1984) Estudios Neuroepidemiologicas
en el Ecuador. CIEN, Fundacion Eugenio Espejo, Quito: Ministerio de Salud Publica. GARCfA-PEDROZA F , RUBIO-DONNADIEU F , GARCfA-RAMOS G, EsCOBEDO-RlOS F , GONZALES-CORTES A
(1983) Prevalence of epilepsy in children: Tlalpan, Mexico City, Mexico. Neuroepidemiology, 2, 16-23. GUTIERREZ H, RUBIO F, ESCOBEDO F, HERON J (1980) Prevalencia de epilepsia en ninos de edad escolar de una comunidad urbana de la Ciudad de Mexico. Gaceta Medica Mexicana, 116, 497 — 501. HAERER AF, ANDERSON DW, SCHOENBERG BS (1986) Prevalence and clinical features of epilepsy in a biracial United States population. Epilepsia, 27, 66—75. HAUSER WA, KURLAND LT (1975) The epidemioloy of epilepsy in Rochester, Minnesota, 1935 through 1967. Epilepsia, 16, 1 - 6 6 . JUUL-JENSEN P, FOLDSPANG A (1983) Natural history of epileptic seizures. Epilepsia, 24, 297—312. Li S, SCHOENBERG BS, WANG C, CHENG X, ZHOU S, BOLIS CL (1985) Epidemiology of epilepsy in urban
areas of the People's Republic of China. Epilepsia, 26, 391—394. LILIENFELD AM, LILIENFELD DE (1980) Foundations of Epidemiology. Second Edition. Oxford University Press, pp. 150-153. MACMAHON B, PUGH TF (1981) Principios y Metados de Epidemiology. Mexico: La Prensa Medica Mexicana, pp. 242-245. MAUSNER JS, BAHN AK (1977) Epidemiology. Mexico: Edit. Interamericana, pp. 223-234. OSUNTOKUN BO, SCHOENBERG BS, NOTTIDGE VA, ADEUJA A, KALE O, ADEYEFA A et al. (1982) Research
protocol for measuring the prevalence of neurological disorders in developing countries: results of a pilot study in Nigeria. Neuroepidemiology, 1, 143 — 153. OSUNTOKUN BO, ADEUJA AO, NOTTIDGE VA, BADEMOSI O, OLUMIDE A, IGE O etal. (1987) Prevalence
of the epilepsies in Nigerian Africans: a community-based study. Epilepsia, 28, 272—279. scale study of epilepsy in Ecuador: methodological aspects. Neuroepidemiology, 11, 7 4 - 8 4 . PLACENCIA M, SHORVON SD, PAREDES V, BIMOS C, SANDER JWAS, SUAREZ J et al. (1992*) Epileptic
seizures in an Andean region of Ecuador: prevalence and incidence and regional variation. Brain, 115, 771-782. PLACENCIA M, SILVA C, CORDOVA M, ALARCON F, RODRIGUEZ G, CASTRO E (1984) Prevalencia de
Enfermedades Neurologicas en una Communidad Rural Andina (Cangahua). Informe final, Quito: CONACYT. PONCE P, FERNANDEZ R, D'SOUSE C, VILLIEGAS A, DE PONCE C, GONZALES B et al. (1985) Estudio
de prevalencia de transtornos neurologicos en Lezama. In: Estudios Neuroepidemiologicos en Venezuela. Caracas: Departamento de Enfermedades Neurologicas, Ministerio de Sanidad y Asistencia Social, pp. 16-38. PRADILLA G, PARDO C, PUENTES F, RIVERA R, BALLESTEROS G, FLANTRAMSKY H et al. (1984) Estudio
Neuroepidemiologico en Giron, Columbia. Comite de Neuroepidemiologica, Bucaramanga: Universidad Industrial de Santander. ROBB JP (1972) A review of epidemiologic concepts of epilepsy. In The Epidemiology of Epilepsy: a Workshop. NINDS Monograph No. 14. Edited by M. Alter and W. A. Hauser. Washington DC: US Department of Health Education, and Welfare, pp. 13-18. ROSE SW, PENRY JK, MARKUSH RE, RADLOFF LA, PUTNAM PL (1973) Prevalence of epilepsy in children.
Epilepsia, 14, 133-152. ROWAN AJ, HYMAN HH (1976) The prevalence of epilepsy in a large heterogeneous urban population (The Bronx, New York, January 8, 1975). Transactions of the American Neurological Association,
101, 281-283. RUMEAU-ROUQUETTE C, BR£ART G, PADIEU R (1981) Mithodes en tpidimiologie:
tchantillonnage,
Investigations, Analyse. Paris: Flammarion Mddecine-Science, pp. 118—119. SANDER JWAS, SHORVON SD (1987) Incidence and prevalence studies in epilepsy and their methodological problems: a review. Journal of Neurology, Neurosurgery and Psychiatry, 50, 829 — 839. SANDER JWAS, HART YM, JOHNSON AL, SHORVON SD (1990) National General Practice Study of Epilepsy: newly diagnosed epileptic seizures in a general population. Lancet, 336, 1267—1271. SCHOENBERG BS (1982a) Clinical neuroepidemiology in developing countries: neurology with few neurologists. Neuroepidemiology, 1, 137—142.
Downloaded from by guest on July 12, 2015
PLACENCIA M, SUAREZ J, CRESPO F, SHORVON SD, SANDER JWAS, ELLISON RH et al. (1992a) A large
794
M. PLACENCIA AND OTHERS
SCHOENBERG BS (1982*) Neuroepidemiologic definitions of diseases. Neuroepidemiology, 1, 197-200. SCHOENBERG BS (1982c) The scope of neuroepidemiology: from stone age to Stockholm. Neuroepidemiology, 1, 1-16. SHORVON SD, FARMER PJ (1988) Epilepsy in developing countries: a review of epidemiological, sociological, and treatment aspects. Epilepsia, 29, Supplement 1, S36—S54. VECCHIO TJ (1966) Predictive value of a single diagnostic test in unselected populations. New England Journal of Medicine, 274, 1171 - 1173. ZIELINSKJ JJ (1974) Epidemiology and Medico-social Problems of Epilepsy in Warsaw. Final Report on Research Program No.l9-P-58325-F-01. Warsaw: Psychoneurological Institute, pp. 5 0 - 5 5 . (Received August 20, 1991. Revised February 6, 1992. Accepted April 6, 1992)
APPENDIX 1 Original screening questionnaire as used during the clinic-based validation
(14) (15) (16) (17) (18) (19) (20)
Have you ever lost consciousness? Have you ever had attacks in which you loose contact with the surroundings? Have you ever had attacks of shaking of the arms or legs which you could not control? Have you ever had attacks in which you fall to the ground with loss of consciousness? Have you ever had attacks in which you fall and bite your tongue? Have you ever had attacks in which you fall and lose control of your bladder? Have you ever had attacks in which you fall and become pale? Have you ever had attacks in which you lose your memory for a short period of time? Have you ever had attacks of strange behaviour and loss of memory? Have you ever had brief attacks of shaking or trembling in one arm or leg or in the face? Have you ever had attacks of tingling or numbness which move up your arm, leg or body? Have you ever had attacks of jerkings which move up your arm, leg or body? Have you ever had attacks in which you lose contact with the surrounding and experience a feeling of unreality or dreaminess? Have you ever had attacks in which you lose contact with the surrounding and experience a sensation in which objects change shape or size? Have you ever had attacks in which you lose contact with the surrounding and experience abnormal visions? Have you ever had attacks in which you lose contact with the surrounding and experience abnormal sounds? Have you ever had attacks in which you lose contact with the surroundings and experience abnormal smells? Have you ever had attacks in which you behave momentarily in a confused fashion? Have you ever had attacks of palpitation? Have you ever been told that you have or had epilepsy or epiliptic fits?
APPENDIX Screening
questionnaire
2
as used during the large-scale
survey in the
community
(1) Have you ever had attacks of shaking of the arms or legs which you could not control? (2) Have you ever had attacks in which you fall and become pale? Both question I and 2 must be affirmative to render the subject positive. (3) (4) (5) (6) (7) (8) (9)
Have Have Have Have Have Have Have
you ever you ever you ever you ever you ever you ever you ever
tost consciousness? had attacks in which you fall with loss of consciousness? had attacks in which you fall and bite your tongue? had attacks in which you fall and lose control of your bladder? had brief attacks of shaking or trembling in one arm or leg or in the face? had attacks in which you lose contact with the surroundings and experience abnormal smells? been told that you have or had epilepsy or epileptic fits?
Any question 3 to 9 if affirmative renders the subject positive.
Downloaded from by guest on July 12, 2015
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13)
VALIDATION OF A SCREENING QUESTIONNAIRE FOR THE DETECTION OF EPILEPTIC SEIZURES IN EPIDEMIOLOGICAL STUDIES by M. PLACENCIA, 1 J. W. A. S. SANDER, 4 S. D. SHORVON, 4 R. H. ELLISON 3 and S. M. CASCANTE 2 {From the ^Community Management of Epilepsy Project, the 2Ciga Geigy Ecuatoriana, Quito, Ecuador, the ^Diseases Control Programme, Ciba Geigy, Basel, Switerland and the ^Epilepsy Research Group, Institute of Neurology, Queen Square, London, and Chalfont Centre for Epilepsy, Gerrards Cross, Bucks, UK) SUMMARY
INTRODUCTION
It has been common practice during the last three decades, for epidemiologists to use a two-staged approach for the identification of cases of a given condition in the community (Mausner and Bahn, 1977). The first step is the application of simplified interviews or screening techniques to identify suspected cases; the procedure can be a simple laboratory test (e.g. blood sugar estimations for diabetes) or a screening questionnaire. This step is designed to be simple, inexpensive and require no sophisticated facilities. The efficacy of this screening procedure depends on the sensitivity and specificity of the screening instrument. The second step involves a specialist examination of all the suspected cases identified by the previous screening. Cases are then categorized into true positives or false positives. Here sophisticated diagnostic resources may be needed. In the majority of epidemiological studies, however, the second step is a simple clinical examination; a perfectly appropriate procedure in epilepsy, which is defined by clinical criteria, and is an entirely clinical diagnosis. Correspondence to: Dr M. Placencia, PO BOX 17-01-693, Quito, Ecuador. © Oxford University Press 1992
Downloaded from by guest on July 12, 2015
A large-scale clinico-epidemiological study of epileptic seizures has been carried out in a highland area in northern Ecuador, South America. This paper describes the design and the two-staged validation procedures undertaken to authenticate the screening questionnaire, which was used to detect epileptic seizures in the community during this project. An initial questionnaire consisting of 20 questions was devised and then validated in a group of healthy subjects and a group of patients with known epileptic seizures. This questionnaire was found to have a sensitivity of 100% and a specificity of 50.8%. A cluster analysis of the responses rates was undertaken and a set of nine questions which presented a specificity of 92% and sensitivity of 98% were chosen to be used during the survey. The validation of this nine-question instrument was then undertaken by direct application to a general population of 72 121 persons. Positive, negative, false positive and false negative rates were derived. The screening instrument was found to have a sensitivity of 79.3%, a specificity of 92.9%, a positive predictive value of 18.3%, a negative predictive value of 99.6% and a Youden's index of 0.79. The methodology of the study and the instruments developed are recommended for future neuro-epidemiological work in epilepsy. The problems of validation are discussed, and previous epidemiological studies of epilepsy reviewed with special emphasis on the handling of questions of validation.
784
M. PLACENCIA AND OTHERS
Downloaded from by guest on July 12, 2015
Methodological problems of epidemiological studies of epilepsy have been extensively reviewed (Alter et al., 1972; Robb, 1972; Rose et al., 1973; Zielinski, 1974; Hauser andKurland, 1975; Baumann, 1982; Schoenberg, \9%2a,b; Haerer etal., 1986; Sander and Shorvon, 1987; Shorvon and Fanner, 1988). Interest has generally centred on case ascertainment, classification and definitions but to date the issues regarding the validity of the screening instruments have not been fully discussed. In response to a screening questionnaire, populations under study are grouped into possible cases {positives) and into apparently healthy persons (negatives). If the sensitivity of the screening instrument is low, many false negatives will be missed in the community. On the other hand, if the screening questionnaire has a low specificity, a large number offalse positives will be included as possible cases, generating serious logistical difficulties for stage two. Thus, the effectiveness of the screening questionnaire is of critical importance to the design of any epidemiological study. Validation is the process of establishing the degree of accuracy of an instrument. MacMahon and Pugh (1981) state that validation is the procedure to assess the ability of an instrument to recognize observations which apply to real situations, established simultaneously by more accurate and precise methods. Two measures of importance are usually quoted; the sensitivity and specificity (Mausner and Bahn, 1977; Lilienfeld and Lilienfeld, 1980; MacMahon and Pugh, 1981; Rumeau-Rouquette et al., 1981; Schoenberg, 1982c). The sensitivity of an instrument is a measure of its ability to distinguish as positive those subjects who have the condition under investigation (i.e. low false negative rates). Specificity is a measure of the ability to correctly identify those subjects who do not have the condition (i.e. low false positive rates). The sensitivity of a test is defined as the number of true positives correctly identified by the test, divided by the sum of the true positives and false negatives ( x 100). The specificity is defined as the number of true negatives correctly identified by the test, divided by the sum of the false positive and true negatives (X100). In addition, the predictive value originally described by Vecchio (1966) is a useful measure in a validation process, which accounts for the prevalence of the condition as well as the specificity and sensitivity of the instrument. The positive predictive value of an instrument is the proportion of subjects who are positive and have the condition under investigation (the number of true positive cases divided by the sum of the numbers of true and false positives x 100). The negative predictive value is the proportion of subjects who are negative and do not have the condition (the number of true negatives divided by the sum of the numbers of true and false negatives x 100). Also useful is an Youden's index (Armitage and Berry, 1987) which is an estimate of the diagnostic value of the instrument; the diagnostic value is greater, the nearer to unity is the index (derived by the subtraction of the probabilities of false positives and false negatives from unity). Validation procedures can be undertaken in two different settings. In the first, two selected samples of subjects who are known to have the condition (cases) and who are healthy (non-cases) are chosen, to whom the instrument under study is applied. The positive and negative rates on screening are then compared with the previously known diagnostic status. This method is effective as data for both healthy persons and cases can be gathered. However, generalization of results is limited by the selection criteria and by the fact that it is carried out in an artificial setting. The power of the method depends on the size of the sample and the level of diagnostic sophistication; cost and
VALIDATION OF AN EPILEPSY SCREEN
785
logistical difficulties may limit these. We shall call this method the clinic-based validation. The second method to validate an instrument is to derive rates directly from its application to a general population. Generalization of findings from this approach are more powerful as they are obtained in a less artificial setting. The main limitation of the method, however, is that it is not usually possible to apply to an accurate diagnostic examination to the whole population being studied due to the high cost of such an exercise. A sample of the population must be examined for validation purposes, and the results extrapolated to the whole population. Data therefore are partly indirect. We have called this method the field validation. Between 1986 and 1987, a large-scale clinico-epidemiological study of epileptic seizures was carried out in a highland area in northern Ecuador, South America, the methodology of which is described in detail elsewhere (Placencia et al., 1991) and is summarized in the following paper. In order to undertake this study, it was imperative to have a screening questionnaire capable of being used in large communities which had a better sensitivity and a higher specificity than existing instruments. In this paper details of the design and of the validation procedures undertaken to authenticate this screening questionnaire are presented. Both clinic-based and field validation were carried out.
CLINIC BASED VALIDATION
Results Table 1 shows the sex and age distribution by years of the healthy controls (no seizures) and patients (cases with seizures). Fifty-five of the 87 patients had partial seizures (six simple partial, 16 complex partial alone, and 33 partial with secondarily generalization). Thirty of the patients had generalized seizures (17 tonic clonic, five other generalized and eight mixed generalized seizures). Of the 63 controls, 32 were screened as negative and 31 positive, and of the 87 patients, all were positive on screening; a sensitivity of 100% and a specificity of 50.8%. In order to improve specificity (and maintain the high sensitivity), a cluster analysis of the responses rates was undertaken, and a set of nine questions
Downloaded from by guest on July 12, 2015
Material and methods A screening instrument of 20 questions was designed, based on the authors' clinical experience (Appendix 1). The screening questionnaire was designed to identify patients with a history of epileptic seizures. An affirmative answer to any of the questions would identify the subject as a positive case. In developed countries where the majority of the population is registered under a health system it is, theoretically at least, relatively easy to choose a homogeneous and comparable group of patients with the same diagnosis and simultaneously select a group of people without the condition under study and apply a screening instrument. In developing countries, such as Ecuador, where medical records are less comprehensive, healthy subjects usually will have to be identified in the community. We employed a strategy utilized previously in another survey carried out locally (Placencia et al., 1984), in which the validation sample was formed from two different sources. First, healthy subjects were identified from a survey in a village. Secondly, known patients with epilepsy, with a matched socio-cultural background, were selected from three hospital clinics in Quito (Hospital Andrade Marin run by the Ecuadorian Institute of Social Security was a source for adult patients, Hospital Eugenio Espejo, a Ministry of Public Health establishment was a source for adult and paediatric patients, and Hospital Baca Ortiz, also run by the Ministry of Public Health was a source for children). Screening was carried out by six high school-leavers, none of whom was a health worker or professional surveyor. They underwent a short training course to accomplish a high level of uniformity. The reference diagnosticians were three consultant neurologists, all of whom had previously attended the training course. A specially prepared diagnostic protocol (algorithm) was used. Diagnosis was always undertaken on clinical grounds, according to stringent clinical criteria, and in the hospital cases with hospital notes.
786
M. PLACENCIA AND OTHERS TABLE 1. AGE AND SEX DISTRIBUTION OF THE CLINIC-BASED VALIDATION GROUP Males
Age in years 0- 9
Patients
>70
18 5 4 8 3 0 2 3
Total
43
10-19 20-29 30-39 40-49 50-59 60-69
Controls 12 5 3 2 1 2 1 1 27
Females Patients 8 12 7 8 6 2 1 0 44
Total
Controls 11 9 6 5 2 0 2 1 36
Patients 26 17 11 16 9 2 3 3 87
Controls 23 14 9 7 3 2 3 2 63
Total
Percentage
49 31 20 23 12 4 6 5 150
32.7 20.7 13.3 15.3 8.0 2.7 4.0 3.3
100.0
were chosen which had a combination of specificity (92%) and sensitivity (98%) (on screening, of the 63 controls, five were positive and 58 negative; and of the 87 patients, 85 were positive and two were negative). No questions that would readily identify absences or myoclonic seizures were included in the final nine questions, but this combination was felt acceptable for general screening purposes. The questionnaire is shown in Appendix 2. FIELD VALIDATION
In addition to this, three 'quality control' steps were undertaken, of which two (steps 5 and 6) allowed field validation of the screening instruments: Step 5. Rural doctor examination of some screening negative cases. Step 6. Neurologists examination of some rural doctor negative cases. Step 7. To assesss inter-observer reliability of the neurological diagnosis, a sample of 349 (33.9%) cases was re-examined with strict clinical criteria. The findings from the seven steps were as follows: Step 1. Of the 72 121 persons surveyed, 6330 (8.8%) people were positive. Step 2. One of the 11 rural doctors examined 6155 (97%) and of these 1235 were considered true positives or uncertain. Step 3. Twelve consultant neurologists then examined 1199 (97%), of whom 958 were confirmed as patients with afebrile seizures and 34 as cases of febrile seizures (these 34 cases slipped through the screening
Downloaded from by guest on July 12, 2015
Material and methods The revised questionnaire (nine questions) was then applied to 72 121 people in a door-to-door population survey, undertaken in the highlands of northern Ecuador. This was combined with a demographic instrument. The questionnaire was designed to identify all cases who had suffered epileptic seizures. All seizure types were included, as were single seizures, acute symptomatic seizures. In stage 2, febrile seizures were designed to be excluded by a simple clinical algorithm, although a small number of cases slip through (see below). The methodology and definitions used and other details of this study are described elsewhere (Placencia et al., 1992a,fc). The study plan is summarized in Fig. 1. A cascade system of diagnostic confirmation was used for each case, as follows: Step 1. Door-to-door survey involving 72 121 people, using the screening medical questionnaire (Appendix 2) and demographic questionnaire. Step 2. Positive cases were examined by rural doctors, specially trained for the use of the clinicoepidemiological definitions of epileptic seizures (see Placencia et al., 1992). Step 3. Cases considered positive or uncertain by the rural doctors were then examined by specialist neurologists, from the Ecuadorian Society of Neurology. Step 4. The research case records from the positive cases from step 3 were subjected to an international panel review.
787
VALIDATION OF AN EPILEPSY SCREEN
Downloaded from by guest on July 12, 2015
o
788
M. PLACENCIA AND OTHERS
algorithm). Two hundred and seven cases were rejected: 184 as false positives and 23 as uncertain cases. In addition, the neurologists also confirmed a history of afebrile seizures in 71 patients who had incomplete documentation from the previous stages. Thirty-eight were positive at the screening but were not seen by the rural doctor, and in the remaining 33 the screening procedure documentation data were incomplete. Therefore, the neurologists diagnosed 1029 (958+71) cases of afebrile seizures, 996 of whom were screen positive. Step 4. A research case record review of the 1029 cases (958+71) was carried out, and 881 (85.62%) were considered to be certain (definite) cases, and 148 probable cases. Step 5. To ascertain false negative rates at screening (defined as people who although having a history of epileptic seizures responded to the screening negatively), a random sample of 904 people (1.37%) out of 65 791 negatives were interviewed by a specially trained rural doctor. Of these four (0.44%) were found to be positive, and these cases were confirmed by a neurologist. If this proportion is projected to the total number of negatives (65 791), a total of 291 false negatives would be expected. Step 6. Two hundred and twenty-seven (4.61 %) people out of the 4920 deemed as false positives (defined as those without epilepsy who responded positively to screening) by the rural doctors were examined by the neurologists. Four (1.76%) were found to have a history of epileptic seizures, and if this proportion is projected to the total (4930), a further 87 cases should be expected. Step 7. In 11.5% of the 349 cases selected for the inter-observer reliability assessment, the diagnosis of epileptic seizures was questioned.
Results A sensitivity of 79.3%, a specificity of 92.9%, a positive predictive value of 18.8%, a negative predictive value of 99.6% and a Youden's index of 0.79 were found (Table 2). Age and sex specific values are shown in Table 3 (based on the crude not adjusted values). Predicted value, crude and adjusted. The calculation of this rate offers some other methodological difficulties. For instance, the overall positive predictive value of the clinic-based validation is 94.4%, but this may vary with varying relative proportions of numerator/denominator, which depend on the way the sample is selected, and this may introduce bias. In an attempt to adapt the sample utilized for the clinicbased validation as much as possible to reality, we adjusted the numbers to the corresponding proportions in the large community, as it was estimated in a previous local study (Placencia et al., 1984). In the field
Downloaded from by guest on July 12, 2015
The numbers for true and false positives, true and false negatives in this field validation are derived in the following way: True positives. As this is a validation of the field procedure, we include all the cases who were screen positives (step 1) and confirmed by the neurologist as (febrile or afebrile) seizures (1030 = 958 + 38 + 34). These cases were not missed by the screening and must therefore be included in the total true positive number. Also, included are the 87 cases who were estimated to exist amongst the 4920 people who the rural doctor deemed as false positives. This estimated (adjusted) total is therefore 1117 (1030+87). This number will be used for calculating validity. The number of cases actually identified (crude number, 1030) will be used for calculating age and sex specific rates. False positives. 6330 were screen positive cases, of whom 1117 were considered to be true positives and 234 were not taken into consideration (175 cases who were not seen by rural doctor +36 cases who were examined by the neurologist +23 cases with uncertain diagnosis). Therefore, the number of false positives is estimated to be 4979 (6330-1117-234). Accordingly, 6096 (4979+1117) screen positives will be used for the raw validity estimations. False negatives and true negatives. The total number of negatives in the screening was 65 791. The number of false negatives within this group, i.e. patients with a history of epileptic seizures, but who were negative in the survey, was estimated, based on a review of a random sample of 904, to be 291 cases out of the total of 65 791 negatives. Therefore the number of true negatives was 65 500 (65 791 —291). To determine age and sex-specific validity rates, the crude values will be used. The age and sex specific predictive value will be calculated as the percentage of sex and age distribution of these 1030 cases, the denominator as the distribution of each group in the total of positives (6330). The rate of positive answers or suspects in the general population will be calculated using the distribution of these 6330 cases and the denominator the distribution of each group in the total population (72 121).
VALIDATION OF AN EPILEPSY SCREEN
789
TABLE 2. FINDINGS FOR THE CLINIC-BASED AND FIELD VALIDATION Clinic-based (n = 150) Sensitivity Specificity Predictive value Positive Negative Total Younden's index
Crude (%) 97.7 92.1
Adjusted (%) 97.7 92.1
Field (n = 72. 121) (%)
94.4 96.7 95.3 0.9
19.7 99.9 92.2
18.3 99.6 92.7 0.79
79.3 92.9
TABLE 3. AGE AND SEX SPECIFIC RATES (PERCENTAGE) FOR POSITIVE ANSWER AND POSITIVE PREDICTIVE VALUE OF THE SCREENING AND THE CONFIRMED CASE RATE (PER 1000) Males Age in years 0-9
PAR
Females 2AA1
10-19
4.39
14.31
20-29
5.89
30-39
Total PPV
3.48
CCR 8.54
CCR 8.58
PPV
22.81
PAR 3.52
32.60
5.41
13.25
23.24
4.88
13.80
28.25
12.32
20.92
9.13
17.19
18.83
7.54
14.79
19.63
10.29
15.18
14.76
13.64
22.17
16.26
12.09
18.93
15.67
40-49
12.17
14.45
11.87
18.35
29.01
15.81
15.37
21.98
14.30
50-59
14.14
15.39
10.88
21.92
25.63
11.69
18.17
20.69
11.39
60-69
16.95
10.53
6.21
24.07
21.62
8.98
20.57
16.16
7.86
>70
20.39
10.85
5.32
25.79
17.43
6.76
23.06
14.10
6.11
Total
7.39
12.32
16.66
10.17
16.23
15.63
8.79
14.28
16.25
PAR
24.35
PAR •= positive answer rate (per 100); CCR = confirmed cases rate (per 1000); PPV = positive predictive value (per 100).
validation, the total number of healthy subjects was estimated as 70 479 (4979 false positives+65 500 true negatives) and the total number of cases estimated as 1408 (1117 true positives +291 false negatives). Thus, the ratio healthy/cases is 50.06 (70 479/1408). Using this value in the respective ratio of the clinicbased validation sample we obtained the adjusted ratio: 4355/87 (instead of the raw 63/87). By applying the raw proportion of five false positives and 58 true negatives to the 4355, the adjusted values of 346 and 4009 are obtained, respectively. With these estimated numbers, the adjusted values of the predictive value of the clinic-based validation were calculated and can be seen in Table 2, comparing them with the crude rates and with the rates obtained in the field survey. DISCUSSION
Screening questionnaires have been used in a number of epidemiological studies of epilepsy published in the past decade. The earliest study using a two-tiered system of diagnosis was that of Rose et al. (1973), who included electroencephalogram (EEG)
Downloaded from by guest on July 12, 2015
PPV
3.57
CCR 8.62
790
M. PLACENCIA AND OTHERS
Downloaded from by guest on July 12, 2015
in the diagnostic definitions. Others since have used variations of this methodology. An influential World Health Organization (WHO) group (Osuntokun et al., 1982; Schoenberg, 1982a,b,c) introduced a standardized neurological instrument for screening for a variety of neurological diseases, without recourse to E E C This instrument had three questions relevant to epilepsy, which are for subjects of 7 yrs or older: (i) have you ever lost consciousness? (ii) have you ever had an episode where you lost contact with your surroundings? (iii) have you ever had shaking of your arms and legs which you could not control? In the last decade there have been a proliferation of neuroepidemiological studies using this protocol (Osuntokun et al., 1982; Cruz et al., 1984; Placencia et al., 1984; Pradilla et al., 1984; Li et al., 1985; Ponce et al., 1985; Osuntokun et al., 1987; Bharucha et al., 1988; Chiofalo et al., 1989). Most investigations have screened using a house-to-house survey, postal or telephone survey, or have used a clinic population. Twenty of the published surveys provide some information about the validation of the screening procedures used. In 12, validation data are actually given {see Table 4 for references) and in the remaining eight validation is mentioned only briefly. In only two are sufficient details given to establish how the validation process was actually carried out (Zielinski, 1974; Placencia et al., 1984). In Table 4 the findings from these 12 studies are shown. They report a sensitivity of the screening questionnaire between 77% and 100%, a specificity between 48% and 99%, and a predictive value between 16.3% and 90.8%. Problems exist for all screening questionnaires in three particular areas. First, is the question of diagnostic confirmation. A particular issue in epilepsy is the use of the EEC As epilepsy is essentially a clinical diagnosis and as many patients with epilepsy have normal EEGs, and many have an EEG which is inconclusive, the EEG is very limited as a screening tool (in contrast to its vital role in clinical practice). Its omission from screening procedures is therefore acceptable, and furthermore, its impracticality in a large-scale field study is clear. However, the clinical criteria on which the diagnosis is based must be clearly articulated, but only in a few studies has this issue been addressed. In the current survey, diagnostic criteria were specified on a proforma, made according to clinical features and were strictly adhered to (for details, see Placencia et al., I992a,b). We emphasize this as we consider it an important design aspect of the study. The definitions of epilepsy used are also important: particularly important are methods to differentiate febrile seizures, and also to account for active and inactive epilepsy, acute symptomatic and single seizures. Demographic details are needed to differentiate febrile seizures, and the clinical algorithms should specifically specify whether single seizures and acute symptomatic seizures are to be surveyed. The second problem is that of sensitivity. In epilepsy, most screening questionnaires have concentrated on the detection of tonic clonic seizures. This is a relatively easy task, and there is usually high sensitivity. The detection of partial seizures is a more difficult problem, as their clinical manifestations are protean. Some screening instruments, used in the past, have not included questions likely to be sensitive to these cases. Often partial seizures coexist with tonic clonic attacks (i.e. secondarily generalized seizures), which may lessen but not abolish this under-reporting. The present questionnaire was piloted specifically among clinic patients with partial epilepsy, and a satisfactory sensitivity was recorded. The WHO questionnaire is an instrument for the detection of neurological morbidity which includes epilepsy as part of a wider screening process. Although sensi-
VALIDATION OF AN EPILEPSY SCREEN
791
TABLE 4. PUBLISHED STUDIES OF INCIDENCE AND PREVALENCE OF EPILEPSY WHICH GAVE DATA ON THE VALIDATION OF SCREENING INSTRUMENTS
Author Bhanicha et al. (1988)
Location Bombay, India
Carpio et al. (1986)
Cumbe, Ecuador
Chiofalo et al. (1989)
Santiago, Chile
Cruz et al. (1984)
Quiroga, Ecuador
Garcfo-Pedroza et al. (1983)
Sensitivity 100.0
Validation rates Specificity Positive predictive value -
-
-
99.9
90.8
90.0
-
51.7
Tlalpan, Mexico
79.0
10.0
19.5
Gutierrez et al. (1980)
Ajusco, Mexico
86.0
80.0
-
Osuntokun et al. (1982)
Aiyete, Nigeria
95.0
80.0
57.0
Osuntokun et al. (1987)
Igbo-Ora, Nigeria
95.0
80.0
-
Placencia et al. (1984)
Cangahua, Ecuador
96.3
48.0
27.1
Pradilla et al. (1984)
Giron, Colombia
90.0
68.0
86.8
Rowan and Hyman (1976)
Bronx, USA
77.0
-
-
Zielinslu (1974)
Warsaw, Poland
85.5
84.0
_
93.0 -
tivities, specificities and predictive values have been calculated for all neurological disease, these have not been given for epilepsy alone. The third problem is the contrasting one of specificity. The difficulty in designing seizure specific instruments is shown clearly, for instance, by the application of the WHO protocol. Apart from the atypical result reported by Chiofalo et al. (1989) of an astounding specificity of 99.9%, reported specificities have varied between 48% and 80% (Table 4). In these studies, therefore, between 20% and 52% of healthy individuals in the community will have been identified as possible cases, requiring specialist examination. The low specificity results in a large logistical effort to sieve out true cases and increasing potential for error. It is because of the problems of obtaining satisfactory sensitivity and specificity that we feel that composite neuro-epidemiological screening questionnaires should be avoided; and that for epilepsy at least, a specific screening instrument for partial and generalized seizures is necessary. Although of great use in general neurological screening, we believe that composite screening instruments such as the WHO questionnaire have only a limited role in epilepsy. We have designed a screening questionnaire specifically for epilepsy, initially of 20 questions, which, after clinic-based validation, was refined to nine questions. These were chosen to provide a good compromise between high specificity and sensitivity, both for partial and generalized seizures. These questions, however, were non-specific for both absence and myoclonic seizures; the inclusion of specific questions would have rendered the field survey impractical and produced a high rate of false positives. Patients
Downloaded from by guest on July 12, 2015
-
792
M. PLACENCIA AND OTHERS
with absence and myoclonic seizures are probably a very small proportion of cases, and many of these patients have coexistent seizure types (generalized convulsions) which would be detected by the questionnaire, it was felt that this would not jeopardize the exercise. Furthermore, the proportion of absence and myoclonic seizures detected (about 3%) is very similar to the proportion found in Western populations (Juul-Jensen and Foldspang, 1983; Sander et al., 1990), suggesting to us that not many cases were missed. The sensitivity, specificity and predictive values of this nine question instrument are presented here (sensitivity of 79.3%, specificity of 92.9%, positive predictive value of 18.3%, negative predictive value of 99.6% and a Youden's index of 0.79). The questionnaire was validated in this very large-scale trial, using quality control steps (steps 5 and 6) to detect false positive and false negative rates in the field. We were gratified to find specificity and sensitivity rates in the field validation to be similar to those of the clinic validation; and both satisfactory sensitivity and high specificity were achieved for the detection of epileptic attacks. The results of this exercise encourage us in the belief that this epidemiological survey instrument is both effective and resilient, and will prove to be a useful addition to the neuro-epidemiologist's armamentarium. ACKNOWLEDGEMENTS
REFERENCES ALTER M, MASLAND RL, KURTZKE JF, REED DM (1972) Proposed definitions and classifications of epilepsy
for epidemiological purposes. In: The Epidemiology of Epilepsy: a Workshop. N1NDS Monograph No. 14. Edited by M. Alter and W. A. Hauser. Washington, DC: US Department of Health, Education, and Welfare, pp. 147-148. ARMITAGE P, BERRY G (1987) Statistical Methods in Medical Research. Second edition. Oxford: Blackwell Scientific. BAUMANN RJ (1982) Classification and population studies of epilepsy. In: Genetic Basis of the Epilepsies. Edited by V. E. Anderson, W. A. Hauser, J. K. Penry and C. F. Singh. New York: Raven Press, pp. 11-20. BHARUCHA NE, BHARUCHA EP, BHARUCHA AE, BHISE AV, SCHOENBERG BS (1988) Prevalence of epilepsy
in the Parsi community of Bombay. Epilepsia, 29, 111 — 115. CARPIO A, MORALES J, CALLE H, TINOCO L, SANTILLAN F (1986) Prevalencia de epilepsia en la parroquia
Cumbe, Azuay, Ecuador. Revista del Instituto de Gencias de Salud Universidade de Cuenca, Ecuador, 1, 10-31. CHIOFALO N, SCHOENBERG BS, KIRSCHBAUM A, OUVARES O, ALVAREZ G, VALENZUELA B (1989) Estudios
epidemiologicos de las enfermedades neurologicas en Santiago Metropolitano, Chile. Abstracts of the IV Pan American Congress of Neuroepidemiology, Cartagena, Columbia.
Downloaded from by guest on July 12, 2015
The authors wish to thank R. Montenegro, F. Alarcon and J. Elsitdie for allowing us to use their hospital patients for the validation (Hospitals: Andrade Marin, Eugenio Espejo and Baca Ortiz); M. Roman, E. Arizaga, A. Madera, V. Paredes and C. Bimos for their neurological examination of the validation patients; IDEA (Instituto de Estudios Avanzados, Quito) for support; J. Suarez for his valuable comments on the manuscript; and G. Buendia and S. Berendsohn for logistic support. The study was supported by a grant from Diseases Control Programme, Ciba Geigy, Basel, Switzerland. The work is reported on behalf of the CME (Community Management of Epilepsy Project, through an agreement between the Ecuadorian Ministry of Public Health, the Ecuadorian League against Epilepsy, the Ecuadorian Society of Neurology and Ciba Geigy) and ICBERG (International Community-based Epilepsy Research Group).
VALIDATION OF AN EPILEPSY SCREEN
793
CRUZ M, RUALES J, BOSSANO F, PROANO J, BARBERIS P, COX L (1984) Estudios Neuroepidemiologicas
en el Ecuador. CIEN, Fundacion Eugenio Espejo, Quito: Ministerio de Salud Publica. GARCfA-PEDROZA F , RUBIO-DONNADIEU F , GARCfA-RAMOS G, EsCOBEDO-RlOS F , GONZALES-CORTES A
(1983) Prevalence of epilepsy in children: Tlalpan, Mexico City, Mexico. Neuroepidemiology, 2, 16-23. GUTIERREZ H, RUBIO F, ESCOBEDO F, HERON J (1980) Prevalencia de epilepsia en ninos de edad escolar de una comunidad urbana de la Ciudad de Mexico. Gaceta Medica Mexicana, 116, 497 — 501. HAERER AF, ANDERSON DW, SCHOENBERG BS (1986) Prevalence and clinical features of epilepsy in a biracial United States population. Epilepsia, 27, 66—75. HAUSER WA, KURLAND LT (1975) The epidemioloy of epilepsy in Rochester, Minnesota, 1935 through 1967. Epilepsia, 16, 1 - 6 6 . JUUL-JENSEN P, FOLDSPANG A (1983) Natural history of epileptic seizures. Epilepsia, 24, 297—312. Li S, SCHOENBERG BS, WANG C, CHENG X, ZHOU S, BOLIS CL (1985) Epidemiology of epilepsy in urban
areas of the People's Republic of China. Epilepsia, 26, 391—394. LILIENFELD AM, LILIENFELD DE (1980) Foundations of Epidemiology. Second Edition. Oxford University Press, pp. 150-153. MACMAHON B, PUGH TF (1981) Principios y Metados de Epidemiology. Mexico: La Prensa Medica Mexicana, pp. 242-245. MAUSNER JS, BAHN AK (1977) Epidemiology. Mexico: Edit. Interamericana, pp. 223-234. OSUNTOKUN BO, SCHOENBERG BS, NOTTIDGE VA, ADEUJA A, KALE O, ADEYEFA A et al. (1982) Research
protocol for measuring the prevalence of neurological disorders in developing countries: results of a pilot study in Nigeria. Neuroepidemiology, 1, 143 — 153. OSUNTOKUN BO, ADEUJA AO, NOTTIDGE VA, BADEMOSI O, OLUMIDE A, IGE O etal. (1987) Prevalence
of the epilepsies in Nigerian Africans: a community-based study. Epilepsia, 28, 272—279. scale study of epilepsy in Ecuador: methodological aspects. Neuroepidemiology, 11, 7 4 - 8 4 . PLACENCIA M, SHORVON SD, PAREDES V, BIMOS C, SANDER JWAS, SUAREZ J et al. (1992*) Epileptic
seizures in an Andean region of Ecuador: prevalence and incidence and regional variation. Brain, 115, 771-782. PLACENCIA M, SILVA C, CORDOVA M, ALARCON F, RODRIGUEZ G, CASTRO E (1984) Prevalencia de
Enfermedades Neurologicas en una Communidad Rural Andina (Cangahua). Informe final, Quito: CONACYT. PONCE P, FERNANDEZ R, D'SOUSE C, VILLIEGAS A, DE PONCE C, GONZALES B et al. (1985) Estudio
de prevalencia de transtornos neurologicos en Lezama. In: Estudios Neuroepidemiologicos en Venezuela. Caracas: Departamento de Enfermedades Neurologicas, Ministerio de Sanidad y Asistencia Social, pp. 16-38. PRADILLA G, PARDO C, PUENTES F, RIVERA R, BALLESTEROS G, FLANTRAMSKY H et al. (1984) Estudio
Neuroepidemiologico en Giron, Columbia. Comite de Neuroepidemiologica, Bucaramanga: Universidad Industrial de Santander. ROBB JP (1972) A review of epidemiologic concepts of epilepsy. In The Epidemiology of Epilepsy: a Workshop. NINDS Monograph No. 14. Edited by M. Alter and W. A. Hauser. Washington DC: US Department of Health Education, and Welfare, pp. 13-18. ROSE SW, PENRY JK, MARKUSH RE, RADLOFF LA, PUTNAM PL (1973) Prevalence of epilepsy in children.
Epilepsia, 14, 133-152. ROWAN AJ, HYMAN HH (1976) The prevalence of epilepsy in a large heterogeneous urban population (The Bronx, New York, January 8, 1975). Transactions of the American Neurological Association,
101, 281-283. RUMEAU-ROUQUETTE C, BR£ART G, PADIEU R (1981) Mithodes en tpidimiologie:
tchantillonnage,
Investigations, Analyse. Paris: Flammarion Mddecine-Science, pp. 118—119. SANDER JWAS, SHORVON SD (1987) Incidence and prevalence studies in epilepsy and their methodological problems: a review. Journal of Neurology, Neurosurgery and Psychiatry, 50, 829 — 839. SANDER JWAS, HART YM, JOHNSON AL, SHORVON SD (1990) National General Practice Study of Epilepsy: newly diagnosed epileptic seizures in a general population. Lancet, 336, 1267—1271. SCHOENBERG BS (1982a) Clinical neuroepidemiology in developing countries: neurology with few neurologists. Neuroepidemiology, 1, 137—142.
Downloaded from by guest on July 12, 2015
PLACENCIA M, SUAREZ J, CRESPO F, SHORVON SD, SANDER JWAS, ELLISON RH et al. (1992a) A large
794
M. PLACENCIA AND OTHERS
SCHOENBERG BS (1982*) Neuroepidemiologic definitions of diseases. Neuroepidemiology, 1, 197-200. SCHOENBERG BS (1982c) The scope of neuroepidemiology: from stone age to Stockholm. Neuroepidemiology, 1, 1-16. SHORVON SD, FARMER PJ (1988) Epilepsy in developing countries: a review of epidemiological, sociological, and treatment aspects. Epilepsia, 29, Supplement 1, S36—S54. VECCHIO TJ (1966) Predictive value of a single diagnostic test in unselected populations. New England Journal of Medicine, 274, 1171 - 1173. ZIELINSKJ JJ (1974) Epidemiology and Medico-social Problems of Epilepsy in Warsaw. Final Report on Research Program No.l9-P-58325-F-01. Warsaw: Psychoneurological Institute, pp. 5 0 - 5 5 . (Received August 20, 1991. Revised February 6, 1992. Accepted April 6, 1992)
APPENDIX 1 Original screening questionnaire as used during the clinic-based validation
(14) (15) (16) (17) (18) (19) (20)
Have you ever lost consciousness? Have you ever had attacks in which you loose contact with the surroundings? Have you ever had attacks of shaking of the arms or legs which you could not control? Have you ever had attacks in which you fall to the ground with loss of consciousness? Have you ever had attacks in which you fall and bite your tongue? Have you ever had attacks in which you fall and lose control of your bladder? Have you ever had attacks in which you fall and become pale? Have you ever had attacks in which you lose your memory for a short period of time? Have you ever had attacks of strange behaviour and loss of memory? Have you ever had brief attacks of shaking or trembling in one arm or leg or in the face? Have you ever had attacks of tingling or numbness which move up your arm, leg or body? Have you ever had attacks of jerkings which move up your arm, leg or body? Have you ever had attacks in which you lose contact with the surrounding and experience a feeling of unreality or dreaminess? Have you ever had attacks in which you lose contact with the surrounding and experience a sensation in which objects change shape or size? Have you ever had attacks in which you lose contact with the surrounding and experience abnormal visions? Have you ever had attacks in which you lose contact with the surrounding and experience abnormal sounds? Have you ever had attacks in which you lose contact with the surroundings and experience abnormal smells? Have you ever had attacks in which you behave momentarily in a confused fashion? Have you ever had attacks of palpitation? Have you ever been told that you have or had epilepsy or epiliptic fits?
APPENDIX Screening
questionnaire
2
as used during the large-scale
survey in the
community
(1) Have you ever had attacks of shaking of the arms or legs which you could not control? (2) Have you ever had attacks in which you fall and become pale? Both question I and 2 must be affirmative to render the subject positive. (3) (4) (5) (6) (7) (8) (9)
Have Have Have Have Have Have Have
you ever you ever you ever you ever you ever you ever you ever
tost consciousness? had attacks in which you fall with loss of consciousness? had attacks in which you fall and bite your tongue? had attacks in which you fall and lose control of your bladder? had brief attacks of shaking or trembling in one arm or leg or in the face? had attacks in which you lose contact with the surroundings and experience abnormal smells? been told that you have or had epilepsy or epileptic fits?
Any question 3 to 9 if affirmative renders the subject positive.
Downloaded from by guest on July 12, 2015
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13)
