Community Dent. Oral Ejidemiol. 1976: 4: 98-101 (Key words: denfal health rurucyr)
Reliability and constancy of information obtained in dental health interviews P. W. NORIIEIM AND L. A. HELOE Department of Prosthodontics and Dental Institute of Experimental Research, University of Oslo, Oslo, Norway
- I n a group of 297 persons intervieweL about dental health, a random sample of 50 persons was reinterviewed by the same person, posing identical questions. T h e average interval between interviews was 9 months. Replies to questions concerning actual conditions, i.e. edentulousness, time of extraction and time of filling, showed a high degree of stability and constancy. The subjects’ attitude to water fluoridation appeared, on the other hand, to have changed somewhat. Similarly, some of the respondents appeared to have acquired knowledge of dental health care as a result of the first interview and had altered their standpoint accordingly. Still, the results of the replies to the interview seemed reliable as defined by stability over time. ARSTRACT
(Received for publication 18 October, accepted 30 November 1975)
In the last decade it has become customary to utilize the interview form in dental health survey^'^. I n dentistry, this method can for example be used to obtain information about knowledge, attitudes, and behavior. Interviews are furthermore used in the collection of various types of epidemiologic data. Often, however, doubt can be raised as to whether the gathering and processing of information give sufficiently reliable information5*Is. One important aspect of reliability is stabilityl7. Stability denotes reliability over time when the same researcher examines the same phenomena using the same measuring took’. A second question is whether the characteristics being measured are consistent or fluctuate from one application of the measure to anThe term constancy‘ or consistency3 of trait is very often confused with reliability5 of measurement. In this investigation, data from two “identical” interviews are compared by means of the so-called test-retest method*. The intention was to assess the degree of stability’ of measurement and the degree of constancy of the measured traits.
MATERIAL AND METHODS I n Lyngseidet, a center of a coastal community in Northern Norway with a total population of 358 in the age group 20-69 years, 297 persons participated in a combined interview and clinical examination carried out at the local public dental clinici3. T h e interview, which used the open question format, dealt with knowledge of and attitudes concerning dental health, dental hygiene, and dental visits. I n August 1973, 50 persons were drawn a t random among the survey participants and reinterviewed by the same interviewer (P.W.N.) using the same questions. This test group deviated only slightly from the group with regard to sex, age, and income. The average interval between the interviews was 9 months. A comparison of replies to strategic questions was carried out for each person. The degree of observed agreement was computed, and correction for possible chance agreement was introduced, i.e. Scott’s pi16, which varies from 0.00 to 1.00: pi =
% observed agreement - agreement expected by chance 100 - agreement expected by chance Care was taken to standardize the interviews, for example by using the same interviewer and by not allowing the presence of outsiders during the interview#. Also, the questions had been pretested in a pilot study14, and possible errors in data processing were sought eliminated through checking of coding’?, 20 and punching5.
Information obtained in dental health surveys
99
Table 1. A comparison of replies given to the question: “DO you feel that fluoride should be added to the drinking-water?” Reply alternatives
Interview 1
Interview 2 Yes
Don’t know
No
Total Interview 1
17 7 2
7 6 1
2 5 3
26 18 6
26
14
10
50
Yes Don’t know NO
Total Interview 2
CHILD DENTAL CARE
RESULTS
Question 3: “ A t what age do you feel that children ought to begin seeing the dentist?” In Interview 1, 31 persons gave an answer of 3 years or under; in Interview 2, 30 of these same persons gave an identical reply. I n the second interview a total of 39 replied 3 years or under. There was an agreement for 36 of the 50 replies and Scott’s pi was 0.41 (the number of cells in the Table was 9).
The actual wording of the various questions as well as the agreement data are given below. For questions 1, 4, and 5 the results of the comparison of the two data sets are presented in Tables 1-3. The questions are grouped according to the following topics: fluorides, child dental care, gum disease, and dental visits. FLUORIDES
GUM DISEASE Question 4: crHaveyou ever had a gum disease?” The distribution of the replies and the correspondence are presented in detail in Table 2. Full agreement was found for 35 data pairs, and Scott’s pi was 0.45.
Question 1 : “Do you think that fluoride should be added to the drinking-water?” From Table 1, 26 subjects out of the 50 gave identical answers in both interviews. The degree of agreement as measured by Scott’s pi was 0.23. Question 2: ”Do you think that it is right for children to receive topical fluoride treatment at school?” Forty-three persons answered “yesY’and two persons replied in both interviews that they did not know. This gave 45 identical answers, giving a Scott’s pi of 0.41 (the number of cells in the Table was 9). There were no negative replies in the first interview; in the second, there were two.
DENTAL VISITS
Question 5: “ H o w old were you when the last upper teeth were pulled?” The agreement in the replies was complete concerning the state of edentulousnes and the possession of natural teeth (Table 3 ) . Of the 24 persons having had all maxillary teeth extracted, there was full
Table 2. A comparison of replies given to the question: “Have you at any time had a gum disease?” Interview 2 Reply alternatives
Interview 1
No, never Yes, bothered somewhat Yes, much bothered Total Interview 2
Total Interview 1
No, never
Yes, bothered somewhat
Yes, much bothered
26
4 4 4
1
4 2
5
31 8 11
32
12
6
50
100 NORHEIM AND HELOE Table 3. A comparison of replies given to the question: “How old were you when the last upper teeth were pulled?” Interview 2 Has natural teeth Has natural teeth 5 19 years Interview 1 2C-24 years 25-29 years 30-34 years 2 35 years Total Interview 2
5 19 years
20-24 years
25-29 years
30-34 years
2 35 years
26
26
1 2
26
Total Interview 1
3
1 4 1
6
correspondence in 17 cases. There was, however, no indication of a tendency to raising or lowering of the age at the second interview (Scott’s pi = 0.79). Question 6: “Have you at any time had your teeth filled?” Of the 50 persons taking part in the test, 41 of the same people replied “yes”, while nine replied “no” in both interviews, i.e. a complete agreement in the distribution of answers, giving a Scott’s pi of 1. Question 7: “Did you go to the school dentist when you attended school?” The “no” and “yes” groups consisted of, respectively, the same 39 and 11 persons in both interviews, giving a complete agreement in the replies and a Scott’s pi of 1.
DISCUSSION The values for Scott’s pi varied in the two questions dealing with fluorides. I t is not likely that this was only due to a lower stability rating for Question 1 than for Question 2. Table 1 shows that although the total distribution of answers to Question 1 was much alike in both interviews, every second respondent had nevertheless changed his/ her answer. Possibly this shift between categories may be ascribed to uncertainty or to a real change of opinion with regard to fluoridations, i.e. low constancy of the observed phenomenal In Question 2, there was a better correspondence between the replies. Brushing with fluoride had been practiced at
1 7
8
2 7 2 1
3
10
4
1 4
4
50
Lyngseidet school for about 2 years, a circumstance which probably had made the population aware of and accept the practice. The reason for the low Scott’s pi was that 43 of the data pairs were located in the same cell, and agreement expected by chance was accordingly large”. Presumably, the transition to a “correct” viewpoint in Interview 2 concerning dental visits of children (Question 3 ) was to some extent due to the influence of Interview 1. Some respondents may have become influenced by the opinion held by the interviewer. The fact that the question was even posed, could have resulted in the respondent informing him/herself of the “correct” answer’>17. The relatively low pi value on this occasion therefore probably also expressed a low “constancy”. The high proportion of negative replies and the low pi value for Question 4 confirmed earlier findings with regard to people’s knowledge about their diseased gingiva*. It is striking that of the 11 persons saying that they were greatly bothered (Interview l ) , o d y five repeated themselves in Interview 2 (Table 2 ) . Many respondents are conceivably not aware of which disease this is and for this reason confuse abscess and gingival disease. I t is likely that Interview 1 to a certain degree had an informative value which resulted in some of the interviewees correcting their reply in Interview 2. One can therefore assume that by repeating Question 5 , stability was only measured to a small degree because the first test was not valid”, i.e. definition of the concept “gum disease” differed between the interviewer and the respondent.
Information obtained in dental health surveys 101 The replies dealing with visits to the dentist showed a high degree of correspondence, a fact which appears to corroborate earlier results regarding reliability and validity of this type of information10’18and, for that matter, the Occurrence of teeth and prostheses’. The declaration of ages for extraction of the maxillary teeth showed small variations, which, however, would have to a great degree been incorporated given a somewhat more liberal graduation of the intervals (Table 3 ) . Summing up, t h i degree of agreement between the replies vaned clearly between the questions: 1 ) The replies given regarding knowledge of facts, e.g. edentulousness, “having had teeth filled”, “been to the school dentist”, showed a high degree of stability and constancy. I t may be that this type of information could be easily remembered or calculated by the interviewees. 2 ) Replies to questions dealing with attitudes, in this connection toward fluorides, showed a lower degree of correspondence; one likely explanation is that many of the respondents changed their opinion. 3 ) Replies to questions dealing with knowledge about the age for commencing dental care for children and awareness of one’s own gingival disease also showed a lower correspondence. I t is likely that the main reason was uncertainty because their knowledge of the subject was vague, and that the interviewees had been influenced by the first interview. Although it is difficult to distinguish between stability and constancy5-‘?I7, the data from the present study suggest that the stability remained high throughout all questions. The constancy, on the other hand, tended to be high for responses dealing with known facts, and moderate for responses dealing with opinions, and when the knowledge was sketchy.
REFERENCES ABRAMSON, J. H.: Survey methods in community medicine. Churchill-Livingstone, Edinburgh and London 1974, p. 85-94. BEFRING, E.: Deskriptiv statistikk og mdlingsproblemer. Scandinavian University Books, Universitetsforlaget, Oslo 1969, p. 35-39. BORCATTA, E. F. & BALES,R. F.: The consistency of subject behavior and the reliability of scoring in interaction process analyses. A m . Sociol. Rev. 1958: 23: 566-569.
4. EKMAN,G.: Reliabilitet och konstans. Ett bidrag ti1 testpsykologiens metodologi. Almquist & Wiksells Boktrykkeri AB, Uppsala 1947. 5. GALTUNG, J.: Theory and methods of social research. Scandinavian University Books, Universitetsforlaget, Oslo 1969, p. 109-128, 169-180. 6. HELLEVIK,0.: Forskningsmetode i sosiologi og statsvitenskap. Scandinavian University Books, Universitetsforlaget, Oslo 1971, p. 140-154. G. C.: Principles of psychological mea7. HELMSTADTER, surements. Methuen & Co. Ltd., London 1966, p. 6366. 8. HELOE,L. A.: Comparison of dental health data obtained from questionnaires, interviews and clinical examination. Scand. J. Dent. Res. 1972: 80: 495499. J. M.: The public opinion 9. HELOE,L. A. & BIRKELAND, in Norway on water fluoridation. Community Dent. Oral Epidemiol. 1974: 2 : 95-97. 10. JENSEN, K.: Tandpleje: Adfard-viden-attituder. En socialodontologisk enqueteundersegelse af 2500 16-45 drige danskere fra udvalgte omrider. Thesis, Odontologisk Forlag, Aarhus 1973, p. 44-46. F.: National Health Interview Surveys. Public 11. LINDER, Health Pap. 1965: 27. B.: The re12. MOUTON,J. S., BLAKE,R. R. & FRUCIITER, liability of sociometric reasons. Sociom. 1955: 18: 748. P. W. & HELOE,L. A.: Comparison between 13. NORHEIM, participants and non-participants in a dental health survey in Northern Norway. Community Dent. Oral Epidemiol. 1975: 3: 56-60. 14. PARTEN,M.: Surveys, polls and samples. Harper & Row, New York 1950, p. 55-59. 15. RICHARDS,N. D. & COHEN,L. K.: Social sciences and dentistry: a critical bibliography. FCderation Dentaire Internationale, The Hague 1971. 16. SCOTT,W. A.: Reliability of central analysis: the case of Nominal Scale Coding. Public Opin. Q. 1955: 19: 321-325. C., IAHODA, M., DEUTSCH, M. & COOK, S. W.: 17. SELLTIZ, Research methods in social relations. Holt, Rinehart and Winston, New York 1958, p. 166-172, 401-406. B.: Tandvdrdsvanor och tandvdrdskostnader. 18. SMEDBY, Statens offentliga utordningar. Stockholm 1965: 4: 167-1 84. F. F. & MCCARTHY, P. J.: Sampling opin19. STEPHEN, ions. A n analysis of survey procedure. Wiley & Sons, New York 1958, p. 360-371. J. L. & RAYMOND, F.: A study of coding 20. WOODWARD, reliability. Publ. Opin. Q. 1948: 12: 253-257.
Address: P . W. Norheim Department of Prosthodontics Dental Faculty 69 Geitmyrsveien Oslo 4 Norway
.
Reliability and constancy of information obtained in dental health interviews P. W. NORIIEIM AND L. A. HELOE Department of Prosthodontics and Dental Institute of Experimental Research, University of Oslo, Oslo, Norway
- I n a group of 297 persons intervieweL about dental health, a random sample of 50 persons was reinterviewed by the same person, posing identical questions. T h e average interval between interviews was 9 months. Replies to questions concerning actual conditions, i.e. edentulousness, time of extraction and time of filling, showed a high degree of stability and constancy. The subjects’ attitude to water fluoridation appeared, on the other hand, to have changed somewhat. Similarly, some of the respondents appeared to have acquired knowledge of dental health care as a result of the first interview and had altered their standpoint accordingly. Still, the results of the replies to the interview seemed reliable as defined by stability over time. ARSTRACT
(Received for publication 18 October, accepted 30 November 1975)
In the last decade it has become customary to utilize the interview form in dental health survey^'^. I n dentistry, this method can for example be used to obtain information about knowledge, attitudes, and behavior. Interviews are furthermore used in the collection of various types of epidemiologic data. Often, however, doubt can be raised as to whether the gathering and processing of information give sufficiently reliable information5*Is. One important aspect of reliability is stabilityl7. Stability denotes reliability over time when the same researcher examines the same phenomena using the same measuring took’. A second question is whether the characteristics being measured are consistent or fluctuate from one application of the measure to anThe term constancy‘ or consistency3 of trait is very often confused with reliability5 of measurement. In this investigation, data from two “identical” interviews are compared by means of the so-called test-retest method*. The intention was to assess the degree of stability’ of measurement and the degree of constancy of the measured traits.
MATERIAL AND METHODS I n Lyngseidet, a center of a coastal community in Northern Norway with a total population of 358 in the age group 20-69 years, 297 persons participated in a combined interview and clinical examination carried out at the local public dental clinici3. T h e interview, which used the open question format, dealt with knowledge of and attitudes concerning dental health, dental hygiene, and dental visits. I n August 1973, 50 persons were drawn a t random among the survey participants and reinterviewed by the same interviewer (P.W.N.) using the same questions. This test group deviated only slightly from the group with regard to sex, age, and income. The average interval between the interviews was 9 months. A comparison of replies to strategic questions was carried out for each person. The degree of observed agreement was computed, and correction for possible chance agreement was introduced, i.e. Scott’s pi16, which varies from 0.00 to 1.00: pi =
% observed agreement - agreement expected by chance 100 - agreement expected by chance Care was taken to standardize the interviews, for example by using the same interviewer and by not allowing the presence of outsiders during the interview#. Also, the questions had been pretested in a pilot study14, and possible errors in data processing were sought eliminated through checking of coding’?, 20 and punching5.
Information obtained in dental health surveys
99
Table 1. A comparison of replies given to the question: “DO you feel that fluoride should be added to the drinking-water?” Reply alternatives
Interview 1
Interview 2 Yes
Don’t know
No
Total Interview 1
17 7 2
7 6 1
2 5 3
26 18 6
26
14
10
50
Yes Don’t know NO
Total Interview 2
CHILD DENTAL CARE
RESULTS
Question 3: “ A t what age do you feel that children ought to begin seeing the dentist?” In Interview 1, 31 persons gave an answer of 3 years or under; in Interview 2, 30 of these same persons gave an identical reply. I n the second interview a total of 39 replied 3 years or under. There was an agreement for 36 of the 50 replies and Scott’s pi was 0.41 (the number of cells in the Table was 9).
The actual wording of the various questions as well as the agreement data are given below. For questions 1, 4, and 5 the results of the comparison of the two data sets are presented in Tables 1-3. The questions are grouped according to the following topics: fluorides, child dental care, gum disease, and dental visits. FLUORIDES
GUM DISEASE Question 4: crHaveyou ever had a gum disease?” The distribution of the replies and the correspondence are presented in detail in Table 2. Full agreement was found for 35 data pairs, and Scott’s pi was 0.45.
Question 1 : “Do you think that fluoride should be added to the drinking-water?” From Table 1, 26 subjects out of the 50 gave identical answers in both interviews. The degree of agreement as measured by Scott’s pi was 0.23. Question 2: ”Do you think that it is right for children to receive topical fluoride treatment at school?” Forty-three persons answered “yesY’and two persons replied in both interviews that they did not know. This gave 45 identical answers, giving a Scott’s pi of 0.41 (the number of cells in the Table was 9). There were no negative replies in the first interview; in the second, there were two.
DENTAL VISITS
Question 5: “ H o w old were you when the last upper teeth were pulled?” The agreement in the replies was complete concerning the state of edentulousnes and the possession of natural teeth (Table 3 ) . Of the 24 persons having had all maxillary teeth extracted, there was full
Table 2. A comparison of replies given to the question: “Have you at any time had a gum disease?” Interview 2 Reply alternatives
Interview 1
No, never Yes, bothered somewhat Yes, much bothered Total Interview 2
Total Interview 1
No, never
Yes, bothered somewhat
Yes, much bothered
26
4 4 4
1
4 2
5
31 8 11
32
12
6
50
100 NORHEIM AND HELOE Table 3. A comparison of replies given to the question: “How old were you when the last upper teeth were pulled?” Interview 2 Has natural teeth Has natural teeth 5 19 years Interview 1 2C-24 years 25-29 years 30-34 years 2 35 years Total Interview 2
5 19 years
20-24 years
25-29 years
30-34 years
2 35 years
26
26
1 2
26
Total Interview 1
3
1 4 1
6
correspondence in 17 cases. There was, however, no indication of a tendency to raising or lowering of the age at the second interview (Scott’s pi = 0.79). Question 6: “Have you at any time had your teeth filled?” Of the 50 persons taking part in the test, 41 of the same people replied “yes”, while nine replied “no” in both interviews, i.e. a complete agreement in the distribution of answers, giving a Scott’s pi of 1. Question 7: “Did you go to the school dentist when you attended school?” The “no” and “yes” groups consisted of, respectively, the same 39 and 11 persons in both interviews, giving a complete agreement in the replies and a Scott’s pi of 1.
DISCUSSION The values for Scott’s pi varied in the two questions dealing with fluorides. I t is not likely that this was only due to a lower stability rating for Question 1 than for Question 2. Table 1 shows that although the total distribution of answers to Question 1 was much alike in both interviews, every second respondent had nevertheless changed his/ her answer. Possibly this shift between categories may be ascribed to uncertainty or to a real change of opinion with regard to fluoridations, i.e. low constancy of the observed phenomenal In Question 2, there was a better correspondence between the replies. Brushing with fluoride had been practiced at
1 7
8
2 7 2 1
3
10
4
1 4
4
50
Lyngseidet school for about 2 years, a circumstance which probably had made the population aware of and accept the practice. The reason for the low Scott’s pi was that 43 of the data pairs were located in the same cell, and agreement expected by chance was accordingly large”. Presumably, the transition to a “correct” viewpoint in Interview 2 concerning dental visits of children (Question 3 ) was to some extent due to the influence of Interview 1. Some respondents may have become influenced by the opinion held by the interviewer. The fact that the question was even posed, could have resulted in the respondent informing him/herself of the “correct” answer’>17. The relatively low pi value on this occasion therefore probably also expressed a low “constancy”. The high proportion of negative replies and the low pi value for Question 4 confirmed earlier findings with regard to people’s knowledge about their diseased gingiva*. It is striking that of the 11 persons saying that they were greatly bothered (Interview l ) , o d y five repeated themselves in Interview 2 (Table 2 ) . Many respondents are conceivably not aware of which disease this is and for this reason confuse abscess and gingival disease. I t is likely that Interview 1 to a certain degree had an informative value which resulted in some of the interviewees correcting their reply in Interview 2. One can therefore assume that by repeating Question 5 , stability was only measured to a small degree because the first test was not valid”, i.e. definition of the concept “gum disease” differed between the interviewer and the respondent.
Information obtained in dental health surveys 101 The replies dealing with visits to the dentist showed a high degree of correspondence, a fact which appears to corroborate earlier results regarding reliability and validity of this type of information10’18and, for that matter, the Occurrence of teeth and prostheses’. The declaration of ages for extraction of the maxillary teeth showed small variations, which, however, would have to a great degree been incorporated given a somewhat more liberal graduation of the intervals (Table 3 ) . Summing up, t h i degree of agreement between the replies vaned clearly between the questions: 1 ) The replies given regarding knowledge of facts, e.g. edentulousness, “having had teeth filled”, “been to the school dentist”, showed a high degree of stability and constancy. I t may be that this type of information could be easily remembered or calculated by the interviewees. 2 ) Replies to questions dealing with attitudes, in this connection toward fluorides, showed a lower degree of correspondence; one likely explanation is that many of the respondents changed their opinion. 3 ) Replies to questions dealing with knowledge about the age for commencing dental care for children and awareness of one’s own gingival disease also showed a lower correspondence. I t is likely that the main reason was uncertainty because their knowledge of the subject was vague, and that the interviewees had been influenced by the first interview. Although it is difficult to distinguish between stability and constancy5-‘?I7, the data from the present study suggest that the stability remained high throughout all questions. The constancy, on the other hand, tended to be high for responses dealing with known facts, and moderate for responses dealing with opinions, and when the knowledge was sketchy.
REFERENCES ABRAMSON, J. H.: Survey methods in community medicine. Churchill-Livingstone, Edinburgh and London 1974, p. 85-94. BEFRING, E.: Deskriptiv statistikk og mdlingsproblemer. Scandinavian University Books, Universitetsforlaget, Oslo 1969, p. 35-39. BORCATTA, E. F. & BALES,R. F.: The consistency of subject behavior and the reliability of scoring in interaction process analyses. A m . Sociol. Rev. 1958: 23: 566-569.
4. EKMAN,G.: Reliabilitet och konstans. Ett bidrag ti1 testpsykologiens metodologi. Almquist & Wiksells Boktrykkeri AB, Uppsala 1947. 5. GALTUNG, J.: Theory and methods of social research. Scandinavian University Books, Universitetsforlaget, Oslo 1969, p. 109-128, 169-180. 6. HELLEVIK,0.: Forskningsmetode i sosiologi og statsvitenskap. Scandinavian University Books, Universitetsforlaget, Oslo 1971, p. 140-154. G. C.: Principles of psychological mea7. HELMSTADTER, surements. Methuen & Co. Ltd., London 1966, p. 6366. 8. HELOE,L. A.: Comparison of dental health data obtained from questionnaires, interviews and clinical examination. Scand. J. Dent. Res. 1972: 80: 495499. J. M.: The public opinion 9. HELOE,L. A. & BIRKELAND, in Norway on water fluoridation. Community Dent. Oral Epidemiol. 1974: 2 : 95-97. 10. JENSEN, K.: Tandpleje: Adfard-viden-attituder. En socialodontologisk enqueteundersegelse af 2500 16-45 drige danskere fra udvalgte omrider. Thesis, Odontologisk Forlag, Aarhus 1973, p. 44-46. F.: National Health Interview Surveys. Public 11. LINDER, Health Pap. 1965: 27. B.: The re12. MOUTON,J. S., BLAKE,R. R. & FRUCIITER, liability of sociometric reasons. Sociom. 1955: 18: 748. P. W. & HELOE,L. A.: Comparison between 13. NORHEIM, participants and non-participants in a dental health survey in Northern Norway. Community Dent. Oral Epidemiol. 1975: 3: 56-60. 14. PARTEN,M.: Surveys, polls and samples. Harper & Row, New York 1950, p. 55-59. 15. RICHARDS,N. D. & COHEN,L. K.: Social sciences and dentistry: a critical bibliography. FCderation Dentaire Internationale, The Hague 1971. 16. SCOTT,W. A.: Reliability of central analysis: the case of Nominal Scale Coding. Public Opin. Q. 1955: 19: 321-325. C., IAHODA, M., DEUTSCH, M. & COOK, S. W.: 17. SELLTIZ, Research methods in social relations. Holt, Rinehart and Winston, New York 1958, p. 166-172, 401-406. B.: Tandvdrdsvanor och tandvdrdskostnader. 18. SMEDBY, Statens offentliga utordningar. Stockholm 1965: 4: 167-1 84. F. F. & MCCARTHY, P. J.: Sampling opin19. STEPHEN, ions. A n analysis of survey procedure. Wiley & Sons, New York 1958, p. 360-371. J. L. & RAYMOND, F.: A study of coding 20. WOODWARD, reliability. Publ. Opin. Q. 1948: 12: 253-257.
Address: P . W. Norheim Department of Prosthodontics Dental Faculty 69 Geitmyrsveien Oslo 4 Norway
.
