Assessing Reliability
clinical signs of temporomandibular of clinical examiners
Samuel F. Dworkin, D.D.S., Ph.D.,* Linda LeResche, Timothy DeRouen, Ph.D.,*** and Michael Von Korff, University of Washington, School of Dentistry, Seattle, Wash.
disorders:
Sc.D.,** Sc.D.****
Data on interrater reliability in assessing a number of clinical signs commonly evaluated in the diagnosis and treatment of temporomandibular disorders (TMD) is presented in this article. Four experienced dental hygienists who were 5eld examiners for a large epidemiologic study of TMD and three experienced clinical TMD specialists (dentists) who are coinvestigators in the same study followed carefully detailed speci5cations and criteria for examination of TMD patients and pain-free controls. Excellent reliability was found for vertical range of motion measures and for summary indices measuring the overall presence of a clinical sign that could arise from several sources (for example, summary indices of muscle palpation pain). However, many clinical signs important in the differential diagnosis ofxubtypes of TMD were not measured with high reliability. In particular, assessment of pain in response to muscle palpation and identi5cation of speci5c temporomandibular joint sounds seemed to be possible only with modest, sometimes marginal, reliability. These modest reliabilities could arise from examiner error because the clinical signs are themselves unreliable, changing spontaneously over time and making it difficult to 5nd the same sign on successive examinations. The finding that, without calibration, experienced clinicians showed low reliability with other clinicians suggests the importance of establishing reliable clinical standards for the examination and diagnostic classification of TMD. (J PROSTAET DENT 1990;63:574-9.)
A
consensusis emerging that persistent pain and dysfunction in the temporomandibular region probably reflects not a single critical entity but a variety of related clinical conditions.’ The term “temporomandibular disorders” (TMD) has been suggestedby the American Dental Association President’s Conference on the Examination, Diagnosis, and Treatment of Temporomandibular Disorders2as a collective label for several related but probably diagnostically distinct states of chronic orofacial pain. Included under the broad TMD category are musclerelated orofacial pain states, such asmyofascial pain dysfunction syndrome (MPD) and other diagnostic subtypes that might reflect pain and dysfunction associatedwith internal derangement(ID) or degenerativedisease(DJD) of
*Professor, Departmentof Oral Medicine;Professor, Department of PsychiatryandBehavioralSciences, Universityof Washington, Schoolof Medicine. **Research AssociateProfessor,Departmentof Oral Medicine. ***Professor,Departmentof DentalPublicHealthSciences, and Departmentof Biostatistics,Schoolof Public Health, University of Washington. ****AssociateScientificInvestigator,GroupHealth Cooperative of PugetSound,Seattle,Wash. 10/l/18982 574
the structural componentsof the temporomandibularjoint (TMJ). It is recognized that the differential diagnosisof TMD accordingto oneor another of its subtypesis often difficult. Signsand symptomsof the different diagnostic categories of TMD may overlap. More than one clinical problem can be present at the sametime (for example, patients may have clinical signsof both MPD and ID), and the clinical signsmay be difficult to measurewith consistency.3l4 This article presentsclinical data regarding the dependability with which important clinical signsof TMD can be measured.The needto know whether signsof TMD can be measuredreliably is obviously of fundamental importance. As dentists gather data to formulate the diagnosisof a particular TMD, it is reasonableto expect that relevant clinical signsshould be measuredwith consistency.If, for example,on the samemeasurementtrial, one clinician heard a joint soundand another clinician did not or, in the same patient, someclinicians found a masticatory muscletender to palpation and others did not, confidence that the clinical signbeingmeasuredwasa valid diagnosticindicator for a particular patient would be diminished. Conversely, clinicians would probably agree that when clinical findings can be measuredwith high agreement,diagnosesare more likely to be valid. MAY1090
VOLUME63
NUMBER5
ASSESSING
CLINICAL
SIGNS
OF TMD
In medical and dental diagnosis, the consistency of measurement of clinical signs by different clinicians, that is, the interrater reliability of clinical signs, is often found to be less than desirable, especially when clinicians have not been carefully trained to use comparable methods or a uniform set of criteria.5,6 In medicine, this observation extends to the reading of radiographs and the measurement of physical signs.7s8 For dentistry, the World Health Organization9 has recommended that data regarding reliability of clinical measurements be a routine part of all oral health survey reports, reliably measuring clinical signs of the most common oral health indicators, such as caries, periodontal health, and oral hygiene status. In previous reports,lO* I1 published studies were reviewed regarding reliability of TMD-related clinical measures and factors were discussed that might significantly influence the reliability of clinical signs commonly measured to evaluate TMD.2 The most critical issues identified included (1) training of clinical examiners; (2) use of appropriate samples of symptomatic and asymptomatic subjects for reliability studies; (3) choosing among alternative statistical approaches for analysis of reliability data; and (4) changes in clinical signs, either spontaneous changes over time or changes due to repeated testing. This article presents the reliability of clinical signs commonly evaluated in the diagnosis and treatment of TMD. These detailed findings are from reliability studies included in a large epidemiologic study of TMD presently in progress.12 The epidemiologic study of TMD entails field examinations to measure clinical signs of TMD in patients under treatment, symptomatic community patients not under treatment, and asymptomatic community controls. The reliability studies discussed in this article were designed to assess the consistency of measurement among the field examiners. The following sections will present the design of the reliability studies and the specific reliability data for critical TMD variables, including range of motion, joint sounds, muscle palpation pain, and occlusal classification. Finally, these findings are discussed in terms of their implications for the differential diagnosis and classification of TMD.
DESIGN Subjects
AND
METHODS
Subjects included asymptomatic, normal, healthy volunteers, selected from the University community, patients in the Clinic of the Department of Orthodontics representing a variety of occlusal disharmonies and malocclusions, and patients experiencing pain and related symptoms of TMD recruited from the Orofacial Pain and Dysfunction Clinic of the Department of Oral Medicine, University of Washington. A total of 64 subjects participated in the reliability studies. The subjects were predominantly women in the 20- to 40-year age group, reflecting the distribution of persons seeking TMD treatment. The subjects represented a balanced sample of asymptomatic and symptomTHE
JOURNAL
OF PROSTHETIC
DENTISTRY
atic persons. TMD patients comprised approximately 60% of the subjects and 40 % of the patients were asymptomatic for TMD, thus maximizing our ability to observe the extent to which examiners could reliably agree in detecting signs when they were present and not detecting them when the signs were absent. All of the subjects provided signed, informed consent and were modestly remunerated for their efforts.
Examiners Examiners included four experienced registered dental hygienists who served as field examiners for a larger epidemiologic study and three experienced clinical TMD specialists (dentists) who were coinvestigators in the same epidemiologic study of TMD. The coinvestigators developed carefully detailed specifications and criteria that were used by the field examiners as their examination protocol. The field examiners underwent 40 hours of supervised training and calibration, including preliminary reliability assessment and retraining where indicated, before conducting actual field examinations with patients and controls. The study team dentists, who were experienced TMD specialists, did not undergo similar training. Hence, they were not calibrated in the use of the TMD examination specifications and criteria they collaborated in developing. The results of the reliability studies were analyzed with regard to the clinical examiner’s familiarity in using a standardized set of examination specifications and criteria, enabling the comparison of trained and calibrated examiners (CE) with uncalibrated examiners (UE).
Clinical
variables
Table I contains all the clinical variables included in the reliability studies reported and a brief outline of the criteria used for measurement. Clinical measurements were obtained for a variety of range-of-motion variables, including vertical, lateral, protrusive, and retrusive jaw movements; joint sounds by palpation and by stethoscope during vertical and other excursions of the jaw; pain elicited in response to palpation of predetermined sites on the muscles of mastication, the TMJ, and adjacent structures; and classification of occlusal relationships. The variables included in these reliability studies are a representative subset of clinical variables measured in the actual field examination. These variables have been included by many workers for their importance in the diagnosis, classification, and treatment of TMD.5y 6$13,I4
Protocol
for reliability
studies
An incomplete Latin square design was used to ensure that each subject was seen by each examiner in a randomized sequence to determine whether the order of examination influenced responses. Twenty-four subjects were assessed in this fashion by four CEs and two UEs on variables labelled as “fixed” because it was presumed that repeated trials would not influence the clinical findings. These vari575
DWORlIIN
Table
I. Clinical signsand criteria for measurement Cliiical
sign
I. Range of motion A Vertical dimension 1’ Unassisted opening, no pain 2 Maximum unassisted opening 3 Ma~mum assisted opening 4 Jaw opening pattern B Excursive movements 1 Lateral excursions 2 Protruded movement 3 Protrusive pattern II. Occlusal and tooth reIatio~~~ A Posterior occlusion B Anterior occlusion C Cross bite/open bite I Posterior crossbite 2 Anterior crossbite 3 Vertical overlap 4 Horizontal overbite D Tooth wear I Posterior tooth wear 2 Canine tooth wear 8 Anterior tooth wear III. TMJ sounds A Joint sounds with vertical opening 1 Palpation 2 Stethoscope 3 Click/pop on opening B Joint sounds with excursive movements 1 Lateral excursion 2 Protruded movement IV. Pain on function and palpation A Pain with function/movement 1 Biting/chewing 2 Lateral excursions 3 Protruded movement B Palpation pain 1 Extraoral muscles
2 Intraoral muscles 3 TMJ
Criteria
for measurement
(mm) Between maxillary and mandibular labioincisal embrasures Straight; R/L corrected deviation; RiL uncorrected deviation
(mm) From upper to lower incisor embrasures (mm) From upper lingual to lower labial incisor surfaces Straight; R/L corrected and uncorrected deviation Angle’s ciass f; class II; class III Division I; division II
Buccal or lingual deviation (mm) Overlap maxillary central incisor (mm) Upper liil to lower labial incisor surface Cusp facets, enamel loss, exposed dentin Wear facets, enamel loss, exposed dentin Same as canines, one or more incisors
Click/pop; soft crepitus; hard grating Click/pop; soft crepitus; hard grating (mm) To sound on vertical jaw opening Accomp~ying joint sounds Accompanying joint sounds
Pain report on R/L bite/chew of cotton roll Pain report on R/L lateral excursions Pain report on maximum protruded movement Pain report on digital palpation pressure (2 lbs) of anterior and posterior temporalis; superficial and deep masseter; posterior neck; sternocleidomastoid, occipital attachment Pain report on digital palpation pressure (1 lb) of tendon of temporalis; lateral pterygoid; masseter; body of tongue Pain report on digital palpation pressure (1 lb) of lateral pole of TMJ and external auditory meatus
ablesincluded ~~ss~~tion of occlusionand ~sessmentof vertical and horizontal overlap. A separate pool of 24 subjectswere assessed by four CEs and three UEs using a similar protocol for the remaining clinical variables that might be expected to fluctuate as a result of repeated measurements(such aspain elicited by repeatedm~~le’p~pation). This study designs~tist~~~y assessed the order effects for so-calledfixed and fluctuating variablesand did not place undue strain on subjectsdue to the length of the study procedures.Finally, a third pool of 16 subjects was assessedby the four CEs only after 576
ET AL
retraining and re~ibration. Recorderswereusedto record all measurementsobtained by each examiner.
Data analysis Data were analyzed for reliability by using two statistical methods.’Intraclass correlation was used where the clinical variable could be measuredon a ~n~muous scale, such as the amount of vertical opening measuredin mm. The intraclass correlation coefficient (ICC) was derived from an analysis of variance of the data. Values for ICC above 0.75to 0.80are consideredacceptableand can be inMAY
1890
VOLUME
63
NUMBER
5
ASSESSING
Table
CLINICAL
II.
SIGNS
Reliability
OF TMD
of TMD clinical signs: range of
motion Clinical
sign
Statistic
Vertical dimension Unassisted opening without pain (mm) Maximim unassisted opening (mm) Maximum assisted opening (mm) Jaw opening pattern Jaw excursions Lateral excursions (mm) Protruded movement (mm) Protrusive pattern
ICC ICC ICC
K ICC ICC
K
IV. Reliability of joint sounds
Reliability
0.90 0.96 0.98 0.70 0.70 0.68 0.38
Clinical
III.
Clinical
sign
Statistic
Classification of occlusion Posterior occlusion (Angle) Anterior occlusion Cross bite/open bite Posterior crossbite Anterior crossbite Anterior vertical overlap (mm) Anterior horizontal overlap (mm) Tooth wear Anterior tooth wear Cuspid tooth wear Posterior tooth wear
Statistic
V.
Reliability
K K
0.62
ICC ICC
0.68 0.74
K
0.30 0.44
0.61
K
Reliability of clinical TMD signs:Assessment
of pain Clinical
sign
Statistic
Reliability
Reliability
K K
.0.78
K K
1.00 0.72
ICC ICC
0.81
K K K
sign
Joint sounds on vertical opening Using palpation Using stethoscope Summary score (any joint sound) Opening to click/pop sound (mm) Joint sounds with excursive movements Lateral excursions Protruded movement
Table
Reliability of clinical TMD signs:Occlusal and tooth relationships Table
of clinical TMD signs: Detection
Table
0.28
0.88 0.60 0.60 0.46
Pain with function/movement Pain on biting/chewing (cotton roll) Pain on lateral excursions Pain on protruded movement Pain on palpation Extraoral muscles: Mean of 7 palpation sites Summary score* Intraoral muscles: Mean of 4 palpation sites Summary score* TMJ: Mean of 2 palpation sites Summary score* *Reliability
terpreted to meanthat clinical examinersare interchangeable. Lower values indicate unsatisfactory levels of agreement among clinical examiners.15 The Kappa statistic (Cohen’sKappa, K) wasusedto assessreliability for clinical variables measuredwith a categorical scale, as when joint sounds are characterized as clicks, soft crepitus, or hard grating. The K statistic reflects the percent of agreement among examiners corrected for chance; the likelihood of examiners agreeingis higher, by chance alone, for events that occur frequently than for events that occur only rarely. Values for K were computed for eachpair of examinersand the meanK value acrossexaminer pairs was determined for each clinical variable. When K is in the 0.4 to 0.6 range, reliability is considered only moderate;K values above 0.6 indicate that examiners are using the sameresponsecategoriesat more acceptable levels of reliability.16
of detecting
any type
of pain
among
K K K
0.46 0.71 0.73
K
0.65 0.91
ICC
K ICC
0.61 0.90
K ICC
0.52 0.94
all sites
palpated.
sounds,especiallyby stethoscope.The CEs for whom reliability data are presentedin Tables II through V are the field examinersin the continuing epidemiologicstudy. For comparison,similar analysesperformed on representative variables for UEs are shownin Table VI.
Range
of motion
The reliability of measurement for range-of-motion clinical signsis given in Table II. Vertical rangeof motion, measuredin millimeters, was associatedwith extremely high reliability levels. Assessingextent in millimeters of lateral and protrusive excursionswasassociatedwith less than desirable reliability. Patterns of jaw movement on vertical opening could be assessed with adequate reliability whereasprotrusive opening patterns could not.
RESULTS
Occlusal
Tables II through V present ICC and meanK values for the four CEs after retraining in instanceswhere preliminary analyses revealed less than acceptable reliability. Variables requiring retraining included classification of
The classification of posterior occlusion into classI, II, and III (Angle) was performed with good reliability but only after retraining (Table III). Table VI reveals marked differences in the reliability with which CEs and URs classified posterior occlusion. Classification of anterior occlu-
posterior THE
JOURNAL
and anterior
occlusion
OF PROSTHETIC
and detection
DENTISTRY
of joint
and tooth
relationships
577
DWORKIN
VI. Comparison of reliability for CEs and UEs on a representative sample of clinical signs
Table
Clinical
sign
Range of motion Maximum unassisted opening (mm) Maximum assisted opening (mm) Jaw opening pattern Protrusivepattern Occlusalendtooth relationships Posteriorocclusion Vertical overlap(mm) Horizontaloverlap(mm) Posteriortooth wear Caninetooth wear TMJ sounds Palpationmethod Stethoscope method Lateralexcursionstethoscope Protrudedmovementstethoscope Painon function or palpation Biting/chewing(cotton roll) Protrudedmovement Extraoral musclepalpation Intraoral musclepalpation TMJ palpation
Reliability statistic
CE
UE
ICC ICC
0.96 0.98
0.90 0.92
K K
0.70 0.30 0.38 0.19
K
K K
0.78 0.81 0.88 0.46 0.60
0.37 0.85 0.79 0.09 0.26
K K K K
0.62 0.61 0.30 0.44
0.30 0.35 0.35 0.34
K K K K K
0.46 0.73 0.65 0.61 0.52
0.30 0.22 0.39 0.13 0.25
ICC ICC
sion (division I and II) remained lessthan satisfactory for CE, even after retraining. Classification of anterior occlusion is the only clinical sign for which reliability did not appreciably increaseas a result of continued retraining. Moderate reliability was achieved in the assessmentof tooth wear.
TMJ
sounds
Reliability ranged from marginally acceptablefor agreement in detecting joint soundsassociatedwith vertical jaw opening (K = 0.62 for detecting joint soundsby palpation) to poor reliability associatedwith detecting joint sounds associated with lateral ox protrusive jaw excursions (K = 0.30 and 0.44, respectively) (Table IV). The joint soundsassessed were clicking-popping, soft crepitus, and hard grating. A summary scorewascomputed for each examiner, indicating the number of sounds, regardlessof type, that were found for a particular subject.‘Reliability of thesecombinedsummary scoresfor agreementon number of joint soundswasalsomarginally acceptable(ICC = 0.74). Again, reliability levels.of UEs were below those of CEs (Table VI).
Pain
assessment
Table V indicates that the ability to detect pain during function (when biting or chewing a cotton roll) wasassociated with low reliability (K = 0.46), whereasgoodreliability wasassociatedwith the ability to detect pain accompa-
nying
lateral
(K = 0.71) and protruded
ET AL
movements
(K = 0.73). With the use of a standardized examination protocol, acceptable mean levels of reliability were achieved when recording if pain was present on palpation of seven extraoral and four intraoral masticatory and adjacent muscles(K = 0.65and 0.61, respectively). Reliability of assessing musclepalpation pain wasappreciably enhancedby use of a summary scoresimilar to that proposedby Fricton and Schiffman,17 which reflected agreement regarding how many, but not which, sites were painful to palpation for each subject (ICC = 0.91 and 0.90 for extraoral and intraoral muscles). Reliability of TMJ palpation wasmarginally acceptable when the scoreswere averagedfor intrameatal and external (lateral pole) palpation. Again, when a combinedTMJ pain palpation score wascomputed, reliability among examinersincreasedappreciably (ICC = 0.94).
CONCLUSION The reliability of the calibrated clinical examinersin assessingclinical signscommonly usedto evaluate TMD was in good agreement with findings of the few clinical researcherswho reported reliability for at least someof their clinical data596*17p l8 Excellent reliability for vertical range of motion measures,which wasfound in the data, has also been reported.5*6~17*la Similarly, summary or combined indices, which measureoverall presenceof a clinical sign that could arisefrom severalsources(such assummary indices of musclepalpation pain), have been shownby others to be highly reliable for someclinical signsof TMD.17 Again, the findings with such index measuresconfirm the increasedreliability for the summary clinical measurealthough they mask the reliability of individual components (such as the reliability of measuring palpation pain in a particular muscle). Many important clinical signsof TMD are not highly reliable. The classificationof anterior occlusion,although not heavily implicated in TMD, remained unreliable even after retraining. The orthodontic literature acknowledges difficulty over classification of division I and division II anterior malocclusion. More important for clinicians attempting to evaluate specific TMD diagnosesasinternal disk derangements,assessmentof pain in responseto musclepalpation and the identification of specific TMJ soundsseempossibleonly with modest, sometimesmarginal, reliability. These conclusions are also in agreement with other reports.5 The modestreliability of clinical signsthat are theoretically and clinically important for the differential diagnosisof TMD may arise becauseof examiner error, but they may also arise becausethe clinical phenomenathemselvesmay be inherently unreliable. The clinical signsmay changespontaneouslyover time, making it difficult to find the samesign on successiveexamination attempts, a phenomenonevery clinician has encountered. For example, data previously
MAY
1990
VOLUME
63
NUMBER
5
ASSESSING
CLINICAL
SIGNS
OF TMD
reported indicate that joint souqds may change from trial to trial approximately 50% of the time.lg Where measures of clinical signs of TMD on palpation or joint sounds have been reported reliable, they are usually composite joint measures, that is, an index score obtained by combining several individual measurements. It is possible for individual clinical measures (such as identifying a particular joint sound) to be only marginally reliable and still contribute to a meaningful overall diagnosis of a complexly determined syndrome. However this has not been demonstrated for TMD. Experienced clinicians reporting clinical signs of TMD within their own experience will show low reliability with other clinicians with whom they do not share a calibrated set of clinical measurement criteria. Because observation is virtually an axiom of clinical reporting, it is advisable that dentists work together to evolve reliable, meaningful clinical standards for examination of the signs and symptoms of TMD and for the classification of diagnostic categories. We gratefully acknowledge the cooperation and assistance of the faculty and the staff of the Department of Orthodontics and the Department of Oral Medicine, School of Dentistry, University of Washington in the collection of data presented here and the assistance of Ms. Laura McDonald with data management.
REFERENCES 1. Rugh JD, Solberg WK. Oral health status in the United States: Temporomandibular disorders. J Dent Educ 1985;49:398-405. 2. American Dental Association: Report of the president’s conference on the examination, diagnosis, and management of temporomsndibular disorders. J Am Dent Assoc 1983;66:75-7. 3. Kopp S, Wenneberg B. Intra- and interobserver variability in the assessment of signs of disorder in the stomatognathic system. Swed Dent J 1983;7:239-46.
Availability
of
JOURNAL
4. Carlsson GE, Egermark-Eriksson I, Magnusson T. Intra- and inter-observer variation in functional examination of the masticatory system. Swed Dent J 1980;4:187-94. 5. Fleiss JL, Slakter MJ, Fischman SL, Park MH, Chilton NW. Inter-examiner reliability in caries trials. J Dent Res 1979;58:604-9. 6. Chilton NW. Reliability studies. In: Design and analysis in dental and oral research. 2nd ed. New York: Praeger, 1982;421. 7. Koran LM. The reliability of clinical methods, data, and judgments. N Engl J Med 1975;293:642-6. 8. Koran LM. The reliability of clinical methods, data and judgments (Part Two). N Engl J Med 1975;293:642-6. 9. World Health Organization: Oral health surveys-basic methods. WHO, Geneva, 1971. 10. Dworkin SF, LeResche L, DeRouen T. Reliability of clinical measurement in temporomandibular disorders. Clin J Pain 1988;4:8999. 11. Dworkin SF, Von Korff M, Truelove E, Sommers E, LeResche L. Reliability of examiners assessing TMD signs and symptoms [Abstract]. J Dent Res 1986;65:820. 12. Dworkin SF, Von Korff M, LeResche L, Kruger A. TMD and other pain complaints: initial screening results from a population-based epidemiologic study [Abstract]. J Dent Res 1987;66:192. 13. Greene CS, Marbach JJ. Epidemiologic studies of mandibular dysfunction. J PROSTHET DENT 1982;48:184-90. 14. Bell WE. Orofacial pain differential diagnosis. Dallas: Denedco, 1973. 15. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psycho1 Bull 1979;86:420-8. 16. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;20:37-46. 17. Fricton JR, Scbiffman EL. Reliability of a rraniomandibular index. J Dent Res 1986;65:1359-64. 18. Duinkerke ASH, Luteijn F, Bouman TK, de Jong HP. Reproducibility of a palpation test for the stomatognathic system. Community Dent Oral Epidemiol 1986;14:80-5. 19. Truelove E, LeResche L, Sommers E, Dworkin S, Huggins K, Reay B. Reliability of TMJ sounds in patients and controls [Abstract]. J Dent Res 1987;66:336. Reprint
requests
to:
DR. SAMUEL F. DWORKIN SCHOOL OF DENTISTRY, SC-63 UNIVERSITY OF WASHINGTON SEATTLE, WA 98195
back issues, 1986-1989
Back issuesof the JOURNALOF PROSTHETIC DENTISTRYare available for purchasefrom the publisher, The C. V. Mosby Co., at a cost of $6.50per issue.(Foreign postageis not included.) The following quantity discountsare available: 25% off on quantities of 12 to 23, and one third off on quantities of 24 or more. Pleasewrite to the C.V. Mosby Co., Circulation Department, 11830Westline Industrial Drive, St. Louis, MO 63146-3318,or call (314)872-8370,ext. 7351for information on availability of particular issuesfor that period from 1978to 1969. If unavailable from the publisher, photocopies of complete issuesare available from University Microforms International, 300 N. Zeeb Rd., Ann Arbor, MI 48106, (313)761-4700.
TEE
JOURNAL
OF PROSTHETIC
DENTISTRY
579
clinical signs of temporomandibular of clinical examiners
Samuel F. Dworkin, D.D.S., Ph.D.,* Linda LeResche, Timothy DeRouen, Ph.D.,*** and Michael Von Korff, University of Washington, School of Dentistry, Seattle, Wash.
disorders:
Sc.D.,** Sc.D.****
Data on interrater reliability in assessing a number of clinical signs commonly evaluated in the diagnosis and treatment of temporomandibular disorders (TMD) is presented in this article. Four experienced dental hygienists who were 5eld examiners for a large epidemiologic study of TMD and three experienced clinical TMD specialists (dentists) who are coinvestigators in the same study followed carefully detailed speci5cations and criteria for examination of TMD patients and pain-free controls. Excellent reliability was found for vertical range of motion measures and for summary indices measuring the overall presence of a clinical sign that could arise from several sources (for example, summary indices of muscle palpation pain). However, many clinical signs important in the differential diagnosis ofxubtypes of TMD were not measured with high reliability. In particular, assessment of pain in response to muscle palpation and identi5cation of speci5c temporomandibular joint sounds seemed to be possible only with modest, sometimes marginal, reliability. These modest reliabilities could arise from examiner error because the clinical signs are themselves unreliable, changing spontaneously over time and making it difficult to 5nd the same sign on successive examinations. The finding that, without calibration, experienced clinicians showed low reliability with other clinicians suggests the importance of establishing reliable clinical standards for the examination and diagnostic classification of TMD. (J PROSTAET DENT 1990;63:574-9.)
A
consensusis emerging that persistent pain and dysfunction in the temporomandibular region probably reflects not a single critical entity but a variety of related clinical conditions.’ The term “temporomandibular disorders” (TMD) has been suggestedby the American Dental Association President’s Conference on the Examination, Diagnosis, and Treatment of Temporomandibular Disorders2as a collective label for several related but probably diagnostically distinct states of chronic orofacial pain. Included under the broad TMD category are musclerelated orofacial pain states, such asmyofascial pain dysfunction syndrome (MPD) and other diagnostic subtypes that might reflect pain and dysfunction associatedwith internal derangement(ID) or degenerativedisease(DJD) of
*Professor, Departmentof Oral Medicine;Professor, Department of PsychiatryandBehavioralSciences, Universityof Washington, Schoolof Medicine. **Research AssociateProfessor,Departmentof Oral Medicine. ***Professor,Departmentof DentalPublicHealthSciences, and Departmentof Biostatistics,Schoolof Public Health, University of Washington. ****AssociateScientificInvestigator,GroupHealth Cooperative of PugetSound,Seattle,Wash. 10/l/18982 574
the structural componentsof the temporomandibularjoint (TMJ). It is recognized that the differential diagnosisof TMD accordingto oneor another of its subtypesis often difficult. Signsand symptomsof the different diagnostic categories of TMD may overlap. More than one clinical problem can be present at the sametime (for example, patients may have clinical signsof both MPD and ID), and the clinical signsmay be difficult to measurewith consistency.3l4 This article presentsclinical data regarding the dependability with which important clinical signsof TMD can be measured.The needto know whether signsof TMD can be measuredreliably is obviously of fundamental importance. As dentists gather data to formulate the diagnosisof a particular TMD, it is reasonableto expect that relevant clinical signsshould be measuredwith consistency.If, for example,on the samemeasurementtrial, one clinician heard a joint soundand another clinician did not or, in the same patient, someclinicians found a masticatory muscletender to palpation and others did not, confidence that the clinical signbeingmeasuredwasa valid diagnosticindicator for a particular patient would be diminished. Conversely, clinicians would probably agree that when clinical findings can be measuredwith high agreement,diagnosesare more likely to be valid. MAY1090
VOLUME63
NUMBER5
ASSESSING
CLINICAL
SIGNS
OF TMD
In medical and dental diagnosis, the consistency of measurement of clinical signs by different clinicians, that is, the interrater reliability of clinical signs, is often found to be less than desirable, especially when clinicians have not been carefully trained to use comparable methods or a uniform set of criteria.5,6 In medicine, this observation extends to the reading of radiographs and the measurement of physical signs.7s8 For dentistry, the World Health Organization9 has recommended that data regarding reliability of clinical measurements be a routine part of all oral health survey reports, reliably measuring clinical signs of the most common oral health indicators, such as caries, periodontal health, and oral hygiene status. In previous reports,lO* I1 published studies were reviewed regarding reliability of TMD-related clinical measures and factors were discussed that might significantly influence the reliability of clinical signs commonly measured to evaluate TMD.2 The most critical issues identified included (1) training of clinical examiners; (2) use of appropriate samples of symptomatic and asymptomatic subjects for reliability studies; (3) choosing among alternative statistical approaches for analysis of reliability data; and (4) changes in clinical signs, either spontaneous changes over time or changes due to repeated testing. This article presents the reliability of clinical signs commonly evaluated in the diagnosis and treatment of TMD. These detailed findings are from reliability studies included in a large epidemiologic study of TMD presently in progress.12 The epidemiologic study of TMD entails field examinations to measure clinical signs of TMD in patients under treatment, symptomatic community patients not under treatment, and asymptomatic community controls. The reliability studies discussed in this article were designed to assess the consistency of measurement among the field examiners. The following sections will present the design of the reliability studies and the specific reliability data for critical TMD variables, including range of motion, joint sounds, muscle palpation pain, and occlusal classification. Finally, these findings are discussed in terms of their implications for the differential diagnosis and classification of TMD.
DESIGN Subjects
AND
METHODS
Subjects included asymptomatic, normal, healthy volunteers, selected from the University community, patients in the Clinic of the Department of Orthodontics representing a variety of occlusal disharmonies and malocclusions, and patients experiencing pain and related symptoms of TMD recruited from the Orofacial Pain and Dysfunction Clinic of the Department of Oral Medicine, University of Washington. A total of 64 subjects participated in the reliability studies. The subjects were predominantly women in the 20- to 40-year age group, reflecting the distribution of persons seeking TMD treatment. The subjects represented a balanced sample of asymptomatic and symptomTHE
JOURNAL
OF PROSTHETIC
DENTISTRY
atic persons. TMD patients comprised approximately 60% of the subjects and 40 % of the patients were asymptomatic for TMD, thus maximizing our ability to observe the extent to which examiners could reliably agree in detecting signs when they were present and not detecting them when the signs were absent. All of the subjects provided signed, informed consent and were modestly remunerated for their efforts.
Examiners Examiners included four experienced registered dental hygienists who served as field examiners for a larger epidemiologic study and three experienced clinical TMD specialists (dentists) who were coinvestigators in the same epidemiologic study of TMD. The coinvestigators developed carefully detailed specifications and criteria that were used by the field examiners as their examination protocol. The field examiners underwent 40 hours of supervised training and calibration, including preliminary reliability assessment and retraining where indicated, before conducting actual field examinations with patients and controls. The study team dentists, who were experienced TMD specialists, did not undergo similar training. Hence, they were not calibrated in the use of the TMD examination specifications and criteria they collaborated in developing. The results of the reliability studies were analyzed with regard to the clinical examiner’s familiarity in using a standardized set of examination specifications and criteria, enabling the comparison of trained and calibrated examiners (CE) with uncalibrated examiners (UE).
Clinical
variables
Table I contains all the clinical variables included in the reliability studies reported and a brief outline of the criteria used for measurement. Clinical measurements were obtained for a variety of range-of-motion variables, including vertical, lateral, protrusive, and retrusive jaw movements; joint sounds by palpation and by stethoscope during vertical and other excursions of the jaw; pain elicited in response to palpation of predetermined sites on the muscles of mastication, the TMJ, and adjacent structures; and classification of occlusal relationships. The variables included in these reliability studies are a representative subset of clinical variables measured in the actual field examination. These variables have been included by many workers for their importance in the diagnosis, classification, and treatment of TMD.5y 6$13,I4
Protocol
for reliability
studies
An incomplete Latin square design was used to ensure that each subject was seen by each examiner in a randomized sequence to determine whether the order of examination influenced responses. Twenty-four subjects were assessed in this fashion by four CEs and two UEs on variables labelled as “fixed” because it was presumed that repeated trials would not influence the clinical findings. These vari575
DWORlIIN
Table
I. Clinical signsand criteria for measurement Cliiical
sign
I. Range of motion A Vertical dimension 1’ Unassisted opening, no pain 2 Maximum unassisted opening 3 Ma~mum assisted opening 4 Jaw opening pattern B Excursive movements 1 Lateral excursions 2 Protruded movement 3 Protrusive pattern II. Occlusal and tooth reIatio~~~ A Posterior occlusion B Anterior occlusion C Cross bite/open bite I Posterior crossbite 2 Anterior crossbite 3 Vertical overlap 4 Horizontal overbite D Tooth wear I Posterior tooth wear 2 Canine tooth wear 8 Anterior tooth wear III. TMJ sounds A Joint sounds with vertical opening 1 Palpation 2 Stethoscope 3 Click/pop on opening B Joint sounds with excursive movements 1 Lateral excursion 2 Protruded movement IV. Pain on function and palpation A Pain with function/movement 1 Biting/chewing 2 Lateral excursions 3 Protruded movement B Palpation pain 1 Extraoral muscles
2 Intraoral muscles 3 TMJ
Criteria
for measurement
(mm) Between maxillary and mandibular labioincisal embrasures Straight; R/L corrected deviation; RiL uncorrected deviation
(mm) From upper to lower incisor embrasures (mm) From upper lingual to lower labial incisor surfaces Straight; R/L corrected and uncorrected deviation Angle’s ciass f; class II; class III Division I; division II
Buccal or lingual deviation (mm) Overlap maxillary central incisor (mm) Upper liil to lower labial incisor surface Cusp facets, enamel loss, exposed dentin Wear facets, enamel loss, exposed dentin Same as canines, one or more incisors
Click/pop; soft crepitus; hard grating Click/pop; soft crepitus; hard grating (mm) To sound on vertical jaw opening Accomp~ying joint sounds Accompanying joint sounds
Pain report on R/L bite/chew of cotton roll Pain report on R/L lateral excursions Pain report on maximum protruded movement Pain report on digital palpation pressure (2 lbs) of anterior and posterior temporalis; superficial and deep masseter; posterior neck; sternocleidomastoid, occipital attachment Pain report on digital palpation pressure (1 lb) of tendon of temporalis; lateral pterygoid; masseter; body of tongue Pain report on digital palpation pressure (1 lb) of lateral pole of TMJ and external auditory meatus
ablesincluded ~~ss~~tion of occlusionand ~sessmentof vertical and horizontal overlap. A separate pool of 24 subjectswere assessed by four CEs and three UEs using a similar protocol for the remaining clinical variables that might be expected to fluctuate as a result of repeated measurements(such aspain elicited by repeatedm~~le’p~pation). This study designs~tist~~~y assessed the order effects for so-calledfixed and fluctuating variablesand did not place undue strain on subjectsdue to the length of the study procedures.Finally, a third pool of 16 subjects was assessedby the four CEs only after 576
ET AL
retraining and re~ibration. Recorderswereusedto record all measurementsobtained by each examiner.
Data analysis Data were analyzed for reliability by using two statistical methods.’Intraclass correlation was used where the clinical variable could be measuredon a ~n~muous scale, such as the amount of vertical opening measuredin mm. The intraclass correlation coefficient (ICC) was derived from an analysis of variance of the data. Values for ICC above 0.75to 0.80are consideredacceptableand can be inMAY
1890
VOLUME
63
NUMBER
5
ASSESSING
Table
CLINICAL
II.
SIGNS
Reliability
OF TMD
of TMD clinical signs: range of
motion Clinical
sign
Statistic
Vertical dimension Unassisted opening without pain (mm) Maximim unassisted opening (mm) Maximum assisted opening (mm) Jaw opening pattern Jaw excursions Lateral excursions (mm) Protruded movement (mm) Protrusive pattern
ICC ICC ICC
K ICC ICC
K
IV. Reliability of joint sounds
Reliability
0.90 0.96 0.98 0.70 0.70 0.68 0.38
Clinical
III.
Clinical
sign
Statistic
Classification of occlusion Posterior occlusion (Angle) Anterior occlusion Cross bite/open bite Posterior crossbite Anterior crossbite Anterior vertical overlap (mm) Anterior horizontal overlap (mm) Tooth wear Anterior tooth wear Cuspid tooth wear Posterior tooth wear
Statistic
V.
Reliability
K K
0.62
ICC ICC
0.68 0.74
K
0.30 0.44
0.61
K
Reliability of clinical TMD signs:Assessment
of pain Clinical
sign
Statistic
Reliability
Reliability
K K
.0.78
K K
1.00 0.72
ICC ICC
0.81
K K K
sign
Joint sounds on vertical opening Using palpation Using stethoscope Summary score (any joint sound) Opening to click/pop sound (mm) Joint sounds with excursive movements Lateral excursions Protruded movement
Table
Reliability of clinical TMD signs:Occlusal and tooth relationships Table
of clinical TMD signs: Detection
Table
0.28
0.88 0.60 0.60 0.46
Pain with function/movement Pain on biting/chewing (cotton roll) Pain on lateral excursions Pain on protruded movement Pain on palpation Extraoral muscles: Mean of 7 palpation sites Summary score* Intraoral muscles: Mean of 4 palpation sites Summary score* TMJ: Mean of 2 palpation sites Summary score* *Reliability
terpreted to meanthat clinical examinersare interchangeable. Lower values indicate unsatisfactory levels of agreement among clinical examiners.15 The Kappa statistic (Cohen’sKappa, K) wasusedto assessreliability for clinical variables measuredwith a categorical scale, as when joint sounds are characterized as clicks, soft crepitus, or hard grating. The K statistic reflects the percent of agreement among examiners corrected for chance; the likelihood of examiners agreeingis higher, by chance alone, for events that occur frequently than for events that occur only rarely. Values for K were computed for eachpair of examinersand the meanK value acrossexaminer pairs was determined for each clinical variable. When K is in the 0.4 to 0.6 range, reliability is considered only moderate;K values above 0.6 indicate that examiners are using the sameresponsecategoriesat more acceptable levels of reliability.16
of detecting
any type
of pain
among
K K K
0.46 0.71 0.73
K
0.65 0.91
ICC
K ICC
0.61 0.90
K ICC
0.52 0.94
all sites
palpated.
sounds,especiallyby stethoscope.The CEs for whom reliability data are presentedin Tables II through V are the field examinersin the continuing epidemiologicstudy. For comparison,similar analysesperformed on representative variables for UEs are shownin Table VI.
Range
of motion
The reliability of measurement for range-of-motion clinical signsis given in Table II. Vertical rangeof motion, measuredin millimeters, was associatedwith extremely high reliability levels. Assessingextent in millimeters of lateral and protrusive excursionswasassociatedwith less than desirable reliability. Patterns of jaw movement on vertical opening could be assessed with adequate reliability whereasprotrusive opening patterns could not.
RESULTS
Occlusal
Tables II through V present ICC and meanK values for the four CEs after retraining in instanceswhere preliminary analyses revealed less than acceptable reliability. Variables requiring retraining included classification of
The classification of posterior occlusion into classI, II, and III (Angle) was performed with good reliability but only after retraining (Table III). Table VI reveals marked differences in the reliability with which CEs and URs classified posterior occlusion. Classification of anterior occlu-
posterior THE
JOURNAL
and anterior
occlusion
OF PROSTHETIC
and detection
DENTISTRY
of joint
and tooth
relationships
577
DWORKIN
VI. Comparison of reliability for CEs and UEs on a representative sample of clinical signs
Table
Clinical
sign
Range of motion Maximum unassisted opening (mm) Maximum assisted opening (mm) Jaw opening pattern Protrusivepattern Occlusalendtooth relationships Posteriorocclusion Vertical overlap(mm) Horizontaloverlap(mm) Posteriortooth wear Caninetooth wear TMJ sounds Palpationmethod Stethoscope method Lateralexcursionstethoscope Protrudedmovementstethoscope Painon function or palpation Biting/chewing(cotton roll) Protrudedmovement Extraoral musclepalpation Intraoral musclepalpation TMJ palpation
Reliability statistic
CE
UE
ICC ICC
0.96 0.98
0.90 0.92
K K
0.70 0.30 0.38 0.19
K
K K
0.78 0.81 0.88 0.46 0.60
0.37 0.85 0.79 0.09 0.26
K K K K
0.62 0.61 0.30 0.44
0.30 0.35 0.35 0.34
K K K K K
0.46 0.73 0.65 0.61 0.52
0.30 0.22 0.39 0.13 0.25
ICC ICC
sion (division I and II) remained lessthan satisfactory for CE, even after retraining. Classification of anterior occlusion is the only clinical sign for which reliability did not appreciably increaseas a result of continued retraining. Moderate reliability was achieved in the assessmentof tooth wear.
TMJ
sounds
Reliability ranged from marginally acceptablefor agreement in detecting joint soundsassociatedwith vertical jaw opening (K = 0.62 for detecting joint soundsby palpation) to poor reliability associatedwith detecting joint sounds associated with lateral ox protrusive jaw excursions (K = 0.30 and 0.44, respectively) (Table IV). The joint soundsassessed were clicking-popping, soft crepitus, and hard grating. A summary scorewascomputed for each examiner, indicating the number of sounds, regardlessof type, that were found for a particular subject.‘Reliability of thesecombinedsummary scoresfor agreementon number of joint soundswasalsomarginally acceptable(ICC = 0.74). Again, reliability levels.of UEs were below those of CEs (Table VI).
Pain
assessment
Table V indicates that the ability to detect pain during function (when biting or chewing a cotton roll) wasassociated with low reliability (K = 0.46), whereasgoodreliability wasassociatedwith the ability to detect pain accompa-
nying
lateral
(K = 0.71) and protruded
ET AL
movements
(K = 0.73). With the use of a standardized examination protocol, acceptable mean levels of reliability were achieved when recording if pain was present on palpation of seven extraoral and four intraoral masticatory and adjacent muscles(K = 0.65and 0.61, respectively). Reliability of assessing musclepalpation pain wasappreciably enhancedby use of a summary scoresimilar to that proposedby Fricton and Schiffman,17 which reflected agreement regarding how many, but not which, sites were painful to palpation for each subject (ICC = 0.91 and 0.90 for extraoral and intraoral muscles). Reliability of TMJ palpation wasmarginally acceptable when the scoreswere averagedfor intrameatal and external (lateral pole) palpation. Again, when a combinedTMJ pain palpation score wascomputed, reliability among examinersincreasedappreciably (ICC = 0.94).
CONCLUSION The reliability of the calibrated clinical examinersin assessingclinical signscommonly usedto evaluate TMD was in good agreement with findings of the few clinical researcherswho reported reliability for at least someof their clinical data596*17p l8 Excellent reliability for vertical range of motion measures,which wasfound in the data, has also been reported.5*6~17*la Similarly, summary or combined indices, which measureoverall presenceof a clinical sign that could arisefrom severalsources(such assummary indices of musclepalpation pain), have been shownby others to be highly reliable for someclinical signsof TMD.17 Again, the findings with such index measuresconfirm the increasedreliability for the summary clinical measurealthough they mask the reliability of individual components (such as the reliability of measuring palpation pain in a particular muscle). Many important clinical signsof TMD are not highly reliable. The classificationof anterior occlusion,although not heavily implicated in TMD, remained unreliable even after retraining. The orthodontic literature acknowledges difficulty over classification of division I and division II anterior malocclusion. More important for clinicians attempting to evaluate specific TMD diagnosesasinternal disk derangements,assessmentof pain in responseto musclepalpation and the identification of specific TMJ soundsseempossibleonly with modest, sometimesmarginal, reliability. These conclusions are also in agreement with other reports.5 The modestreliability of clinical signsthat are theoretically and clinically important for the differential diagnosisof TMD may arise becauseof examiner error, but they may also arise becausethe clinical phenomenathemselvesmay be inherently unreliable. The clinical signsmay changespontaneouslyover time, making it difficult to find the samesign on successiveexamination attempts, a phenomenonevery clinician has encountered. For example, data previously
MAY
1990
VOLUME
63
NUMBER
5
ASSESSING
CLINICAL
SIGNS
OF TMD
reported indicate that joint souqds may change from trial to trial approximately 50% of the time.lg Where measures of clinical signs of TMD on palpation or joint sounds have been reported reliable, they are usually composite joint measures, that is, an index score obtained by combining several individual measurements. It is possible for individual clinical measures (such as identifying a particular joint sound) to be only marginally reliable and still contribute to a meaningful overall diagnosis of a complexly determined syndrome. However this has not been demonstrated for TMD. Experienced clinicians reporting clinical signs of TMD within their own experience will show low reliability with other clinicians with whom they do not share a calibrated set of clinical measurement criteria. Because observation is virtually an axiom of clinical reporting, it is advisable that dentists work together to evolve reliable, meaningful clinical standards for examination of the signs and symptoms of TMD and for the classification of diagnostic categories. We gratefully acknowledge the cooperation and assistance of the faculty and the staff of the Department of Orthodontics and the Department of Oral Medicine, School of Dentistry, University of Washington in the collection of data presented here and the assistance of Ms. Laura McDonald with data management.
REFERENCES 1. Rugh JD, Solberg WK. Oral health status in the United States: Temporomandibular disorders. J Dent Educ 1985;49:398-405. 2. American Dental Association: Report of the president’s conference on the examination, diagnosis, and management of temporomsndibular disorders. J Am Dent Assoc 1983;66:75-7. 3. Kopp S, Wenneberg B. Intra- and interobserver variability in the assessment of signs of disorder in the stomatognathic system. Swed Dent J 1983;7:239-46.
Availability
of
JOURNAL
4. Carlsson GE, Egermark-Eriksson I, Magnusson T. Intra- and inter-observer variation in functional examination of the masticatory system. Swed Dent J 1980;4:187-94. 5. Fleiss JL, Slakter MJ, Fischman SL, Park MH, Chilton NW. Inter-examiner reliability in caries trials. J Dent Res 1979;58:604-9. 6. Chilton NW. Reliability studies. In: Design and analysis in dental and oral research. 2nd ed. New York: Praeger, 1982;421. 7. Koran LM. The reliability of clinical methods, data, and judgments. N Engl J Med 1975;293:642-6. 8. Koran LM. The reliability of clinical methods, data and judgments (Part Two). N Engl J Med 1975;293:642-6. 9. World Health Organization: Oral health surveys-basic methods. WHO, Geneva, 1971. 10. Dworkin SF, LeResche L, DeRouen T. Reliability of clinical measurement in temporomandibular disorders. Clin J Pain 1988;4:8999. 11. Dworkin SF, Von Korff M, Truelove E, Sommers E, LeResche L. Reliability of examiners assessing TMD signs and symptoms [Abstract]. J Dent Res 1986;65:820. 12. Dworkin SF, Von Korff M, LeResche L, Kruger A. TMD and other pain complaints: initial screening results from a population-based epidemiologic study [Abstract]. J Dent Res 1987;66:192. 13. Greene CS, Marbach JJ. Epidemiologic studies of mandibular dysfunction. J PROSTHET DENT 1982;48:184-90. 14. Bell WE. Orofacial pain differential diagnosis. Dallas: Denedco, 1973. 15. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psycho1 Bull 1979;86:420-8. 16. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;20:37-46. 17. Fricton JR, Scbiffman EL. Reliability of a rraniomandibular index. J Dent Res 1986;65:1359-64. 18. Duinkerke ASH, Luteijn F, Bouman TK, de Jong HP. Reproducibility of a palpation test for the stomatognathic system. Community Dent Oral Epidemiol 1986;14:80-5. 19. Truelove E, LeResche L, Sommers E, Dworkin S, Huggins K, Reay B. Reliability of TMJ sounds in patients and controls [Abstract]. J Dent Res 1987;66:336. Reprint
requests
to:
DR. SAMUEL F. DWORKIN SCHOOL OF DENTISTRY, SC-63 UNIVERSITY OF WASHINGTON SEATTLE, WA 98195
back issues, 1986-1989
Back issuesof the JOURNALOF PROSTHETIC DENTISTRYare available for purchasefrom the publisher, The C. V. Mosby Co., at a cost of $6.50per issue.(Foreign postageis not included.) The following quantity discountsare available: 25% off on quantities of 12 to 23, and one third off on quantities of 24 or more. Pleasewrite to the C.V. Mosby Co., Circulation Department, 11830Westline Industrial Drive, St. Louis, MO 63146-3318,or call (314)872-8370,ext. 7351for information on availability of particular issuesfor that period from 1978to 1969. If unavailable from the publisher, photocopies of complete issuesare available from University Microforms International, 300 N. Zeeb Rd., Ann Arbor, MI 48106, (313)761-4700.
TEE
JOURNAL
OF PROSTHETIC
DENTISTRY
579
