232
Journal of Public Health Dentistry
Improving Dental Epidemiologic Data Collection with Computers Anil Joshi, BDS, MPH Instructor Department of Dental Care Administration Harvard School of Dental Medicine Susan McDermott, MPH, RN Project Director Elder Dental Health New England Research Institute Philip Marcus, DMD Instructor Department of Dental Care Administration Harvard School of Dental Medicine Chester W. Douglass, DMD, PhD Professor and Chairman Department of Dental Care Administration Harvard School of Dental Medicine 188 Longwood Avenue Boston, MA 021 15 Alan Jette, PhD Senior Research Scientist New England Research Institute Watertown, MA
Abstract A computerized dental data recording system (DDRS) was developed for the New England Elder Dental Study to improve data quality and increase field staff efficiency. The DDRS displays video screens similar tc traditional paper forms to record data on coronal and root caries, dentate and denture status, subacute bacterial endocarditis screening, gingival bleeding, calculus, and periodontalattachmentlevel. DDRS provides facilities fordate and exam-component time tracking, on-line contextual comments, random record retrieval, editing, data backup, and data output in various data formats. This study compared the DDRS with a paper-form system for data entry accuracy. Dental caries and periodontal disease measurement data from38 subjects were recorded on paper forms and independentlyenteredusingDDRS. The DDRSidentified 150 illogical data errors, 39 inconsistent data errors, Send correspondence and reprint requests to Dr. Josh. This work was supported by NIA grant #AG07139. Manusaipt received: 7/31 /91; returned to authors for revision: 9/10/91; accepted for publication: 1 /14/92.
J Public Health Dent 1992;52(4):232-8
7 invalid data and 34 miscellaneous data errors. Four technicians with field experience using both paper forms and DDRS reported time savings using DDRS in the field. DDRS has the potential for additional time savings by minimizing the time for data coding, cleaning, and management. Results demonstratethat DDRS could improve the quality of oral epidemiologic data by mandating strict adherence to protocols,preventing errors, and increasing field efficiency.
Key words: dental, epidemiology, computer, data collection, field studies, in-home surveys.
The role of computers in dental education, research, and practice has increased at a rapid rate (1-6).Computer softwarehas been developed to aid cliniciansin differential diagnosis of pulp-relatedpain and oral radiographic lesions (7,8). Computer-aided design and computeraided manufacturing(CAD-CAM) systems are available for designing and preparing dental restorations (9). Other computer programs allow efficientrecord keeping
233
Vol. 52, No. 4, Summer 1992 FIGURE 1 Screen-Page 1 of DDRS: Identifying Information on Dentist and Interviewer-recorders Are Recorded on this Page
Enter ID information: Dent-ID 201 FS-SIGN PASS
R-ID EXAM-DT
CODE TABLE ID Information:
410 03/09/90
Field Tech ID ... You Know It. Dentist ID:
THE RECORD IS ALREADY IN THE DATABASE.
Dentist A ... 201 Dentist B ... 301 Dentist C ... 401 Dentist D ... 501
ENTER 0TO RETRIEVE THE RECORD ENTER (N) TO CHANGE RESPONDENT ID
FS -SIGN ... Type EXAM-DT.. .Examination Date
FIGURE 2 Tooth Table: Screen Page 4 of DDRS 1
M M 17
2 M M 18
3
T M 19
4 P M 20
5 M M 21
6 T M 22
7 P M 23
8
P M 24
9 P M 25
10
11
P M 26
P M 27
12 T M 28
13
14
M M 29
P M 30
15 M M 31
16
M M 32
Tooth status: tooth present?, tooth missing=M, unable to assess=Y,root tip only=T
in dental offices, including periodontal charting (10). Innovative use of computer technology such as memory glucometry, which electronically transfers glucose values from the patient's home glucometer to a physician's computer via telephone-modem, has been reported (11,121. There has been increased use of clinical data validation through computer systems (131, as well as computer-aided data collection and validation in multicenter epidemiologic studies and clinical trials. (14) There has been little empirical work on the impact of using computers for direct data collection. In comparing a computer-assisted data collection system for recording the responses from personal interviews with the paperbased data collection system, Christiansen et al. (15) found no systematic differences between the two methods. An evaluation of computer-aided personal interviewing revealed a slightly higher error rate for computerized systems when compared to the paper-pencil methods (16). Use of microcomputers in collecting dental epidemiologic data in the field, however, has become common since the early work of Eklund at the University of Michigan. There are, however, no published reports describing the functionality of computerized dental systems, their usefulness, or how they compare with paper-form systems in terms of efficiency and data quality. This paper describes the development and testing of a computerized dental data recording system (DDRS)used in the New England Elder Dental Study (NEEDS) to collect
in-home dental examinationdata. The dental component of this study included soft tissue, coronal and root caries, and periodontal examinations. Methods The DDRS is a comprehensive data recording system that integrates data collection for soft tissue examination, dentate and denture status, and caries and periodontal examination. The DDRS operates on IBM Personal Computers running under MS-DOS 3.0 and later versions. In NEEDS the DDRS was used on Zenith Supersport 286 laptop computers to collect dental examination data in the homes of study participants. The program's "computer screens" were designed to simulate the paper forms developed for collectingdental examination data. Twelve screens made up the complete dental examination protocol. Pages 1-3 recorded general information including field protocol reminders, respondent, examiner and interviewer-recorder identification numbers, exam-type and exam-component timings (Figure 1). Time fields were not displayed on the screen, but the DDFS automatically recorded the times of starting and completion for each section of the examination. This section also recorded the preexamination subacute bacterial endocarditis screening responses, which determined whether or not a subject was to be included in the periodontal examination. Requirement of a field supervisor signature, ''%-sign," allowed only authorized personnel to update the files in the master data base.
Journal of Public Health Dentistry
234
FIGURE 3 Screen Page 5 of DDRS: Information on Coronal Surfaces of Teeth 1-16 Are Recorded on this Screen Page
CROWN 1-16
R-ID: 99999
T H C C M 0 ~
- _ _ _ - - - ~ -
1 2 3 4 5 6
M M T P M T
M M T F M T
M M T F M T
M M T R M T
M M T S M T
M M T S M T
7
S
S
A
S
S
S
I
9 P S 10 P S 11 P S 1 2 T T 1 3 M M 14 P N 15 M M 16 M M
A A
D
CORONAL CARIES CODES
B L ~
F S A S T T M M F F M M M M
7 S S T M S M M
~ Valid Crown Code (CC):
F S
D T M S M M
1
Comment: Press
NEXT PAGE "()'
Screen pages 4-12 of DDRS were designed to record epidemiologic dental examination data, including soft tissue and denture-wearing status. The tooth table, page 4, allowed dentate status of each subject to be recorded (Figure 2). In addition, the information entered in the tooth table dictated which fields were displayed on subsequent screens. For example, if tooth #27 was missing, and was so noted on the tooth table, this tooth was automatically eliminated from the examination, making it unavailable for recording caries or periodontal measures in the rest of the examination. A default code of "M" (missing)was applied, preventing inadvertent data entry for this tooth. Coronal and root caries examination information was recorded on screen pages 5-8 of DDRS. Only valid, predefined codes that were displayed on each of these pages were accepted for various measurements (Figure 3). The final two pages of the DDRS program were for recording periodontal examination information (Figure 4). These pages have fields for recording measures of gngival bleeding, calculus accumulation, the distance from free gingival margin to cemento-enamel junction (CEJ), and probing depths. Periodontal measurements were routinely recorded for the mesial, buccal, and distal sites for each tooth. A fourth site was also recorded if it was greater than the three established sites. Each of the two periodontal screens represented a maxillary or mandibular arch, and contained 176fields in which data were recorded. When any invalid code was entered or any illogical datum was entered for any field on any page,
'
Present Crowned Crowned (deficient) Sound Valid Surface Codes: Decayed FailedResto.Non-car Filled Filled (C/R) Sound Filled (C/R) Crwn Decay Filled (R/C) Root Decay Filled (R/C) Sound Filled (R/C) Crwn Decay Filled (R/C) Root Decay Recurrent Caries Unable to Assess
P C
H S D
N F 1
2 3
7 8 9
R Y
then DDRS flagged the recorder by producing an audible tone and displaying a message on top of the screen. The DDRS integrates data collection for soft tissue examination, dentate and denture status, and caries and periodontal examination in one program. For soft tissue examination, however, it only records presence or absence of a soft tissue lesion and whether a referral was made at the time of examination. In brief, the functionality of DDRS can be summarized as follows: 1.The DDRS is not only a data entry system, but rather a data base system capable of appending, deleting, browsing and editing records. Full-screen editing capability allows the recorder to move back and forth among different examination screens. 2. The system also keeps track of the date and time spent on each section of an examination. 3. The DDRS can prevent common errors such as invalid code entry and inconsistent data across fields. 4. The system skips fields based on the information entered on previous observations. A default value is automatically assigned to the skipped fields. 5. A commenting feature of the DDRS allows the interviewer-recorder to write comments related to any information during each examination section.The availability of a "comment" data base allows systematic documentation of unusual findings and permits better coding decisions for clinical findingsunresolved at the time of examination. 6. The data output can be developed in many formats such as in ASCII, dBase I11 Cdbf format), or SAS data set
-
235
Vol. 52, No. 4, Summer 1992
FIGURE 4 Screen Page 7 of DDRS: Periodontal Assessment Information on Teeth 1-16 Are Recorded on this Screen Page R I D : 99999 T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
GA M CAL M B-CEJ M B-PD M M-CEJ M M-PD M DL-CEJ M DL-PD M GT-SIT M GT-CEJ M GT-PDM
M M M M M M M M M M M
B G -2 2 -2 3 0 4 ML -2 6
B G 2 3 0 3 O 3 A A A
M M M M M M M M M M M
N S 0 2 0 3 0 3 A A A
N S 0
N N 0
1
1
0 2 0 4 A A
0 2 0 2 A A A
N N 0 2 0 2 0 3 A A A
N B N G - 1 - 3 1 4 0 0 2 3 0 0 2 3 DB A -1 A 4 A
~ - - _ _ _ - - - ~ - - - - - ~ - - -
A
TI1
T12
T13
T14
T15
T16
B G 0 2 O 3 O 3 DI -2 5
M M M M M M M M M M M
B G
M M M M M M M M M M M
M M M M M M M M M M M
-
3 3
-
2 5
-
2 5 ML -1
6
PER10 CODE TABLE 1.Gingival Assessment 2. Calculus Assessment 3. CEJ or PD 4. GT Site
No Bleeding=N No calculus=N Buccal=BU Mesial=ME Distal=CI
Bleeding+B Unable to Assess=Y Supra-Gv=S Sub-Gv o r Both=G Put (-1 sign where appropriate with data Distobuccal=DB Distolingual=DL Lingual=LI Mesiolingual=ML Mesiobuccal=MB N/A=A
(.SSDformat). 7. The modularity of the design facilitatesmodification and makes it easy to maintain the software. We estimated DDRS’ potential to prevent data entry errors in comparison to a traditional paper-form system. Dental data were collected in the field (in subjects’ homes) on paper forms similar in format to the screens of the DDRS program. Four examiners and three recorders were involved in collecting data on paper forms in the field. Subsequently, one person other than examiner or recorders used DDRS to enter the same data into a computer. By flagging any invalid or inconsistent data that were transcribed from paper forms to the computer system, the DDRS identified the errors that would have been prevented if DDRS had been used directly in the field in place of the paper forms. The NEEDS was a community-based epidemiologic survey of a representative sample of noninstitutionalized elders aged 70 and older living in the six New England states. The data used for this analysis were obtained from the pilot study for NEEDS conducted in an urban community in the Greater Boston Metropolitan area in 198889. Data were collected on coronal caries, root caries, gingival bleeding, calculus accumulation, and periodontal attachment level of 38 study participants who had 12 or more teeth. The dental examinations were carried out by dentists in the participants’ homes using a portable examination lamp, mouth mirror, explorer, and a peri-
odontal probe. Results A mean number of 19 teeth were reported for the 38 study participants. The total number of measurements recorded on paper forms was 14392. Table 1presents the types and frequency of errors flagged when data were entered from paper forms to a computer using DDRS. Overall, 230 errors were identified (1.5%of all measurements), an average of six errors per subject examined. Illogical data ~ Y Y O Y S ,defined as inconsistencies between the measurements made on gingival recession and those made from the free gingival margin to the cementoenamel junctions, were identified in 75 pairs, resulting in 150 data errors. Two forms of inconsistencies were observed: (1) the root surface was not exposed, but a negative score in millimeters was recorded for the distance between the free gingival margin and cemento-enamel junction; or (2) the root surface was exposed but a positive score in millimeters was recorded for distance between the free gingival margin and cementosnamel junction. Znconsistent codes, cases where different codes were used to represent the same condition, were noted on only seven instances. For example, for an unexposed sound root surface the following different codes/symbols were used: NA, N/a, slash only (”/”), ”U”,“9dash , (”-”). Conversely, recorders used one code to represent differ-
Journal of Public Health Dentistry
236
ent situations in the same context. For instance, a blank space on the paper form might have meant any of the following:tooth missing, sound coronal surface, sound root surface, unable to assess. Unclear protocols and nondefrned situations (23 cases) were errors resulting from the use of various codes or clinical situations for which there were no codes available at the time of the examination. For example, codes were unavailable for crowned teeth with deficient margins and for restorations crossing the cemento-enamel junction with decay on the root. These errors per se were independent of paper form or computer-based system. (However, availability of computer-based system with commenting features will result in better documentation of such problems). There were 39 cases of invalid data where an unavailable code was used for a data field. Some aspect of six records wasillegible, creating the potential for many data errors per record. For the purpose of this evaluation we counted only one error per record. The six cases of illegibility included overwriting, strikeouts, and similarity of symbolsslanted “1” and ”/”. The remaining five errors were included in the miscellaneous category. Examples in this category include errors in recording identifying information such as respondent, and interviewer-recorder ID or date of examination. In one instance data for tooth number three was recorded in the space for tooth number four. In another instance, during caries examination a tooth was designated first premolar while during periodontal examinationit was designated second premolar. Discussion DDRS offers numerous advantages for the collection of dental epidemiologic data compared with traditional methods of data collection (Table 2). Dental epidemiologic data are typically collected on paper forms and subsequently entered into a computer for data management and statistical analysis. Traditional methods also include tape recording of actual exams and interviews, and use of specially designed paper-sheets that may be scanned electronically for data entry. All three methods that have been used extensively in the past have inherent disadvantages. In a paper-form system recording errors occur when out-of-range data, incorrect codes for responses, and inconsistent data are recorded. (Recording errors also include errors of omission and illegibility.) Keystroke errors occur when the data collected on paper forms is entered into a computer. Keystroke errors are most common when a data set is large with many data variables or when the data involves a complex coding and transcribing scheme. For instance, theNEEDS dental exam data set has 1,200 subjects with as many as 750 variables per subject. Tape-recorded dental e x a m and interviews incorporate the same problems as a paperform system. In addition, transcribing and coding problems in tape medium are even more complex if the cor-
TABLE 1 Frequency of Data Recording Error
Error Type 1.Illogical
2. Invalid 3. Unclear protocol and undefined situations 4.Inconsistent 5. Illegibility 6. Miscellaneous Total
__
Count
150 39 23 7 6 5 230
rections of previous measurements are made during the examination. Use of scanners to read the data eliminates keystroke errors, but requires clear demarcation of the appropriate marks on the sheet and often a complex marking sheet. This medium, like paper forms and tape recording, is inherently prone to recording errors. Computerized, on-site DDRS is most valuable for minimizing recording errors. Logical checks built into the computer program allow the data collector to identify inconsistenciesor out-of-rangedata in the field, allowing for correction when both patient and examinersare present. In all other means of data collection, incorrect data are usually identified in the “data cleaning” phase, which not only involves greater time and labor, but increases the cost of correcting the identified errors. The result is that these data have to be either excluded from the analysis or an arbitrary decision made for their inclusion. Computer systems such as DDRS allow programming of any logical checks or procedures used in the traditional data “cleaningphase,” allowing errors to be caught when the data are being collected. DDRS also helps in recording difficult measurements such as consistent scoring of gingival recession. The DDRS computer program mandates that the examiner-recorder team conduct and record data in a predefined and standardized manner. This eliminateserrors of omission and allows for clearer communication between dentists and field technicians during the data collection procedure. It also helps interview-recorders to work with a consistent examination style. The most striking advantage of the DDRSis that it provides the research team with immediate access to data for analysis. The system minimizes the need for coding and data entry and the quality controls that are otherwise necessary steps when paper or other systems are used. In the NEEDS study, field staff reported that time was saved in recording the dental examinationdata by DDRS. Two published reports, one comparing a computerized system with a paper-based system for personal interviews (16) and another for telephone interviews (17), have reported that the computer-aided methods took longer than paper systems to conduct the interviews. The
237
Vol. 52, No. 4, Summer 1992
TABLE 2 Comparative Evaluation of Computerized Data Recording System and Paper-Form System Computer-based System
Paper-based System
1. Minimized recording errors (e.g., checks for out-of-range and inconsistent data). 2. Transcribing error (not applicable).
1.Not feasible.
3. Minimized effort required at "data cleaning" phase as a result of Items 1.and 2. in this table. 4. Better coordination between dentist and recorder at the time of the examination due to fixed computer mandated protocol. 5. Immediate access to data for analysis feasible. 6. Initial cost of program development and testing is substantial. 7. Some computer training required for recorders. 8. Better documentation: no illegibility problem. 9. Possible malfunctioning in the field.
main reasons cited for increased time using a computer included questions requiring date responses (with correct punctuation), typing speed, questions that required more data entry in the computer than was required on paper forms for the same response, and the quality of hardware and software-slow systems took longer to display the next question or to verify a response. An objectiveassessment of time to use each system was beyond the scope of the current investigation. Although not determined in this study, time saving using DDRS could come from several sources. The interviewer-recorder enters fewer data with DDRS. For example, there is no need to enter subsequent data once a tooth is coded as missing. If there are eight teeth missing, an interviewer-recorderdoes not have to enter 160codes (20 x 8). Several other initial codes mandate a default code for subsequent data fields; the DDRS automatically records these data. For instance, a sound ( S ) crown code mandates the " S code for each of five tooth surfaces. Beyond data collection, several time-consumingda ta-processing steps are eliminated with DDRS. There is no need for data coding (i.e., transferring data from instrument to code sheet in preparation for data entry): there is no need for data entry or key punching data into the computer. Finally, time spent on "cleaning" the data is minimized, since the data collection program is designed to be self cleaning to the extent possible. The DDRS program is written in Clipper, a data base language superset of dBASE I11 Plus. Although the same application could have been written in conventional languages such as BASIC or C, the advantage of Clipper is the availability of data base features and functions. A program written in Clipper may be compiled and run on any DOS system. The program has been designed in
2. Potential for keystroke error when data are transcribed from paper form to computer for analysis. 3. "Data cleaning" phase requires substantial effort. Effort increases proportional to size and complexity of data set. 4. Change in examination pattern possible, thus potential for error of omission and incorrect recording. 5 . Not feasible. 6. No such costs involved. 7. Computer training not needed. 8. Illegibility of data possible. 9. No such concern.
modules, so any modifications are comparatively simple to make. Any method selected for data collection has its limitations. A computerized system would be efficient if field supervisors and interviewers were computer literate. While these skills can be included in comprehensive start-up training, it would be ideal if the staff were hired with typing skills. The program must be user friendly. A program such as DDRS could be improved with contextsensitive help screens. This function would be especially helpful for nondental interviewer-recorderswho are less familiar with dental conditions and codes. In addition, if a program is developed from scratch, project management should keep a research computer programmer on the team who is available to troubleshoot problems as they arise. After a program has stood the rigor of an actual study, this need is minimal. Moreover, a clear distinction must be made between two levels of such a system: one with data-entry function only, and the other with data-entry capabilities combined with a data management system (full-screen editing, browsing, editing, appending of records).The initial time and cost involved are linked to the additional functionality of the system. Also, the data-entry function and data-management function do not necessarily have to be linked in the same system. This is a choice that a given project has to make. In our experience, adding a data-management function, although it adds complexity at the design and coding level, provides better data handling at the user level. Although cost savings can be realized using a computerized DDRS, there are other associated expenses such as the cost of the hardware and necessary software. The laptop computers used in the NEEDS were subjected to an extreme amount of use due to the nature of the project.
Journal of Public Health Dentistry
238
The field staff were required to travel throughout New England, mostly by car. Finally, one may question whether such technology is warranted, given the frequency of errors detected in this comparison with a paper-form system. The answer to this question is based on two fundamental issues: (1) What error rates are tolerable without compromising the quality of data? and (2) What time savings, if any, are needed before investment in the development and use of such a system starts to pay off? While our initial experience and evaluation results are encouraging, we do not have data on time and costs of using DDRS to assess the superiorityof computer-based systems over paper-based systems. We strongly feel that more studies are needed that answer these questions of cost-benefit and impact on quality of data in a direct way. An important component of such cost-benefit analysis would be whether a study team has a turnkey system, such as the data-entry system developed by Eklund at the University of Michigan, or DDRS, available to them or if they have to develop the computerized system from scratch. In summary, the dentaldata recording system (DDRS) developed for NEEDS showed that a computerized data collection system could minimize many types of data recording and keystroke errors that are not possible to handle on a paper system. DDRS may save time during data collection and certainly saves time during data processing. By eliminating the need for data coding and subsequent cleaning, immediate access to data for analysis was possible. Acknowledgments We acknowledge with thanks the contribution of Dr.Steve Eklund for his pioneering work in this area and the idea of direct data-entry systems in dental epidemiologic studies.
References 1. JeffcoatMJ, Entin E, Douglass CW.Computer education for dental students. J Dent Educ 1986;50(5):260-3. 2. Feldman CA. Microcomputer elective for senior dental students. J Dent Educ 1990;54(9):57@5. 3. Zimmerman IL, Landsman HM, Bilan JP.Study of computer applications in dental practice. Dent Clin North Am 19868:731-8. 4. Irvine RS,Moore RN. Computer-assisted instruction in mixed dentition analysis. J Dent Educ 1986;50(6):312-15. 5. Pao YC, Reinhart RF, Kreja RF, Taylor DT. Computer-graphics aidedinstructionof three-dimensional dentalanatomy. J Dent Educ 1984;48:315-17. 6. Luffingham JK. An assessment of computer-assisted learning in orthodontics. Br J Orthod 1984;11:205-8. 7. Monteith BD. Computerized expert system for the diagnosis of pulprelated pain. Int J Prosthodont 1991Jan-Feb;4(1):30-6. 8. White SC. Computer-aided Uferential dlagnmis of oral radiographic lesions. Dentomaxillofac Radio1 1989May;l8(2):53-9. 9. Leinfelder KF, Isenberg BP, Essig ME. A new method for generatingceramicrestorations: a CADCAMSystem 1989June.J Am Dent ASWC1989 J~11;118:7M-7. 10. Combs R. Dental office computerization-the next generation. Dent Econ 1989;20(7):69-79. 11. Arbogast JG, Dodril WH. Diabetes home monitoring by telephone data etry. Primary Care 1985Sept;12(3):573-9. 12. Khoury JC, Hertzberg VS, Miodovnik M, Khoury P, Berk M. The SAS system: an aid in caring for the pregnant diabetic patient. In: Proceedings of the 14th Annual SAS Users Group International Conference,San Francisco: 1989;1049-52. 13. Holdbrook,MJ. Determining theroleofSASsystem anda database management system in clinical research. In: Proceedingsof the 14th Annual SASUsersGroup International Conference,San Franscisco: 19898a3-7. 14. Hawkins BS, Singer SW. Design, development, and implementation of a data processing system for multiple controlled trials and epidemiologic studies. Controlled Clin Trials 1986J~n;7(2):89-117. 15. Christiansen, DH, Hosking JD, Dannenberg AL, Williams OD. Computer-assisted data collection in multicenter epidemiologic research. Theatherosderosisriskin communitiesstudy. Controlled Clin Trials 1990Apr;11(2):101-5. 16. Birkett NJ. Computer-aided personal interviewing. A new technique for data collection in epidemiologic surveys. Am J Epidemiol 1988Mar;127(3):684-90. 17. Harlow BL, Rosenthall JF, Ziegler RG. A comparison of computerassisted and hard copy telephone interviewing. Am J Epidemiol 1985 Feb;122(2):335-40.
Journal of Public Health Dentistry
Improving Dental Epidemiologic Data Collection with Computers Anil Joshi, BDS, MPH Instructor Department of Dental Care Administration Harvard School of Dental Medicine Susan McDermott, MPH, RN Project Director Elder Dental Health New England Research Institute Philip Marcus, DMD Instructor Department of Dental Care Administration Harvard School of Dental Medicine Chester W. Douglass, DMD, PhD Professor and Chairman Department of Dental Care Administration Harvard School of Dental Medicine 188 Longwood Avenue Boston, MA 021 15 Alan Jette, PhD Senior Research Scientist New England Research Institute Watertown, MA
Abstract A computerized dental data recording system (DDRS) was developed for the New England Elder Dental Study to improve data quality and increase field staff efficiency. The DDRS displays video screens similar tc traditional paper forms to record data on coronal and root caries, dentate and denture status, subacute bacterial endocarditis screening, gingival bleeding, calculus, and periodontalattachmentlevel. DDRS provides facilities fordate and exam-component time tracking, on-line contextual comments, random record retrieval, editing, data backup, and data output in various data formats. This study compared the DDRS with a paper-form system for data entry accuracy. Dental caries and periodontal disease measurement data from38 subjects were recorded on paper forms and independentlyenteredusingDDRS. The DDRSidentified 150 illogical data errors, 39 inconsistent data errors, Send correspondence and reprint requests to Dr. Josh. This work was supported by NIA grant #AG07139. Manusaipt received: 7/31 /91; returned to authors for revision: 9/10/91; accepted for publication: 1 /14/92.
J Public Health Dent 1992;52(4):232-8
7 invalid data and 34 miscellaneous data errors. Four technicians with field experience using both paper forms and DDRS reported time savings using DDRS in the field. DDRS has the potential for additional time savings by minimizing the time for data coding, cleaning, and management. Results demonstratethat DDRS could improve the quality of oral epidemiologic data by mandating strict adherence to protocols,preventing errors, and increasing field efficiency.
Key words: dental, epidemiology, computer, data collection, field studies, in-home surveys.
The role of computers in dental education, research, and practice has increased at a rapid rate (1-6).Computer softwarehas been developed to aid cliniciansin differential diagnosis of pulp-relatedpain and oral radiographic lesions (7,8). Computer-aided design and computeraided manufacturing(CAD-CAM) systems are available for designing and preparing dental restorations (9). Other computer programs allow efficientrecord keeping
233
Vol. 52, No. 4, Summer 1992 FIGURE 1 Screen-Page 1 of DDRS: Identifying Information on Dentist and Interviewer-recorders Are Recorded on this Page
Enter ID information: Dent-ID 201 FS-SIGN PASS
R-ID EXAM-DT
CODE TABLE ID Information:
410 03/09/90
Field Tech ID ... You Know It. Dentist ID:
THE RECORD IS ALREADY IN THE DATABASE.
Dentist A ... 201 Dentist B ... 301 Dentist C ... 401 Dentist D ... 501
ENTER 0TO RETRIEVE THE RECORD ENTER (N) TO CHANGE RESPONDENT ID
FS -SIGN ... Type EXAM-DT.. .Examination Date
FIGURE 2 Tooth Table: Screen Page 4 of DDRS 1
M M 17
2 M M 18
3
T M 19
4 P M 20
5 M M 21
6 T M 22
7 P M 23
8
P M 24
9 P M 25
10
11
P M 26
P M 27
12 T M 28
13
14
M M 29
P M 30
15 M M 31
16
M M 32
Tooth status: tooth present?, tooth missing=M, unable to assess=Y,root tip only=T
in dental offices, including periodontal charting (10). Innovative use of computer technology such as memory glucometry, which electronically transfers glucose values from the patient's home glucometer to a physician's computer via telephone-modem, has been reported (11,121. There has been increased use of clinical data validation through computer systems (131, as well as computer-aided data collection and validation in multicenter epidemiologic studies and clinical trials. (14) There has been little empirical work on the impact of using computers for direct data collection. In comparing a computer-assisted data collection system for recording the responses from personal interviews with the paperbased data collection system, Christiansen et al. (15) found no systematic differences between the two methods. An evaluation of computer-aided personal interviewing revealed a slightly higher error rate for computerized systems when compared to the paper-pencil methods (16). Use of microcomputers in collecting dental epidemiologic data in the field, however, has become common since the early work of Eklund at the University of Michigan. There are, however, no published reports describing the functionality of computerized dental systems, their usefulness, or how they compare with paper-form systems in terms of efficiency and data quality. This paper describes the development and testing of a computerized dental data recording system (DDRS)used in the New England Elder Dental Study (NEEDS) to collect
in-home dental examinationdata. The dental component of this study included soft tissue, coronal and root caries, and periodontal examinations. Methods The DDRS is a comprehensive data recording system that integrates data collection for soft tissue examination, dentate and denture status, and caries and periodontal examination. The DDRS operates on IBM Personal Computers running under MS-DOS 3.0 and later versions. In NEEDS the DDRS was used on Zenith Supersport 286 laptop computers to collect dental examination data in the homes of study participants. The program's "computer screens" were designed to simulate the paper forms developed for collectingdental examination data. Twelve screens made up the complete dental examination protocol. Pages 1-3 recorded general information including field protocol reminders, respondent, examiner and interviewer-recorder identification numbers, exam-type and exam-component timings (Figure 1). Time fields were not displayed on the screen, but the DDFS automatically recorded the times of starting and completion for each section of the examination. This section also recorded the preexamination subacute bacterial endocarditis screening responses, which determined whether or not a subject was to be included in the periodontal examination. Requirement of a field supervisor signature, ''%-sign," allowed only authorized personnel to update the files in the master data base.
Journal of Public Health Dentistry
234
FIGURE 3 Screen Page 5 of DDRS: Information on Coronal Surfaces of Teeth 1-16 Are Recorded on this Screen Page
CROWN 1-16
R-ID: 99999
T H C C M 0 ~
- _ _ _ - - - ~ -
1 2 3 4 5 6
M M T P M T
M M T F M T
M M T F M T
M M T R M T
M M T S M T
M M T S M T
7
S
S
A
S
S
S
I
9 P S 10 P S 11 P S 1 2 T T 1 3 M M 14 P N 15 M M 16 M M
A A
D
CORONAL CARIES CODES
B L ~
F S A S T T M M F F M M M M
7 S S T M S M M
~ Valid Crown Code (CC):
F S
D T M S M M
1
Comment: Press
NEXT PAGE "()'
Screen pages 4-12 of DDRS were designed to record epidemiologic dental examination data, including soft tissue and denture-wearing status. The tooth table, page 4, allowed dentate status of each subject to be recorded (Figure 2). In addition, the information entered in the tooth table dictated which fields were displayed on subsequent screens. For example, if tooth #27 was missing, and was so noted on the tooth table, this tooth was automatically eliminated from the examination, making it unavailable for recording caries or periodontal measures in the rest of the examination. A default code of "M" (missing)was applied, preventing inadvertent data entry for this tooth. Coronal and root caries examination information was recorded on screen pages 5-8 of DDRS. Only valid, predefined codes that were displayed on each of these pages were accepted for various measurements (Figure 3). The final two pages of the DDRS program were for recording periodontal examination information (Figure 4). These pages have fields for recording measures of gngival bleeding, calculus accumulation, the distance from free gingival margin to cemento-enamel junction (CEJ), and probing depths. Periodontal measurements were routinely recorded for the mesial, buccal, and distal sites for each tooth. A fourth site was also recorded if it was greater than the three established sites. Each of the two periodontal screens represented a maxillary or mandibular arch, and contained 176fields in which data were recorded. When any invalid code was entered or any illogical datum was entered for any field on any page,
'
Present Crowned Crowned (deficient) Sound Valid Surface Codes: Decayed FailedResto.Non-car Filled Filled (C/R) Sound Filled (C/R) Crwn Decay Filled (R/C) Root Decay Filled (R/C) Sound Filled (R/C) Crwn Decay Filled (R/C) Root Decay Recurrent Caries Unable to Assess
P C
H S D
N F 1
2 3
7 8 9
R Y
then DDRS flagged the recorder by producing an audible tone and displaying a message on top of the screen. The DDRS integrates data collection for soft tissue examination, dentate and denture status, and caries and periodontal examination in one program. For soft tissue examination, however, it only records presence or absence of a soft tissue lesion and whether a referral was made at the time of examination. In brief, the functionality of DDRS can be summarized as follows: 1.The DDRS is not only a data entry system, but rather a data base system capable of appending, deleting, browsing and editing records. Full-screen editing capability allows the recorder to move back and forth among different examination screens. 2. The system also keeps track of the date and time spent on each section of an examination. 3. The DDRS can prevent common errors such as invalid code entry and inconsistent data across fields. 4. The system skips fields based on the information entered on previous observations. A default value is automatically assigned to the skipped fields. 5. A commenting feature of the DDRS allows the interviewer-recorder to write comments related to any information during each examination section.The availability of a "comment" data base allows systematic documentation of unusual findings and permits better coding decisions for clinical findingsunresolved at the time of examination. 6. The data output can be developed in many formats such as in ASCII, dBase I11 Cdbf format), or SAS data set
-
235
Vol. 52, No. 4, Summer 1992
FIGURE 4 Screen Page 7 of DDRS: Periodontal Assessment Information on Teeth 1-16 Are Recorded on this Screen Page R I D : 99999 T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
GA M CAL M B-CEJ M B-PD M M-CEJ M M-PD M DL-CEJ M DL-PD M GT-SIT M GT-CEJ M GT-PDM
M M M M M M M M M M M
B G -2 2 -2 3 0 4 ML -2 6
B G 2 3 0 3 O 3 A A A
M M M M M M M M M M M
N S 0 2 0 3 0 3 A A A
N S 0
N N 0
1
1
0 2 0 4 A A
0 2 0 2 A A A
N N 0 2 0 2 0 3 A A A
N B N G - 1 - 3 1 4 0 0 2 3 0 0 2 3 DB A -1 A 4 A
~ - - _ _ _ - - - ~ - - - - - ~ - - -
A
TI1
T12
T13
T14
T15
T16
B G 0 2 O 3 O 3 DI -2 5
M M M M M M M M M M M
B G
M M M M M M M M M M M
M M M M M M M M M M M
-
3 3
-
2 5
-
2 5 ML -1
6
PER10 CODE TABLE 1.Gingival Assessment 2. Calculus Assessment 3. CEJ or PD 4. GT Site
No Bleeding=N No calculus=N Buccal=BU Mesial=ME Distal=CI
Bleeding+B Unable to Assess=Y Supra-Gv=S Sub-Gv o r Both=G Put (-1 sign where appropriate with data Distobuccal=DB Distolingual=DL Lingual=LI Mesiolingual=ML Mesiobuccal=MB N/A=A
(.SSDformat). 7. The modularity of the design facilitatesmodification and makes it easy to maintain the software. We estimated DDRS’ potential to prevent data entry errors in comparison to a traditional paper-form system. Dental data were collected in the field (in subjects’ homes) on paper forms similar in format to the screens of the DDRS program. Four examiners and three recorders were involved in collecting data on paper forms in the field. Subsequently, one person other than examiner or recorders used DDRS to enter the same data into a computer. By flagging any invalid or inconsistent data that were transcribed from paper forms to the computer system, the DDRS identified the errors that would have been prevented if DDRS had been used directly in the field in place of the paper forms. The NEEDS was a community-based epidemiologic survey of a representative sample of noninstitutionalized elders aged 70 and older living in the six New England states. The data used for this analysis were obtained from the pilot study for NEEDS conducted in an urban community in the Greater Boston Metropolitan area in 198889. Data were collected on coronal caries, root caries, gingival bleeding, calculus accumulation, and periodontal attachment level of 38 study participants who had 12 or more teeth. The dental examinations were carried out by dentists in the participants’ homes using a portable examination lamp, mouth mirror, explorer, and a peri-
odontal probe. Results A mean number of 19 teeth were reported for the 38 study participants. The total number of measurements recorded on paper forms was 14392. Table 1presents the types and frequency of errors flagged when data were entered from paper forms to a computer using DDRS. Overall, 230 errors were identified (1.5%of all measurements), an average of six errors per subject examined. Illogical data ~ Y Y O Y S ,defined as inconsistencies between the measurements made on gingival recession and those made from the free gingival margin to the cementoenamel junctions, were identified in 75 pairs, resulting in 150 data errors. Two forms of inconsistencies were observed: (1) the root surface was not exposed, but a negative score in millimeters was recorded for the distance between the free gingival margin and cemento-enamel junction; or (2) the root surface was exposed but a positive score in millimeters was recorded for distance between the free gingival margin and cementosnamel junction. Znconsistent codes, cases where different codes were used to represent the same condition, were noted on only seven instances. For example, for an unexposed sound root surface the following different codes/symbols were used: NA, N/a, slash only (”/”), ”U”,“9dash , (”-”). Conversely, recorders used one code to represent differ-
Journal of Public Health Dentistry
236
ent situations in the same context. For instance, a blank space on the paper form might have meant any of the following:tooth missing, sound coronal surface, sound root surface, unable to assess. Unclear protocols and nondefrned situations (23 cases) were errors resulting from the use of various codes or clinical situations for which there were no codes available at the time of the examination. For example, codes were unavailable for crowned teeth with deficient margins and for restorations crossing the cemento-enamel junction with decay on the root. These errors per se were independent of paper form or computer-based system. (However, availability of computer-based system with commenting features will result in better documentation of such problems). There were 39 cases of invalid data where an unavailable code was used for a data field. Some aspect of six records wasillegible, creating the potential for many data errors per record. For the purpose of this evaluation we counted only one error per record. The six cases of illegibility included overwriting, strikeouts, and similarity of symbolsslanted “1” and ”/”. The remaining five errors were included in the miscellaneous category. Examples in this category include errors in recording identifying information such as respondent, and interviewer-recorder ID or date of examination. In one instance data for tooth number three was recorded in the space for tooth number four. In another instance, during caries examination a tooth was designated first premolar while during periodontal examinationit was designated second premolar. Discussion DDRS offers numerous advantages for the collection of dental epidemiologic data compared with traditional methods of data collection (Table 2). Dental epidemiologic data are typically collected on paper forms and subsequently entered into a computer for data management and statistical analysis. Traditional methods also include tape recording of actual exams and interviews, and use of specially designed paper-sheets that may be scanned electronically for data entry. All three methods that have been used extensively in the past have inherent disadvantages. In a paper-form system recording errors occur when out-of-range data, incorrect codes for responses, and inconsistent data are recorded. (Recording errors also include errors of omission and illegibility.) Keystroke errors occur when the data collected on paper forms is entered into a computer. Keystroke errors are most common when a data set is large with many data variables or when the data involves a complex coding and transcribing scheme. For instance, theNEEDS dental exam data set has 1,200 subjects with as many as 750 variables per subject. Tape-recorded dental e x a m and interviews incorporate the same problems as a paperform system. In addition, transcribing and coding problems in tape medium are even more complex if the cor-
TABLE 1 Frequency of Data Recording Error
Error Type 1.Illogical
2. Invalid 3. Unclear protocol and undefined situations 4.Inconsistent 5. Illegibility 6. Miscellaneous Total
__
Count
150 39 23 7 6 5 230
rections of previous measurements are made during the examination. Use of scanners to read the data eliminates keystroke errors, but requires clear demarcation of the appropriate marks on the sheet and often a complex marking sheet. This medium, like paper forms and tape recording, is inherently prone to recording errors. Computerized, on-site DDRS is most valuable for minimizing recording errors. Logical checks built into the computer program allow the data collector to identify inconsistenciesor out-of-rangedata in the field, allowing for correction when both patient and examinersare present. In all other means of data collection, incorrect data are usually identified in the “data cleaning” phase, which not only involves greater time and labor, but increases the cost of correcting the identified errors. The result is that these data have to be either excluded from the analysis or an arbitrary decision made for their inclusion. Computer systems such as DDRS allow programming of any logical checks or procedures used in the traditional data “cleaningphase,” allowing errors to be caught when the data are being collected. DDRS also helps in recording difficult measurements such as consistent scoring of gingival recession. The DDRS computer program mandates that the examiner-recorder team conduct and record data in a predefined and standardized manner. This eliminateserrors of omission and allows for clearer communication between dentists and field technicians during the data collection procedure. It also helps interview-recorders to work with a consistent examination style. The most striking advantage of the DDRSis that it provides the research team with immediate access to data for analysis. The system minimizes the need for coding and data entry and the quality controls that are otherwise necessary steps when paper or other systems are used. In the NEEDS study, field staff reported that time was saved in recording the dental examinationdata by DDRS. Two published reports, one comparing a computerized system with a paper-based system for personal interviews (16) and another for telephone interviews (17), have reported that the computer-aided methods took longer than paper systems to conduct the interviews. The
237
Vol. 52, No. 4, Summer 1992
TABLE 2 Comparative Evaluation of Computerized Data Recording System and Paper-Form System Computer-based System
Paper-based System
1. Minimized recording errors (e.g., checks for out-of-range and inconsistent data). 2. Transcribing error (not applicable).
1.Not feasible.
3. Minimized effort required at "data cleaning" phase as a result of Items 1.and 2. in this table. 4. Better coordination between dentist and recorder at the time of the examination due to fixed computer mandated protocol. 5. Immediate access to data for analysis feasible. 6. Initial cost of program development and testing is substantial. 7. Some computer training required for recorders. 8. Better documentation: no illegibility problem. 9. Possible malfunctioning in the field.
main reasons cited for increased time using a computer included questions requiring date responses (with correct punctuation), typing speed, questions that required more data entry in the computer than was required on paper forms for the same response, and the quality of hardware and software-slow systems took longer to display the next question or to verify a response. An objectiveassessment of time to use each system was beyond the scope of the current investigation. Although not determined in this study, time saving using DDRS could come from several sources. The interviewer-recorder enters fewer data with DDRS. For example, there is no need to enter subsequent data once a tooth is coded as missing. If there are eight teeth missing, an interviewer-recorderdoes not have to enter 160codes (20 x 8). Several other initial codes mandate a default code for subsequent data fields; the DDRS automatically records these data. For instance, a sound ( S ) crown code mandates the " S code for each of five tooth surfaces. Beyond data collection, several time-consumingda ta-processing steps are eliminated with DDRS. There is no need for data coding (i.e., transferring data from instrument to code sheet in preparation for data entry): there is no need for data entry or key punching data into the computer. Finally, time spent on "cleaning" the data is minimized, since the data collection program is designed to be self cleaning to the extent possible. The DDRS program is written in Clipper, a data base language superset of dBASE I11 Plus. Although the same application could have been written in conventional languages such as BASIC or C, the advantage of Clipper is the availability of data base features and functions. A program written in Clipper may be compiled and run on any DOS system. The program has been designed in
2. Potential for keystroke error when data are transcribed from paper form to computer for analysis. 3. "Data cleaning" phase requires substantial effort. Effort increases proportional to size and complexity of data set. 4. Change in examination pattern possible, thus potential for error of omission and incorrect recording. 5 . Not feasible. 6. No such costs involved. 7. Computer training not needed. 8. Illegibility of data possible. 9. No such concern.
modules, so any modifications are comparatively simple to make. Any method selected for data collection has its limitations. A computerized system would be efficient if field supervisors and interviewers were computer literate. While these skills can be included in comprehensive start-up training, it would be ideal if the staff were hired with typing skills. The program must be user friendly. A program such as DDRS could be improved with contextsensitive help screens. This function would be especially helpful for nondental interviewer-recorderswho are less familiar with dental conditions and codes. In addition, if a program is developed from scratch, project management should keep a research computer programmer on the team who is available to troubleshoot problems as they arise. After a program has stood the rigor of an actual study, this need is minimal. Moreover, a clear distinction must be made between two levels of such a system: one with data-entry function only, and the other with data-entry capabilities combined with a data management system (full-screen editing, browsing, editing, appending of records).The initial time and cost involved are linked to the additional functionality of the system. Also, the data-entry function and data-management function do not necessarily have to be linked in the same system. This is a choice that a given project has to make. In our experience, adding a data-management function, although it adds complexity at the design and coding level, provides better data handling at the user level. Although cost savings can be realized using a computerized DDRS, there are other associated expenses such as the cost of the hardware and necessary software. The laptop computers used in the NEEDS were subjected to an extreme amount of use due to the nature of the project.
Journal of Public Health Dentistry
238
The field staff were required to travel throughout New England, mostly by car. Finally, one may question whether such technology is warranted, given the frequency of errors detected in this comparison with a paper-form system. The answer to this question is based on two fundamental issues: (1) What error rates are tolerable without compromising the quality of data? and (2) What time savings, if any, are needed before investment in the development and use of such a system starts to pay off? While our initial experience and evaluation results are encouraging, we do not have data on time and costs of using DDRS to assess the superiorityof computer-based systems over paper-based systems. We strongly feel that more studies are needed that answer these questions of cost-benefit and impact on quality of data in a direct way. An important component of such cost-benefit analysis would be whether a study team has a turnkey system, such as the data-entry system developed by Eklund at the University of Michigan, or DDRS, available to them or if they have to develop the computerized system from scratch. In summary, the dentaldata recording system (DDRS) developed for NEEDS showed that a computerized data collection system could minimize many types of data recording and keystroke errors that are not possible to handle on a paper system. DDRS may save time during data collection and certainly saves time during data processing. By eliminating the need for data coding and subsequent cleaning, immediate access to data for analysis was possible. Acknowledgments We acknowledge with thanks the contribution of Dr.Steve Eklund for his pioneering work in this area and the idea of direct data-entry systems in dental epidemiologic studies.
References 1. JeffcoatMJ, Entin E, Douglass CW.Computer education for dental students. J Dent Educ 1986;50(5):260-3. 2. Feldman CA. Microcomputer elective for senior dental students. J Dent Educ 1990;54(9):57@5. 3. Zimmerman IL, Landsman HM, Bilan JP.Study of computer applications in dental practice. Dent Clin North Am 19868:731-8. 4. Irvine RS,Moore RN. Computer-assisted instruction in mixed dentition analysis. J Dent Educ 1986;50(6):312-15. 5. Pao YC, Reinhart RF, Kreja RF, Taylor DT. Computer-graphics aidedinstructionof three-dimensional dentalanatomy. J Dent Educ 1984;48:315-17. 6. Luffingham JK. An assessment of computer-assisted learning in orthodontics. Br J Orthod 1984;11:205-8. 7. Monteith BD. Computerized expert system for the diagnosis of pulprelated pain. Int J Prosthodont 1991Jan-Feb;4(1):30-6. 8. White SC. Computer-aided Uferential dlagnmis of oral radiographic lesions. Dentomaxillofac Radio1 1989May;l8(2):53-9. 9. Leinfelder KF, Isenberg BP, Essig ME. A new method for generatingceramicrestorations: a CADCAMSystem 1989June.J Am Dent ASWC1989 J~11;118:7M-7. 10. Combs R. Dental office computerization-the next generation. Dent Econ 1989;20(7):69-79. 11. Arbogast JG, Dodril WH. Diabetes home monitoring by telephone data etry. Primary Care 1985Sept;12(3):573-9. 12. Khoury JC, Hertzberg VS, Miodovnik M, Khoury P, Berk M. The SAS system: an aid in caring for the pregnant diabetic patient. In: Proceedings of the 14th Annual SAS Users Group International Conference,San Francisco: 1989;1049-52. 13. Holdbrook,MJ. Determining theroleofSASsystem anda database management system in clinical research. In: Proceedingsof the 14th Annual SASUsersGroup International Conference,San Franscisco: 19898a3-7. 14. Hawkins BS, Singer SW. Design, development, and implementation of a data processing system for multiple controlled trials and epidemiologic studies. Controlled Clin Trials 1986J~n;7(2):89-117. 15. Christiansen, DH, Hosking JD, Dannenberg AL, Williams OD. Computer-assisted data collection in multicenter epidemiologic research. Theatherosderosisriskin communitiesstudy. Controlled Clin Trials 1990Apr;11(2):101-5. 16. Birkett NJ. Computer-aided personal interviewing. A new technique for data collection in epidemiologic surveys. Am J Epidemiol 1988Mar;127(3):684-90. 17. Harlow BL, Rosenthall JF, Ziegler RG. A comparison of computerassisted and hard copy telephone interviewing. Am J Epidemiol 1985 Feb;122(2):335-40.
