British Journal of Dermalohgii {} 992} 127, 614-619.
DERMis: a computer system for assisting primary-care physicians with dermatological diagnosis G.J.BROOKS,* R.E.ASHTONt AND R.J.PETHYBRIDGEij: Royal Naval Medical Service.'^ Royal Naval Hospital Haslar, ^Institute of Naval Medicine. Gosport. Hampshire P012 2AA. U.K. Accepted for publication 26 |une 1992
Summary
is a computerized skin disease diagnostic prompting system which has been derived from the prospective study of 5205 cases. It has been designed for use by non-dermatologists such as general practitioners. The program produces a list of reasonable diagnoses based on probabilities calculated using Bayes' theorem. Out of 221 precise diagnoses made hy the dermatologist in the clinic, 42 groupings were created to encompass the most common or important diseases encountered in genera! practice. Eour 'remainder' or 'send to specialist' groups were included for the 13% of uncommon conditions. The program, when tested by "one out' analysis on the original cases, placed the correct diagnosis first on 76% of occasions, and within the first three on 9 S% of occasions. In 76 of 1 2 5 cases randomly selected from the data base, a request for diagnostic assistance had been made by the referring general practitioner. It has been estimated that in 54 of these 76 cases the DERMIS system could have provided differential advice with the correct diagnosis appearing at the top of the list. DERMIS
The system has been written in MUMPS and runs on an IBM-compatible desk-top computer. The software allows real time data entry. Arrangements are in hand for performing clinicai trials of the system in general practice. If current performance is maintained, and the response to the system's output is favourable, then DRRMIS might well enhance performance in medical decision making for the benefit of patients, medical services and budget holders. During the last two decades, numerous computer programs have been designed to offer medical advice on a variety of topics.'^ Unfortunately, the creator's enthusiasm has rarely been matched by any compelling medical need for the product. Even when projects have appeared appropriate, many have failed to find widespread use. as there has been no easy way of implementing them. The current emphasis on audit and budget control has led to the installation of computer terminals in many general practitioners' consulting rooms. The hardware is being used to store clinical information and could also run diagnostic prompting software. Skin disease accounts for about 7%i of all consultations in general practice, yet many doctors find the accurate diagnosis of skin disease difficult. This should not be the case, as skin physical signs are visible for all to see. Standard terminology can be used to describe the physical signs, but the interpretation of these is learned by experience and not from standard textbooks. The Correspondence: Surgeon Oinimander K.lvAshton KN, dermatology Deparlmenl, Royal Nava! Elospitul. (losporl, Hampshire F012 2AA. U,K, * Present iidtlress; Fgton Medical Information Services. 77 Back Lane, Horsforth. l^eds LSI H 4Rr, U.K.
614
objective of this study has been to produce a tool which can analyse physical signs and assist with the formulation of a differential diagnosis. The potential value of the prompting system to users has been assessed from a sample of cases by analysing the reason for referral, to identify whether diagnostic advice or treatment is requested.
Methods The subjects included 5203 consecutive patients (new referrals only) seen by a dermatologist (R.E.A.) from 1985 to 1989. Any omissions occurred randomly due to logistic reasons. Data was collected from each case according to a comprehensive list of dermatology symptoms and signs (Fig. 1). Data collection was performed by the consultant, or clinical assistant/general practice trainee. Signs and symptoms were, however, checked by the consultant for consistency. The terms employed had previously been defined and tested through practical use by students and general practitioner trainees. The collection sheet was self-copying and was in two parts. One of these became the patient record. The other was coded to allow data base entry by staff of the Institute of
COMPUTER SYSTEM IN UERMATOLOGICAL DIAGNOSIS
Dale
/
Male/ Female
/
ComDieted by AGE
. yrs/
mths
HOSPITAL No:
ASSOCIATED FEATURES
NAME:
19 SCALP not involved papules scaling hair loss uniform patchy palpable impalpable scarnng extends beyond hair margin remains vulhin hair margin
HISTORY
TIME since onset of tash/lesiori DURATION of rash when present SIZE gradual increase in Ves/Nc PETS: dog/ cat/ other ( 2. HISTORY of alopic eczema other eczema asthma hay fever psoriasis dandruff skin disease
family family family family family family
years/ mtha/ days •fieeks/ days/ , hours PIGMENT any change in Yes/No ITCHING. none mild moderate seveie day night family
20. MOUTH not involved white streaks ulcers
4. RACE, Caucasian negro asian oriental SKIN TYPE, I Hi II IV
21. GENITALS: no! inuQived muolued
5 TREATMENT used previously. steroids improved improved anli-fungals improved
6. MADE WORSE BY: 7, SKIN HYDRATION soap/delergent normal oil/grease greasy water dry sunshirie veiy Ory lichthyosisl exercise/hot tjaths OINTMENTS/CREAMS given by GP DRUGS laKer in last 3 months
DIAGNOSIS: TnEATMENT:
12. COLOUR: normal pink red
pink purple white cream orange yeltow golden light brown dark brown black grey hypo pig men ted hyperpigmwitea other (
_
_J
HOSPITAL No. 13 BORDER/EDGE: definite variable tndislinct raised above centre active edge other 1 \ \ 1 M. SHAPE (definite lesions) round oval annular linear pedunculated irregular other
15 SURFACE FEATURES
normal warty scaly ex u date crust macerated friable atrophic flats shiny white streaks lichen if led excoriated umbiiicated nairy keralin fissures oltier (—
16 VASCULAR FEATURES erythema purpura telangiectasia varicose veins 17 PALPATION ideep< (surface) normal smooth soft uneven firm rough hard hot tender
)
1 f f f
f f f f f
18. SCRATCH TEST no Change mild scale profuse scale wnal
Figure 1. Data collection sheet used during patient interview.
not involved fine pitting coarse pitting onyc ho lysis subungual hyperkeratosis nail thickening loss of nail plate transverse nOges longitubinsi ridges
23 PALMS S SOLES: nol involved vesicles pustules fissures hyperkeratosis scaling in creases scaling in finger webs maceration between toes burrows on fingers/wrists
MYCOLOGY PERFORMED VES/NO BIOPSY PERFORMED YES/NO
OCCUPATION EXAMINATION OF RASH/LESION 8 NUMBER. 11 TYPE: Single iesion macule 2-5 lesions patch 6-20 lesions papule 21 /rash nodule no rash plaque vesicle 9. DISTRIBUTION buNa symmetrical F^stule asymmetrical weal grouped erosion linear ulcer sun exposed crater scar 10 SIZE: comedone anqiooeaerna mrti average other variable to mm
22 NAILS.
RESULT organism
+ve''-ve
615
616
C.j.BROOKS et «i.
Naval Medicine on io a mainframe computer. This information was later transferred to an IBM-compatible desk-top computer for analysis. The diagnosis in each case was made clinically by the dermatologist at the time of the examination, or modified later based on histological or mycological findings. Bayes' formula was used to calculate the likelihood of each disease being present given a set of symptoms and signs. The probability of a disease being present (i.e. the computer's suggestion or diagnosis) can be calculated if we know the probability of the disease occurring in the population (here our data base), the probability of the sign occurring in that disease (measured by collecting the cases) and the probability of the sign occurring in all diseases. In mathematical terms this is expressed as the following formula: If disease D (e.g. psoriasis) and sign S (e.g. scaling) are independent events then:' PiD\S) =
P iS\D) X F (D) PjS)
where P {D\S) is the probability of a disease (e.g. psoria• . sis) being present given the sign (e.g. scaling) elicited; P(S\D) is the probability of the sign (scaling) occurring in psoriasis: P (D) is the prior probability of the disease •; occurring, i.e. how commonly psoriasis occurs in the data base: P (S) is the probability of the sign occurring at ali. By using this calculation for each disease in turn, relative likelihoods (as a percentage) of all diseases are produced. The diseases can be placed in order by percentage likelihood to give a differential list. It was apparent from examining cases that combinations of signs occurred within categories (e.g. two colours or two types of lesion are present). Single probability estimates have been formed for these combinations and used in the Bayes' equation instead of the separate 'independent' component estimates. There were 221 separate diagnoses. Where appropriate, diagnoses were collated into groups (Table 1). For example, psoriasis has been used as a general term to describe cases of plaque, guttate. intertriginous. nail and scalp psoriasis. For some other diseases it was appropriate to retain the original diagnosis. For example, malignant melanoma is represented in both nodular and superficial spreading forms, as the clinical features are so different. Rare diseases have been represented in four remainder' or 'refer to specialist' groups.
Tdbic 1. Accuracy of assignment of correct diagnosis to top of differential list in 'one-out' analysis of 5203 cases in data base
Group title 1. Alopecia areata' 2. Acnet 3. Basal cell carcinoniLi 4. Superficial BCC 5. Bowen's disease 6. Chondrodermatitis 7. Cyst epidermoid 8. Dermatotibroma 9. Eczema^ 10. Foiliculitis 11. Granuloma anniilare 12. Herpes simplex 11. Insect bite§ 14. Keratoacanthoma 15. Lentigo 16. Lichen planusi 17. Malignant melanoma 18. Malig, melan, superficial 19. Milia 20. Molluscum contagiosum 21. Naevus** 22. Naevus spider 23. Pyogenic granuloma 24. Pityriasis rosea 2 5. Psoriasis palm/plantar 2fi. Psoriasis*! 27. Pityriasis versicolor 28. Rosacea 29, Scabies 30. Seborrhoeic wart 31. Solar keratosis$t 32. Skin tags 3 3. Squamous cell carcinoma 34. Tinea§§ 35. Urticaria 56. Vitiligo !7. VVartsiH 38. Verruca*" 59. Other single lesion 40. Other multiple lesions 41. Other rash 42. Other no rash or lesion
No. cases data base 40
253 288 24 27 31 59 90
1124 23
47 23 31 25 38 64 21 51 23
47 396 36 47 19 10 414 6S 60 52 266 269 33 35 82 69
24 277 50 217 190 253 7
No, (%) of times top differential 39(98) 232 (92) 201 (70)" 2 3 (96) 22 (Hl)^ 28(90) 51(86) 77(86) 847(75) 19 I8J) 42 190) 22 (96) 24(77) 22 (88) 28(74)
54(84) 18 (86)' 49 (96)'' 20(87) 45(96) 250(63) 36(100) 46(98) 18 (95) 10(100) 379(91)
54(79) 53 (88) 51 (98) 183 ((.9) 187(69) 32(97) 26(74)'' 64(78) 65(94) 23(96) 222 (80) 49(98) 109 (50) 101 (53) 104 (41) 6 (861
* Includes cases of alopecia totalis. t Includes cases of acne excoriee, perioral dermatitis. % Includes cases of eczema; atopic, contact allergic, discoid, foot and hand. intertriginous. impetiginized. pompholyx, craquelc, seborrhoeic. varicose, papular, acute, irritant; lichen simplex chronicus. juvenile plantar dermalosis, ^ Includes cases of papular urticaria, •! Includes cases of lichen planus hypertrophic, " Includes cases of naevus; [unctional. blue, compound, halo, hairy, pigmented. intradermat. linear epidermal, warty epidermal, •|"t Includes cases of plaque, guttate, intertriginous, nail, scalp, :^if Includes cases of cutaneous horn, §§ Includes cases of tinea: corporis. cruris, incognito, manuum, pedis, unguium. ' ' Includes cases of common, filiform, genital, plane. """ Includes cases of corn, exostosis, ^ Not included 49 cases diagnosed as other malignancy, ^ Not included four cases diagnosed as other malignancy. '" Excluding one case diagnosed as superficial spreading melanoma. '' Not included one case diagnosed as other malignancy.
COMPUTER SYSTEM IN DERMATOLOGICAL DIAGNOSIS
A 'one-out' method was adopted^ to test the accuracy of the computer modeL Each case in turn was excluded from the data base, and the frequency of symptoms and signs being present for each diagnosis was calculated using the remaining cases. An estimate of the likelihood of occurrence of each disease with the symptoms and signs of the excluded case was calculated by multiplication of the appropriate frequency and prior probability information. The diseases were then ranked according to their likelihood scores to give a differential list. Assessment of predictive accuracy using the algorithm involved: 1. the original 221 diagnoses; 2. 42 diagnostic groups created using the diagnosis reached in 1: 3. combination of diagnoses into 42 groups before calculation using the algorithm; 4. as for 3. using a reduced data sheet. Where the predicted diagnosis has disagreed with the actual diagnosis, the computer's differential diagnosis was investigated in order to assess where the correct diagnosis appeared in the differential list. The clinical records from a random sample of 125 collected cases were scrutinized to determine the reason for referral, in order to determine when a diagnostic advice system might be of use.
Results Within the data base the number of cases allocated to each of 221 specific diagnoses varied considerably, from the hundreds of 'eczema' cases to one or two representatives of extremely rare diseases. The diagnosis appearing at the top of the differential was correct in 60% of cases when all 221 diagnoses were used. However, if a correct diagnosis was more loosely confined to one of the 42 major diagnostic groups, the overall predictive value increased to 76%. In a further test, where the 42 groups were formed prior to use of the predictive algorithm, the diagnostic accuracy was again found to be 76%, but the pattern of success and failure of individual cases varied between the two methods. On 95% of occasions the correct diagnosis appeared in the top three of the differential list. The original data sheet (Fig. 1) allowed 270 options for description of cases. The list was reviewed in the light of clinical experience and analysis of the predictive value of each item. It was found that the list could be reduced to 11 5 items without loss of predictive accuracy.
617
Table 2. Most common errors: confusion between specific disease groups
Actual diagnosis Eczema Send to specialist Fx:xema Eczema Naevus Eczema Eczema Naevus Basal cell carcinoma Naevus Send to specialist Seborrhoeic wart Acne
Top of differential
Number of cases (%)
Psoriasis Eczema Scabies Tinea Seborrhoeic wart Pityriiisis rosea Send to specialist Superlicial melanoma Squamous cell ctircinomii Skin tug Pityriasis versicolor N'aevus Rosticea
84(7) 70(3) 63(6) 38 (J) 21 (5) 20(2) 20(2) 1815) IS (6) 18(5) 17(7) 15(6) 15(6)
The accuracy with which the diagnosis was predicted varied between the groups (Table 1). Eor example. 847 (75%) of the 1124 cases of eczema were correctly identified by the program, compared with 49 (96%) of the 51 cases of superficial spreading melanoma. Cases of rarer diseases, e.g. pemphigoid. mycosis fungoides, which were assigned to "remainder' groups, made up 13% of the total database. These groups had the highest failure rates, i.e. 41-53% (Table 1). The system output was studied in order to determine any patterns of success and failure. Certain diseases tended to be confused with one another (see Table 2). For example, the commonest type of failure occurred in 84 cases of eczema where the system placed psoriasis above eczema in the differential list. In 49 cases of the papular/ nodular presentation of basal cell carcinoma, a malignant tumour other than basal cell carcinoma (or 'remainder' group in four cases) was selected by the system. The management of these cases is the same— either excision biopsy or referral to a specialist. A sample of 12 5 randomly selected case records were scrutinized in detail, in order to determine the reason for referral and the outcome following the clinic visit. The results are presented in Table 3. In 76 (61%,) cases (summarized in Table 4) the family practitioner requested assistance with diagnosis. The final diagnosis by the dermatologist matched one of the 38 main DKRMIS disease group headings in 71 (9 3%) of these cases. The number of cases in which the DERMIS system placed the correct diagnosis at the top of its differential list was 54 (71%).
618
G.J,BROOKS et al.
Table J. A random sample of 12 5 cases referred to dermatology clinics by primary care physicians: reason for referral vs. outcome
Outcome
Keason for referral Diagnosis and management (a) diagnosis unknown (bi ,- malignant (c) ? infection Diagnosis correct, for second opinion Management only Re-referral, further treatment Removal/biopsy
Malignant tumour
Benign lesion(s)
Rash no infection
5* 6* — — _
19*
21*
— 4* 4
3* 3* 2* 2 15
Infection/ infestation
7* — — 1' 3 9
10
Indicates the cases shown in Table 4.
Table 4. Breakdown by diagnosis of 76 randomly selected cases referred by primary care physicians to dermatology clinics for diagnosis and management
Final diagnosis Basal cell carcinoma Bowen's disease Discoid lupus Eczema Erysipelas In.sect bites Lentigo Lichen planus Naevus Rare tumours Pyogenic granuloma Pityriasis rosea Psoriasis Pityriasis versicolor Rosacea Scabies Solar keratosis Squamous cell carcinoma Tinea Urticaria Vasculitis Viral wart Verruca
Number of cases 9 1 1 15 1 2 2 2 12 2 1 1 4 1 1 2 7 1 5 2 1 1 2
Discussion Prospective collection of case data is one of the techniques commonly used to acquire knowledge for computer-based medical prompting and expert systems. The method relies on the clinician defining the diagnosis and ensuring that all relevant information is collected. Case collection is a lengthy task which needs to he carefully planned and conscientiously carried out hefore any
predictions can be made. However, the effort frequently brings rewards as previous gaps in knowledge are tilled and uncertain boundaries between diseases are clarified. Haberman et al.'' chose not to use case collection in creating a diagnostic system for dermatologists. Instead, they asked dermatologists to estimate the importance of varions clinical features in diseases. These estimates were then used as the basis of weightings and rules which formed the diagnostic algorithm. The system which they produced has been tested, with mixed results. In the later stages of development case information was fed back into the system in order to adjust the original weightings. Other methods which have been used in attempts to create diagnostic advice systems for dermatology have been described by Stoecker.'' The statistical approach to pattern matching adopted in this study has been popular for many years. The requirements, advantages and disadvantages of Bayesian methodology are well known.' There are several sources of variation which are likely to determine the performance of the algorithm in practical use. Disease prevalence can alter with geographic area. If DERMIS was to be tested in general practice, appropriate prevalence rates would have to be used to prime the system. The presentation of a disease can vary between patients, and the content of data collected from the same patient can vary between doctors. In this study the cases were all collected by one of the authors (R.E.A.), and occasionally by GP trainees or medical students. There was no allowance for inter-observer variation. A further weakness in this study has been that the dermatologist (R.E.A,) could have introduced substantial bias by subconsciously fitting symptoms and signs to diseases rather than faithfully recording new case details. This could lead to increased accuracy of the
COMPUTER SYSTEM IN DERMATOLOGICAL DIAGNOSIS
program when tested against itself. Testing the program against other dermatologists would help reveal any bias. Data has been collected mainly from white Caucasian patients. Rashes in patients with pigmented skin may look different, but although the colour of the rash may be different, other features such as surface changes and border characteristics should be the same. The accuracy of DERMIS using the original 221 diagnoses is comparable with similar systems in other subjects.' Many of the errors were eliminated by combining different forms of the same disease (i.e. combining ail the eczemas), to produce a predictedaccuracy of 95%for the first three in the differential list. It has not been possible to eliminate all DHKMIS system errors by the combination of subgroups. The clinical appearance of eczema, for example, can be identical to that of tinea infection, and that of a solar keratosis visually similar to a basal cell carcinoma. Combining two such diseases into the same group is of no immediate value to any system user, as the treatments are different. In these cases a dermatologist would investigate further to establish the diagnosis. The prompting system could be set to recommend further investigation when two diseases are known to cause problems. The group combinations of Table 1 were tested in two ways. Combining these before application of the algorithm led to a dilution effect of the important diagnostic features of the subgroups. Whereas, combination of the groups following application of the algorithm led to under-diagnosis of the small subgroups. The outcome of the two types of combination was virtually the same, with an overall diagnostic accuracy of 76%. The accuracy with which the correct diagnosis was placed at the top of the differential list by the system appeared to be related both to group size, and variability of presentation. Increasing the number of cases of a disease in the data base will improve the accuracy with which the disease is identified. However, if there are few distinctive features, or the presentation varies from case to case, then performance will be degraded. In such cases dermatologists often have a distinct advantage, as their expertise enables them to sift the available information
619
for important clues which may have variable weighting according to circumstance, DtRMis was not designed to replace dermatologists, but to offer general practitioners assistance with the formulation of a reasonable differential diagnosis. The system was able to place the correct diagnosis in the top three of its differential output on 9 5% of occasions. It also demonstrated a high accuracy in identification of malignancy, even if the correct tumour was not always placed at the top of the list. It is important to recognize that computer-based diagnostic prompting or expert systems are unlikely ever to recommend the correct diagnosis on all occasions. When a random sample of 12S cases from the data base was reviewed it was found that 76 (61%) had been referred for initial diagnosis or a second opinion. The system offered a differential listing with the correct diagnosis at the top in 54 of these 76 cases. It is not known what influence this type of information would have on referral patterns. Currently, patients may have to wait many months before being seen in a dermatology clinic. Use of a system such as DKRMis might encourage general practitioners to treat more cases themselves and to make more appropriate referrals.
References 1 KunzJC, Shortliflc I'.W, Buchanan BG. rt'igeiibaum RA. Computerassisted decision making in medicine. / Med I'hilos I yS4:9:l iS-fiO, 2 Miller PL. The evaluation of artificial intelligence sy.stcms in medicine. Comput Methods Protframs Biomed I4Sf):22:S-l 1. 1 Cood IJ. The Estimation of Probahilitifs: An Essan on Modern Hayesian Melhods. Cambridge, Miiss: MIT Press, 19f»'5. 4 Ingram D, Bloch RF, Medical stalistics: advanced techniques and computation. In: Mathemat'ual Models in Medicine. Chichester: I. Wiley and Sons Ltd., 1984. 5 tlaberman HI'". Norwich KH. Diehl DL cl al. DIAC: A computerassisted dermati)logic diagnostic system clinical expericnct- and insight. / Am Acad Dermatol iySS;12:l i 2 - 4 i . (l Stoeker WV. Computer-aided diagnosis of dermatologic disorders. Dermalol Clin 198fi:4:fi()7-25. 7 Carroll B. ArtiHcial intelligence. Expert systems for clinical diagnosis: are they worth the effort.' Hehav Sci 1987:52:274-92.
DERMis: a computer system for assisting primary-care physicians with dermatological diagnosis G.J.BROOKS,* R.E.ASHTONt AND R.J.PETHYBRIDGEij: Royal Naval Medical Service.'^ Royal Naval Hospital Haslar, ^Institute of Naval Medicine. Gosport. Hampshire P012 2AA. U.K. Accepted for publication 26 |une 1992
Summary
is a computerized skin disease diagnostic prompting system which has been derived from the prospective study of 5205 cases. It has been designed for use by non-dermatologists such as general practitioners. The program produces a list of reasonable diagnoses based on probabilities calculated using Bayes' theorem. Out of 221 precise diagnoses made hy the dermatologist in the clinic, 42 groupings were created to encompass the most common or important diseases encountered in genera! practice. Eour 'remainder' or 'send to specialist' groups were included for the 13% of uncommon conditions. The program, when tested by "one out' analysis on the original cases, placed the correct diagnosis first on 76% of occasions, and within the first three on 9 S% of occasions. In 76 of 1 2 5 cases randomly selected from the data base, a request for diagnostic assistance had been made by the referring general practitioner. It has been estimated that in 54 of these 76 cases the DERMIS system could have provided differential advice with the correct diagnosis appearing at the top of the list. DERMIS
The system has been written in MUMPS and runs on an IBM-compatible desk-top computer. The software allows real time data entry. Arrangements are in hand for performing clinicai trials of the system in general practice. If current performance is maintained, and the response to the system's output is favourable, then DRRMIS might well enhance performance in medical decision making for the benefit of patients, medical services and budget holders. During the last two decades, numerous computer programs have been designed to offer medical advice on a variety of topics.'^ Unfortunately, the creator's enthusiasm has rarely been matched by any compelling medical need for the product. Even when projects have appeared appropriate, many have failed to find widespread use. as there has been no easy way of implementing them. The current emphasis on audit and budget control has led to the installation of computer terminals in many general practitioners' consulting rooms. The hardware is being used to store clinical information and could also run diagnostic prompting software. Skin disease accounts for about 7%i of all consultations in general practice, yet many doctors find the accurate diagnosis of skin disease difficult. This should not be the case, as skin physical signs are visible for all to see. Standard terminology can be used to describe the physical signs, but the interpretation of these is learned by experience and not from standard textbooks. The Correspondence: Surgeon Oinimander K.lvAshton KN, dermatology Deparlmenl, Royal Nava! Elospitul. (losporl, Hampshire F012 2AA. U,K, * Present iidtlress; Fgton Medical Information Services. 77 Back Lane, Horsforth. l^eds LSI H 4Rr, U.K.
614
objective of this study has been to produce a tool which can analyse physical signs and assist with the formulation of a differential diagnosis. The potential value of the prompting system to users has been assessed from a sample of cases by analysing the reason for referral, to identify whether diagnostic advice or treatment is requested.
Methods The subjects included 5203 consecutive patients (new referrals only) seen by a dermatologist (R.E.A.) from 1985 to 1989. Any omissions occurred randomly due to logistic reasons. Data was collected from each case according to a comprehensive list of dermatology symptoms and signs (Fig. 1). Data collection was performed by the consultant, or clinical assistant/general practice trainee. Signs and symptoms were, however, checked by the consultant for consistency. The terms employed had previously been defined and tested through practical use by students and general practitioner trainees. The collection sheet was self-copying and was in two parts. One of these became the patient record. The other was coded to allow data base entry by staff of the Institute of
COMPUTER SYSTEM IN UERMATOLOGICAL DIAGNOSIS
Dale
/
Male/ Female
/
ComDieted by AGE
. yrs/
mths
HOSPITAL No:
ASSOCIATED FEATURES
NAME:
19 SCALP not involved papules scaling hair loss uniform patchy palpable impalpable scarnng extends beyond hair margin remains vulhin hair margin
HISTORY
TIME since onset of tash/lesiori DURATION of rash when present SIZE gradual increase in Ves/Nc PETS: dog/ cat/ other ( 2. HISTORY of alopic eczema other eczema asthma hay fever psoriasis dandruff skin disease
family family family family family family
years/ mtha/ days •fieeks/ days/ , hours PIGMENT any change in Yes/No ITCHING. none mild moderate seveie day night family
20. MOUTH not involved white streaks ulcers
4. RACE, Caucasian negro asian oriental SKIN TYPE, I Hi II IV
21. GENITALS: no! inuQived muolued
5 TREATMENT used previously. steroids improved improved anli-fungals improved
6. MADE WORSE BY: 7, SKIN HYDRATION soap/delergent normal oil/grease greasy water dry sunshirie veiy Ory lichthyosisl exercise/hot tjaths OINTMENTS/CREAMS given by GP DRUGS laKer in last 3 months
DIAGNOSIS: TnEATMENT:
12. COLOUR: normal pink red
pink purple white cream orange yeltow golden light brown dark brown black grey hypo pig men ted hyperpigmwitea other (
_
_J
HOSPITAL No. 13 BORDER/EDGE: definite variable tndislinct raised above centre active edge other 1 \ \ 1 M. SHAPE (definite lesions) round oval annular linear pedunculated irregular other
15 SURFACE FEATURES
normal warty scaly ex u date crust macerated friable atrophic flats shiny white streaks lichen if led excoriated umbiiicated nairy keralin fissures oltier (—
16 VASCULAR FEATURES erythema purpura telangiectasia varicose veins 17 PALPATION ideep< (surface) normal smooth soft uneven firm rough hard hot tender
)
1 f f f
f f f f f
18. SCRATCH TEST no Change mild scale profuse scale wnal
Figure 1. Data collection sheet used during patient interview.
not involved fine pitting coarse pitting onyc ho lysis subungual hyperkeratosis nail thickening loss of nail plate transverse nOges longitubinsi ridges
23 PALMS S SOLES: nol involved vesicles pustules fissures hyperkeratosis scaling in creases scaling in finger webs maceration between toes burrows on fingers/wrists
MYCOLOGY PERFORMED VES/NO BIOPSY PERFORMED YES/NO
OCCUPATION EXAMINATION OF RASH/LESION 8 NUMBER. 11 TYPE: Single iesion macule 2-5 lesions patch 6-20 lesions papule 21 /rash nodule no rash plaque vesicle 9. DISTRIBUTION buNa symmetrical F^stule asymmetrical weal grouped erosion linear ulcer sun exposed crater scar 10 SIZE: comedone anqiooeaerna mrti average other variable to mm
22 NAILS.
RESULT organism
+ve''-ve
615
616
C.j.BROOKS et «i.
Naval Medicine on io a mainframe computer. This information was later transferred to an IBM-compatible desk-top computer for analysis. The diagnosis in each case was made clinically by the dermatologist at the time of the examination, or modified later based on histological or mycological findings. Bayes' formula was used to calculate the likelihood of each disease being present given a set of symptoms and signs. The probability of a disease being present (i.e. the computer's suggestion or diagnosis) can be calculated if we know the probability of the disease occurring in the population (here our data base), the probability of the sign occurring in that disease (measured by collecting the cases) and the probability of the sign occurring in all diseases. In mathematical terms this is expressed as the following formula: If disease D (e.g. psoriasis) and sign S (e.g. scaling) are independent events then:' PiD\S) =
P iS\D) X F (D) PjS)
where P {D\S) is the probability of a disease (e.g. psoria• . sis) being present given the sign (e.g. scaling) elicited; P(S\D) is the probability of the sign (scaling) occurring in psoriasis: P (D) is the prior probability of the disease •; occurring, i.e. how commonly psoriasis occurs in the data base: P (S) is the probability of the sign occurring at ali. By using this calculation for each disease in turn, relative likelihoods (as a percentage) of all diseases are produced. The diseases can be placed in order by percentage likelihood to give a differential list. It was apparent from examining cases that combinations of signs occurred within categories (e.g. two colours or two types of lesion are present). Single probability estimates have been formed for these combinations and used in the Bayes' equation instead of the separate 'independent' component estimates. There were 221 separate diagnoses. Where appropriate, diagnoses were collated into groups (Table 1). For example, psoriasis has been used as a general term to describe cases of plaque, guttate. intertriginous. nail and scalp psoriasis. For some other diseases it was appropriate to retain the original diagnosis. For example, malignant melanoma is represented in both nodular and superficial spreading forms, as the clinical features are so different. Rare diseases have been represented in four remainder' or 'refer to specialist' groups.
Tdbic 1. Accuracy of assignment of correct diagnosis to top of differential list in 'one-out' analysis of 5203 cases in data base
Group title 1. Alopecia areata' 2. Acnet 3. Basal cell carcinoniLi 4. Superficial BCC 5. Bowen's disease 6. Chondrodermatitis 7. Cyst epidermoid 8. Dermatotibroma 9. Eczema^ 10. Foiliculitis 11. Granuloma anniilare 12. Herpes simplex 11. Insect bite§ 14. Keratoacanthoma 15. Lentigo 16. Lichen planusi 17. Malignant melanoma 18. Malig, melan, superficial 19. Milia 20. Molluscum contagiosum 21. Naevus** 22. Naevus spider 23. Pyogenic granuloma 24. Pityriasis rosea 2 5. Psoriasis palm/plantar 2fi. Psoriasis*! 27. Pityriasis versicolor 28. Rosacea 29, Scabies 30. Seborrhoeic wart 31. Solar keratosis$t 32. Skin tags 3 3. Squamous cell carcinoma 34. Tinea§§ 35. Urticaria 56. Vitiligo !7. VVartsiH 38. Verruca*" 59. Other single lesion 40. Other multiple lesions 41. Other rash 42. Other no rash or lesion
No. cases data base 40
253 288 24 27 31 59 90
1124 23
47 23 31 25 38 64 21 51 23
47 396 36 47 19 10 414 6S 60 52 266 269 33 35 82 69
24 277 50 217 190 253 7
No, (%) of times top differential 39(98) 232 (92) 201 (70)" 2 3 (96) 22 (Hl)^ 28(90) 51(86) 77(86) 847(75) 19 I8J) 42 190) 22 (96) 24(77) 22 (88) 28(74)
54(84) 18 (86)' 49 (96)'' 20(87) 45(96) 250(63) 36(100) 46(98) 18 (95) 10(100) 379(91)
54(79) 53 (88) 51 (98) 183 ((.9) 187(69) 32(97) 26(74)'' 64(78) 65(94) 23(96) 222 (80) 49(98) 109 (50) 101 (53) 104 (41) 6 (861
* Includes cases of alopecia totalis. t Includes cases of acne excoriee, perioral dermatitis. % Includes cases of eczema; atopic, contact allergic, discoid, foot and hand. intertriginous. impetiginized. pompholyx, craquelc, seborrhoeic. varicose, papular, acute, irritant; lichen simplex chronicus. juvenile plantar dermalosis, ^ Includes cases of papular urticaria, •! Includes cases of lichen planus hypertrophic, " Includes cases of naevus; [unctional. blue, compound, halo, hairy, pigmented. intradermat. linear epidermal, warty epidermal, •|"t Includes cases of plaque, guttate, intertriginous, nail, scalp, :^if Includes cases of cutaneous horn, §§ Includes cases of tinea: corporis. cruris, incognito, manuum, pedis, unguium. ' ' Includes cases of common, filiform, genital, plane. """ Includes cases of corn, exostosis, ^ Not included 49 cases diagnosed as other malignancy, ^ Not included four cases diagnosed as other malignancy. '" Excluding one case diagnosed as superficial spreading melanoma. '' Not included one case diagnosed as other malignancy.
COMPUTER SYSTEM IN DERMATOLOGICAL DIAGNOSIS
A 'one-out' method was adopted^ to test the accuracy of the computer modeL Each case in turn was excluded from the data base, and the frequency of symptoms and signs being present for each diagnosis was calculated using the remaining cases. An estimate of the likelihood of occurrence of each disease with the symptoms and signs of the excluded case was calculated by multiplication of the appropriate frequency and prior probability information. The diseases were then ranked according to their likelihood scores to give a differential list. Assessment of predictive accuracy using the algorithm involved: 1. the original 221 diagnoses; 2. 42 diagnostic groups created using the diagnosis reached in 1: 3. combination of diagnoses into 42 groups before calculation using the algorithm; 4. as for 3. using a reduced data sheet. Where the predicted diagnosis has disagreed with the actual diagnosis, the computer's differential diagnosis was investigated in order to assess where the correct diagnosis appeared in the differential list. The clinical records from a random sample of 125 collected cases were scrutinized to determine the reason for referral, in order to determine when a diagnostic advice system might be of use.
Results Within the data base the number of cases allocated to each of 221 specific diagnoses varied considerably, from the hundreds of 'eczema' cases to one or two representatives of extremely rare diseases. The diagnosis appearing at the top of the differential was correct in 60% of cases when all 221 diagnoses were used. However, if a correct diagnosis was more loosely confined to one of the 42 major diagnostic groups, the overall predictive value increased to 76%. In a further test, where the 42 groups were formed prior to use of the predictive algorithm, the diagnostic accuracy was again found to be 76%, but the pattern of success and failure of individual cases varied between the two methods. On 95% of occasions the correct diagnosis appeared in the top three of the differential list. The original data sheet (Fig. 1) allowed 270 options for description of cases. The list was reviewed in the light of clinical experience and analysis of the predictive value of each item. It was found that the list could be reduced to 11 5 items without loss of predictive accuracy.
617
Table 2. Most common errors: confusion between specific disease groups
Actual diagnosis Eczema Send to specialist Fx:xema Eczema Naevus Eczema Eczema Naevus Basal cell carcinoma Naevus Send to specialist Seborrhoeic wart Acne
Top of differential
Number of cases (%)
Psoriasis Eczema Scabies Tinea Seborrhoeic wart Pityriiisis rosea Send to specialist Superlicial melanoma Squamous cell ctircinomii Skin tug Pityriasis versicolor N'aevus Rosticea
84(7) 70(3) 63(6) 38 (J) 21 (5) 20(2) 20(2) 1815) IS (6) 18(5) 17(7) 15(6) 15(6)
The accuracy with which the diagnosis was predicted varied between the groups (Table 1). Eor example. 847 (75%) of the 1124 cases of eczema were correctly identified by the program, compared with 49 (96%) of the 51 cases of superficial spreading melanoma. Cases of rarer diseases, e.g. pemphigoid. mycosis fungoides, which were assigned to "remainder' groups, made up 13% of the total database. These groups had the highest failure rates, i.e. 41-53% (Table 1). The system output was studied in order to determine any patterns of success and failure. Certain diseases tended to be confused with one another (see Table 2). For example, the commonest type of failure occurred in 84 cases of eczema where the system placed psoriasis above eczema in the differential list. In 49 cases of the papular/ nodular presentation of basal cell carcinoma, a malignant tumour other than basal cell carcinoma (or 'remainder' group in four cases) was selected by the system. The management of these cases is the same— either excision biopsy or referral to a specialist. A sample of 12 5 randomly selected case records were scrutinized in detail, in order to determine the reason for referral and the outcome following the clinic visit. The results are presented in Table 3. In 76 (61%,) cases (summarized in Table 4) the family practitioner requested assistance with diagnosis. The final diagnosis by the dermatologist matched one of the 38 main DKRMIS disease group headings in 71 (9 3%) of these cases. The number of cases in which the DERMIS system placed the correct diagnosis at the top of its differential list was 54 (71%).
618
G.J,BROOKS et al.
Table J. A random sample of 12 5 cases referred to dermatology clinics by primary care physicians: reason for referral vs. outcome
Outcome
Keason for referral Diagnosis and management (a) diagnosis unknown (bi ,- malignant (c) ? infection Diagnosis correct, for second opinion Management only Re-referral, further treatment Removal/biopsy
Malignant tumour
Benign lesion(s)
Rash no infection
5* 6* — — _
19*
21*
— 4* 4
3* 3* 2* 2 15
Infection/ infestation
7* — — 1' 3 9
10
Indicates the cases shown in Table 4.
Table 4. Breakdown by diagnosis of 76 randomly selected cases referred by primary care physicians to dermatology clinics for diagnosis and management
Final diagnosis Basal cell carcinoma Bowen's disease Discoid lupus Eczema Erysipelas In.sect bites Lentigo Lichen planus Naevus Rare tumours Pyogenic granuloma Pityriasis rosea Psoriasis Pityriasis versicolor Rosacea Scabies Solar keratosis Squamous cell carcinoma Tinea Urticaria Vasculitis Viral wart Verruca
Number of cases 9 1 1 15 1 2 2 2 12 2 1 1 4 1 1 2 7 1 5 2 1 1 2
Discussion Prospective collection of case data is one of the techniques commonly used to acquire knowledge for computer-based medical prompting and expert systems. The method relies on the clinician defining the diagnosis and ensuring that all relevant information is collected. Case collection is a lengthy task which needs to he carefully planned and conscientiously carried out hefore any
predictions can be made. However, the effort frequently brings rewards as previous gaps in knowledge are tilled and uncertain boundaries between diseases are clarified. Haberman et al.'' chose not to use case collection in creating a diagnostic system for dermatologists. Instead, they asked dermatologists to estimate the importance of varions clinical features in diseases. These estimates were then used as the basis of weightings and rules which formed the diagnostic algorithm. The system which they produced has been tested, with mixed results. In the later stages of development case information was fed back into the system in order to adjust the original weightings. Other methods which have been used in attempts to create diagnostic advice systems for dermatology have been described by Stoecker.'' The statistical approach to pattern matching adopted in this study has been popular for many years. The requirements, advantages and disadvantages of Bayesian methodology are well known.' There are several sources of variation which are likely to determine the performance of the algorithm in practical use. Disease prevalence can alter with geographic area. If DERMIS was to be tested in general practice, appropriate prevalence rates would have to be used to prime the system. The presentation of a disease can vary between patients, and the content of data collected from the same patient can vary between doctors. In this study the cases were all collected by one of the authors (R.E.A.), and occasionally by GP trainees or medical students. There was no allowance for inter-observer variation. A further weakness in this study has been that the dermatologist (R.E.A,) could have introduced substantial bias by subconsciously fitting symptoms and signs to diseases rather than faithfully recording new case details. This could lead to increased accuracy of the
COMPUTER SYSTEM IN DERMATOLOGICAL DIAGNOSIS
program when tested against itself. Testing the program against other dermatologists would help reveal any bias. Data has been collected mainly from white Caucasian patients. Rashes in patients with pigmented skin may look different, but although the colour of the rash may be different, other features such as surface changes and border characteristics should be the same. The accuracy of DERMIS using the original 221 diagnoses is comparable with similar systems in other subjects.' Many of the errors were eliminated by combining different forms of the same disease (i.e. combining ail the eczemas), to produce a predictedaccuracy of 95%for the first three in the differential list. It has not been possible to eliminate all DHKMIS system errors by the combination of subgroups. The clinical appearance of eczema, for example, can be identical to that of tinea infection, and that of a solar keratosis visually similar to a basal cell carcinoma. Combining two such diseases into the same group is of no immediate value to any system user, as the treatments are different. In these cases a dermatologist would investigate further to establish the diagnosis. The prompting system could be set to recommend further investigation when two diseases are known to cause problems. The group combinations of Table 1 were tested in two ways. Combining these before application of the algorithm led to a dilution effect of the important diagnostic features of the subgroups. Whereas, combination of the groups following application of the algorithm led to under-diagnosis of the small subgroups. The outcome of the two types of combination was virtually the same, with an overall diagnostic accuracy of 76%. The accuracy with which the correct diagnosis was placed at the top of the differential list by the system appeared to be related both to group size, and variability of presentation. Increasing the number of cases of a disease in the data base will improve the accuracy with which the disease is identified. However, if there are few distinctive features, or the presentation varies from case to case, then performance will be degraded. In such cases dermatologists often have a distinct advantage, as their expertise enables them to sift the available information
619
for important clues which may have variable weighting according to circumstance, DtRMis was not designed to replace dermatologists, but to offer general practitioners assistance with the formulation of a reasonable differential diagnosis. The system was able to place the correct diagnosis in the top three of its differential output on 9 5% of occasions. It also demonstrated a high accuracy in identification of malignancy, even if the correct tumour was not always placed at the top of the list. It is important to recognize that computer-based diagnostic prompting or expert systems are unlikely ever to recommend the correct diagnosis on all occasions. When a random sample of 12S cases from the data base was reviewed it was found that 76 (61%) had been referred for initial diagnosis or a second opinion. The system offered a differential listing with the correct diagnosis at the top in 54 of these 76 cases. It is not known what influence this type of information would have on referral patterns. Currently, patients may have to wait many months before being seen in a dermatology clinic. Use of a system such as DKRMis might encourage general practitioners to treat more cases themselves and to make more appropriate referrals.
References 1 KunzJC, Shortliflc I'.W, Buchanan BG. rt'igeiibaum RA. Computerassisted decision making in medicine. / Med I'hilos I yS4:9:l iS-fiO, 2 Miller PL. The evaluation of artificial intelligence sy.stcms in medicine. Comput Methods Protframs Biomed I4Sf):22:S-l 1. 1 Cood IJ. The Estimation of Probahilitifs: An Essan on Modern Hayesian Melhods. Cambridge, Miiss: MIT Press, 19f»'5. 4 Ingram D, Bloch RF, Medical stalistics: advanced techniques and computation. In: Mathemat'ual Models in Medicine. Chichester: I. Wiley and Sons Ltd., 1984. 5 tlaberman HI'". Norwich KH. Diehl DL cl al. DIAC: A computerassisted dermati)logic diagnostic system clinical expericnct- and insight. / Am Acad Dermatol iySS;12:l i 2 - 4 i . (l Stoeker WV. Computer-aided diagnosis of dermatologic disorders. Dermalol Clin 198fi:4:fi()7-25. 7 Carroll B. ArtiHcial intelligence. Expert systems for clinical diagnosis: are they worth the effort.' Hehav Sci 1987:52:274-92.
