Mutation Research, 272 (1992) 59-71 © 1992 Elsevier Science Publishers B.V. All rights reserved 0165-1161/92/$05.00
59
MUTENV 08831
Testing by artificial intelligence: Computational alternatives to the determination of mutagenicity Gilles Klopman and Herbert S. Rosenkranz b a
a Department of Chemistry, Case Western Reserve University, Cleveland, OH 44106, USA and # Department of Environmental and Occupational Health, University of Pittsburgh, Pittsburgh, PA 15261, USA (Received 19 February 1991) (Revision received 17 January 1992) (Accepted 4 March 1992)
Keywords: Artificial intelligence; Intelligence, artificial; Computational Structure-activity methods; CASE; MULTICASE; CASE/GI
alternatives,
determination
of mutagenicity;
Summary In order to develop methods for evaluating the predictive performance of computer-driven structureactivity methods (SAR) as well as to determine the limits of predictivity, we investigated the behavior of two Salmonella mutagenicity data bases: (a) a subset from the Genetox Program and (b) one from the U.S. National Toxicology Program (NTP). For molecules common to the two data bases, the experimental concordance was 76% when "marginals" were included and 81% when they were excluded. Three SAR methods were evaluated: CASE, MULTICASE and C A S E / G r a p h Indices (CASE/GI). The programs "learned" the Genetox data base and used it to predict NTP molecules that were not present in the Genetox compilation. The concordances were 72, 80 and 47% respectively. Obviously, the MULTICASE version is superior and approaches the 85% interlaboratory variability observed for the Salmonella mutagenicity assays when the latter was carried out under carefully controlled conditions.
The development of computer driven knowledge-based structure-activity techniques has raised considerable expectations among some researchers as well as serious doubts among others regarding their potential to predict "in video" the pharmacological a n d / o r toxicological properties of organic compounds. These methods are intended to be used in situations where a certain
Correspondence, Prof. H.S. Rosenkranz, Department of Environmental and Occupational Health, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA.
type of biological activity is observed for a series of "diverse" molecules and where traditional QSAR methods, which require congenericity, are not applicable. The diversity of the structures of the molecules producing the activity may be due to multiple mechanisms, or to an "obscure" commonality between the structures of the active compounds. The purpose of this communication is to investigate the usefulness of three such techniques, the CASE program, the C A S E / G I program which is a variation of the CASE program based on graph indices (Raychaudhury and Klopman, 1990), and MULTICASE (Klopman, 1992), a new artificial
60
intelligence program. All three programs were developed in our laboratory and used extensively to study the potential genotoxic activity of chemicals The CASE method, which has been described on a number of occasions (Klopman et al., 1990; Rosenkranz and Klopman, 1989, 1990a), employs an artificial intelligence algorithm for the evaluation of the relevance of chemical substructures. Recently, the program's predictions of the activity of antibacterial agents have been compared with those of human experts (Klopman and Kolossvary, 1990). The second of these programs is based on graph theoretical descriptors. We have used it, with a somewhat different underlying graph index, to study a number of biological activities. Finally, the third method, MULTICASE, is a new methodology which, as CASE, uses structural fragments as descriptors, but is based on a totally different algorithm, incorporating extensive hierarchy in the selection of its descriptors. The method has already been used to study a number of toxicological data bases and shows considerable promise. We have now compiled sufficient data to feel confident that a reliable evaluation of the usefulness of these methods can be made. In view of the fact that the largest data bases available to us consists of results of the Salmonella mutagenicity assay and because that assay occupies a pivotal role in defining a "genotoxic carcinogen" (Ashby and Tennant, 1988, 1991; Zeiger et al., 1990), we conducted a study to establish the reasonable expectations one might have about the results of this type of analysis and compared'them to experimental results. However it should be stressed that the conclusions drawn from the present analyses are not restricted to the Salmonella mutagenicity assay, but are applicable to the results of biological activities analyzed with the CASE program. We selected two data bases for our study; one consisting of 808 organic molecules tested for mutagenic activity by the Environmental Protection Agency under the G E N E T O X program (Klopman et al., 1990) and another of 262 chemicals tested under the aegis of the U.S. National Toxicology Program (NTP) (Rosenkranz and Klopman, 1990a). These two data bases consist of largely diverse compounds and are thus not
amenable to analysis by conventional QSAR techniques. Both databases list the results of Salmonella typhimurium mutagenicity assays but differ with respect to criteria used for the acceptability of the results. The G E N E T O X data arc derived from a consensus analysis of published data (Kier et al., 1986) while the NTP results were all obtained from a limited number of laboratories using coded samples of known purity and employing pre-established protocol (Haworth et al., 1983; Zeiger et al., 1988). Overall, the NTP data base is probably superior as the results were obtained under carefully controlled conditions. Each of the data bases was the subject of previous CASE analyses and the results have been published (Klopman et al., 1990; Rosenkranz and Klopman, 1990a). Results and discussion
Analysis of the data bases In the study reported, we used the 808 compounds of the G E N E T O X data base as the learning set for each of the three candidate methodologies and then used the acquired expertise to predict the activity of the compounds listed in the NTP compilation. Each of the methods was found to be capable of learning from the data. Indeed, the entire set of 808 molecules was retrofitted satisfactorily by each of the three methods, with an 86.2% accuracy for CASE, 98.4% for MULTICASE and 78.4% accuracy for C A S E / G I when molecules for which "inconclusive" experimental results were obtained were excluded from consideration. When, however, the latter were included, the accuracy was 76.5% for CASE and MULTICASE and 55% for C A S E / G I (Table 1). It is to be noted that in the G E N E T O X Salmonella data base the majority of the chemicals listed as "inconclusive" were not marginally active but contained protocol defects (Klopman et al., 1990; Kier et al., 1986) The NTP data base consisted of the 262 compounds Of these, 117 were also present in the G E N E T O X data base. They are listed in Table 2, together with their reported activity in both the G E N E T O X and the NTP compilations. The activities of the 145 chemicals of the NTP compilation that were not included in the G E N E T O X
61 TABLE 1 R E T R O F I T OF G E N E T O X L E A R N I N G SET BY VARIOUS M E T H O D O L O G I E S
(a) CASE Methodology Activ.
Inact.
Total
Pred. + Pred. -
425 25
17 106
442 131
Total
450
123
573
Sens. 94.4%, Spec. 86.2% Observed correct Predictions (OCP) = 92.67% 87 out of the 235 marginal molecules are predicted correctly Observed Correct Predictions (Including marginals) = 76.49%
(b) MUL TICA SE Methodology Activ.
Inact.
Total
Pred. + Pred.-
446 4
5 118
451 122
Total
450
123
573
Sens. 99.1%, Spec. 95.9% Observed correct Predictions (OCP) = 98.43% 54 out of the 235 marginal molecules are predicted correctly Observed correct Predictions (Including marginals) = 76.49%
(c) GRAPH THEORETICAL Methodology Activ.
Inact.
Total
Pred. + Pred. -
427 23
101 22
528 45
Total
450
123
573
Sens. 94.9%, Spec. 17.9% Observed correct Predictions (OCP) = 78.36% 0 out of the 235 marginal molecules are predicted correctly Observed Correct Predictions (Including marginals) = 55.56%
data base, are shown in Table 3. The distribution of the data among activity classes is shown in Table 4. Although there is general agreement between G E N E T O X and NTP results (see Table 2), there are also frequent discrepancies and even a difference in definitions with respect to activity. Thus " m " is defined as "inconclusive" for G E N E T O X and as "marginal" for NTP. The disagreement between experimental values is undesirable and certainly complicates the evaluation. However, the relationship between these two independently derived sets of experimental activities provides a
standard by which the accuracy of the computerbased predicted activities can be compared. Therefore, as the first stage of the analysis, the 117 compounds (test set 1), which are common to both data bases, can be used to evaluate how well one protocol reproduces the results of the other. The results are shown in Table 5. As can be seen, the agreement between the G E N E T O X and NTP classifications is affected considerably by inclusion or omission of the marginal activity compounds (activity "m"). Although this may be due to the inconsistent definition associated with this activity class, it remains that the experimental reproducibility, as measured by the percentage of observed correct predictions (OCP), (defined as the ratio of the sum of the correct predictions of carcinogenicity (TP) or lack thereof (TN) divided by the total number of predictions made) is 87% when "marginals" and "inconclusives" are not considered. An overall concordance (i.e. correct predictions) of 87% thus appears to reflect the current experimental reproducibility and thus the absolute limit one may expect any knowledge-based methodology to reach. In fact analysis of the interlaboratory reproducibility of the Salmonella mutagenicity assay as performed under the aegis of the NTP indicates a reproducibility of approximately 85% (Piegorsch and Zeiger, 1991). It is therefore unrealistic to expect that any structure-activity method using Salmonella data as its training set can predict the mutagenic activity of unknown molecules with a concordance of better than 76%, unless the data set contains no marginal compounds, in which case the concordance can, at best, reach 87%.
Structure-actiuity analyses The three methods were next used to predict the " N T P " activity of the 117 compound data set using the knowledge acquired from the G E N E T O X learning set. The G E N E T O X , NTP and the predicted activities are listed in Table 2. Columns 3 and 4 list the experimental results as reported by G E N E T O X and NTP, respectively. Columns 5 and 6 summarize the CASE predictions, 7 and 8 the M U L T I C A S E predictions and column 9 the C A S E / G I predictions. Columns 6 and 8 denote the probabilities that the molecule is active ac-
62 TABLE
2
MOLECULES
EXISTING
IN
BOTH
Molecules
THE
NTP
Exp.
AND
THE
Activity
GENETOX
COMPILATIONS
CASE
Graphics
MULTICASE
Gr. Ind. GEN (Cytoxan)
NTP
CASE
Prob.
M.C.
Prob.
1
Cyclophosphamide
+
+
+
98.7
+
87.9
+
2
Benzo[a]pyrene
+
+
+
93.5
+
77.3
+
3
Phenylbutazone
m
-
-
0.0
0.0
+
4
N-Hydroxy-2-acetylaminoflu
+
+
+
97.8
+
92.8
+
5
N-2-Acetylaminofluorene
+
+
+
91.8
+
87.4
+
6
Diethylnitrosamine
+
-
+
86.0
+
87.3
+
7
Fenthion
m
+
-
84.5
-
0.0
+
8
Ethyl
m
+
+
94.0
+
69.8
-
9
3-Methylcholanthrene
+
+
+
95.7
+
77.3
+
Diethylstilbestrol
-
-
-
40.0
-
0.0
+
+
+
+
77.0
+
77.3
+
+
+
+
99.6
+
95.0
+
10
parathion
11
Benz[a]anthracene
12
4-Nitroquinoline-
13
/3-Propiolactone
+
+
+
87.0
+
87.0
+
14
7,12-Dimethylbenz[a]anthra
+
+
+
70.0
+
77.3
+
15
Caffeine
-
-
94.7
-
0.0
+
16
5-(4-Chlorophenyl)-6-ethyl
m
-
-
0.0
-
0.0
+
17
p-Dimethylaminoazobenzene
+
-
-
67.0
+
80.0
+
18
Methylhydrazine
+
-
+
83.0
+
83.3
+
19
Acetamide
-
-
-
0.0
-
0.0
-
20
Dieldrin
-
-
-
0.0
0.0
-
21
Amitrole
-
-
-
0.0
0.0
-
22
Ethyl
+
+
+
87.0
+
86.7
+
23
Aniline
-
-
0.0
-
0.0
-
24
Thioacetamide
-
-
-
0.0
-
0.0
+
25
Thiourea
-
-
-
0.0
-
0.0
-
26
Dimethylnitrosamine
+
+
+
89.0
+
87.3
+
27
Methyl
+
+
+
87.0
+
86.7
+
28
Uracil
+
+
+
99.3
+
96.1
+
29
Nitrofurantoin
+
+
+
100.0
+
89.7
+
30
Hexachloroethane
-
-
-
0.0
-
0.0
-
31
N-Methyl-N
+
+
+
89.0
+
87.3
+
32
Methoxychlor
-
-
70.0
-
0.0
-
33
p,p'-DDE
-
-
-
11.0
-
0.(l
-
34
Vinylidene
+
-
-
80.0
+
66.7
+
35
Propylenimine
+
+
+
75.0
+
66.7
+
1-oxide
(3-amino-l,2,4-tr
methanesulfonate
(EM
(DMN)
methanesulfonate
(M
mustard
'-nitro-N-nitros
chloride
36
Propylene
+
+
+
86.0
+
91.4
+
37
Trichloroethylene
chloride
m
-
-
83.0
-
0.0
+
38
Chloroacetic
-
-
83.3
-
54.7
-
39
Dimethylcarbamoyl
+
+
-
73.0
+
66.7
+
40
Saccharin
-
-
+
77.9
-
46.7
-
41
Pentachloronitrobenzene
m
-
+
74.1
-
0.0
+
42
Phenanthrene
-
-
+
77.0
+
77.3
+
43
/3-Naphthoquinoline
+
+
+
93.5
+
86.9
+
44
Dipheny~itrosamine
-
-
-
76.7
-
52.3
+
45
Pentachlorophenol
-
-
-
30.0
-
0.0
+
46
2,4,6-Trichlorophenol
-
-
-
0.0
-
0.0
-
47
Guaiacol
m
-
-
70.0
-
0.0
+
48
2-Aminobiphenyl
+
+
+
99.4
+
91.3
+
49
9-Aminoacridine
+
+
+
96.4
+
86.9
+
5O
Naphthalene
-
-
-
57.1
-
46.7
-
acid chloride
63 TABLE 2 (continued) Molecules
E x p . Activity
CASE
MULTICASE
GEN
NTP
CASE
Prob.
M.C.
Prob.
Graphics G r . Ind.
51 52 53 54 55
Quinoline o-Nitroanisole 2-Naphthylamine 4 - A m i n o - 1,1 ' - b i p h e n y l Benzidine
+ m + + +
+ + + + +
+ + + + -
93.5 94.0 90.1 98.4 87.0
+ + + + +
86.9 69.8 77.3 75.0 75.0
+ + + + +
56 57 58 59 60
Sulfallate 2,4-Diaminotoluene N,N'-Ethylenethiourea Furfural 4-Nitro-o-phenylenediamine
+ + + + +
+ + + +
+ + + +
95.1 99.5 83.0 99.8 99.9
+ + + + +
66.7 95.7 66.7 80.0 95.4
+ + + + +
61 62 63 64 65
5-Nitro-o-anisidine p-Nitrophenol Styrene N-Nitrosopiperidine 4,4'-Methylenebis(2-chloro
+ + + +
+ + +
+ + +
99.9 76.7 67.0 86.0 95.0
+ + + +
95.4 53.7 66.7 87.3 91.3
+ + + +
66 67 68 69 70
N-Phenyl-p-phenylenediamin
-
-
-
29.0
-
0.0
+
trans-
Stilbene
-
-
-
0.0
-
0.0
-
Azobenzene Acetaminophen 1,2-Epoxybutane
+ m +
+
-
67.0 0.0 94.0
+
80.0
+
-
0.0
+
+
87.0
+
71 72 73 74 75
1,2-Dibromoethane Ethylene chlorohydrin Resorcinol Bromobenzene C~clohexane
Ill +
86.0 94.0 0.0 0.0 0.0
+ +
85.7 91.4
+
-
0.0
-
-
0.0
+
-
0.0
-
76 77 78 79 8O
3,3'-Dimethyoxybenzidine Tolidine Anthracene Malathion Maleic hydrazine
+ +
99.2 99.2 77.0 0.0 0.0
+
+ +
91.3 91.3 77.3
+ +
-
0.0
+
-
0.0
+
81 82 83 84 85
Tris(2,3-dibromopropyl)pho Butylated hydroxytoluene Pyrene E D T A t r i s o d i u m salt Monuron
+
86.0 0.0 81.0 0.0 73.0
+
87.0
+
-
0.0
+
+
77.3
+
-
0.0
-
-
0.0
+
86 87 88 89 90
2 - B r o m o - 2 - c h l o r o - 1,1,1-tri Benzo[e]pyrene Methyl parathion Chloral hydrate Diazinon
0.0 77.0 94.0 75.0 0.0
+ + + +
80.0 77.3 69.8 73.7
+ + +
-
0.0
+
91 92 93 94 95
Natulan 1,8,9-Trihydroxyanthracene 1 -Naphthylisothiocyanate meso-l,2 :2,4-Diepoxybutane Pararosaniline
0.0 62.2 53.5 94.0 11.0
-
0.0
+
-
46.7
+
-
25.0
+
+
87.0
-
-
0.0
+
96 97 98 99 100
Methylazoxymethanol acetat Methyl carbamate 5-Nitroacenapthene 2-Nitrofluorene 2-Anthramine
0.0 0.0 99.7 98.7 91.8
-
0.0
-
0.0
+ + +
95.0 94.1 77.3
+ + +
+ + +
+ +
in m
+ 111
+
+ + +
+ +
+
m
+ + +
+ + +
-
-
-
+ +
+ + +
64 TABLE 2 (continued) Molecules
Exp. Activity
CASE
GEN
NTP
CASE
MULTICASE Prob.
M.C.
Prob.
Graphics Gr. Ind.
101 102
Glycidaldehyde Proflavin h y d r o c h l o r i d e
+ +
+ +
+ +
94.0 97.5
+ +
87.0 88.6
+ +
103 104 105
7,9-Dimethylbenz(c)acridin Trifluralin C.I. A c i d red. 14 ( A z o rubi
+ +
+ + -
+ +
99.2 76.7 70.0
+ +
92.8 0.0 50.0
+ + +
106
2-(2-Furanyl)-3-(5-nitro-2
+
+
+
100.0
+
89.7
+
107 108 109
N-Hydroxy-4-aminoquinoline 2-Nitro-p-phenylenediamine 7-Bromomethyl-I2-methylben
+ + +
+ + +
+ + +
94.0 99.9 93.5
+ + +
88.2 94.4 87.8
+ + +
110
Xearalenone
m
-
-
0.0
-
0.0
+
111 112 113
N-(4-(5-Nitro-2-furyl)-2-t P o l y e t h y l e n e glycol Azoxymethane
+ +
.+ -
+ -
100.0 0.0 67.0
+ -
89.7 0.0 0.0
+ +
114 115
N-4-Fluorenylacetamide Hydrochlorothiazide
+ m
+ m
+ -
77.0 13.6
+ -
77.3 0.0
+ +
116 117
E v a n s Blue, C.I. direct bl Crystal violet
m m
m m
-
32.0 11.0
-
0.0 0.0
+ +
+ , - , m indicates activity, inactivity a n d m a r g i n a l activity, respectively. Prob is c a l c u l a t e d p r o b a b i l i t y o f activity. " E x p t . A c t i v i t y " r e f e r s to the e x p e r i m e n t a l results r e p o r t e d by the G e n e T o x S a l m o n e l l a M u t a g e n i c i t y W o r k i n g G r o u p ( G E N ) or g e n e r a t e d u n d e r the aegis of the U.S. N a t i o n a l Toxicology P r o g r a m ( N T P ) .
cording to CASE and MULTICASE, respectively. Table 6 summarizes the results of the analyses for the three structure-activity methods and shows the associated statistical analyses. As expected, the overall accuracy of the predictions is quite similar to that obtained using the experimental G E N E T O X results to predict NTP resuits. The accuracy (OCP) is 86% (84% when including the "marginals/inconclusives") for CASE; it rises to 89% (86%) for M U L T I C A S E but only reaches 65% (65%) for C A S E / G I . It was surprising to find that MULTICASE, trained with G E N E T O X , predicted NTP activities better than G E N E T O X itself (88.6% vs. 86.87%). This may be due to the fact that inconsistencies are automatically weeded out when M U L T I C A S E tries to rationalize the "admittedly inferior" G E N E T O X data, yielding a better correlation with the NTP results. This is similar to what would be expected when a straight line correlating experimental data points is used to predict the outcome of further experiments.
Table 6 also shows the sensitivity and specificity of the three methods, as well as the expected percentage of correct predictions (ECP), the Phi 2 and the Chi 2 values for the correlation. As was shown previously (Klopman and Rosenkranz, 1991a), ECP is defined as the percent "correct" predictions to be expected when a data base (e.g. NTP) containing a fraction X of active molecules is predicted by a method producing a fraction Y of actives (e.g. CASE). ECP can be calculated by the formula: ECP = (1 + 2 X Y - X -
Y ) * 100
(1)
where X and Y values are between 0 and 1. ( X = 61/114 for NTP, Y = 63/114 for CASE) The probability that the observed concordance, i.e. percentage of correct predictions,
65 TABLE
3
NTP MOLECULES
UNKNOWN
TO THE
Molecules
GENETOX
Exp. Activity NTP
LEARNING
SET
CASE
MULTICASE
Graphics
CASE
Prob.
M.C.
Prob.
Gr. Ind.
1
Reserpine
-
+
88.7
-
67.0
2
Acetylsalicylic acid
-
-
70.0
-
0.0
+
3
Piperonyl butoxide
-
-
0.0
-
0.0
+
4
op-DDD
(Mitotane)
-
-
0.0
-
0.0
+
5
Myleran (Busulfan)
+
+
87.0
+
86.7
+
+
88.7
-
0.0
+
33.0
-
0.0
+
0.0
-
0.0
+
6
Coumaphos
w -
7
Allyl isothiocyanate (must
w +
8
Urea
w -
9
Phenytoin
-
+
w -
-
3.0
-
41.2
+
10
1,2-Propanediol
-
-
0.0
-
0.0
+
11
Ethynylestradiol
-
+
77.0
-
0.0
+
12
Chlordane
-
-
0.0
-
0.0
+
13
Progesterone
-
-
0.0
-
0.0
+
14
y-Lindane
-
-
0.0
-
0.0
-
15
Dimethoate
+
-
0.0
-
0.0
+
16
Phenacetin
-
-
70.0
-
0.0
+
17
Tolbutamide
-
-
26.5
-
0.0
+
18
Colchicine
-
-
70.0
+
66.7
+
19
Choline chloride
-
-
67.0
-
0.0
+
20
Dimethylformamide
w -
-
0.0
-
0.0
-
21
Hexachlorophene
-
-
30.0
-
0.0
+
22
2,4-Dinitrofluorobenzene
+
+
87.0
+
89.7
+
23
1,1,1-Trichloroethane
w -
-
0.0
-
0.0
-
24
p,p'-DDD
-
-
0.0
-
0.0
+
25
p,p'-Ethyl-DDD( p e r t h a n e )
+
0.0
-
0.0
+
+
26
1-Bromoethane
w -
+
86.0
+
85.7
27
Iodoform
w +
-
0.0
-
0.0
-
28
Chloropicrin
w +
-
0.0
-
0.0
+
29
Phenolphthalein
-
+
98.8
-
55.1
+
30
Bromdiethylacetylurea (car
w -
0.0
-
0.0
+
31
Dioxathion
w +
-
0.0
-
0.0
+
32
1,1,2,2-Tetrachloroethane
-
-
0.0
-
0.0
+
33
Dapsone
-
-
0.0
+
60.0
+
34
1-Nitronaphthalene
+
+
99.6
+
95.0
+
35
Cinnamyl anthranilate
-
+
95.0
+
91.3
+
36
Picric acid
+
+
87.0
+
62.5
-
37
C.I. Developer 1
-
-
0.0
-
0.0
+
38
Michler's Ketone
+
-
0.0
-
0.0
+
39
Coumarin
+
+
77.0
+
77.3
+
40
3,3'-Dimethoxy-4,4'-biphen
+
0.0
-
0.0
+
41
6-Methylcoumarin
-
+
77.0
+
81.8
+
42
Biphenyl
-
+
90.0
-
0.0
+
43
Phenothiazifie
-
+
77.0
+
77.3
+
44
6-Nitrobenzimidazole
+
+
99.0
+
69.8
+
45
1,2,3-Benzotriazole
w +
+
77.0
+
77.3
+
46
3-Chloro-p-toluidine
-
-
91.7
-
0.0
+
47
5-Chloro-o-toluidine
-
+
99.1
+
91.3
+
48
4-Chloro-o-phenylenediamin
+
+
99.1
+
91.3
+
49
y-Butyrolactone
-
-
0.0
-
0.0
+
50
Eugenol
-
-
0.0
-
0.0
+
66
TABLE
3 (continued) Molecules
Exp. Activity NTP
CASE
MULTICASE
CASE
Prob.
Graphics
M.C.
Prob.
+
95.4
Gr. Ind.
51
5-Nitro-o-toluidine
+
+
99.9
52
Benzyl alcohol
-
-
0.0
-
0.0
+ +
53
Anilazine
-
-
29.0
-
0.0
+
54
Bis(p-(dimethylamino)pheny
-
-
0.0
-
0.0
+
55
Dicyclohexylamine
-
+
80.0
-
0.0
+
56
m-Cresidine
+
57
1-Phenyl-2-thiourea
58
p-Benzoquinone
-
+
97.8
+
91.3
w -
-
29.0
+
66.7
+
+
-
0.0
-
0.0
+
dioxime
59
1,3-Diethyl-2-thiourea
-
-
0.0
+
66.7
+
60
p-Chloroaniline
+
-
0.0
+
0.0
+
61
p-Benzoquinone
-
-
0.0
-
62
1- E p o x y e t h y l - 3 , 4 - e p o x y c y c l
+
+
94.0
+
68.5
63
Bis(2-chloro-l-methylethyl
+
+
86.0
+
91.4
64
1,3,5-Trichlorobenzene
-
-
0.0
-
0.0
-
65
Melamine
-
-
0.0
-
0.0
+
66
Toluene
-
-
0.0
-
0.0
+
67
Cyclohexylamine
-
-
0.0
-
0.0
+
68
Furfuran
-
+
79.0
-
0.0
+
69
Pyridine
-
-
0.0
-
0.0
+
70
Morpholine
-
-
83.0
-
0.0
+
71
Sym-Dichloromethyle t h e r
+
+
94.0
+
91.4
+
72
N-Dibutylamine
-
-
83.0
-
0.0
+
0.0 + +
-
-
80.0
+
66.7
+
w -
-
0.0
-
0.0
+
2-Amino-9,10-anthracenedio
+
+
97.3
+
77.3
+
76
4-Amino-2-nit rophenol
+
+
98.7
+
87.5
+
77
Benzophenone
-
-
0.0
-
0.0
+
78
Piperonyl sulfoxide
w -
+
77.0
-
0.0
+
79
p-Cresidine
+
+
99.6
+
91.3
+
80
2,4-Dinitrotoluene
+
+
87.0
+
89.7
+
81
Enheptin
-
+
87.0
-
0.0
+
82
Hydrazine,
w +
-
29.0
-
0.0
+
83
1,4-Dioxane
-
-
0.0
-
0.0
+
84
Hydroxyurea
+
+
87.0
+
87.5
-
85
Tetrachloroethylene
-
-
0.0
-
0.0
-
86
Sulfisoxazole
-
-
6.9
-
0.0
+
87
2-Methyl- 1-nitro-9,10-anth
+
+
99.6
+
88.2
+
88
S u l p h a n b l u e , C . I . a c i d bl
+
-
1.8
-
0.0
+
89 90
Chloramben Benzenamine,
+ +
+
98.3 97.8
+ +
91.3 91.3
+ +
91
Cupferron
w +
+
93.0
+
92.7
+
92
2,4,5-Trimethylaniline
+
+
99.1
+
91.3
+
93
Nithiazide
94
N-Chloroacetyl-n-phenylace
+ +
+ +
87.0 91.0
+ +
58.9 91.4
+ +
95
2,5-Dithiobiurea
w -
-
0.0
+
66.7
+
96 97
Chlordecone Sodium diethyldithiocarbam
-
0.0
-
98
p-Nitrosodiphenylamine
99
Calcium cyanamide
73
1,1-Bis(p-chlorophenyl)-2,
74
Aldicarb
75
100
Lasiocarpine
1,2-diphenyl
w
2-methoxy-hyd
0.0
+
w +
-
+
0.0 62.0
+ +
66.7 70.6
+ +
w + +
-
0.0 87.0
+
0.0 66.7
+ +
67
TABLE 3 (continued) Molecules
101 102
Exp. Activity
CASE
MULTICASE
Graphics
NTP
CASE
Prob.
M.C.
Prob.
G r . Ind.
+ w + -
+ +
94.0 0.0 0.0 80.0 94.6
+ + -
91.4 0.0 0.0 73.7 46.7
+ + + + +
105
Chlorambucil Lithocholic acid 3 - N i t r o p r o p i o n i c acid Chlorobenzilate Sodium fluorescein
106 107
Semicarbazide hydrochlorid
-
+
83.0
+
83.3
-
N-Ethyl-N-nitrosobenzenami
-
+
75.2
-
68.7
+
108
p - N i t r o a n t h r a n i l i c acid
+
+
99.9
+
95.4
+
109 110
p-Phenylenediamine dihydro Fuchsin
+ +
+
0.0 70.1
+
0.0 91.3
+ +
111 112 113 114 115
Dimethoxane Acetohexamide
+ -
+
0.0 22.8
+ -
70.6 0.0
+ +
(2-Chloroethyl)t rimethylam 2-(1-aziridinyl)ethanol 1,3-Bis(cyclohexyl)thioure
+ w -
+ +
94.0 0.0 80.0
+ + +
91.4 66.7 66.7
+ + +
116 117 118 119
N-(1-Naphthyl)ethylenediam Succinic acid 2,2-dimethyl Acetamide Nitrophen
+ w + +
+ + +
91.9 0.0 94.0 94.0
+ + +
77.3 0.0 87.5 89.7
+ + + +
120
Tetrachloroisonaphthonitri
-
-
30.0
-
0.0
+
121 122
Picloram D i r e c t b l a c k 38
+
+
30.0 99.8
+
0.0 95.7
+ +
123 124 125
2 - O x e t h a n o n e , 3,3-dimethylCotoran 1,5-Naphthalenediamine
+ +
+ +
87.0 57.1 77.0
+ + +
87.0 66.7 77.3
+ + +
126 127 128 129 130
Mirex 2,3,5,6-Tetrachloro-4-nitr 1,1,3-Trimethyl-2-thiourea C.I. d i r e c t b l u e 6 2,4-Dimethoxyaniline
w +
+ +
0.0 74.1 0.0 76.8 97.8
+ +
0.0 0.0 66.7 0.0 91.3
+ + + + +
131 132 133 134 135
4-Chloro-o-toluidine hydro S t r e p t o m y c i n sulfate
w +
+ -
95.0 0.0
+ -
91.3 0.0
+ +
Groton BK 4-Chloro-m -phenylenediamin T o l u e n e - 2 , 5 - d i a m i n e sulfat
w + + +
+ +
67.0 99.5 99.5
+ +
0.0 95.7 91.3
+ + +
136 137 138 139 140
2-(Chloromethyl)pyridine h Pyridine 3-(chloromethyl)Phosphamidon N - ( 3 - A m i n o - 4 - e thoxyphenyl) Trisodium nitrolotriacetat
+ + w + + -
+ + + -
98.9 91.0 0.0 99.6 0.0
+ + + + -
96.1 91.4 45.5 91.3 0.0
+ + + + +
141 142 143
p-Anisidine hydrochloride 2-Chloro-l,4-benzenediamin Phenazopyridine hydrochlor
+ + m
+ +
70.0 99.5 86.0
+ +
0.0 91.3 80.0
+ + +
103 104
+ , - , m indicates activity, inactivity a n d m a r g i n a l activity, respectively. Prob is c a l c u l a t e d probability o f activity. W
indicates t h a t the p r o g r a m h a d identified an unsufficiently d o c u m e n t e d f e a t u r e o f the m o l e c u l e , t h e r e b y w a r n i n g that the p r e d i c t i o n is " I n c o n c l u s i v e " .
" E x p t . A c t i v i t y " r e f e r s to the e x p e r i m e n t a l results g e n e r a t e d u n d e r the aegis of t h e U.S. N a t i o n a l Toxicology P r o g r a m ( N T P ) .
68 TABLE 4 SUMMARY OF THE ACTIVITIES IN THE GENETOX AND NTP COMPILATIONS activity + GENETOXlearningset NTP test set 1 test set 2
m
-
total number of compounds
450 235 123 808 61 3 53 117 63 1 61 145
The scale used for biological activity is as follows: + Compound mutagenic m Compound activity inconclusive (GENETOX) Compound activity marginal (NTP) - Compound inactive. O C P , is d u e to c h a n c e is given by s t a n d a r d Chi 2 d i s t r i b u t i o n f u n c t i o n t a b l e s (Bailey, 1971): Chi 2 = N * Phi 2
(2)
w h e r e N is the n u m b e r of m o l e c u l e s in t h e d a t a base. Phi 2 m e a s u r e s the a c c u r a c y o f t h e m e t h o d with r e s p e c t to e x p e c t a t i o n s f r o m r a n d o m assignm e n t o f activity, a n d is e q u a l to 0 if O C P = E C P a n d to 1 for a p e r f e c t fit. TP 2
Phi 2 =
TN 2
+ SC1 * SR1
SC2" SR2
Fp 2 +
FN 2 +
SC1 * S R 2
1
(3)
SC2*SR1
w h e r e T P a r e t r u e positives, T N a r e true n e g a tives, F P a r e false positive, F N a r e false n e g a tives a n d SR1 = T P + F N ;
t i o n e d above, this set of c o m p o u n d s was not i n c l u d e d in the G E N E T O X study and was previously u n s e e n by t h e p r o g r a m . T h e e x p e r i m e n t a l l y o b s e r v e d N T P d a t a a r e shown in c o l u m n 4; the results of the s t r u c t u r e - a c t i v i t y analyses are shown, as b e f o r e , in c o l u m n s 5 to 9. T h e statistical r e s u l t s for C A S E , M U L T I C A S E and C A S E / G I a r e r e p o r t e d in T a b l e s 7, 8 a n d 9 respectively. T w e n t y eight of the m o l e c u l e s were flagged by the p r o g r a m s as b e i n g insufficiently d o c u m e n t e d ; they are m a r k e d with t h e symbol " w " in c o l u m n 3 of T a b l e 3. T h e p r e d i c t i o n s for t h e s e m o l e c u l e s a r e q u e s t i o n a b l e a n d the statistical analyses a r e given for d a t a i n c l u d i n g as well as excluding t h e s e molecules. A c t u a l l y , the o b s e r v e d c o r r e c t p r e d i c t i o n s for these 28 m o l e c u l e s is only 50% for C A S E a n d 46.4% for b o t h M U L T I CASE and CASE/GI. Overall, we find t h a t the p r e d i c t i o n s for activity of m o l e c u l e s u n k n o w n to the t r a i n i n g set are somewhat i n f e r i o r to t h o s e o b t a i n e d by r e t r o f i t t i n g the d a t a (see also K l o p m a n a n d R o s e n k r a n z , 1991b; R o s e n k r a n z et at., 1991). Exc l u d i n g the 28 inconclusive results, we find that C A S E ' s c o n c o r d a n c e d r o p p e d to 72% (it was 86% for the m o l e c u l e s that existed in the training set). M U L T I C A S E is still d o i n g very well; its c o n c o r d a n c e , ( O C P ) is still 80%, (89% b e f o r e ) b u t C A S E / G I ' s p e r f o r m a n c e , 47%, is b a r e l y better t h a n o n e might have e x p e c t e d f r o m a r a n d o m selection of active molecules, i.e. 45%. TABLE 5 CORRELATION BETWEEN GENETOX AND NTP EXPERIMENTAL DETERMINATIONS
S R 2 = F P + T N ; SC1 = T P + F P
Activ.
lnact.
Total
a n d SC2 = F N + T N
Pred.+ Pred.-
52 4
9 34
61 38.
A m i n i m u m value o f Chi 2 o f 6.63 is n e e d e d to i n d i c a t e a c o n f i d e n c e level o f 99% (i.e. 1% c h a n c e t h a t t h e o b s e r v e d c o n c o r d a n c e is d u e to chance). Overall, the results o f M U L T I C A S E a r e generally s u p e r i o r to t h o s e o b t a i n e d with C A S E while the results o f the C A S E / G I analysis a r e conside r a b l y less impressive. Finally, we e v a l u a t e d t h e ability o f the m e t h o d o l o g y to p r e d i c t the N T P activity o f the 145 c o m p o u n d s of test set 2 ( T a b l e 3). A s m e n -
Total
56
43
99
% Predicted + Sens.
92.9% 92.9%
Spec.
20.9% 79.1%
Observed Correct Predictions (OCP)=86.87% (excluding marginals). 3 out of 18 marginally active molecules are predicted correctly. Observed Correct Predictions (OCP)=76.07% (including marginals).
69
General conclusions
TABLE 7
Of the three methodologies evaluated herein, MULTICASE s t a n d s o u t b o t h f o r its a b i l i t y t o
CASE PREDICTIONS (BASED ON GENETOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES UNKNOWN TO THE GENETOX LEARNING SET
TABLE 6 COMPUTER GENERATED PREDICTIONS (BASED ON GENETOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES COMMON TO NTP AND GENETOX COMPILATIONS
(a) CASE methodology Activ.
Inact.
Pred. + Pred. -
54 7
9 44
63 51
Total
61
93
114
% Predicted + Sens.
88.5% 88.5%
17.0% 83.0%
Spec.
Total
Activ.
lnact.
Total
Pred.+ Pred.
40 23
23 56
63 79
Total
63
79
142
% Predicted + Sens.
63.5% 63.5%
Spec.
29.1% 70.9%
Observed correct Predictions (OCP) = 67.61% Expected Correct Predictions (ECP) = 50.63% Phi sq. = 0.118 Chi sq. = 16.782 0 out of 1 marginal molecule is predicted correctly Observed Correct Predictions (including marginals)= 67.13%
Excluding the 28 inconclusives, we find: Observed Correct Predictions (OCP) = 85.96% Expected Correct Predictions (ECP) = 50.37% Phi sq. = 0.515 Chi sq. = 58.714 0 out of 3 marginal molecule, are predicted correctly Observed Correct Predictions (including marginals) = 83.76%
(b) MUL TICASE methodology Activ.
Inact.
Total
Pred. + Pred. -
58 3
10 43
68 46
Total
61
53
114
% Predicted+ Sens.
95.1% 95.1%
Spec.
18.9% 81.1%
Observed Correct Predictions (OCP)= 88.60% (excluding marginals) Expected Correct Predictions (ECP) = 50.68% Phi sq. = 0.600 Chi sq. = 68.441 0 out of the 3 marginal molecule, are predicted correctly Observed Correct Predictions (OCP) = 86.32% (including marginals)
(c) Graph theory prediction Activ.
Inact.
Pred. + Pred. -
56 5
35 18
91 23
Total
61
53
114
% Predicted + Sens.
91.8% 91.8%
Spec.
Total
66.0% 34.0%
Observed Correct Predictions (OCP)= 64.91% Phi sq. = 0.103 Chi sq. = 11.690 Expected Correct Predictions (ECP) = 52.09% 0 out of the 3 marginal molecules are predicted correctly Observed Correct Predictions (including the 3 marginals) = 63.25%
Activ.
Inact.
Total
Pred.+ Pred.-
38 13
19 44
57 57
Total
51
63
114
% predicted + Sens.
74.5% 74.5%
Spec.
30.2% 69.8%
Observed Correct Predictions (OCP)= 71.93% Expected Correct Predictions (ECP) = 50.00% Phi sq. = 0.195 Chi sq. = 22.176 Observed Correct Predictions (including 1 marginal) = 67.61% Observed Correct Predictions for the 28 inconclusives = 50.00%
r e t r o f i t t h e d a t a it w a s t r a i n e d w i t h , a n d t o p r e d i c t t h e a c t i v i t y o f u n k n o w n m o l e c u l e s . T h e exp e c t e d c o n c o r d a n c e f o r its p r e d i c t i o n s o f u n k n o w n m o l e c u l e s is a b o u t 8 0 % , w h i c h is c l o s e t o t h e 76 t o 8 7 % a c c u r a c y l e v e l e x p e c t e d f r o m experimental results (see above). The CASE methodology follows with an expected concordance of 72%, while our method based on Graph Indices barely exceeds random predictions. At best, one can expect a Salmonella test to provide information about Salmonella mutagenicity w i t h a c o n c o r d a n c e o f 8 7 % , w h i c h is n o t significantly different from the interlaboratory rep r o d u c i b i l i t y f o r t h e N T P d a t a is e s t i m a t e d t o b e a p p r o x i m a t e l y 8 5 % ( P i e g o r s c h a n d Z e i g e r , 1991). The MULTICASE methodology, when applied to molecules that were seen before (retrofit) provides the same kind of accuracy, or better. On the
70
other hand, for molecules not encountered before by the program, the rate of correct classifications falls to about 80%. The accuracy of the predictions, although not as good as experimental measurements is still significant. The ability of the Salmonella assay to predict rodent carcinogens is about 62% (Ashby et al., 1989). Therefore, the expectation for an expert system used to predict carcinogenesis based upon an analysis of the G E N E T O X data base is a 50% success rate (80%*62% = 50%). Although this result is less than impressive, it is to be noted it is not much worse than that expected from the predic-
TABLE 8 MULTICASE PREDICTIONS (BASED ON GENETOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES UNKNOWN TO GENETOX
TABLE 9 GRAPH THEORY PREDICTIONS (BASED ON GENE~ TOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES UNKNOWN TO THE GENETOX COMPILATIONS Activ.
Inact.
Total
Pred. + Pred. -
60 3
72 7
132 10
Total
63
79
142
% Predicted+ Sens.
95.2% 95.2%
Spec.
91.1% 8.9%
O b s e r v e d C o r r e c t P r e d i c t i o n s ( O C P ) = 47.18% E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 45.16% Phi sq. = 0.006 Chi sq. = 0.900 0 out of the 1 marginal molecule is predicted correctly O b s e r v e d C o r r e c t Predictions (including 1 marginal) = 46.85%
Excluding the 28 inconclusices, we find: Total
Activ.
Inact.
Pred.+ Pred.-
45 18
20 59
65 77
Total
63
79
142
% Predicted + Sens.
71.4% 71.4%
25.3% 74.7%
Spec.
O b s e r v e d correct P r e d i c t i o n s ( O C P ) = 73.24% (excluding
marginals) E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 50.48% Phi sq. = 0.211 Chi sq. = 30.025 0 out of the 1 marginal molecule was predicted correctly O b s e r v e d correct P r e d i c t i o n s ( O C P ) = 72.72% (including 1 marginal)
Excluding the 28 inconclusiues, we find:
Inact.
Total
Pred. + Pred. -
49 2
58 5
107 7
Total
51
63.
114
% Predicted+ Sens.
96.1% 96.1%
Spec.
92.1% 7.9%
O b s e r v e d C o r r e c t P r e d i c t i o n s ( O C P ) = 47.37% E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 45.38% Phi sq. = 0.007 Chi sq. = 0.788 O b s e r v e d correct Predictions (including 1 marginal) = 46.96% O b s e r v e d C o r r e c t Predictions, of the 28 inconclusives = 46.43%
Total
Activ.
Inact.
Pred.+ Pred.-
42 9
14 49
56 58
Total
51
63
114
% Predicted + Sens.
82.4% 82.4%
Spec.
Activ.
22.2% 77.8%
O b s e r v e d correct P r e d i c t i o n s ( O C P ) = 79.82% (excluding
marginals) E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 50.09% Phi sq. = 0.358 Chi sq. = 40.775 O b s e r v e d C o r r e c t P r e d i c t i o n s ( O C P ) = 79.13% (including 1 marginal) O b s e r v e d correct Predictions for the 28 inconclusives = 46.43%
tion of carcinogenesis from Salmonella directly (87% * 62% = 54%).
The interesting point though is that one can expect that the 80% concordance obtained for mutagenicity will also be obtained if the MULTICASE program is trained with a data base of rodent carcinogens, thus yielding far superior predictivity of carcinogenicity than "indirect" predictions based on the results of the Salmonella assay. That this expectation is realistic is supported by the finding that we have already shown that the CASE version of the program is highly predictive of rodent carcinogenicity (Rosenkranz and Klopman, 1990b).
71
Acknowledgements This investigation was supported by the National Institute of Environmental Health Sciences (ES04659) and The U.S. Environmental Protection Agency (R818275-01-0).
References Ashby, J., and R.W. Tennant (1988) Chemical structure, Salmonella mutagenicity and extent of carcinogenicity as indicators of genotoxic carcinogenesis among 222 chemicals tested in rodents by the U.S. NCI/NTP, Mutation Res., 204, 17-115. Ashby, J., and R.W. Tennant (1991) Definitive relationships among chemical structure, carcinogenicity and mutagenicity for 301 chemicals tested by the U.S. NTP, Mutation Res., 257, 229-306. Ashby, J., R.W. Tennant, E. Zeiger and S. Stasiewicz (1989) Classification according to chemical structure, mutagenicity to Salmonella and level of carcinogenicity of a further 42 chemicals tested for carcinogenicity by the U.S. National Toxicology Program, Mutation Res., 223, 73-103. Bailey, D.E. (1971) Probability and Statistics Models for Research, Wiley, New York. Haworth, S., T. Lawlor, K. Mortelmans, W. Speck and E. Zeiger (1983) Salmonella mutagenicity test results for 250 chemicals, Environ. Mutagen., 5, Suppl. 1, 3-142. Kier, L.D., D.J. Brusick, A.E. Auletta, E. Von Halle, M.M. Brown, V.F. Simmon, V. Dunkel, J. McCann, K. Mortelmans, M. Prival, T.K. Rap and V. Ray (1986) The Salmonella typhimuriurn/mammalian microsomal assay, Mutation Res., 168, 69-240. Klopman, G. (1992) MultiCASE I, QSAR, in press. Klopman, G., and I. Kolossvary (1990) Evaluation of quantitative structure-activity predictions. Comparison of the pre-
dictive power of an artificial intelligence system with human experts, J. Math. Chem., 5, 389-401. Klopman, G., and H.S. Rosenkranz (1991a) Quantification of the predictivity of some short term assays carcinogenicity in rodents, Mutation Res., 253, 237-240. Klopman, G., and H.S. Rosenkranz (1991b) Structure-activity relations, Maximizing the usefulness of mutagenicity and carcinogenicity data bases, Environ. Health Perspect., in press. Klopman, G., M.R. Frierson and H.S. Rosenkranz (1990) The structural basis of the mutagenicity of chemicals in Salmonella typhimuriurn: The Gene-Tox Data Base, Mutation Res., 228, 1-50. Piegorsch, W.W., and E. Zeiger (1991) Measuring intra-assay agreement for the Ames Salmonella assay, in: O. Rienhoff and D.A.B. Lindberg (Eds.), Lecture Notes in Medical Information (Statistical Methods in Toxicology), Springer, Berlin, pp. 35-41. Raychaudhury, C., and G. Klopman (1990) New vertex indices and their applications in evaluating antileukemic activity of 9-anilinoacridines and the activity of 2',3'-dideoxynucleosides against HIV, Bull. Soc. Chim. Belg., 99, 255264. Rosenkranz, H.S., and G. Klopman (1989) Structural basis of the mutagenicity of phenylazoaniline dyes, Mutation Res., 221,217-239. Rosenkranz, H.S., and G. Klopman (1990) Structural basis of carcinogenicity in rodents of genotoxicants and nongenotoxicants, Mutation Res., 228, 105-124. Rosenkranz, H.S., N. Takihi and G. Klopman (1991) Structure-activity-based predictive toxicology, An efficient and economical method for generating non-congeneric data bases, Mutagenesis, 5, 391-394. Zeiger, E., J.K. Haseman, M.D. Shelby, B.H. Margolin and R.W. Tennant (1990) Evaluation of four in vitro genetic toxicity tests for predicting rodent carcinogenicity, Confirmation of earlier results with 41 additional chemicals, Environ. Mol. Mutagen., 16, Suppl. 18, 1-14.
59
MUTENV 08831
Testing by artificial intelligence: Computational alternatives to the determination of mutagenicity Gilles Klopman and Herbert S. Rosenkranz b a
a Department of Chemistry, Case Western Reserve University, Cleveland, OH 44106, USA and # Department of Environmental and Occupational Health, University of Pittsburgh, Pittsburgh, PA 15261, USA (Received 19 February 1991) (Revision received 17 January 1992) (Accepted 4 March 1992)
Keywords: Artificial intelligence; Intelligence, artificial; Computational Structure-activity methods; CASE; MULTICASE; CASE/GI
alternatives,
determination
of mutagenicity;
Summary In order to develop methods for evaluating the predictive performance of computer-driven structureactivity methods (SAR) as well as to determine the limits of predictivity, we investigated the behavior of two Salmonella mutagenicity data bases: (a) a subset from the Genetox Program and (b) one from the U.S. National Toxicology Program (NTP). For molecules common to the two data bases, the experimental concordance was 76% when "marginals" were included and 81% when they were excluded. Three SAR methods were evaluated: CASE, MULTICASE and C A S E / G r a p h Indices (CASE/GI). The programs "learned" the Genetox data base and used it to predict NTP molecules that were not present in the Genetox compilation. The concordances were 72, 80 and 47% respectively. Obviously, the MULTICASE version is superior and approaches the 85% interlaboratory variability observed for the Salmonella mutagenicity assays when the latter was carried out under carefully controlled conditions.
The development of computer driven knowledge-based structure-activity techniques has raised considerable expectations among some researchers as well as serious doubts among others regarding their potential to predict "in video" the pharmacological a n d / o r toxicological properties of organic compounds. These methods are intended to be used in situations where a certain
Correspondence, Prof. H.S. Rosenkranz, Department of Environmental and Occupational Health, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA.
type of biological activity is observed for a series of "diverse" molecules and where traditional QSAR methods, which require congenericity, are not applicable. The diversity of the structures of the molecules producing the activity may be due to multiple mechanisms, or to an "obscure" commonality between the structures of the active compounds. The purpose of this communication is to investigate the usefulness of three such techniques, the CASE program, the C A S E / G I program which is a variation of the CASE program based on graph indices (Raychaudhury and Klopman, 1990), and MULTICASE (Klopman, 1992), a new artificial
60
intelligence program. All three programs were developed in our laboratory and used extensively to study the potential genotoxic activity of chemicals The CASE method, which has been described on a number of occasions (Klopman et al., 1990; Rosenkranz and Klopman, 1989, 1990a), employs an artificial intelligence algorithm for the evaluation of the relevance of chemical substructures. Recently, the program's predictions of the activity of antibacterial agents have been compared with those of human experts (Klopman and Kolossvary, 1990). The second of these programs is based on graph theoretical descriptors. We have used it, with a somewhat different underlying graph index, to study a number of biological activities. Finally, the third method, MULTICASE, is a new methodology which, as CASE, uses structural fragments as descriptors, but is based on a totally different algorithm, incorporating extensive hierarchy in the selection of its descriptors. The method has already been used to study a number of toxicological data bases and shows considerable promise. We have now compiled sufficient data to feel confident that a reliable evaluation of the usefulness of these methods can be made. In view of the fact that the largest data bases available to us consists of results of the Salmonella mutagenicity assay and because that assay occupies a pivotal role in defining a "genotoxic carcinogen" (Ashby and Tennant, 1988, 1991; Zeiger et al., 1990), we conducted a study to establish the reasonable expectations one might have about the results of this type of analysis and compared'them to experimental results. However it should be stressed that the conclusions drawn from the present analyses are not restricted to the Salmonella mutagenicity assay, but are applicable to the results of biological activities analyzed with the CASE program. We selected two data bases for our study; one consisting of 808 organic molecules tested for mutagenic activity by the Environmental Protection Agency under the G E N E T O X program (Klopman et al., 1990) and another of 262 chemicals tested under the aegis of the U.S. National Toxicology Program (NTP) (Rosenkranz and Klopman, 1990a). These two data bases consist of largely diverse compounds and are thus not
amenable to analysis by conventional QSAR techniques. Both databases list the results of Salmonella typhimurium mutagenicity assays but differ with respect to criteria used for the acceptability of the results. The G E N E T O X data arc derived from a consensus analysis of published data (Kier et al., 1986) while the NTP results were all obtained from a limited number of laboratories using coded samples of known purity and employing pre-established protocol (Haworth et al., 1983; Zeiger et al., 1988). Overall, the NTP data base is probably superior as the results were obtained under carefully controlled conditions. Each of the data bases was the subject of previous CASE analyses and the results have been published (Klopman et al., 1990; Rosenkranz and Klopman, 1990a). Results and discussion
Analysis of the data bases In the study reported, we used the 808 compounds of the G E N E T O X data base as the learning set for each of the three candidate methodologies and then used the acquired expertise to predict the activity of the compounds listed in the NTP compilation. Each of the methods was found to be capable of learning from the data. Indeed, the entire set of 808 molecules was retrofitted satisfactorily by each of the three methods, with an 86.2% accuracy for CASE, 98.4% for MULTICASE and 78.4% accuracy for C A S E / G I when molecules for which "inconclusive" experimental results were obtained were excluded from consideration. When, however, the latter were included, the accuracy was 76.5% for CASE and MULTICASE and 55% for C A S E / G I (Table 1). It is to be noted that in the G E N E T O X Salmonella data base the majority of the chemicals listed as "inconclusive" were not marginally active but contained protocol defects (Klopman et al., 1990; Kier et al., 1986) The NTP data base consisted of the 262 compounds Of these, 117 were also present in the G E N E T O X data base. They are listed in Table 2, together with their reported activity in both the G E N E T O X and the NTP compilations. The activities of the 145 chemicals of the NTP compilation that were not included in the G E N E T O X
61 TABLE 1 R E T R O F I T OF G E N E T O X L E A R N I N G SET BY VARIOUS M E T H O D O L O G I E S
(a) CASE Methodology Activ.
Inact.
Total
Pred. + Pred. -
425 25
17 106
442 131
Total
450
123
573
Sens. 94.4%, Spec. 86.2% Observed correct Predictions (OCP) = 92.67% 87 out of the 235 marginal molecules are predicted correctly Observed Correct Predictions (Including marginals) = 76.49%
(b) MUL TICA SE Methodology Activ.
Inact.
Total
Pred. + Pred.-
446 4
5 118
451 122
Total
450
123
573
Sens. 99.1%, Spec. 95.9% Observed correct Predictions (OCP) = 98.43% 54 out of the 235 marginal molecules are predicted correctly Observed correct Predictions (Including marginals) = 76.49%
(c) GRAPH THEORETICAL Methodology Activ.
Inact.
Total
Pred. + Pred. -
427 23
101 22
528 45
Total
450
123
573
Sens. 94.9%, Spec. 17.9% Observed correct Predictions (OCP) = 78.36% 0 out of the 235 marginal molecules are predicted correctly Observed Correct Predictions (Including marginals) = 55.56%
data base, are shown in Table 3. The distribution of the data among activity classes is shown in Table 4. Although there is general agreement between G E N E T O X and NTP results (see Table 2), there are also frequent discrepancies and even a difference in definitions with respect to activity. Thus " m " is defined as "inconclusive" for G E N E T O X and as "marginal" for NTP. The disagreement between experimental values is undesirable and certainly complicates the evaluation. However, the relationship between these two independently derived sets of experimental activities provides a
standard by which the accuracy of the computerbased predicted activities can be compared. Therefore, as the first stage of the analysis, the 117 compounds (test set 1), which are common to both data bases, can be used to evaluate how well one protocol reproduces the results of the other. The results are shown in Table 5. As can be seen, the agreement between the G E N E T O X and NTP classifications is affected considerably by inclusion or omission of the marginal activity compounds (activity "m"). Although this may be due to the inconsistent definition associated with this activity class, it remains that the experimental reproducibility, as measured by the percentage of observed correct predictions (OCP), (defined as the ratio of the sum of the correct predictions of carcinogenicity (TP) or lack thereof (TN) divided by the total number of predictions made) is 87% when "marginals" and "inconclusives" are not considered. An overall concordance (i.e. correct predictions) of 87% thus appears to reflect the current experimental reproducibility and thus the absolute limit one may expect any knowledge-based methodology to reach. In fact analysis of the interlaboratory reproducibility of the Salmonella mutagenicity assay as performed under the aegis of the NTP indicates a reproducibility of approximately 85% (Piegorsch and Zeiger, 1991). It is therefore unrealistic to expect that any structure-activity method using Salmonella data as its training set can predict the mutagenic activity of unknown molecules with a concordance of better than 76%, unless the data set contains no marginal compounds, in which case the concordance can, at best, reach 87%.
Structure-actiuity analyses The three methods were next used to predict the " N T P " activity of the 117 compound data set using the knowledge acquired from the G E N E T O X learning set. The G E N E T O X , NTP and the predicted activities are listed in Table 2. Columns 3 and 4 list the experimental results as reported by G E N E T O X and NTP, respectively. Columns 5 and 6 summarize the CASE predictions, 7 and 8 the M U L T I C A S E predictions and column 9 the C A S E / G I predictions. Columns 6 and 8 denote the probabilities that the molecule is active ac-
62 TABLE
2
MOLECULES
EXISTING
IN
BOTH
Molecules
THE
NTP
Exp.
AND
THE
Activity
GENETOX
COMPILATIONS
CASE
Graphics
MULTICASE
Gr. Ind. GEN (Cytoxan)
NTP
CASE
Prob.
M.C.
Prob.
1
Cyclophosphamide
+
+
+
98.7
+
87.9
+
2
Benzo[a]pyrene
+
+
+
93.5
+
77.3
+
3
Phenylbutazone
m
-
-
0.0
0.0
+
4
N-Hydroxy-2-acetylaminoflu
+
+
+
97.8
+
92.8
+
5
N-2-Acetylaminofluorene
+
+
+
91.8
+
87.4
+
6
Diethylnitrosamine
+
-
+
86.0
+
87.3
+
7
Fenthion
m
+
-
84.5
-
0.0
+
8
Ethyl
m
+
+
94.0
+
69.8
-
9
3-Methylcholanthrene
+
+
+
95.7
+
77.3
+
Diethylstilbestrol
-
-
-
40.0
-
0.0
+
+
+
+
77.0
+
77.3
+
+
+
+
99.6
+
95.0
+
10
parathion
11
Benz[a]anthracene
12
4-Nitroquinoline-
13
/3-Propiolactone
+
+
+
87.0
+
87.0
+
14
7,12-Dimethylbenz[a]anthra
+
+
+
70.0
+
77.3
+
15
Caffeine
-
-
94.7
-
0.0
+
16
5-(4-Chlorophenyl)-6-ethyl
m
-
-
0.0
-
0.0
+
17
p-Dimethylaminoazobenzene
+
-
-
67.0
+
80.0
+
18
Methylhydrazine
+
-
+
83.0
+
83.3
+
19
Acetamide
-
-
-
0.0
-
0.0
-
20
Dieldrin
-
-
-
0.0
0.0
-
21
Amitrole
-
-
-
0.0
0.0
-
22
Ethyl
+
+
+
87.0
+
86.7
+
23
Aniline
-
-
0.0
-
0.0
-
24
Thioacetamide
-
-
-
0.0
-
0.0
+
25
Thiourea
-
-
-
0.0
-
0.0
-
26
Dimethylnitrosamine
+
+
+
89.0
+
87.3
+
27
Methyl
+
+
+
87.0
+
86.7
+
28
Uracil
+
+
+
99.3
+
96.1
+
29
Nitrofurantoin
+
+
+
100.0
+
89.7
+
30
Hexachloroethane
-
-
-
0.0
-
0.0
-
31
N-Methyl-N
+
+
+
89.0
+
87.3
+
32
Methoxychlor
-
-
70.0
-
0.0
-
33
p,p'-DDE
-
-
-
11.0
-
0.(l
-
34
Vinylidene
+
-
-
80.0
+
66.7
+
35
Propylenimine
+
+
+
75.0
+
66.7
+
1-oxide
(3-amino-l,2,4-tr
methanesulfonate
(EM
(DMN)
methanesulfonate
(M
mustard
'-nitro-N-nitros
chloride
36
Propylene
+
+
+
86.0
+
91.4
+
37
Trichloroethylene
chloride
m
-
-
83.0
-
0.0
+
38
Chloroacetic
-
-
83.3
-
54.7
-
39
Dimethylcarbamoyl
+
+
-
73.0
+
66.7
+
40
Saccharin
-
-
+
77.9
-
46.7
-
41
Pentachloronitrobenzene
m
-
+
74.1
-
0.0
+
42
Phenanthrene
-
-
+
77.0
+
77.3
+
43
/3-Naphthoquinoline
+
+
+
93.5
+
86.9
+
44
Dipheny~itrosamine
-
-
-
76.7
-
52.3
+
45
Pentachlorophenol
-
-
-
30.0
-
0.0
+
46
2,4,6-Trichlorophenol
-
-
-
0.0
-
0.0
-
47
Guaiacol
m
-
-
70.0
-
0.0
+
48
2-Aminobiphenyl
+
+
+
99.4
+
91.3
+
49
9-Aminoacridine
+
+
+
96.4
+
86.9
+
5O
Naphthalene
-
-
-
57.1
-
46.7
-
acid chloride
63 TABLE 2 (continued) Molecules
E x p . Activity
CASE
MULTICASE
GEN
NTP
CASE
Prob.
M.C.
Prob.
Graphics G r . Ind.
51 52 53 54 55
Quinoline o-Nitroanisole 2-Naphthylamine 4 - A m i n o - 1,1 ' - b i p h e n y l Benzidine
+ m + + +
+ + + + +
+ + + + -
93.5 94.0 90.1 98.4 87.0
+ + + + +
86.9 69.8 77.3 75.0 75.0
+ + + + +
56 57 58 59 60
Sulfallate 2,4-Diaminotoluene N,N'-Ethylenethiourea Furfural 4-Nitro-o-phenylenediamine
+ + + + +
+ + + +
+ + + +
95.1 99.5 83.0 99.8 99.9
+ + + + +
66.7 95.7 66.7 80.0 95.4
+ + + + +
61 62 63 64 65
5-Nitro-o-anisidine p-Nitrophenol Styrene N-Nitrosopiperidine 4,4'-Methylenebis(2-chloro
+ + + +
+ + +
+ + +
99.9 76.7 67.0 86.0 95.0
+ + + +
95.4 53.7 66.7 87.3 91.3
+ + + +
66 67 68 69 70
N-Phenyl-p-phenylenediamin
-
-
-
29.0
-
0.0
+
trans-
Stilbene
-
-
-
0.0
-
0.0
-
Azobenzene Acetaminophen 1,2-Epoxybutane
+ m +
+
-
67.0 0.0 94.0
+
80.0
+
-
0.0
+
+
87.0
+
71 72 73 74 75
1,2-Dibromoethane Ethylene chlorohydrin Resorcinol Bromobenzene C~clohexane
Ill +
86.0 94.0 0.0 0.0 0.0
+ +
85.7 91.4
+
-
0.0
-
-
0.0
+
-
0.0
-
76 77 78 79 8O
3,3'-Dimethyoxybenzidine Tolidine Anthracene Malathion Maleic hydrazine
+ +
99.2 99.2 77.0 0.0 0.0
+
+ +
91.3 91.3 77.3
+ +
-
0.0
+
-
0.0
+
81 82 83 84 85
Tris(2,3-dibromopropyl)pho Butylated hydroxytoluene Pyrene E D T A t r i s o d i u m salt Monuron
+
86.0 0.0 81.0 0.0 73.0
+
87.0
+
-
0.0
+
+
77.3
+
-
0.0
-
-
0.0
+
86 87 88 89 90
2 - B r o m o - 2 - c h l o r o - 1,1,1-tri Benzo[e]pyrene Methyl parathion Chloral hydrate Diazinon
0.0 77.0 94.0 75.0 0.0
+ + + +
80.0 77.3 69.8 73.7
+ + +
-
0.0
+
91 92 93 94 95
Natulan 1,8,9-Trihydroxyanthracene 1 -Naphthylisothiocyanate meso-l,2 :2,4-Diepoxybutane Pararosaniline
0.0 62.2 53.5 94.0 11.0
-
0.0
+
-
46.7
+
-
25.0
+
+
87.0
-
-
0.0
+
96 97 98 99 100
Methylazoxymethanol acetat Methyl carbamate 5-Nitroacenapthene 2-Nitrofluorene 2-Anthramine
0.0 0.0 99.7 98.7 91.8
-
0.0
-
0.0
+ + +
95.0 94.1 77.3
+ + +
+ + +
+ +
in m
+ 111
+
+ + +
+ +
+
m
+ + +
+ + +
-
-
-
+ +
+ + +
64 TABLE 2 (continued) Molecules
Exp. Activity
CASE
GEN
NTP
CASE
MULTICASE Prob.
M.C.
Prob.
Graphics Gr. Ind.
101 102
Glycidaldehyde Proflavin h y d r o c h l o r i d e
+ +
+ +
+ +
94.0 97.5
+ +
87.0 88.6
+ +
103 104 105
7,9-Dimethylbenz(c)acridin Trifluralin C.I. A c i d red. 14 ( A z o rubi
+ +
+ + -
+ +
99.2 76.7 70.0
+ +
92.8 0.0 50.0
+ + +
106
2-(2-Furanyl)-3-(5-nitro-2
+
+
+
100.0
+
89.7
+
107 108 109
N-Hydroxy-4-aminoquinoline 2-Nitro-p-phenylenediamine 7-Bromomethyl-I2-methylben
+ + +
+ + +
+ + +
94.0 99.9 93.5
+ + +
88.2 94.4 87.8
+ + +
110
Xearalenone
m
-
-
0.0
-
0.0
+
111 112 113
N-(4-(5-Nitro-2-furyl)-2-t P o l y e t h y l e n e glycol Azoxymethane
+ +
.+ -
+ -
100.0 0.0 67.0
+ -
89.7 0.0 0.0
+ +
114 115
N-4-Fluorenylacetamide Hydrochlorothiazide
+ m
+ m
+ -
77.0 13.6
+ -
77.3 0.0
+ +
116 117
E v a n s Blue, C.I. direct bl Crystal violet
m m
m m
-
32.0 11.0
-
0.0 0.0
+ +
+ , - , m indicates activity, inactivity a n d m a r g i n a l activity, respectively. Prob is c a l c u l a t e d p r o b a b i l i t y o f activity. " E x p t . A c t i v i t y " r e f e r s to the e x p e r i m e n t a l results r e p o r t e d by the G e n e T o x S a l m o n e l l a M u t a g e n i c i t y W o r k i n g G r o u p ( G E N ) or g e n e r a t e d u n d e r the aegis of the U.S. N a t i o n a l Toxicology P r o g r a m ( N T P ) .
cording to CASE and MULTICASE, respectively. Table 6 summarizes the results of the analyses for the three structure-activity methods and shows the associated statistical analyses. As expected, the overall accuracy of the predictions is quite similar to that obtained using the experimental G E N E T O X results to predict NTP resuits. The accuracy (OCP) is 86% (84% when including the "marginals/inconclusives") for CASE; it rises to 89% (86%) for M U L T I C A S E but only reaches 65% (65%) for C A S E / G I . It was surprising to find that MULTICASE, trained with G E N E T O X , predicted NTP activities better than G E N E T O X itself (88.6% vs. 86.87%). This may be due to the fact that inconsistencies are automatically weeded out when M U L T I C A S E tries to rationalize the "admittedly inferior" G E N E T O X data, yielding a better correlation with the NTP results. This is similar to what would be expected when a straight line correlating experimental data points is used to predict the outcome of further experiments.
Table 6 also shows the sensitivity and specificity of the three methods, as well as the expected percentage of correct predictions (ECP), the Phi 2 and the Chi 2 values for the correlation. As was shown previously (Klopman and Rosenkranz, 1991a), ECP is defined as the percent "correct" predictions to be expected when a data base (e.g. NTP) containing a fraction X of active molecules is predicted by a method producing a fraction Y of actives (e.g. CASE). ECP can be calculated by the formula: ECP = (1 + 2 X Y - X -
Y ) * 100
(1)
where X and Y values are between 0 and 1. ( X = 61/114 for NTP, Y = 63/114 for CASE) The probability that the observed concordance, i.e. percentage of correct predictions,
65 TABLE
3
NTP MOLECULES
UNKNOWN
TO THE
Molecules
GENETOX
Exp. Activity NTP
LEARNING
SET
CASE
MULTICASE
Graphics
CASE
Prob.
M.C.
Prob.
Gr. Ind.
1
Reserpine
-
+
88.7
-
67.0
2
Acetylsalicylic acid
-
-
70.0
-
0.0
+
3
Piperonyl butoxide
-
-
0.0
-
0.0
+
4
op-DDD
(Mitotane)
-
-
0.0
-
0.0
+
5
Myleran (Busulfan)
+
+
87.0
+
86.7
+
+
88.7
-
0.0
+
33.0
-
0.0
+
0.0
-
0.0
+
6
Coumaphos
w -
7
Allyl isothiocyanate (must
w +
8
Urea
w -
9
Phenytoin
-
+
w -
-
3.0
-
41.2
+
10
1,2-Propanediol
-
-
0.0
-
0.0
+
11
Ethynylestradiol
-
+
77.0
-
0.0
+
12
Chlordane
-
-
0.0
-
0.0
+
13
Progesterone
-
-
0.0
-
0.0
+
14
y-Lindane
-
-
0.0
-
0.0
-
15
Dimethoate
+
-
0.0
-
0.0
+
16
Phenacetin
-
-
70.0
-
0.0
+
17
Tolbutamide
-
-
26.5
-
0.0
+
18
Colchicine
-
-
70.0
+
66.7
+
19
Choline chloride
-
-
67.0
-
0.0
+
20
Dimethylformamide
w -
-
0.0
-
0.0
-
21
Hexachlorophene
-
-
30.0
-
0.0
+
22
2,4-Dinitrofluorobenzene
+
+
87.0
+
89.7
+
23
1,1,1-Trichloroethane
w -
-
0.0
-
0.0
-
24
p,p'-DDD
-
-
0.0
-
0.0
+
25
p,p'-Ethyl-DDD( p e r t h a n e )
+
0.0
-
0.0
+
+
26
1-Bromoethane
w -
+
86.0
+
85.7
27
Iodoform
w +
-
0.0
-
0.0
-
28
Chloropicrin
w +
-
0.0
-
0.0
+
29
Phenolphthalein
-
+
98.8
-
55.1
+
30
Bromdiethylacetylurea (car
w -
0.0
-
0.0
+
31
Dioxathion
w +
-
0.0
-
0.0
+
32
1,1,2,2-Tetrachloroethane
-
-
0.0
-
0.0
+
33
Dapsone
-
-
0.0
+
60.0
+
34
1-Nitronaphthalene
+
+
99.6
+
95.0
+
35
Cinnamyl anthranilate
-
+
95.0
+
91.3
+
36
Picric acid
+
+
87.0
+
62.5
-
37
C.I. Developer 1
-
-
0.0
-
0.0
+
38
Michler's Ketone
+
-
0.0
-
0.0
+
39
Coumarin
+
+
77.0
+
77.3
+
40
3,3'-Dimethoxy-4,4'-biphen
+
0.0
-
0.0
+
41
6-Methylcoumarin
-
+
77.0
+
81.8
+
42
Biphenyl
-
+
90.0
-
0.0
+
43
Phenothiazifie
-
+
77.0
+
77.3
+
44
6-Nitrobenzimidazole
+
+
99.0
+
69.8
+
45
1,2,3-Benzotriazole
w +
+
77.0
+
77.3
+
46
3-Chloro-p-toluidine
-
-
91.7
-
0.0
+
47
5-Chloro-o-toluidine
-
+
99.1
+
91.3
+
48
4-Chloro-o-phenylenediamin
+
+
99.1
+
91.3
+
49
y-Butyrolactone
-
-
0.0
-
0.0
+
50
Eugenol
-
-
0.0
-
0.0
+
66
TABLE
3 (continued) Molecules
Exp. Activity NTP
CASE
MULTICASE
CASE
Prob.
Graphics
M.C.
Prob.
+
95.4
Gr. Ind.
51
5-Nitro-o-toluidine
+
+
99.9
52
Benzyl alcohol
-
-
0.0
-
0.0
+ +
53
Anilazine
-
-
29.0
-
0.0
+
54
Bis(p-(dimethylamino)pheny
-
-
0.0
-
0.0
+
55
Dicyclohexylamine
-
+
80.0
-
0.0
+
56
m-Cresidine
+
57
1-Phenyl-2-thiourea
58
p-Benzoquinone
-
+
97.8
+
91.3
w -
-
29.0
+
66.7
+
+
-
0.0
-
0.0
+
dioxime
59
1,3-Diethyl-2-thiourea
-
-
0.0
+
66.7
+
60
p-Chloroaniline
+
-
0.0
+
0.0
+
61
p-Benzoquinone
-
-
0.0
-
62
1- E p o x y e t h y l - 3 , 4 - e p o x y c y c l
+
+
94.0
+
68.5
63
Bis(2-chloro-l-methylethyl
+
+
86.0
+
91.4
64
1,3,5-Trichlorobenzene
-
-
0.0
-
0.0
-
65
Melamine
-
-
0.0
-
0.0
+
66
Toluene
-
-
0.0
-
0.0
+
67
Cyclohexylamine
-
-
0.0
-
0.0
+
68
Furfuran
-
+
79.0
-
0.0
+
69
Pyridine
-
-
0.0
-
0.0
+
70
Morpholine
-
-
83.0
-
0.0
+
71
Sym-Dichloromethyle t h e r
+
+
94.0
+
91.4
+
72
N-Dibutylamine
-
-
83.0
-
0.0
+
0.0 + +
-
-
80.0
+
66.7
+
w -
-
0.0
-
0.0
+
2-Amino-9,10-anthracenedio
+
+
97.3
+
77.3
+
76
4-Amino-2-nit rophenol
+
+
98.7
+
87.5
+
77
Benzophenone
-
-
0.0
-
0.0
+
78
Piperonyl sulfoxide
w -
+
77.0
-
0.0
+
79
p-Cresidine
+
+
99.6
+
91.3
+
80
2,4-Dinitrotoluene
+
+
87.0
+
89.7
+
81
Enheptin
-
+
87.0
-
0.0
+
82
Hydrazine,
w +
-
29.0
-
0.0
+
83
1,4-Dioxane
-
-
0.0
-
0.0
+
84
Hydroxyurea
+
+
87.0
+
87.5
-
85
Tetrachloroethylene
-
-
0.0
-
0.0
-
86
Sulfisoxazole
-
-
6.9
-
0.0
+
87
2-Methyl- 1-nitro-9,10-anth
+
+
99.6
+
88.2
+
88
S u l p h a n b l u e , C . I . a c i d bl
+
-
1.8
-
0.0
+
89 90
Chloramben Benzenamine,
+ +
+
98.3 97.8
+ +
91.3 91.3
+ +
91
Cupferron
w +
+
93.0
+
92.7
+
92
2,4,5-Trimethylaniline
+
+
99.1
+
91.3
+
93
Nithiazide
94
N-Chloroacetyl-n-phenylace
+ +
+ +
87.0 91.0
+ +
58.9 91.4
+ +
95
2,5-Dithiobiurea
w -
-
0.0
+
66.7
+
96 97
Chlordecone Sodium diethyldithiocarbam
-
0.0
-
98
p-Nitrosodiphenylamine
99
Calcium cyanamide
73
1,1-Bis(p-chlorophenyl)-2,
74
Aldicarb
75
100
Lasiocarpine
1,2-diphenyl
w
2-methoxy-hyd
0.0
+
w +
-
+
0.0 62.0
+ +
66.7 70.6
+ +
w + +
-
0.0 87.0
+
0.0 66.7
+ +
67
TABLE 3 (continued) Molecules
101 102
Exp. Activity
CASE
MULTICASE
Graphics
NTP
CASE
Prob.
M.C.
Prob.
G r . Ind.
+ w + -
+ +
94.0 0.0 0.0 80.0 94.6
+ + -
91.4 0.0 0.0 73.7 46.7
+ + + + +
105
Chlorambucil Lithocholic acid 3 - N i t r o p r o p i o n i c acid Chlorobenzilate Sodium fluorescein
106 107
Semicarbazide hydrochlorid
-
+
83.0
+
83.3
-
N-Ethyl-N-nitrosobenzenami
-
+
75.2
-
68.7
+
108
p - N i t r o a n t h r a n i l i c acid
+
+
99.9
+
95.4
+
109 110
p-Phenylenediamine dihydro Fuchsin
+ +
+
0.0 70.1
+
0.0 91.3
+ +
111 112 113 114 115
Dimethoxane Acetohexamide
+ -
+
0.0 22.8
+ -
70.6 0.0
+ +
(2-Chloroethyl)t rimethylam 2-(1-aziridinyl)ethanol 1,3-Bis(cyclohexyl)thioure
+ w -
+ +
94.0 0.0 80.0
+ + +
91.4 66.7 66.7
+ + +
116 117 118 119
N-(1-Naphthyl)ethylenediam Succinic acid 2,2-dimethyl Acetamide Nitrophen
+ w + +
+ + +
91.9 0.0 94.0 94.0
+ + +
77.3 0.0 87.5 89.7
+ + + +
120
Tetrachloroisonaphthonitri
-
-
30.0
-
0.0
+
121 122
Picloram D i r e c t b l a c k 38
+
+
30.0 99.8
+
0.0 95.7
+ +
123 124 125
2 - O x e t h a n o n e , 3,3-dimethylCotoran 1,5-Naphthalenediamine
+ +
+ +
87.0 57.1 77.0
+ + +
87.0 66.7 77.3
+ + +
126 127 128 129 130
Mirex 2,3,5,6-Tetrachloro-4-nitr 1,1,3-Trimethyl-2-thiourea C.I. d i r e c t b l u e 6 2,4-Dimethoxyaniline
w +
+ +
0.0 74.1 0.0 76.8 97.8
+ +
0.0 0.0 66.7 0.0 91.3
+ + + + +
131 132 133 134 135
4-Chloro-o-toluidine hydro S t r e p t o m y c i n sulfate
w +
+ -
95.0 0.0
+ -
91.3 0.0
+ +
Groton BK 4-Chloro-m -phenylenediamin T o l u e n e - 2 , 5 - d i a m i n e sulfat
w + + +
+ +
67.0 99.5 99.5
+ +
0.0 95.7 91.3
+ + +
136 137 138 139 140
2-(Chloromethyl)pyridine h Pyridine 3-(chloromethyl)Phosphamidon N - ( 3 - A m i n o - 4 - e thoxyphenyl) Trisodium nitrolotriacetat
+ + w + + -
+ + + -
98.9 91.0 0.0 99.6 0.0
+ + + + -
96.1 91.4 45.5 91.3 0.0
+ + + + +
141 142 143
p-Anisidine hydrochloride 2-Chloro-l,4-benzenediamin Phenazopyridine hydrochlor
+ + m
+ +
70.0 99.5 86.0
+ +
0.0 91.3 80.0
+ + +
103 104
+ , - , m indicates activity, inactivity a n d m a r g i n a l activity, respectively. Prob is c a l c u l a t e d probability o f activity. W
indicates t h a t the p r o g r a m h a d identified an unsufficiently d o c u m e n t e d f e a t u r e o f the m o l e c u l e , t h e r e b y w a r n i n g that the p r e d i c t i o n is " I n c o n c l u s i v e " .
" E x p t . A c t i v i t y " r e f e r s to the e x p e r i m e n t a l results g e n e r a t e d u n d e r the aegis of t h e U.S. N a t i o n a l Toxicology P r o g r a m ( N T P ) .
68 TABLE 4 SUMMARY OF THE ACTIVITIES IN THE GENETOX AND NTP COMPILATIONS activity + GENETOXlearningset NTP test set 1 test set 2
m
-
total number of compounds
450 235 123 808 61 3 53 117 63 1 61 145
The scale used for biological activity is as follows: + Compound mutagenic m Compound activity inconclusive (GENETOX) Compound activity marginal (NTP) - Compound inactive. O C P , is d u e to c h a n c e is given by s t a n d a r d Chi 2 d i s t r i b u t i o n f u n c t i o n t a b l e s (Bailey, 1971): Chi 2 = N * Phi 2
(2)
w h e r e N is the n u m b e r of m o l e c u l e s in t h e d a t a base. Phi 2 m e a s u r e s the a c c u r a c y o f t h e m e t h o d with r e s p e c t to e x p e c t a t i o n s f r o m r a n d o m assignm e n t o f activity, a n d is e q u a l to 0 if O C P = E C P a n d to 1 for a p e r f e c t fit. TP 2
Phi 2 =
TN 2
+ SC1 * SR1
SC2" SR2
Fp 2 +
FN 2 +
SC1 * S R 2
1
(3)
SC2*SR1
w h e r e T P a r e t r u e positives, T N a r e true n e g a tives, F P a r e false positive, F N a r e false n e g a tives a n d SR1 = T P + F N ;
t i o n e d above, this set of c o m p o u n d s was not i n c l u d e d in the G E N E T O X study and was previously u n s e e n by t h e p r o g r a m . T h e e x p e r i m e n t a l l y o b s e r v e d N T P d a t a a r e shown in c o l u m n 4; the results of the s t r u c t u r e - a c t i v i t y analyses are shown, as b e f o r e , in c o l u m n s 5 to 9. T h e statistical r e s u l t s for C A S E , M U L T I C A S E and C A S E / G I a r e r e p o r t e d in T a b l e s 7, 8 a n d 9 respectively. T w e n t y eight of the m o l e c u l e s were flagged by the p r o g r a m s as b e i n g insufficiently d o c u m e n t e d ; they are m a r k e d with t h e symbol " w " in c o l u m n 3 of T a b l e 3. T h e p r e d i c t i o n s for t h e s e m o l e c u l e s a r e q u e s t i o n a b l e a n d the statistical analyses a r e given for d a t a i n c l u d i n g as well as excluding t h e s e molecules. A c t u a l l y , the o b s e r v e d c o r r e c t p r e d i c t i o n s for these 28 m o l e c u l e s is only 50% for C A S E a n d 46.4% for b o t h M U L T I CASE and CASE/GI. Overall, we find t h a t the p r e d i c t i o n s for activity of m o l e c u l e s u n k n o w n to the t r a i n i n g set are somewhat i n f e r i o r to t h o s e o b t a i n e d by r e t r o f i t t i n g the d a t a (see also K l o p m a n a n d R o s e n k r a n z , 1991b; R o s e n k r a n z et at., 1991). Exc l u d i n g the 28 inconclusive results, we find that C A S E ' s c o n c o r d a n c e d r o p p e d to 72% (it was 86% for the m o l e c u l e s that existed in the training set). M U L T I C A S E is still d o i n g very well; its c o n c o r d a n c e , ( O C P ) is still 80%, (89% b e f o r e ) b u t C A S E / G I ' s p e r f o r m a n c e , 47%, is b a r e l y better t h a n o n e might have e x p e c t e d f r o m a r a n d o m selection of active molecules, i.e. 45%. TABLE 5 CORRELATION BETWEEN GENETOX AND NTP EXPERIMENTAL DETERMINATIONS
S R 2 = F P + T N ; SC1 = T P + F P
Activ.
lnact.
Total
a n d SC2 = F N + T N
Pred.+ Pred.-
52 4
9 34
61 38.
A m i n i m u m value o f Chi 2 o f 6.63 is n e e d e d to i n d i c a t e a c o n f i d e n c e level o f 99% (i.e. 1% c h a n c e t h a t t h e o b s e r v e d c o n c o r d a n c e is d u e to chance). Overall, the results o f M U L T I C A S E a r e generally s u p e r i o r to t h o s e o b t a i n e d with C A S E while the results o f the C A S E / G I analysis a r e conside r a b l y less impressive. Finally, we e v a l u a t e d t h e ability o f the m e t h o d o l o g y to p r e d i c t the N T P activity o f the 145 c o m p o u n d s of test set 2 ( T a b l e 3). A s m e n -
Total
56
43
99
% Predicted + Sens.
92.9% 92.9%
Spec.
20.9% 79.1%
Observed Correct Predictions (OCP)=86.87% (excluding marginals). 3 out of 18 marginally active molecules are predicted correctly. Observed Correct Predictions (OCP)=76.07% (including marginals).
69
General conclusions
TABLE 7
Of the three methodologies evaluated herein, MULTICASE s t a n d s o u t b o t h f o r its a b i l i t y t o
CASE PREDICTIONS (BASED ON GENETOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES UNKNOWN TO THE GENETOX LEARNING SET
TABLE 6 COMPUTER GENERATED PREDICTIONS (BASED ON GENETOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES COMMON TO NTP AND GENETOX COMPILATIONS
(a) CASE methodology Activ.
Inact.
Pred. + Pred. -
54 7
9 44
63 51
Total
61
93
114
% Predicted + Sens.
88.5% 88.5%
17.0% 83.0%
Spec.
Total
Activ.
lnact.
Total
Pred.+ Pred.
40 23
23 56
63 79
Total
63
79
142
% Predicted + Sens.
63.5% 63.5%
Spec.
29.1% 70.9%
Observed correct Predictions (OCP) = 67.61% Expected Correct Predictions (ECP) = 50.63% Phi sq. = 0.118 Chi sq. = 16.782 0 out of 1 marginal molecule is predicted correctly Observed Correct Predictions (including marginals)= 67.13%
Excluding the 28 inconclusives, we find: Observed Correct Predictions (OCP) = 85.96% Expected Correct Predictions (ECP) = 50.37% Phi sq. = 0.515 Chi sq. = 58.714 0 out of 3 marginal molecule, are predicted correctly Observed Correct Predictions (including marginals) = 83.76%
(b) MUL TICASE methodology Activ.
Inact.
Total
Pred. + Pred. -
58 3
10 43
68 46
Total
61
53
114
% Predicted+ Sens.
95.1% 95.1%
Spec.
18.9% 81.1%
Observed Correct Predictions (OCP)= 88.60% (excluding marginals) Expected Correct Predictions (ECP) = 50.68% Phi sq. = 0.600 Chi sq. = 68.441 0 out of the 3 marginal molecule, are predicted correctly Observed Correct Predictions (OCP) = 86.32% (including marginals)
(c) Graph theory prediction Activ.
Inact.
Pred. + Pred. -
56 5
35 18
91 23
Total
61
53
114
% Predicted + Sens.
91.8% 91.8%
Spec.
Total
66.0% 34.0%
Observed Correct Predictions (OCP)= 64.91% Phi sq. = 0.103 Chi sq. = 11.690 Expected Correct Predictions (ECP) = 52.09% 0 out of the 3 marginal molecules are predicted correctly Observed Correct Predictions (including the 3 marginals) = 63.25%
Activ.
Inact.
Total
Pred.+ Pred.-
38 13
19 44
57 57
Total
51
63
114
% predicted + Sens.
74.5% 74.5%
Spec.
30.2% 69.8%
Observed Correct Predictions (OCP)= 71.93% Expected Correct Predictions (ECP) = 50.00% Phi sq. = 0.195 Chi sq. = 22.176 Observed Correct Predictions (including 1 marginal) = 67.61% Observed Correct Predictions for the 28 inconclusives = 50.00%
r e t r o f i t t h e d a t a it w a s t r a i n e d w i t h , a n d t o p r e d i c t t h e a c t i v i t y o f u n k n o w n m o l e c u l e s . T h e exp e c t e d c o n c o r d a n c e f o r its p r e d i c t i o n s o f u n k n o w n m o l e c u l e s is a b o u t 8 0 % , w h i c h is c l o s e t o t h e 76 t o 8 7 % a c c u r a c y l e v e l e x p e c t e d f r o m experimental results (see above). The CASE methodology follows with an expected concordance of 72%, while our method based on Graph Indices barely exceeds random predictions. At best, one can expect a Salmonella test to provide information about Salmonella mutagenicity w i t h a c o n c o r d a n c e o f 8 7 % , w h i c h is n o t significantly different from the interlaboratory rep r o d u c i b i l i t y f o r t h e N T P d a t a is e s t i m a t e d t o b e a p p r o x i m a t e l y 8 5 % ( P i e g o r s c h a n d Z e i g e r , 1991). The MULTICASE methodology, when applied to molecules that were seen before (retrofit) provides the same kind of accuracy, or better. On the
70
other hand, for molecules not encountered before by the program, the rate of correct classifications falls to about 80%. The accuracy of the predictions, although not as good as experimental measurements is still significant. The ability of the Salmonella assay to predict rodent carcinogens is about 62% (Ashby et al., 1989). Therefore, the expectation for an expert system used to predict carcinogenesis based upon an analysis of the G E N E T O X data base is a 50% success rate (80%*62% = 50%). Although this result is less than impressive, it is to be noted it is not much worse than that expected from the predic-
TABLE 8 MULTICASE PREDICTIONS (BASED ON GENETOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES UNKNOWN TO GENETOX
TABLE 9 GRAPH THEORY PREDICTIONS (BASED ON GENE~ TOX LEARNING SET) OF NTP ACTIVITIES OF MOLECULES UNKNOWN TO THE GENETOX COMPILATIONS Activ.
Inact.
Total
Pred. + Pred. -
60 3
72 7
132 10
Total
63
79
142
% Predicted+ Sens.
95.2% 95.2%
Spec.
91.1% 8.9%
O b s e r v e d C o r r e c t P r e d i c t i o n s ( O C P ) = 47.18% E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 45.16% Phi sq. = 0.006 Chi sq. = 0.900 0 out of the 1 marginal molecule is predicted correctly O b s e r v e d C o r r e c t Predictions (including 1 marginal) = 46.85%
Excluding the 28 inconclusices, we find: Total
Activ.
Inact.
Pred.+ Pred.-
45 18
20 59
65 77
Total
63
79
142
% Predicted + Sens.
71.4% 71.4%
25.3% 74.7%
Spec.
O b s e r v e d correct P r e d i c t i o n s ( O C P ) = 73.24% (excluding
marginals) E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 50.48% Phi sq. = 0.211 Chi sq. = 30.025 0 out of the 1 marginal molecule was predicted correctly O b s e r v e d correct P r e d i c t i o n s ( O C P ) = 72.72% (including 1 marginal)
Excluding the 28 inconclusiues, we find:
Inact.
Total
Pred. + Pred. -
49 2
58 5
107 7
Total
51
63.
114
% Predicted+ Sens.
96.1% 96.1%
Spec.
92.1% 7.9%
O b s e r v e d C o r r e c t P r e d i c t i o n s ( O C P ) = 47.37% E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 45.38% Phi sq. = 0.007 Chi sq. = 0.788 O b s e r v e d correct Predictions (including 1 marginal) = 46.96% O b s e r v e d C o r r e c t Predictions, of the 28 inconclusives = 46.43%
Total
Activ.
Inact.
Pred.+ Pred.-
42 9
14 49
56 58
Total
51
63
114
% Predicted + Sens.
82.4% 82.4%
Spec.
Activ.
22.2% 77.8%
O b s e r v e d correct P r e d i c t i o n s ( O C P ) = 79.82% (excluding
marginals) E x p e c t e d C o r r e c t P r e d i c t i o n s ( E C P ) = 50.09% Phi sq. = 0.358 Chi sq. = 40.775 O b s e r v e d C o r r e c t P r e d i c t i o n s ( O C P ) = 79.13% (including 1 marginal) O b s e r v e d correct Predictions for the 28 inconclusives = 46.43%
tion of carcinogenesis from Salmonella directly (87% * 62% = 54%).
The interesting point though is that one can expect that the 80% concordance obtained for mutagenicity will also be obtained if the MULTICASE program is trained with a data base of rodent carcinogens, thus yielding far superior predictivity of carcinogenicity than "indirect" predictions based on the results of the Salmonella assay. That this expectation is realistic is supported by the finding that we have already shown that the CASE version of the program is highly predictive of rodent carcinogenicity (Rosenkranz and Klopman, 1990b).
71
Acknowledgements This investigation was supported by the National Institute of Environmental Health Sciences (ES04659) and The U.S. Environmental Protection Agency (R818275-01-0).
References Ashby, J., and R.W. Tennant (1988) Chemical structure, Salmonella mutagenicity and extent of carcinogenicity as indicators of genotoxic carcinogenesis among 222 chemicals tested in rodents by the U.S. NCI/NTP, Mutation Res., 204, 17-115. Ashby, J., and R.W. Tennant (1991) Definitive relationships among chemical structure, carcinogenicity and mutagenicity for 301 chemicals tested by the U.S. NTP, Mutation Res., 257, 229-306. Ashby, J., R.W. Tennant, E. Zeiger and S. Stasiewicz (1989) Classification according to chemical structure, mutagenicity to Salmonella and level of carcinogenicity of a further 42 chemicals tested for carcinogenicity by the U.S. National Toxicology Program, Mutation Res., 223, 73-103. Bailey, D.E. (1971) Probability and Statistics Models for Research, Wiley, New York. Haworth, S., T. Lawlor, K. Mortelmans, W. Speck and E. Zeiger (1983) Salmonella mutagenicity test results for 250 chemicals, Environ. Mutagen., 5, Suppl. 1, 3-142. Kier, L.D., D.J. Brusick, A.E. Auletta, E. Von Halle, M.M. Brown, V.F. Simmon, V. Dunkel, J. McCann, K. Mortelmans, M. Prival, T.K. Rap and V. Ray (1986) The Salmonella typhimuriurn/mammalian microsomal assay, Mutation Res., 168, 69-240. Klopman, G. (1992) MultiCASE I, QSAR, in press. Klopman, G., and I. Kolossvary (1990) Evaluation of quantitative structure-activity predictions. Comparison of the pre-
dictive power of an artificial intelligence system with human experts, J. Math. Chem., 5, 389-401. Klopman, G., and H.S. Rosenkranz (1991a) Quantification of the predictivity of some short term assays carcinogenicity in rodents, Mutation Res., 253, 237-240. Klopman, G., and H.S. Rosenkranz (1991b) Structure-activity relations, Maximizing the usefulness of mutagenicity and carcinogenicity data bases, Environ. Health Perspect., in press. Klopman, G., M.R. Frierson and H.S. Rosenkranz (1990) The structural basis of the mutagenicity of chemicals in Salmonella typhimuriurn: The Gene-Tox Data Base, Mutation Res., 228, 1-50. Piegorsch, W.W., and E. Zeiger (1991) Measuring intra-assay agreement for the Ames Salmonella assay, in: O. Rienhoff and D.A.B. Lindberg (Eds.), Lecture Notes in Medical Information (Statistical Methods in Toxicology), Springer, Berlin, pp. 35-41. Raychaudhury, C., and G. Klopman (1990) New vertex indices and their applications in evaluating antileukemic activity of 9-anilinoacridines and the activity of 2',3'-dideoxynucleosides against HIV, Bull. Soc. Chim. Belg., 99, 255264. Rosenkranz, H.S., and G. Klopman (1989) Structural basis of the mutagenicity of phenylazoaniline dyes, Mutation Res., 221,217-239. Rosenkranz, H.S., and G. Klopman (1990) Structural basis of carcinogenicity in rodents of genotoxicants and nongenotoxicants, Mutation Res., 228, 105-124. Rosenkranz, H.S., N. Takihi and G. Klopman (1991) Structure-activity-based predictive toxicology, An efficient and economical method for generating non-congeneric data bases, Mutagenesis, 5, 391-394. Zeiger, E., J.K. Haseman, M.D. Shelby, B.H. Margolin and R.W. Tennant (1990) Evaluation of four in vitro genetic toxicity tests for predicting rodent carcinogenicity, Confirmation of earlier results with 41 additional chemicals, Environ. Mol. Mutagen., 16, Suppl. 18, 1-14.
