Journal o f Psyeholinguistic Research, Vol. 2, No. 2, 1973
Semantic Similarity Between Sentences Richard P. H o n e c k I
Received June 14, 1972
The present study investigated the effects of deep, lexical, and surface structure relationships between sentences on ]udgments o f these sentences' semantic similarity. Ten sentence conditions, four paraphrases and six nonparaphrases, were derived from a base sentence. The four paraphrase types were transformational (T), a passive form of the base, lexical (L), containing synonyms for base content words, formalexic (F), a combination o f T and L types, and parasyntactic (P), one of several alternative interpretations o f the base. The six nonparaphrases consisted of three sets o f two sentences each: the false permutation sentences retained the base lexicon, the false synonymous sentences contained synonyms, and the unrelated sentences' lexicon was completely unrelated to the base. One sentence in each nonparaphrase set retained the base surface form and the other, a passivization, did not. Using a modified paired comparisons task, the following rank order o f conditions, in terms of preference, was obtained: T > L > F > P > false permutation > false synonymous > unrelated. It was concluded that deep structure similarity had potent effects but that a more complete description of the data required the postulation o f additional factors such as "propositional structure" and "semantic structure."
INTRODUCTION Imagine there exists a continuum ranging from complete identity to complete dissimilarity in meaning of connected discourse (sentences, in this discussion). The question then must be raised as to what aspect(s) of sentences form the basis for judgments of similarity on such a continuum. To the best of the author's knowledge, no study has explored this question in a comprehensive empirical manner. Traditional psychological approaches to semantic similarity have little bearing on this question as they have focused on the isolated word 1psychology Department, University of Cincinnati, Cincinnati, Ohio. 137 9 1973 Plenum Publishing Corporation, 227 West 17th Street, New York, N.Y. 10011.
138
Honeck
as the unit of analysis. For example, papers (Osgood, 1968; Fillenbaum, 1969; Anisfeld and Knapp, 1968, to mention only a few) abound in the literature which treat the word as a bundle of distinctive semantic features. On this view, word similarity is a function of feature overlap. Similarly, word synonymity has been analyzed in terms of associative overlap (Deese, 1965), images, concepts (Schaeffer and Wallace, 1970), branching trees (Collins and QuiUian, 1969), etc. By contrast, Miller (1969) specifies other bases of judgments of word similarity. Miller asked Ss to sort nouns, each defined by a short sentence, on the basis of semantic similarity. He attempted to account for the resulting clusters on the basis of "shared presuppositions" of the nouns and of "assertions" made by the sentence definitions. Thus, Miller seems to argue that, even on the word level, a list of word features or associations, etc., cannot explain similarity. Similarly, Anisfeld (1970) argues that phrases are "propositional entities" while the isolated word is less prescribed. Anisfeld found that more false recognition memory errors occurred to phrases (e.g., fat book) which were synonymous with preceding phrases (thick book) than to control phrases (history book) or antonymous phrases (thin book). The results of the Miller and the Anisfeld studies suggest that sentence similarity is a complex affair which cannot be handled solely in terms of the properties of words. Some extra-word, relational, gestaltlike factors must clearly be operating. A suggestion as to the nature of these factors is provided by Chomsky's (1965) linguistic theory, particularly by the concept of deep structure. Deep structure is an output of the base rules, which Chomsky defines as a set of categorial rules which "define grammatical functions and grammatical relations and determine an abstract underlying order," and a lexicon which, "characterizes the individual properties of particular lexical items that are inserted in specified positions in base phrase-markers" (p. 136). Deep structure is, in turn, an input to the semantic component, which renders a semantic interpretation, and to the syntactic transformation rules whose application results in the surface structure. Surface structure, which can be represented by an immediate constituent (phrase) structure analysis, is, in turn, an input to the phonological rules which render a phonetic interpretation. According to Chomsky's view, two sentences should theoretically receive identical interpretations when their deep structures are identical and different interpretations to the extent that their deep structures diverge [though Chomsky (1969) now feels that aspects of surface structure may affect meaning]. Aside from mere sentence repetition, which also repeats the lexical and surface structure, Katz and Postal (1964), who have elaborated on the semantic component in Chomsky's theory, assert that grammatical transformations preserve deep structure; hence, two transformationally related sentences
Semantic Similarity Between Sentences
139
should receive the same interpretation and, by implication, be judged identical in meaning. However, there appear to be forms of sentential synonymy whose existence does not rely on deep structure identity. This is so because many such cases involve either minimal or no lexical overlap between the sentences and, by Chomsky's definition, lexicat entries are part of the deep structure. The author (Honeck, 1971) has developed a taxonomy of paraphrases which bears on this issue. Aside from the transformational (7") type paraphrase, three additional types of paraphrase are delineated by the taxonomy. Substitution of synonyms for major vocabulary in a source (base) sentence results in a lexical (L) paraphrase. A formalexic (F) paraphrase is a combination of T and L types, being formed by both transformation and lexical substitution operations. A parasyntactic (P) paraphrase admits of no operational definition at present but can best be considered one of many possible interpretations of a base sentence. Note that only T preserves the major vocabulary of the base and only L preserves its surface structure. These four types are illustrated in Table I. Honeck found that college students rated visually presented T type paraphrases most similar in meaning to their base, F types least similar, with L types intermediate. Moreover, these paraphrases had generally parallel effects when presented auditorily in a memory-comprehension task. The P paraphrase was not investigated. And, unfortunately , neither experiment included nonparaphrase conditions. For this reason, a word matching strategy interpretation of the ratings could not be precluded. This strategy would entail Ss rating as most similar to the base that paraphrase, T, which retained the base content words. Therefore, the present study included nonparaphrase conditions which changed the deep structure of their base and either maintained or changed its content words, and, orthogonally, either maintained or changed Table I. A Base Sentence and Derived Conditions
Base: The struggle evoked the feelings that changed the lad. T: The feelings that changed the lad were evoked by the struggle. L: The fight produced the emotions that altered the boy. F: The emotions that altered the boy were produced by the fight. P: Because the lad was in the war he became cold and unemotional. False Perm Same: The lad evoked the struggle that changed the feelings. False Perm Dill: The struggle that changed the feelings was evoked by the lad. False Syn Same: The boy elicited the fight that altered the emotions. False Syn Diff: The fight that altered the emotions was elicited by the boy. Unrelated Same: The lunatic trapped the person that annoyed the onlookers. Unrelated Dill: The person that annoyed the onlookers was trapped by the lunatic.
140
Honeck
its surface structure. The base lexicon was changed in two ways, either by replacing it with synonyms or with completely unrelated words. Six nonparaphrase sentence conditions resulted from this procedure. Two, the false permutation conditions, retained the base lexicon, two, the false synonymous conditions, contained synonyms, and two, the unrelated conditions, contained content items completely unrelated to those of the base. One of the sentences within each of these three sets of nonparaphrase retained the basic surface structure and one did not. Briefly, the S's task on a given trial was to decide which of two visually presented comparison sentences (conditions) best preserved the basic information of the base (standard) sentence. Expectations of results are based on Honeck's (1971) data and on the premise that deep structure should have by far the strongest effect, lexicat structure the next strongest effect, while surface structure effects should be confined to comparison sentences located in the same region on the similarity continuum. Moreover, L and F as well as T seem to retain the logical or "propositional structure" of the base whereas P and all nonparaphrases do not. These considerations predict the the following general ranking of conditions in terms of preference: T > L > F > P > f a l s e perm" > false synonymous > unrelated The rank of P is predicted, frankly, on commonsense grounds.
METHOD Subjects The Ss were 94 students from introductory psychology courses at the University of Cincinnati, 3 graduate students, 2 professors, and 1 secretary-a total of 100. All Ss were native speakers of English whose ages ranged from 18 to 40 years.
Materials Four base (standard) sentences with identical right branching surface structures were constructed. Each had the form the-noun1 - verb - the-noun2 that - verb - the-noun3. The mean log Thorndike-Lorge content (noun and verb) word frequency was 2.43, 2.58, 2.67, and 1.93 occurrences per 489 million for base sentences 1, 2, 3, and 4, respectively. The bases were made somewhat complex by minimizing intrasentence word associations and semantic constraints. Four types of paraphrase T, L, F, and P, and six types of nonparaphrase of each base were also constructed. These 10 conditions are described below. One base sentence and the 10 conditions relating to it are illustrated in Table I.
Semantic Similarity Between Sentences
141
Transformational (T) This paraphrase was a passive form of the base. Thus, the deep structure and lexical (content word) structure of the base is retained but its surface structure is changed.
Lexical (L) The surface structure of the base is retained but synonyms were substituted for its content words. Only content words are substituted for because they carry most of the cognitive load in a sentence (Perfetti, 1969). Moreover, it is much more difficult to find a good one-word synonym for function words in a sentence context. The synonyms were selected with the help of Roget's Thesaurus (Mawson, 1931) with the requirement that they be on the same semantic level as their source. That is, source and synonym did not appear to contain different semantic markers, at least at a high level in the semantic tree. For example, boy rather than male or person was substituted for lad. The mean log frequency of occurrence per 489 million was 2.71, 2.52, 2.72, and 2.07, respectively, for the four corresponding L paraphrases of bases 1, 2, 3, and 4.
Formalexic CF) This paraphrase was a passive form of the base with synonyms substituted for its content words. The surface structure was the same as that used for the T paraphrase and the content words were the same as those used in the L paraphrase.
Parasyntactic (PJ Although it is impossible at present to specify formally the semantic relationship between base and paraphrase, this paraphrase can be considered one of many alternative concrete interpretations of the base. That is, a number of different P paraphrases of each base might have been constructed, each rendering a totally different interpretation of the base. This P paraphrase had a different surface structure and a different deep stracture than the base. It was 1-5 words longer than the base and shared, at most, two content words with it. The mean log frequency of the content words was 2.95, 2.71, 2.79, and 2.79 respectively for the corresponding P paraphrases of bases 1, 2, 3, and 4.
142
Honeck
False Permutation, Same Surface Structure (False Perm Same) The three nouns in the base sentence were permuted such that noun1 was moved to the noun2 position, noun2 to noun3, and noun3 to the noun1 position. The base verbs had been chosen so that selection restriction rules (Chomsky, 1965) were not violated by the permutation, thereby avoiding semantically anomalous sentences. The permutation naturally changed the deep structure of the base but retained its surface and lexical structure. False Permutation, Different Surface Structure (False Perm Dill) This condition represents a passive form of the false perm same condition. These sentences therefore retained the lexical structure of their base but not its deep or surface structure. False Synonymous, Same Surface Structure (False Syn Same) Synonyms were substituted for the content words in the false perm same sentences. The synonyms were the same as those used in the L paraphrase of the base. In effect, the propositional structure of the base was changed, its surface structure was retained, and synonyms replaced its content words. False Synonymous, Different Surface Structure (False Syn Diff) These sentences represent a passive form of the false syn same sentences. Thus, the propositional and surface structures of the base were changed and synonyms replaced its content words.
Unrelated, Same Surface Structure (Unrelated Same) These sentences were completely unrelated in propositional or lexical structure to the base but retained its surface structure. They were, in fact, the other base sentences. Unrelated, Different Surface Structure (Unrelated l)iff) These sentences represent a passive form of the unrelated same sentences. Thus, their propositional, surface, and lexical structures were totally unrelated to that of their base. As a final point of clarification note that L and F are related
Semantic Similarity Between Sentences
143
transformationally and presumably, therefore, do not differ in deep structure. The same applies to the paired sentences in the false perm, the false syn, and the unrel conditions. Design and Procedure
A mixed design was used. Four groups of 25 Ss each contributed data to each of the 10 conditions relating to a particular base. Thus, base sentence was the between-S factor and was crossed with sentence conditions, the within-S factor. A modified paired comparisons procedure was used. The Ss received an 11-page booklet. The first page instructed them to decide which sentence in each of a series of sentence pairs best preserved the basic information in the standard (base) sentence. Full comprehension of the standard and comparison pair was emphasized as was the importance of regarding each standard and adjoining pair as a separate problem. Five problem pairs were presented on each of nine subsequent pages. The problems were numbered from 1 to 45 with the standard typed alongside each number. The pairs were typed immediately beneath the standard and were labeled A and B. The S entered his choice (A or B) on an answer sheet at the back of the booklet. The entire task was self-paced (Ss took the booklet home) in order to facilitate comprehension and required approximately 30-45 min to complete. The 45 possible pairs of the 10 sentence conditions were presented in random order with the exception that no condition occurred successively in more than two pairs. The order of pairs was the same within a group but varied between groups. For each group a sentence condition occurred equally often in the A (upper) and B (lower) positions across pairs. An attempt was also made to present each sentence condition within each block of five problems.
RESULTS AND DISCUSSION
The degree of preference for the conditions was tabulated for each of the four groups. Inspection of these data revealed very little variance between groups for judgments involving only paraphrases. Although the between group variance appeared larger for judgments involving only nonparaphrases, an analysis of variance revealed no differences between the groups, F(3,56) < 1.00. The data from the four groups was therefore pooled to form a single percentage preference value for each of the 45 comparisons. These values are presented in Table 2I. Complementary values appear above and
144
Honeck
Table II. Percent Preference of Row Condition over Column Conditiona T T L F P FPS FPD FSS FSD US UD
14 7 6 1 3 1 2 1 0
L
F
P
FPS
FPD
FSS
FSD
US
UD
86 b
93 b 86b
94 b 80b 89b
99b 96 b 94 b 86b
97 b 97 b 97 b 85 b 56
99 b 99 b 91b 84b 77b 73b
98 b 96b 100 b 84b 66 c 78b 51
99 b 97 b 96 b 91 b 65 c 56 63 c 59 d
100b 99 b 97 b 87 b 62 c 64 c 59d 64 c 76 b
14 20 4 3 1 4 3 1
11 6 3 9 0 4 3
14 15 16 16 9 13
44 23 34 35 38
27 22 44 36
49 37 41
41 36
24
aThe abbreviations for the nonparaphrase conditions are FPS (false perm same), FPD (false perm diff), FSS (false syn same), FSD (false syn diff), US (unrelated same), UD (unrelated diff). bp < 0.001. cp < 0.01, one tailed. dp < 0.05, one tailed. below the major diagonal and have been included for convenience. The P levels are based on Z-test values for correlated measures. Inspection o f the table reveals some general patterns. First, the rank order of the conditions conformed to expectations and was as follows: T > L
> F > P > false perm same = false perm cliff > false syn same = false syn diff > unrelated same > unrelated diff. Transitivity holds in every case with the exception that false perm diff is not significantly perferred over unrelated same. The table also indicates that all paraphrases are strongly preferred over all nonparaphrases. The T, L, and F paraphrases are preferred, at 'a minimum, 91% o f the time over nonparaphrases, this minimum dropping slightly to 84% for the P paraphrase. The ordering of the paraphrases themselves also reflects strong preferences. In fact, the range of preference is 80%, for L over P, to 94% for T over P. The preferences within the nonparaphrases are generally weaker as the maximum value is 78% for false perm d i f f over false syn diff. The general pattern of results replicate, extend, and clarify Honeck's (1971) rating data on paraphrases. Both studies found T superior to L and L to F . The present study also demonstrated that P is the least preferred type of paraphrase although it is much preferred over all nonparaphrases. The paraphrase vs. nonparaphrase results also rule out a word matching interpretation o f Honeck's rating data. The content words in the T and false perm conditions were identical yet T was highly preferred. Similarly, the L, F, and false syn conditions shared content words yet L and F were preferred.
Semantic Similarity Between Sentences
145
Obviously, T, L, and F have characteristics which overshadow the factor of lexical similarity between base and derived sentence. As such, the present data provide additional empirical justification for the proposed taxonomy of paraphrases. We can now consider in more detail the effects of deep structure, lexical structure, and surface structure similarities between the standard and the derived sentence conditions. As expected, the deep structure factor was a potent influence on Ss judgments, the lexical factor ran a weak second, and surface structure had only "local" effects. These factors interacted, but we can first consider their effects in relative isolation. Concerning deep structure, Katz and Postal's (1964) theory specifies T as the only condition which retains the deep structure, and therefore, the meaning of the base. The clear superiority of T over all other conditions vindicates their analysis. The superiority of T is consonant with the findings of other investigators, all of whom have used different tasks. For example, using ranking and recognition procedures, Clifton and Odom (1966) found that passive sentences were very similar to their simple - active - affirmative declarative (SAAD) counterparts. And Martin and Roberts' (1966) Ss often recalled a T paraphrase of the input sentence. Similarly, after 30 sec, Sachs' (1967) Ss could more easily recognize a change in the deep structure than the surface structure of a sentence which had been embedded in paragraph context. Bregman and Strasberg (1968) analyzed Ss' second attempts to identify input sentences on a forced-choice recognition test and found that Ss tended to guess T paraphrases for the input (e.g., passive for SAAD). Begg (1971) also reports that in a recognition test, sentences identical to the original were judged no more faithful in meaning to the original than were (T) paraphrases. In sum, the data indicate that T paraphrases are often confused with and substituted for their source in memory tasks. That these paraphrases are judged most similar in meaning to their source in nonmemorial tasks suggests that the memory effects may be obligatory. In any case, the mind seems to regard surface structure variation in a cavalier fashion whenever semantic information processing is emphasized, as it was in the present task. The effect of lexical structure similarity between base and derived sentence is evident for paraphrases and nonparaphrases alike. Among the paraphrases, T and F have the same surface yet T is preferred. In general, that paraphrase condition, T, which retains the base lerdcon is most preferred, those conditions, L and F, containing synonyms of base words are next most preferred, and P, most of whose content words are totally unrelated to the base's, is least preferred, though P supplants all nonparaphrases. A similar trend is apparent among the nonparaphrases. The false perm conditions, which retain the base words, are most preferred, the ,false syn conditions are
146
Honeck
intermediate, and the unrelated conditions are least preferred. Moreover, when surface structure is held constant among the nonparaphrases, the results false perm same >false syn same > unrelated same, and false perm diff >false syn d i f f > unrelated diff clearly demonstrate the effects of the lexical factor. Since the false perm conditions are equivalent and both are significantly preferred over the false syn conditions, which are also equivalent, the conclusion follows that the lexical factor is solely responsible for the ordering of these four conditions. The general finding that lexical structure takes precedence over surface structure hardly offends common sense and accords with the spirit of Chomsky's (1965) theory. The results also illustrate the effects of surface structure similarity between base and derived sentence. In general, these effects were "local," that is, restricted to a small range on the similarity "continuum." The effects are clearly seen for problems involving the direct comparison of two sentences differing only in surface structure. For these cases, the effects are particularly noticeable at the extremes of the semantic similarity continuum. Thus, L is preferred over F, and unrelated same over unrelated diff. In the "middle" of the continuum, however, surface structure similarity made no difference to the rank ordering, as the false perm conditions were equivalent as were the false syn conditions. This "effect at the extremes" seems to be due to two quite different factors. The superiority of L over F can be attributed to a performance factor, the visual-spatial and surface structure equivalence of the L synonyms with thek base counterparts. This probably facilitated visual and memory processing and allowed Ss to more rapidly confirm the overall semantic similarity of L o A corollary explanation is that the different surface in F had a negative effect on Ss' judgments, independent of the rapidity of confirmation of similarity. By contrast, unrel same was apparently preferred to unrel diff because Ss were "cornered" and having no other basis for choice, opted for surface structure identity. Of course, Ss had no clearer choice in the "middle" of the continuum, where, the similarity to or identity of content words with the base, suppressed the influence of surface structure. Such suppression notwithstanding, lexical and surface structure also interacted. Recall, in this connection, that each judgment made by an S involved three sentences, a standard and two comparison sentences. This contextual feature of the task was highlighted when two conditions which differed only in surface structure were judged in different contexts. For example, in the context T, L fares better than F (Z = 1.70, P < 0.05), but in the context P this is reversed (Z = -2.50, P < 0 . 0 5 ) . Similar reversals in preference due to context can be observed among the nonparaphrases. Fkst, in the context false syn dill, false perm diff fares better than false perm same, but in the context unrelated same a reversal occurs (both Z s > 2 . 0 6 ,
Semantic Similarity Between Sentences
147
P < 0.05). Second, in the context F, false syn same does better than false syn dill, but a reversal occurs in the context false perm same (both Zs > 2.33, P < 0.05). Third, in the context false perm dill, unrelated same fares better than unrelated diff (Z = -2.16, P < 0 . 0 5 ) ; in fact, false perm diff is not significantly preferred over unrelated same. This last finding suggests a trade-off between surface and lexical similarity to the base. Lexical and surface structure also interacted to produce contrastlkke effects. As with the reversals, these effects tended to be limited to comparisons involving only nonparaphrases. For example, false sentences fare best against other nonparaphrases whose surface structure is the same but whose content items are next most similar to the base's. That is, false perm same fares best against false syn same, false perm diff against false syn dill, false syn same against unrelated same, and false syn cliff against unrelated diff. In general, interaction effects between lexical and surface structure can be described as follows. When comparison sentences shared a surface structure, Ss preferred whichever sentence contained content items more similar to base content items. Furthermore, they chose that sentence more often than its transformational counterpart when the latter was judged against the same context sentence. This analysis implies that Ss used surface structure and/or its written realization as a framework for making comparisons. This framework undoubtedly facilitated comparison of lexical meanings and therefore of sentential meaning. When surface structures differed, processing load probably increased, thereby increasing subject uncertainty and confusion regarding a choice. In short, it is suggested that processing load was highly influenced by surface similarity and that the load acted as a decision factor. This explanation appears to account for the reversal and contrast effects described above.
GENERAL DISCUSSION
The results indicate that surface structure similarity affects judgments of meaning differently than deep or lexical structure similarity. Surface form acts as a performance factor, as Ss used it as a perceptual framework for comparing content word similarity. By contrast, deep and lexical structure act as competence factors in the sense that they exerted powerful structural effects on judgments. Nevertheless, these three factors cannot completely account for the results. The results present at least two major difficulties whose nature will be sketched briefly. First, consider the fact that T, L, and F were highly preferred over all nonparaphrases. The superiority of T is presumably due to preservation of deep structure. But preservation of deep structure in Chomsky's (1965, 1969)
148
Honeck
theory requires preservation of major vocabulary items, and therefore L and F must retain something else. This something else seems to be the logical or "propositional structure" of the base. By this analysis, L and F are preferred over P because substitution of synonyms on a one-to-one basis does not unduly disturb intrinsic relationships between lexical features. T is preferred over L and F because it shares more lexical features with the base, not because it better preserves the relationship between these features. In other words, T preserves the propositional structure of the base by maintaining its deep structure. More generally, deep structure identity can be considered a special case providing for semantic similarity. On one level, this view is similar to McCawley's (1971) that the meaning of a statement can be divided into a proposition and a set of noun phrase descriptions. For example, the sentence, The man kissed the woman, consists of the proposition, X1 kissed X~, plus the noun phrase descriptions, X1 is a man, Xz is a woman. This view also blends with certain philosophical analyses of sentences. Lemmon (1971) says, for example, that, "inasmuch a s . . . two sentences are not synonymous, two distinct propositions have been expressed" (p. 240). However, McCawley believes that it is unnecessary to postulate an independent linguistic level between meaning and surface structure, whereas the present study seems to suggest such a level. Moreover, in an unreported study the author replicated the findings for T even though nonsense syllables were substituted for the content words under conditions which made it unlikely that the syllables could be recoded into real words. In other words, Ss were able to recognize logical identities between active and passive statements even though no real world reference (meaning?) was involved. Beret (1970) also argues for an "underlying logical structure of sentences" which is different somehow from the meaning of the sentence (p. 190). By his analysis, the same logical relations between actor-action-object may be expressed by a wide variety of surface forms. Bever further argues that complex transformation rules could relate these forms, including forms which differ greatly in lexical structure. Thus, his analysis may be capable of "accounting for" the paraphrastic status of L and F paraphrases. It is unlikely, however, that even the most liberalized grammatical transformation theory could account for the P paraphrase data, and this point brings us to the second major difficulty raised by the data. The P paraphrase was highly preferred over all nonparaphrases. This occurs even though P shares no propositional structure and only minimal texical structure with the base, whereas some nonparaphrases do share a lexicon with the base. Obviously some ways of deviating from the propositional structure of a source are more important than others. Olson (1970) has defined a paraphrase as, "an utterance which specifies the same intended referent as the statement for which it is substituted" (p. 260). The definition
Semantic Similarity Between Sentences
149
is somewhat restrictive but would characterize a P paraphrase as a more acceptable intended referent than any nonparaphrase. Exactly how P retains the "semantic structure" of the base is not clear, but certainly not by retaining the basic information in the base. P's meaning is extrinsic rather than intrinsic to the base. Its acceptability is dependent on Ss' grasp of the logical structure of the base, but it "goes beyond the information given" in the base. For this reason, P is a special sort of paraphrase, ff it is properly considered a paraphrase at all. In considering supposed (P) paraphrases such as, John's uncle, and, the person who is the brother o f John's mother or father, Chafe (1971), a linguist, makes a similar argument. He says these two utterances have quite different, "semantic structures whose meanings happen to be equivalent for some communicative purposes, though not for all . . . " (p. 17). Similarly, Chomsky (1965) says of apparent (P) paraphrases such as, John struck Bill, and Bill received a blow at the hands o f John, that "there is some still more abstract notion of 'semantic function' still unexplained" (p. 163). The same can be said of proverbs and their interpretations as, for example, A stitch in time saves nine, and an interpretation such as, l f you do your work now you'll save yourself a lot o f work later. In any case, the P data appear to support Bever's (1970) contention that, "Under the right circumstances (having more t i m e . . . ) both subjects and linguists make use of a deeper analysis of the logical forms internal to the sentences" (p. 195). Needless to say, it is not clear at this point how either specimen discovers the relationship between internal logical form and external semantic representation. A final word about future research in this area. The generalizability of the data is limited in at least two important ways. First, visual presentation of materials confounds auditory-linguistic surface form with its visual realization. Auditory presentation could help clarify the role of surface form but has the drawback of making heavy memory demands. The second limitation concerns materials. First, base sentences were constructed so that sensible nonparaphrases could be generated from them by noun rotation. However, Hamilton and Deese (1971) suggest that sentences in which all combinations of subjects and verbs are sensible may cause interference. Second, only passive transformations were used and, quite obviously, different L, F, and P paraphrases could have been used. Further investigation of "subspecies" of the paraphrases, particularly P, should prove rewarding. Finally, the experimental conditions were treated as though they were on a unidimensional continuum of semantic similarity. Whether such a continuum exists or not is unclear. The analysis of P paraphrases, in particular, suggests that unidimensionality is an unrealistic assumption. Kreitler and Kreitler (1968) define 13 dimensions of Iexical meaning. Hopefully, things are less complex for more definitive forms of linguistic communication such as the sentence.
150
Honeek
ACKNOWLEDGMENTS The author wishes to thank Patricia Perin Mandel for coding and analyzing data, for her comments on the paper, and for her general enthusiasm. Thanks are also due Donald A. Schumsky for his comments on the paper.
REFERENCES
Anisfeld, M. (1970). False recognition of adjective-noun phrases. J. Exp. Psychol. 86: 120-122. Anisfeld, M., and Knapp, M. (t968). Association, synonymity, and directionality in false recognition. J. Exp. Psychol. 77:171-179. Bach, E., and Harms, R. (eds.) Universals in Linguistic Theory. Holt, Rinehart, and Winston, New York. Begg, I. (1971). Recognition memory for sentence meaning and wording. J. Verb. Learn. Verb. Behav. 10: 114-119. Beret, T. (1970). The integrated study of language behavior, In Morton, J. (ed.), Biological and Social Factors in Psycholinguistics. Univ. of Illinois Press. Urbana. Bregman, A., and Strasberg, R. (1968). Memory for the syntactic form of sentences. J. Verb. Learn. Verb. Behav. 7: 396--403. Chafe, W. (1971). Directionality and paraphrase. Language 47: 1-26. Chomsky, N. (1965). Aspects o f the Theory of Syntax. M.I.T. Press, Cambridge, Mass. Chomsky, N. (1969). Deep structure, surface structure, and semantic interpretation. In Steinberg, D., and Jakobovits, L. (eds.), Semantics: an Interdisciplinary Reader in Philosophy, Linguistics, Anthropology, and Psychology. Cambridge Univ. Press, London. Clifton, C., and Odom, P. (1966). Similarity relations among certain English sentence constructions. Psychological Monographs 80 (5, Entire No. 613). Collins, A., and Quillian, M. (1969). Retrieval time from semantic memory. s Verb. Learn. Verb. Behav. 8: 240-247. Deese, J. (1965). The Structure of Associations in Language and Thought. Johns Hopkins Press, Baltimore. Fillenbaum, S. (1969). Words as feature complexes: False recognition of antonyms and synonyms. s Exp. Psyehol. 82: 400-402. Hamilton, H., and Deese, J. (1971). Comprehensibility and subject-verb relations in complex sentences. J. Verb. Learn. Verb. Behav. 10: 163-170. Honeck, R. (1971). A study of paraphrases. J. Verb~ Learn. Verb. Behav. 10: 367-381. Katz, J., and Postal, P. (1964). An Integrated Theory o f Linguistic Descriptions. M.I.T. Press, Cambridge, Mass. Kreitler, S., and Kreitler, H. (1968). Dimensions of meaning and their measurement. Psychol. Rep. 23: 1307-1329. Lemmon, E. (1971). Sentences, statements, and propositions. In Rosenberg, J.,and Travis, C. (eds.), Readings in the Philosophy o f Language. Prentice-Hall, Englewood Cliffs, N.J. Martin, E., and Roberts, K. (1966). Grammatical factors in sentence retention. J. Verb. Learn. Verb. Behav. 5: 211-218. Mawson, C. (1931). Roger's Thesaurus o f the English Language in Dictionary Form. Garden City Publishing, New York. McCawley, J. (1971). Meaning and the description of languages. In Rosenberg, J., and Travis, C. (eds.), Readings in the Philosophy of Language. Prentice-Hall, Englewood Cliffs, N.J.
Semantic Similarity Between Sentences
151
Miller, G. (1969). A psychological method to investigate verbal concepts. Z Math. Psychol. 6: 169-191. Olson, D. (1970). Language and thought: Aspects of a cognitive theory of semantics. Psychol. Rev. 77: 257-273. Osgood, C. (1968). Toward a wedding of insufficiencies. In Dixon, T., and Horton, D. (eds.), Verbal Behavior and General Behavior Theory. Prentice-Hal/, Englewood Cliffs, N.J. Perfetti, C. (1969). Lexical density and phrase structure depth as variables in sentence retention. J. Verb. Learn. Verb. Behav. 8: 719-724. Sachs, J. (1967). Recognition memory for syntactic and semantic aspects of connected discourse. Percep. Psychophys. 2: 437-442. Schaeffer, B., and Wallace, R. (1970). The comparison of word meaning. J. Exp.Psychol. 86: 144-152.
Semantic Similarity Between Sentences Richard P. H o n e c k I
Received June 14, 1972
The present study investigated the effects of deep, lexical, and surface structure relationships between sentences on ]udgments o f these sentences' semantic similarity. Ten sentence conditions, four paraphrases and six nonparaphrases, were derived from a base sentence. The four paraphrase types were transformational (T), a passive form of the base, lexical (L), containing synonyms for base content words, formalexic (F), a combination o f T and L types, and parasyntactic (P), one of several alternative interpretations o f the base. The six nonparaphrases consisted of three sets o f two sentences each: the false permutation sentences retained the base lexicon, the false synonymous sentences contained synonyms, and the unrelated sentences' lexicon was completely unrelated to the base. One sentence in each nonparaphrase set retained the base surface form and the other, a passivization, did not. Using a modified paired comparisons task, the following rank order o f conditions, in terms of preference, was obtained: T > L > F > P > false permutation > false synonymous > unrelated. It was concluded that deep structure similarity had potent effects but that a more complete description of the data required the postulation o f additional factors such as "propositional structure" and "semantic structure."
INTRODUCTION Imagine there exists a continuum ranging from complete identity to complete dissimilarity in meaning of connected discourse (sentences, in this discussion). The question then must be raised as to what aspect(s) of sentences form the basis for judgments of similarity on such a continuum. To the best of the author's knowledge, no study has explored this question in a comprehensive empirical manner. Traditional psychological approaches to semantic similarity have little bearing on this question as they have focused on the isolated word 1psychology Department, University of Cincinnati, Cincinnati, Ohio. 137 9 1973 Plenum Publishing Corporation, 227 West 17th Street, New York, N.Y. 10011.
138
Honeck
as the unit of analysis. For example, papers (Osgood, 1968; Fillenbaum, 1969; Anisfeld and Knapp, 1968, to mention only a few) abound in the literature which treat the word as a bundle of distinctive semantic features. On this view, word similarity is a function of feature overlap. Similarly, word synonymity has been analyzed in terms of associative overlap (Deese, 1965), images, concepts (Schaeffer and Wallace, 1970), branching trees (Collins and QuiUian, 1969), etc. By contrast, Miller (1969) specifies other bases of judgments of word similarity. Miller asked Ss to sort nouns, each defined by a short sentence, on the basis of semantic similarity. He attempted to account for the resulting clusters on the basis of "shared presuppositions" of the nouns and of "assertions" made by the sentence definitions. Thus, Miller seems to argue that, even on the word level, a list of word features or associations, etc., cannot explain similarity. Similarly, Anisfeld (1970) argues that phrases are "propositional entities" while the isolated word is less prescribed. Anisfeld found that more false recognition memory errors occurred to phrases (e.g., fat book) which were synonymous with preceding phrases (thick book) than to control phrases (history book) or antonymous phrases (thin book). The results of the Miller and the Anisfeld studies suggest that sentence similarity is a complex affair which cannot be handled solely in terms of the properties of words. Some extra-word, relational, gestaltlike factors must clearly be operating. A suggestion as to the nature of these factors is provided by Chomsky's (1965) linguistic theory, particularly by the concept of deep structure. Deep structure is an output of the base rules, which Chomsky defines as a set of categorial rules which "define grammatical functions and grammatical relations and determine an abstract underlying order," and a lexicon which, "characterizes the individual properties of particular lexical items that are inserted in specified positions in base phrase-markers" (p. 136). Deep structure is, in turn, an input to the semantic component, which renders a semantic interpretation, and to the syntactic transformation rules whose application results in the surface structure. Surface structure, which can be represented by an immediate constituent (phrase) structure analysis, is, in turn, an input to the phonological rules which render a phonetic interpretation. According to Chomsky's view, two sentences should theoretically receive identical interpretations when their deep structures are identical and different interpretations to the extent that their deep structures diverge [though Chomsky (1969) now feels that aspects of surface structure may affect meaning]. Aside from mere sentence repetition, which also repeats the lexical and surface structure, Katz and Postal (1964), who have elaborated on the semantic component in Chomsky's theory, assert that grammatical transformations preserve deep structure; hence, two transformationally related sentences
Semantic Similarity Between Sentences
139
should receive the same interpretation and, by implication, be judged identical in meaning. However, there appear to be forms of sentential synonymy whose existence does not rely on deep structure identity. This is so because many such cases involve either minimal or no lexical overlap between the sentences and, by Chomsky's definition, lexicat entries are part of the deep structure. The author (Honeck, 1971) has developed a taxonomy of paraphrases which bears on this issue. Aside from the transformational (7") type paraphrase, three additional types of paraphrase are delineated by the taxonomy. Substitution of synonyms for major vocabulary in a source (base) sentence results in a lexical (L) paraphrase. A formalexic (F) paraphrase is a combination of T and L types, being formed by both transformation and lexical substitution operations. A parasyntactic (P) paraphrase admits of no operational definition at present but can best be considered one of many possible interpretations of a base sentence. Note that only T preserves the major vocabulary of the base and only L preserves its surface structure. These four types are illustrated in Table I. Honeck found that college students rated visually presented T type paraphrases most similar in meaning to their base, F types least similar, with L types intermediate. Moreover, these paraphrases had generally parallel effects when presented auditorily in a memory-comprehension task. The P paraphrase was not investigated. And, unfortunately , neither experiment included nonparaphrase conditions. For this reason, a word matching strategy interpretation of the ratings could not be precluded. This strategy would entail Ss rating as most similar to the base that paraphrase, T, which retained the base content words. Therefore, the present study included nonparaphrase conditions which changed the deep structure of their base and either maintained or changed its content words, and, orthogonally, either maintained or changed Table I. A Base Sentence and Derived Conditions
Base: The struggle evoked the feelings that changed the lad. T: The feelings that changed the lad were evoked by the struggle. L: The fight produced the emotions that altered the boy. F: The emotions that altered the boy were produced by the fight. P: Because the lad was in the war he became cold and unemotional. False Perm Same: The lad evoked the struggle that changed the feelings. False Perm Dill: The struggle that changed the feelings was evoked by the lad. False Syn Same: The boy elicited the fight that altered the emotions. False Syn Diff: The fight that altered the emotions was elicited by the boy. Unrelated Same: The lunatic trapped the person that annoyed the onlookers. Unrelated Dill: The person that annoyed the onlookers was trapped by the lunatic.
140
Honeck
its surface structure. The base lexicon was changed in two ways, either by replacing it with synonyms or with completely unrelated words. Six nonparaphrase sentence conditions resulted from this procedure. Two, the false permutation conditions, retained the base lexicon, two, the false synonymous conditions, contained synonyms, and two, the unrelated conditions, contained content items completely unrelated to those of the base. One of the sentences within each of these three sets of nonparaphrase retained the basic surface structure and one did not. Briefly, the S's task on a given trial was to decide which of two visually presented comparison sentences (conditions) best preserved the basic information of the base (standard) sentence. Expectations of results are based on Honeck's (1971) data and on the premise that deep structure should have by far the strongest effect, lexicat structure the next strongest effect, while surface structure effects should be confined to comparison sentences located in the same region on the similarity continuum. Moreover, L and F as well as T seem to retain the logical or "propositional structure" of the base whereas P and all nonparaphrases do not. These considerations predict the the following general ranking of conditions in terms of preference: T > L > F > P > f a l s e perm" > false synonymous > unrelated The rank of P is predicted, frankly, on commonsense grounds.
METHOD Subjects The Ss were 94 students from introductory psychology courses at the University of Cincinnati, 3 graduate students, 2 professors, and 1 secretary-a total of 100. All Ss were native speakers of English whose ages ranged from 18 to 40 years.
Materials Four base (standard) sentences with identical right branching surface structures were constructed. Each had the form the-noun1 - verb - the-noun2 that - verb - the-noun3. The mean log Thorndike-Lorge content (noun and verb) word frequency was 2.43, 2.58, 2.67, and 1.93 occurrences per 489 million for base sentences 1, 2, 3, and 4, respectively. The bases were made somewhat complex by minimizing intrasentence word associations and semantic constraints. Four types of paraphrase T, L, F, and P, and six types of nonparaphrase of each base were also constructed. These 10 conditions are described below. One base sentence and the 10 conditions relating to it are illustrated in Table I.
Semantic Similarity Between Sentences
141
Transformational (T) This paraphrase was a passive form of the base. Thus, the deep structure and lexical (content word) structure of the base is retained but its surface structure is changed.
Lexical (L) The surface structure of the base is retained but synonyms were substituted for its content words. Only content words are substituted for because they carry most of the cognitive load in a sentence (Perfetti, 1969). Moreover, it is much more difficult to find a good one-word synonym for function words in a sentence context. The synonyms were selected with the help of Roget's Thesaurus (Mawson, 1931) with the requirement that they be on the same semantic level as their source. That is, source and synonym did not appear to contain different semantic markers, at least at a high level in the semantic tree. For example, boy rather than male or person was substituted for lad. The mean log frequency of occurrence per 489 million was 2.71, 2.52, 2.72, and 2.07, respectively, for the four corresponding L paraphrases of bases 1, 2, 3, and 4.
Formalexic CF) This paraphrase was a passive form of the base with synonyms substituted for its content words. The surface structure was the same as that used for the T paraphrase and the content words were the same as those used in the L paraphrase.
Parasyntactic (PJ Although it is impossible at present to specify formally the semantic relationship between base and paraphrase, this paraphrase can be considered one of many alternative concrete interpretations of the base. That is, a number of different P paraphrases of each base might have been constructed, each rendering a totally different interpretation of the base. This P paraphrase had a different surface structure and a different deep stracture than the base. It was 1-5 words longer than the base and shared, at most, two content words with it. The mean log frequency of the content words was 2.95, 2.71, 2.79, and 2.79 respectively for the corresponding P paraphrases of bases 1, 2, 3, and 4.
142
Honeck
False Permutation, Same Surface Structure (False Perm Same) The three nouns in the base sentence were permuted such that noun1 was moved to the noun2 position, noun2 to noun3, and noun3 to the noun1 position. The base verbs had been chosen so that selection restriction rules (Chomsky, 1965) were not violated by the permutation, thereby avoiding semantically anomalous sentences. The permutation naturally changed the deep structure of the base but retained its surface and lexical structure. False Permutation, Different Surface Structure (False Perm Dill) This condition represents a passive form of the false perm same condition. These sentences therefore retained the lexical structure of their base but not its deep or surface structure. False Synonymous, Same Surface Structure (False Syn Same) Synonyms were substituted for the content words in the false perm same sentences. The synonyms were the same as those used in the L paraphrase of the base. In effect, the propositional structure of the base was changed, its surface structure was retained, and synonyms replaced its content words. False Synonymous, Different Surface Structure (False Syn Diff) These sentences represent a passive form of the false syn same sentences. Thus, the propositional and surface structures of the base were changed and synonyms replaced its content words.
Unrelated, Same Surface Structure (Unrelated Same) These sentences were completely unrelated in propositional or lexical structure to the base but retained its surface structure. They were, in fact, the other base sentences. Unrelated, Different Surface Structure (Unrelated l)iff) These sentences represent a passive form of the unrelated same sentences. Thus, their propositional, surface, and lexical structures were totally unrelated to that of their base. As a final point of clarification note that L and F are related
Semantic Similarity Between Sentences
143
transformationally and presumably, therefore, do not differ in deep structure. The same applies to the paired sentences in the false perm, the false syn, and the unrel conditions. Design and Procedure
A mixed design was used. Four groups of 25 Ss each contributed data to each of the 10 conditions relating to a particular base. Thus, base sentence was the between-S factor and was crossed with sentence conditions, the within-S factor. A modified paired comparisons procedure was used. The Ss received an 11-page booklet. The first page instructed them to decide which sentence in each of a series of sentence pairs best preserved the basic information in the standard (base) sentence. Full comprehension of the standard and comparison pair was emphasized as was the importance of regarding each standard and adjoining pair as a separate problem. Five problem pairs were presented on each of nine subsequent pages. The problems were numbered from 1 to 45 with the standard typed alongside each number. The pairs were typed immediately beneath the standard and were labeled A and B. The S entered his choice (A or B) on an answer sheet at the back of the booklet. The entire task was self-paced (Ss took the booklet home) in order to facilitate comprehension and required approximately 30-45 min to complete. The 45 possible pairs of the 10 sentence conditions were presented in random order with the exception that no condition occurred successively in more than two pairs. The order of pairs was the same within a group but varied between groups. For each group a sentence condition occurred equally often in the A (upper) and B (lower) positions across pairs. An attempt was also made to present each sentence condition within each block of five problems.
RESULTS AND DISCUSSION
The degree of preference for the conditions was tabulated for each of the four groups. Inspection of these data revealed very little variance between groups for judgments involving only paraphrases. Although the between group variance appeared larger for judgments involving only nonparaphrases, an analysis of variance revealed no differences between the groups, F(3,56) < 1.00. The data from the four groups was therefore pooled to form a single percentage preference value for each of the 45 comparisons. These values are presented in Table 2I. Complementary values appear above and
144
Honeck
Table II. Percent Preference of Row Condition over Column Conditiona T T L F P FPS FPD FSS FSD US UD
14 7 6 1 3 1 2 1 0
L
F
P
FPS
FPD
FSS
FSD
US
UD
86 b
93 b 86b
94 b 80b 89b
99b 96 b 94 b 86b
97 b 97 b 97 b 85 b 56
99 b 99 b 91b 84b 77b 73b
98 b 96b 100 b 84b 66 c 78b 51
99 b 97 b 96 b 91 b 65 c 56 63 c 59 d
100b 99 b 97 b 87 b 62 c 64 c 59d 64 c 76 b
14 20 4 3 1 4 3 1
11 6 3 9 0 4 3
14 15 16 16 9 13
44 23 34 35 38
27 22 44 36
49 37 41
41 36
24
aThe abbreviations for the nonparaphrase conditions are FPS (false perm same), FPD (false perm diff), FSS (false syn same), FSD (false syn diff), US (unrelated same), UD (unrelated diff). bp < 0.001. cp < 0.01, one tailed. dp < 0.05, one tailed. below the major diagonal and have been included for convenience. The P levels are based on Z-test values for correlated measures. Inspection o f the table reveals some general patterns. First, the rank order of the conditions conformed to expectations and was as follows: T > L
> F > P > false perm same = false perm cliff > false syn same = false syn diff > unrelated same > unrelated diff. Transitivity holds in every case with the exception that false perm diff is not significantly perferred over unrelated same. The table also indicates that all paraphrases are strongly preferred over all nonparaphrases. The T, L, and F paraphrases are preferred, at 'a minimum, 91% o f the time over nonparaphrases, this minimum dropping slightly to 84% for the P paraphrase. The ordering of the paraphrases themselves also reflects strong preferences. In fact, the range of preference is 80%, for L over P, to 94% for T over P. The preferences within the nonparaphrases are generally weaker as the maximum value is 78% for false perm d i f f over false syn diff. The general pattern of results replicate, extend, and clarify Honeck's (1971) rating data on paraphrases. Both studies found T superior to L and L to F . The present study also demonstrated that P is the least preferred type of paraphrase although it is much preferred over all nonparaphrases. The paraphrase vs. nonparaphrase results also rule out a word matching interpretation o f Honeck's rating data. The content words in the T and false perm conditions were identical yet T was highly preferred. Similarly, the L, F, and false syn conditions shared content words yet L and F were preferred.
Semantic Similarity Between Sentences
145
Obviously, T, L, and F have characteristics which overshadow the factor of lexical similarity between base and derived sentence. As such, the present data provide additional empirical justification for the proposed taxonomy of paraphrases. We can now consider in more detail the effects of deep structure, lexical structure, and surface structure similarities between the standard and the derived sentence conditions. As expected, the deep structure factor was a potent influence on Ss judgments, the lexical factor ran a weak second, and surface structure had only "local" effects. These factors interacted, but we can first consider their effects in relative isolation. Concerning deep structure, Katz and Postal's (1964) theory specifies T as the only condition which retains the deep structure, and therefore, the meaning of the base. The clear superiority of T over all other conditions vindicates their analysis. The superiority of T is consonant with the findings of other investigators, all of whom have used different tasks. For example, using ranking and recognition procedures, Clifton and Odom (1966) found that passive sentences were very similar to their simple - active - affirmative declarative (SAAD) counterparts. And Martin and Roberts' (1966) Ss often recalled a T paraphrase of the input sentence. Similarly, after 30 sec, Sachs' (1967) Ss could more easily recognize a change in the deep structure than the surface structure of a sentence which had been embedded in paragraph context. Bregman and Strasberg (1968) analyzed Ss' second attempts to identify input sentences on a forced-choice recognition test and found that Ss tended to guess T paraphrases for the input (e.g., passive for SAAD). Begg (1971) also reports that in a recognition test, sentences identical to the original were judged no more faithful in meaning to the original than were (T) paraphrases. In sum, the data indicate that T paraphrases are often confused with and substituted for their source in memory tasks. That these paraphrases are judged most similar in meaning to their source in nonmemorial tasks suggests that the memory effects may be obligatory. In any case, the mind seems to regard surface structure variation in a cavalier fashion whenever semantic information processing is emphasized, as it was in the present task. The effect of lexical structure similarity between base and derived sentence is evident for paraphrases and nonparaphrases alike. Among the paraphrases, T and F have the same surface yet T is preferred. In general, that paraphrase condition, T, which retains the base lerdcon is most preferred, those conditions, L and F, containing synonyms of base words are next most preferred, and P, most of whose content words are totally unrelated to the base's, is least preferred, though P supplants all nonparaphrases. A similar trend is apparent among the nonparaphrases. The false perm conditions, which retain the base words, are most preferred, the ,false syn conditions are
146
Honeck
intermediate, and the unrelated conditions are least preferred. Moreover, when surface structure is held constant among the nonparaphrases, the results false perm same >false syn same > unrelated same, and false perm diff >false syn d i f f > unrelated diff clearly demonstrate the effects of the lexical factor. Since the false perm conditions are equivalent and both are significantly preferred over the false syn conditions, which are also equivalent, the conclusion follows that the lexical factor is solely responsible for the ordering of these four conditions. The general finding that lexical structure takes precedence over surface structure hardly offends common sense and accords with the spirit of Chomsky's (1965) theory. The results also illustrate the effects of surface structure similarity between base and derived sentence. In general, these effects were "local," that is, restricted to a small range on the similarity "continuum." The effects are clearly seen for problems involving the direct comparison of two sentences differing only in surface structure. For these cases, the effects are particularly noticeable at the extremes of the semantic similarity continuum. Thus, L is preferred over F, and unrelated same over unrelated diff. In the "middle" of the continuum, however, surface structure similarity made no difference to the rank ordering, as the false perm conditions were equivalent as were the false syn conditions. This "effect at the extremes" seems to be due to two quite different factors. The superiority of L over F can be attributed to a performance factor, the visual-spatial and surface structure equivalence of the L synonyms with thek base counterparts. This probably facilitated visual and memory processing and allowed Ss to more rapidly confirm the overall semantic similarity of L o A corollary explanation is that the different surface in F had a negative effect on Ss' judgments, independent of the rapidity of confirmation of similarity. By contrast, unrel same was apparently preferred to unrel diff because Ss were "cornered" and having no other basis for choice, opted for surface structure identity. Of course, Ss had no clearer choice in the "middle" of the continuum, where, the similarity to or identity of content words with the base, suppressed the influence of surface structure. Such suppression notwithstanding, lexical and surface structure also interacted. Recall, in this connection, that each judgment made by an S involved three sentences, a standard and two comparison sentences. This contextual feature of the task was highlighted when two conditions which differed only in surface structure were judged in different contexts. For example, in the context T, L fares better than F (Z = 1.70, P < 0.05), but in the context P this is reversed (Z = -2.50, P < 0 . 0 5 ) . Similar reversals in preference due to context can be observed among the nonparaphrases. Fkst, in the context false syn dill, false perm diff fares better than false perm same, but in the context unrelated same a reversal occurs (both Z s > 2 . 0 6 ,
Semantic Similarity Between Sentences
147
P < 0.05). Second, in the context F, false syn same does better than false syn dill, but a reversal occurs in the context false perm same (both Zs > 2.33, P < 0.05). Third, in the context false perm dill, unrelated same fares better than unrelated diff (Z = -2.16, P < 0 . 0 5 ) ; in fact, false perm diff is not significantly preferred over unrelated same. This last finding suggests a trade-off between surface and lexical similarity to the base. Lexical and surface structure also interacted to produce contrastlkke effects. As with the reversals, these effects tended to be limited to comparisons involving only nonparaphrases. For example, false sentences fare best against other nonparaphrases whose surface structure is the same but whose content items are next most similar to the base's. That is, false perm same fares best against false syn same, false perm diff against false syn dill, false syn same against unrelated same, and false syn cliff against unrelated diff. In general, interaction effects between lexical and surface structure can be described as follows. When comparison sentences shared a surface structure, Ss preferred whichever sentence contained content items more similar to base content items. Furthermore, they chose that sentence more often than its transformational counterpart when the latter was judged against the same context sentence. This analysis implies that Ss used surface structure and/or its written realization as a framework for making comparisons. This framework undoubtedly facilitated comparison of lexical meanings and therefore of sentential meaning. When surface structures differed, processing load probably increased, thereby increasing subject uncertainty and confusion regarding a choice. In short, it is suggested that processing load was highly influenced by surface similarity and that the load acted as a decision factor. This explanation appears to account for the reversal and contrast effects described above.
GENERAL DISCUSSION
The results indicate that surface structure similarity affects judgments of meaning differently than deep or lexical structure similarity. Surface form acts as a performance factor, as Ss used it as a perceptual framework for comparing content word similarity. By contrast, deep and lexical structure act as competence factors in the sense that they exerted powerful structural effects on judgments. Nevertheless, these three factors cannot completely account for the results. The results present at least two major difficulties whose nature will be sketched briefly. First, consider the fact that T, L, and F were highly preferred over all nonparaphrases. The superiority of T is presumably due to preservation of deep structure. But preservation of deep structure in Chomsky's (1965, 1969)
148
Honeck
theory requires preservation of major vocabulary items, and therefore L and F must retain something else. This something else seems to be the logical or "propositional structure" of the base. By this analysis, L and F are preferred over P because substitution of synonyms on a one-to-one basis does not unduly disturb intrinsic relationships between lexical features. T is preferred over L and F because it shares more lexical features with the base, not because it better preserves the relationship between these features. In other words, T preserves the propositional structure of the base by maintaining its deep structure. More generally, deep structure identity can be considered a special case providing for semantic similarity. On one level, this view is similar to McCawley's (1971) that the meaning of a statement can be divided into a proposition and a set of noun phrase descriptions. For example, the sentence, The man kissed the woman, consists of the proposition, X1 kissed X~, plus the noun phrase descriptions, X1 is a man, Xz is a woman. This view also blends with certain philosophical analyses of sentences. Lemmon (1971) says, for example, that, "inasmuch a s . . . two sentences are not synonymous, two distinct propositions have been expressed" (p. 240). However, McCawley believes that it is unnecessary to postulate an independent linguistic level between meaning and surface structure, whereas the present study seems to suggest such a level. Moreover, in an unreported study the author replicated the findings for T even though nonsense syllables were substituted for the content words under conditions which made it unlikely that the syllables could be recoded into real words. In other words, Ss were able to recognize logical identities between active and passive statements even though no real world reference (meaning?) was involved. Beret (1970) also argues for an "underlying logical structure of sentences" which is different somehow from the meaning of the sentence (p. 190). By his analysis, the same logical relations between actor-action-object may be expressed by a wide variety of surface forms. Bever further argues that complex transformation rules could relate these forms, including forms which differ greatly in lexical structure. Thus, his analysis may be capable of "accounting for" the paraphrastic status of L and F paraphrases. It is unlikely, however, that even the most liberalized grammatical transformation theory could account for the P paraphrase data, and this point brings us to the second major difficulty raised by the data. The P paraphrase was highly preferred over all nonparaphrases. This occurs even though P shares no propositional structure and only minimal texical structure with the base, whereas some nonparaphrases do share a lexicon with the base. Obviously some ways of deviating from the propositional structure of a source are more important than others. Olson (1970) has defined a paraphrase as, "an utterance which specifies the same intended referent as the statement for which it is substituted" (p. 260). The definition
Semantic Similarity Between Sentences
149
is somewhat restrictive but would characterize a P paraphrase as a more acceptable intended referent than any nonparaphrase. Exactly how P retains the "semantic structure" of the base is not clear, but certainly not by retaining the basic information in the base. P's meaning is extrinsic rather than intrinsic to the base. Its acceptability is dependent on Ss' grasp of the logical structure of the base, but it "goes beyond the information given" in the base. For this reason, P is a special sort of paraphrase, ff it is properly considered a paraphrase at all. In considering supposed (P) paraphrases such as, John's uncle, and, the person who is the brother o f John's mother or father, Chafe (1971), a linguist, makes a similar argument. He says these two utterances have quite different, "semantic structures whose meanings happen to be equivalent for some communicative purposes, though not for all . . . " (p. 17). Similarly, Chomsky (1965) says of apparent (P) paraphrases such as, John struck Bill, and Bill received a blow at the hands o f John, that "there is some still more abstract notion of 'semantic function' still unexplained" (p. 163). The same can be said of proverbs and their interpretations as, for example, A stitch in time saves nine, and an interpretation such as, l f you do your work now you'll save yourself a lot o f work later. In any case, the P data appear to support Bever's (1970) contention that, "Under the right circumstances (having more t i m e . . . ) both subjects and linguists make use of a deeper analysis of the logical forms internal to the sentences" (p. 195). Needless to say, it is not clear at this point how either specimen discovers the relationship between internal logical form and external semantic representation. A final word about future research in this area. The generalizability of the data is limited in at least two important ways. First, visual presentation of materials confounds auditory-linguistic surface form with its visual realization. Auditory presentation could help clarify the role of surface form but has the drawback of making heavy memory demands. The second limitation concerns materials. First, base sentences were constructed so that sensible nonparaphrases could be generated from them by noun rotation. However, Hamilton and Deese (1971) suggest that sentences in which all combinations of subjects and verbs are sensible may cause interference. Second, only passive transformations were used and, quite obviously, different L, F, and P paraphrases could have been used. Further investigation of "subspecies" of the paraphrases, particularly P, should prove rewarding. Finally, the experimental conditions were treated as though they were on a unidimensional continuum of semantic similarity. Whether such a continuum exists or not is unclear. The analysis of P paraphrases, in particular, suggests that unidimensionality is an unrealistic assumption. Kreitler and Kreitler (1968) define 13 dimensions of Iexical meaning. Hopefully, things are less complex for more definitive forms of linguistic communication such as the sentence.
150
Honeek
ACKNOWLEDGMENTS The author wishes to thank Patricia Perin Mandel for coding and analyzing data, for her comments on the paper, and for her general enthusiasm. Thanks are also due Donald A. Schumsky for his comments on the paper.
REFERENCES
Anisfeld, M. (1970). False recognition of adjective-noun phrases. J. Exp. Psychol. 86: 120-122. Anisfeld, M., and Knapp, M. (t968). Association, synonymity, and directionality in false recognition. J. Exp. Psychol. 77:171-179. Bach, E., and Harms, R. (eds.) Universals in Linguistic Theory. Holt, Rinehart, and Winston, New York. Begg, I. (1971). Recognition memory for sentence meaning and wording. J. Verb. Learn. Verb. Behav. 10: 114-119. Beret, T. (1970). The integrated study of language behavior, In Morton, J. (ed.), Biological and Social Factors in Psycholinguistics. Univ. of Illinois Press. Urbana. Bregman, A., and Strasberg, R. (1968). Memory for the syntactic form of sentences. J. Verb. Learn. Verb. Behav. 7: 396--403. Chafe, W. (1971). Directionality and paraphrase. Language 47: 1-26. Chomsky, N. (1965). Aspects o f the Theory of Syntax. M.I.T. Press, Cambridge, Mass. Chomsky, N. (1969). Deep structure, surface structure, and semantic interpretation. In Steinberg, D., and Jakobovits, L. (eds.), Semantics: an Interdisciplinary Reader in Philosophy, Linguistics, Anthropology, and Psychology. Cambridge Univ. Press, London. Clifton, C., and Odom, P. (1966). Similarity relations among certain English sentence constructions. Psychological Monographs 80 (5, Entire No. 613). Collins, A., and Quillian, M. (1969). Retrieval time from semantic memory. s Verb. Learn. Verb. Behav. 8: 240-247. Deese, J. (1965). The Structure of Associations in Language and Thought. Johns Hopkins Press, Baltimore. Fillenbaum, S. (1969). Words as feature complexes: False recognition of antonyms and synonyms. s Exp. Psyehol. 82: 400-402. Hamilton, H., and Deese, J. (1971). Comprehensibility and subject-verb relations in complex sentences. J. Verb. Learn. Verb. Behav. 10: 163-170. Honeck, R. (1971). A study of paraphrases. J. Verb~ Learn. Verb. Behav. 10: 367-381. Katz, J., and Postal, P. (1964). An Integrated Theory o f Linguistic Descriptions. M.I.T. Press, Cambridge, Mass. Kreitler, S., and Kreitler, H. (1968). Dimensions of meaning and their measurement. Psychol. Rep. 23: 1307-1329. Lemmon, E. (1971). Sentences, statements, and propositions. In Rosenberg, J.,and Travis, C. (eds.), Readings in the Philosophy o f Language. Prentice-Hall, Englewood Cliffs, N.J. Martin, E., and Roberts, K. (1966). Grammatical factors in sentence retention. J. Verb. Learn. Verb. Behav. 5: 211-218. Mawson, C. (1931). Roger's Thesaurus o f the English Language in Dictionary Form. Garden City Publishing, New York. McCawley, J. (1971). Meaning and the description of languages. In Rosenberg, J., and Travis, C. (eds.), Readings in the Philosophy of Language. Prentice-Hall, Englewood Cliffs, N.J.
Semantic Similarity Between Sentences
151
Miller, G. (1969). A psychological method to investigate verbal concepts. Z Math. Psychol. 6: 169-191. Olson, D. (1970). Language and thought: Aspects of a cognitive theory of semantics. Psychol. Rev. 77: 257-273. Osgood, C. (1968). Toward a wedding of insufficiencies. In Dixon, T., and Horton, D. (eds.), Verbal Behavior and General Behavior Theory. Prentice-Hal/, Englewood Cliffs, N.J. Perfetti, C. (1969). Lexical density and phrase structure depth as variables in sentence retention. J. Verb. Learn. Verb. Behav. 8: 719-724. Sachs, J. (1967). Recognition memory for syntactic and semantic aspects of connected discourse. Percep. Psychophys. 2: 437-442. Schaeffer, B., and Wallace, R. (1970). The comparison of word meaning. J. Exp.Psychol. 86: 144-152.
