Journal of Consulting and Clinical Psychology 1978, Vol. 46, No. 4, 620-628
Common Methodological Problems in MMPI Research James N. Butcher and Auke Tellegen University of Minnesota Research with the Minnesota Multiphasic Personality Inventory (MMPI) continues at a high rate. Unfortunately, too many articles submitted and even accepted for publication are methodologically weak. In this article we discuss some common methodological problems involving the use of the MMPI, encountered in the course of reviewing articles submitted for publication. A number of relevant issues are discussed, and some suggestions for improving research designs are made. Research with the Minnesota Multiphasic Personality Inventory (MMPI) has continued at a high level in recent years. As a self-report inventory that includes several measures of psychopathology, the MMPI, or some derivative of it, has become widely used as a descriptive instrument or a criterion measure in a vast array of clinical and research investigations. In 1972 Euros reported that over 200 books and articles on the MMPI are published annually. Dahlstrom, Welsh, and Dahlstrom (197S) cited over 6,000 references to the MMPI; Butcher and Pancheri (1976) reported over 600 recent references in cross-national MMPI research alone. Butcher and Owen (in press) recently reviewed and classified MMPI research for the past 5 years. As shown in Table 1, they found that over one fourth of the research was focused on two areas of "popular" investigation: alcohol and drug abuse, and crime and delinquency. Approximately 37% of the studies focused on the use of the MMPI to study nonpsychiatric populations—medical patients, parents, women, ethnic groups, college students, and the aged. The MMPI seems to have been widely accepted as a criterion measure of psychopathology by researchers who want to We wish to thank David Lykkcn for commenting on an earlier version of this article. Requests for reprints should be sent to James N. Butcher, Department of Psychology, Elliott Hall, University of Minnesota, 75 East River Road, Minneapolis, Minnesota S545S.
measure psychological problems in a variety of nonpsychiatric groups. But studies pointing to the need for revision or modification of the MMPI are scarce. This absence of revisionoriented research is impressive, since the MMPI is around 40 years old and can be expected to show some signs of aging. The only serious efforts to modify the MMPI have been directed at shortening it: About 12% of the published research in the past 5 years concerned the development or use of shorter versions of the instrument. There are many reasons for the extensive use of the MMPI: Its administration is relatively effortless; its scoring is objective; generally straightforward objective interpretive procedures are available; and its validity as a criterion measure is comparatively well founded. But some of the factors encouraging researchers to use a self-report personality measure such as the MMPI also result in the kinds of methodological flaws that this article may help to prevent. The MMPI is so easy to administer as a part of an ongoing study that researchers (even if they have little or no research background with the instrument) feel encouraged to include it. Furthermore, since good criteria in psychopathology or personality research are scarce, the MMPI scales are sometimes too readily adopted as expedient proximate criteria. (In some respects, one might almost consider the MMPI an "attractive nuisance" in the legal sense, to be approached with appropriate caution.) The remainder of this article will focus on
Copyright 1978 by the American Psychological Association, Inc. 0022-006X/78/4604-0620$00.75
620
METHODOLOGICAL PROBLEMS
some methodological problems that are often associated with research using the MMPI. We hope to provide some guidelines that could help avoid these difficulties, by providing minimal criteria. Some of these criteria emerged as a result of our experiences as reviewers of MMPI manuscripts that were submitted for publication in several psychological journals. It will not be possible to provide an exhaustive review of all pertinent issues nor to avoid seeming at times somewhat dogmatic. Yet it is not our intention to "dictate" which areas of investigation are important or which ones should be avoided, or to impose idiosyncratic preferences for certain research methods when several defensible ones are available. What Kind oj Instrument is the MMPI? The MMPI is often mistakenly considered to be an all-purpose personality assessment instrument that is sensitive to "normal range" personality attributes. Consequently, some researchers use the MMPI with groups for which a different instrument might be more appropriate. The standard clinical MMPI scales are measures of psychopathology, not general personality. The MMPI should not be made to do what it is not designed to do. Rather than use the MMPI inappropriately, it would be better to consider alternative or supplementary measures such as the California Psychological Inventory (CPI) or Personality Research Form, which focus on normal-range personality dimensions. What kind of MMPI measures can or should be used? The eight clinical scales and the three validity scales plus Mi and Si are the best established and most widely used MMPI scales.1 In view of the many problems that we have encountered in the computation of clinical scale scores, two suggestions are in order: 1. In research computations more understandable results will often be obtained if one does not use the K correction, but uses K as a separate indicator. This simply means that K'& validity as a "suppressor" is not assumed. Of course, appropriate multivariate analysis of one's data will permit determination whether K in fact does function as a suppressor in one's particular data set. The K
621
Table 1 Classification of Recent Minnesota Multiphasic Personality Inventory Research Category Alcohol and drug abuse: detection and treatment Short forms Use with other tests or rating scales Crime and delinquency Diagnostic considerations: rules, profiles, and code types Psychometric characteristics Medical patients: physical disorders and
symptoms Correlation with diverse criteria Women Treatment variables and therapy outcome Depression and suicide College students and adolescents Parents, couples, and families Drug therapy: choice and effect Race: cross-cultural, ethnic Automated interpretation Employment screening and job performance Anxiety and stress Brain damage Sleep Sexual deviation Aging
No. articles
96 66 62 56 SO 49
47 46 42 38 38 34 33 31 31 26 24 18 18 10 9 5
Note. Adapted from Butcher and Owen (in press).
score will often be found to be an additional measure of good adjustment rather than a measure of invalid variance. If non-K-corrected T scores are used, use the correct conversion tables. Occasionally manuscripts are received containing figures with non-/£-corrected profiles drawn according to incorrect norms. Do not use the raw score indications printed on the standard profile sheet unless your scores are actually K corrected! If you are plotting non-A"-corrected scores for Scales Hs, Pd, Pt, Sc, and Ma and plan to display these on the standard profile sheet, it is necessary to obtain the correct T-score elevation from a conversion table (e.g., in Dahlstrom, Welsh, & Dahlstrom, 1972) and draw the appropriate elevation from the T-score indica1
Researchers interested in obtaining access to the item response data of the original Minnesota standardization sample should contact W. Grant Dahlstrom, Department of Psychology, University of North Carolina, Chapel Hill, North Carolina 27S14.
622
JAMES N. BUTCHER AND AUKE TELLEGEN
tions on the side of the profile sheet. In spite of the above general recommendation about using non-^-corrected scores, it is sometimes useful for comparisons with other studies to do a parallel set of analyses using ^-corrected scores. 2. Use raw scores in research computations rather than T scores unless the use of T scores is specifically indicated, for example, in the analysis of code types or when combining data from both sexes. The eight clinical scales were developed primarily by means of empirical keying methods, using clinical diagnoses as criteria. Because of the primary reliance on external criteria rather than on internal structure, these scales are for the most part quite heterogeneous. Consequently, although these scales, particularly when used in combination, provide useful information as to the diagnostic status of a person (Dahlstrom et al., 1972; Overall, Note 1), they do not in any simple way reflect "what the patient is saying," that is, how the patient is describing herself/himselj through the medium of the MMPI items. This, too, is important information. Nowadays we recognize more clearly than in the heyday of empirical keying and "dust bowl empiricism" that important and clinically relevant direct relations exist between the manifest content of a person's self-description and clinical symptomatology and behavior in general (e.g., Hase & Goldberg, 1967; Jackson, 1971; Koss & Butcher, 1973; Payne & Wiggins, 1972). In addition, in clinical use the client/patient's reply to MMPI questions, impersonal as the process may seem, is still a communication that he/she might expect the psychologist to receive. We therefore need to take steps to make this possible. Some organized way of "coding" the respondent's MMPI "message" is clearly called for. Two approaches are now open: 1. use of content homogeneous scales, including factor or cluster scales (Block, 1965; Eichman, 1962; Overall, Hunter, & Butcher, 1973; Stein, 1968; Welsh, 1956; Wiggins, 1969) and 2. use of so-called "critical items" (Grayson, 1951; Koss & Butcher, 1973; Koss, Butcher, & Hoffmann, 1976).
Both approaches can serve a very useful role complementary to that of the clinical scales. Scale Proliferation The development of additional MMPI scales has been a rather popular pastime for psychologists. There are presently more MMPI scales than there are items on the inventory! Unfortunately, much of the MMPI scale development does not derive from a sound conceptual framework. Many scales have been constructed by contrasting different "samples of convenience" (often of heterogeneous makeup or with important characteristics unknown). Often these scales are not cross-validated, and more often than not their psychometric properties and interrelations with other scales are not reported. Many new scales, since they are, after all, derived from the same item pool, prove to be largely redundant alternative versions of existing scales, although sometimes of poorer quality. A researcher who is tempted to add to this plethora of scales would find his or her contribution to the MMPI literature better received and used if: 1. The investigator has very carefully considered the question of whether the MMPI item pool adequately covers the domain of content of the construct to be measured. If the answer is not affirmative, then the MMPI will obviously not serve the investigator's purpose very well, and the new scale will reflect more the characteristics of the MMPI item pool than the domain of the intended construct. We believe that this is in fact the case with some new MMPI scales. 2. The investigator can show the scale to be conceptually interesting. 3. The scale is developed and cross-validated on reasonably large, well-defined samples (whether it is an empirically keyed scale or one derived by internal consistency methods, e.g., factor analysis). 4. The internal structure of the scale is reported (internal consistency, possibly factor structure), and the relation to other MMPI scales is reported. These relations along with validity data would ideally reveal that the scale either is a superior alternative to an
METHODOLOGICAL
existing MMPI scale or is a measure of a disposition or state not measured by existing scales. S. In addition to publishing statistics on scale relationships, the author reports substantial external validational data. It is not sufficient to know that a scale correlates appreciably with a criterion, since, ideally, both convergent and discriminant validity information should be reported (Campbell & Fiske, 1959) for the new scale. It is also important to know how well a scale predicts behavior for the individual in various contexts. Thus some data reporting prediction success and failure in specific settings (e.g., in the form of hitmiss tables) are desirable. New MMPI scales, like many of the old ones, should not be assumed to measure the characteristics suggested by its name or by its author. The researcher should be aware of the track record of a particular scale before he or she gambles on it. Block (1965) provided an interesting illustrative example: However, ease of analysis should not mean casualness in regard to the scales chosen for study. On occasion, scales have been employed in circumstances which can only mean ignorance or naivete. These are harsh words but consider the following: several correlational studies in the response set domain have employed both the Prejudice (Pr) and Tolerance (To) scales of the MMPI. Both of these scales are due to Gough (1951; 1952). The Pr scale was developed and validated as an MMPI measure to correlate with the California Ethnocentrism-Fascism Scale (Adorno et al., 1950). Later, Gough decided to revise the Pr scale slightly and for entirely appropriate conceptual reasons took the occasion to relabel the scale as a measure of tolerance, reversing the direction of scoring of the items. In the To scale, 29 of the 30 items overlap with the Pr scale but scored oppositely. Since the sample of scales selected for study can determine the shape of the results obtained, simple considerations of availability must be bolstered by actual knowledge of candidate scales. To use both the Pr and the To scales in the same study and draw weighty conclusions from their opposite but identical factor loadings is informative only about the investigators, (p. 118)
The MMPI wears other clothes as well. Many scales in the personality research domain have been generated from the MMPI item pool or from constructs so close to those covered by the MMPI that the items are virtually the same. Often the new researcher
PROBLEMS
623
is unaware of the original relationships or similarities and then conducts an "empirical" study rediscovering the affinity of the tests in question. A few instruments that are wholly or in part derived from the MMPI or are closely related are the CPI (over 200 items are common), the Taylor Manifest Anxiety Scale, and Lanyon's Psychological Screening Inventory. MMPI Short Forms A large number of recent studies have used one of the several MMPI short forms. It should be noted that the only MMPI short form recognized by the test authors and the publishers consists of a reduced number of items (around 400) that include all of the items required to score the 3 validity and 10 standard scales. The items excluded are those at the end of the booklet that are not scored on the basic scales. Some short forms have been the object of a large amount of recent MMPI work and have been developed by varying methods and for different purposes (the Mini-Mult, Midi-Mult, the Faschingbauer Abbreviated MMPI, MMPI-168, etc.) Several recent studies have pointed out the limitations of MMPI short forms (Fillenbaum & Pfeiffer, 1976; Hedlund, Won Cho, & Powell, 1975; Hoffmann & Butcher, 1975). Although MMPI short forms may correlate significantly with the full MMPI, the resulting code-type congruence (hit rate) between the two forms is quite low (from 33% to 49%), too low to result in very similar individual clinical decisions. Although these studies indicate the need for caution in not accepting short forms at face value as adequate substitutions for full-length MMPI measures, many investigators continue to consider an MMPI short form as a near-equivalent set of scales. It is conceivable that some short forms could serve in a much more limited way as measures of global psychopathology. How to Find Relationships: Problems in Discovering and Appraising Relations Between the MMPI and Other Variables All MMPI research can be viewed as essentially correlational research, whether the
624
JAMES N. BUTCHER AND AUKE TELLEGEN
MMPI scales are the "dependent" or "independent" variables and whether we compute correlations explicitly or, for example, do t tests, comparing one group with another on certain MMPI measures (e.g., patients vs. normals), or determine hit rates obtained with certain MMPI-based diagnostic rules. Sometimes the strength of the relation investigated is immediately given by our statistics (e.g., correlation coefficient, hit rate), sometimes it is not (t test, F ratio). It is known but easily forgotten that some measure of strength of relation is always necessary if we are to determine the importance of our findings. Demonstrating statistical significance is necessary, but it is not enough. Not only should the magnitude of, for example, a correlation coefficient be considered but also its confidence interval. Yet one suspects that an inverse relation exists between the sample sizes used in published studies and the strength of the relations that are claimed to exist between certain variables. The reason, of course, is that a good way of "discovering" dramatically strong relationships in a set of data is to collect several measures, preferably those that can be expected to show some moderate interrelations, using a small sample. Chance alone will move a few of these expectable relations up into the "strikingly high" range (the "trade-off" being that one can expect to "lose" some of the other relationships by fluctuations in the opposite direction). In short, sample sizes should be large enough to permit conclusions about the strength of the relationships under consideration that go beyond the inference that the null hypothesis is false. The rejection of this null hypothesis is a trivial contribution, particularly if the observed relationship is in the (more or less) expected direction, because, as has been pointed out, we practically know the null hypothesis to be false—almost everything is somewhat related to everything (Lykken, 1968; Meehl, 1967). Example: Suppose one collects data on a (small) sample of 19 subjects and "discovers" a somewhat unexpected correlation of .50 between Scale 7 and a psychophysiological measure of "arousal," skin conductance level. The corresponding z value of .50 is .55 with a standard error of .25.
The .05 (two-tailed) confidence interval for this correlation, then, in 2 values, is .55 ± .49. In correlational values the corresponding range is between .06 and .78. The finding of a possibly very weak correlation, one that is only significantly greater than .06, we submit, is hardly a contribution (particularly not in this case, since the direction of the relation was more or less anticipated), in spite of its statistical significance and observed magnitude. Large sample sizes are essential to achieve less trivial results. Admittedly, if an observed relation is definitely not in the expected direction, and is definitely "significant" (even though the estimate of its strength still has a large margin of error), and involves a truly interesting relationship, then we might speak of a finding that deserves publication at this point to stimulate further research in what now appears to be a promising area. Nevertheless, it would still be better to follow up one's own findings with a replication permitting a more precise estimate and demonstrating a relationship of nontrivial strength. (After all, the more interesting our findings are to us, the less expected they must have been, and therefore the less reason we apparently have to assume their replicability without further test!) One fairly common practice is to calculate multiple correlations, often on a small sample, and to announce their statistical significance without reminding the reader that the magnitude of the obtained multiple correlations may be substantially inflated. Without cross-validation or an appropriate "shrinkage" estimate, this procedure can be grossly misleading. (See Schmitt, Coyle, & Rauschenberger, 1977, for a recent discussion of this topic.) Almost all MMPI research is multivariate (whether or not multivariate statistics are actually used). This, in combination with the frequent availability of "samples of convenience" (consisting, e.g., of individuals who have completed the MMPI as part of a screening procedure in some clinical or counseling setting), inevitably invites "look-see" or "shotgun" studies, guided only by the investigator's vague hope of finding something somewhere among the many possible relations that might be subjected to scrutiny. Particularly if the sample is small, chance findings
METHODOLOGICAL PROBLEMS
will often occur, and if one's sample of convenience is, in addition, poorly denned, then even nonchance findings may well prove not replicable in the absence of clear understanding of to what population the results are to be generalized. Unfortunately, a rather large number of manuscripts describing "one shot" studies using small and haphazardly collected samples continue to be submitted for publication—and to appear in print. Another undesirable practice is to boost sample sizes by lumping together subjects differing on some potentially important characteristics such as sex, age, race, and socioeconomic status (Carlson, 1971). Sometimes these variables may in fact prove not to be important for the relations of interest to the investigator. But this cannot be assumed. At the very least sex ought to be included among one's variables. However, this would still not allow for the possibility that relationships between the MMPI and other variables, for example, response to treatment or diagnostic status, might differ for the two sexes, for different age groups, and so on. If sample sizes permit, the simplest first step would be to subdivide one's sample into relatively homogeneous subgroups in respect to major differentiators (sex, age, and so forth) and to evaluate the relationships in the different subgroups for possible differences. Because of the MMPI "code type" tradition, it is not uncommon to analyze relationships by presenting MMPI correlates in the form of relevant base rates or means associated with each of several distinct code types. (The actuarial atlases of Marks, Seeman, & Haller, 1974, and Gilberstadt & Duker, 1965, are well-known examples.) One distinct potential advantage of this approach is that certain "configural" relations, reflecting interactions among MMPI scales in relation to criterion variables, will be used. For example, if correlates of Scale 7 differ depending on the subject's scores on additional scales, say 2, 4, or 8, then the use of different code types such as 2-7, 4-7, 2-7-8, and so on, permits one to capitalize on these interactions. But reliance on code types has disadvantages. One is that some information is inevitably lost when a profile is assigned to a code type. Another disadvantage is the small
625
number of classifiable subjects for many code types, even in rather extensive studies. These classification problems are often unnecessary, since many important relations between MMPI and external criteria are often not configural but essentially linear. That is, important relations may be adequately described by the linear relations existing in the total sample between one or more MMPI variables and other measures. One example is Goldberg's (1965) analysis of methods of distinguishing psychotic from neurotic patients on the basis of the MMPI. Goldberg showed that a simple linear combination of several clinical scales did just as well as the highly configural Meehl-Dahlstrom rules. Although the Goldberg analysis is one of the few well-documented examples, we have little doubt that (e.g., in the material from which Marks et al. 1974, and Gilberstadt & Duker, 1965, atlases have been derived) a number of linear relations between single or several MMPI scales and important nontest data are hidden, other than the psychotic versus neurotic distinction. One example is an apparent straightforward (linear) relationship between scores on Scale 4 and alcoholism in the Gilberstadt and Duker (1965) sample (pointed out by them). This example illustrates another disadvantage of exclusively relying on the "codetype approach": The method may exploit but does not reveal the nature of the relations, linear or nonlinear, that may exist in one's data. Goldberg's monograph is one example of how such relations can be pursued systematically, albeit in this case with negative results with respect to nonlinear relations. However, we believe that even nonlinear relations may emerge from systematic analyses provided the search is guided by a certain degree of theorizing permitting a focus on a smaller set of possible configurations, thus reducing the probability of drowning a few actual configural relations in a sea of unreplicable chance relationships. The reason is that without some theoretical constraints, the number of configural patterns that would have to be considered in an exhaustive search easily becomes extremely and unmanageably large, thus requiring unattainably large sample sizes to minimize the occurrences of chance patterns.
626
JAMES N. BUTCHER AND AUKE TELLEGEN
We find, then, that code and profile types provide convenient ways of assimilating the information contained in a set of MMPI scores. But in spite of the continuing advocacy (and plausibility) of typological classifications, acceptance of such classifications should be provisional rather than uncritical. We still do not know a great deal about how much and when type of membership would add to the information already contained in a (more parsimonious and psychometrically conservative) dimensional predictor, for example, an appropriate linear regression formula. Research on this question seems quite feasible and could give interesting results. Assessing Profile Change The measurement of change is methodologically complex, and the 'researcher engaged in such measurement should become aware of the difficulties involved (Cronbach & Furby, 1970; Fiske & Rice, 1955; Mauger, 1972; see especially Dahlstrom et al., 1972, chap. 7). We cannot review all of the relevant issues in the limited space of this article. However, it is important to keep in mind that a simple difference in MMPI mean scale scores on retest may not indicate that the individuals involved have necessarily changed as a result of some interposed treatment. Familiarity with and critical understanding of such concepts as "residual gain" are essential (e.g., Cronbach & Furby, 1970). To balance somewhat the note of pessimism in Cronbach and Furby's article, it should be pointed out that experimental designs containing comparison groups can be effective tools for understanding change. One substantive issue that the investigator should keep in mind is that MMPI measures appear to tap a mixture of changeable and stable characteristics. Some of the scales (e.g., D) appear to be reflective of "state" as well as "trait" characteristics and are sensitive to mood changes. Other scales focus on "biographical" factors (5* or Pd) and seem relatively more stable. What is the meaning of a particular MMPI scale change? If not a statistical artifact, does it reflect a change in affective state, a permanent change, a change in self-presentation? In addition, a change on a particular scale
is not necessarily best described in terms of the name of the scale in question. For example, a significant change on the Pd scale may be due primarily to changes on a subset of Pd items reflecting a negative mood state; content-homogeneous MMPI scales are certainly less potentially misleading in the interpretation of change than the more heterogeneous clinical scales. These problems should be addressed in any thoughtful discussion of MMPI data pertaining to change. As a general point, it is good to keep in mind that some scales may be useful for reflecting change, say, resulting from intervention, but other scales may be more useful for predicting response to treatment (without necessarily reflecting the effects of treatment). On the whole it may be true that the original method of developing the MMPI favored the selection and construction of relatively stable items and scales, respectively. None of the original work was directly focused on measuring change. Subsequent efforts at using MMPI items to detect changeability or to develop scales that would give clues to potential for change have not met with much success (Mauger, 1972; Pepper, 1964). The "stability" of the MMPI is shown by the fairly consistent finding that about 9,1% of the MMPI items are answered in the same direction on retest (Butcher & Gur, 1974; Goldberg & Jones, 1969; Schofield, 1948; Ullmann & Wiggins, 1962). Consequently, the number of items that can be expected to vary on retest is relatively small. Reporting Group Data What do we know from a group mean profile that summarizes a set of MMPI scores? Can we assume that if the group average is a 278" profile type that the established personality correlates for the 278" profile necessarily fit the group as a whole or some subset of individuals in the group? No, sometimes we fall into the trap of drawing such a conclusion. It may actually be the case that no individual in the group has the code type corresponding to the group mean score. Researchers who are not content to present merely the means and standard deviations of individual scales could report the percentages
METHODOLOGICAL PROBLEMS
of the different code or profile types occurring in their sample. In summary, the MMPI continues to be a widely used research instrument as well as a clinical assessment inventory. Reasons for its wide application include its easy administration, its objective scoring, the volume of data demonstrating its validity, and the automation of its interpretation. The MMPI is such an easily used research instrument that it is sometimes misapplied or the data obtained with it are incorrectly analyzed by researchers who are not familiar with some of its limitations or peculiarities. The MMPI can play and has played a useful role in significant research. The main prerequisites are the investigator's long-term and genuine concern with an important problem and her/his ability to draw on the store of substantial methodological knowledge and wisdom, accumulated in the field of personality assessment in general (e.g., Wiggins, 1973) and in the area of MMPI research in particular (e.g., Dahlstrom et al., 1972, 1975). Reference Note 1. Overall, J. E. Implementation of an actuarial diagnostic program in a clinical, setting. Paper presented at the llth annual symposium on recent developments in the use of the MMPI. Minneapolis, April 1976.
References Block, J. The challenge of response sets: Vnconfounding meaning, acquiescence and social desirability in the MMPI. New York: Appleton-CenturyCrofts, 196S. Euros, 0. Seventh mental measurements yearbook. Highland Park, N.J.: Gryphon Press, 1972. Butcher, J. N., & Gur, R. Hebrew translation of the MMPI: An assessment of translation adequacy and preliminary validation. Journal of Cross-cultural Psychology, 1974, 5, 220-227. Butcher, J. N., & Owen, P, L. Objective personality inventories. In B. B. Wolman (Ed.), Clinical diagnosis of mental disorders: A handbook. New York: Plenum Press, in press. Butcher, J. N., & Panchcri, P. Handbook oj crossnational MMPI research. Minneapolis: University of Minnesota Press, 1976, Carlson, R. Where is the person in personality research? Psychological Bulletin, 1971, 75, 203-219. Campbell, D. T., & Fiske, D. W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 1959, 56, 81-105.
627
Cronbach, L. J,, & Furby, L. How we should measure "change"—Or should we? Psychological Bulletin, 1970, 74, 68-80. Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. An MMPI handbook (Vol. 1). Minneapolis: University of Minnesota Press, 1972. Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. An MMPI handbook (Vol. 2). Minneapolis: University of Minnesota Press, 1975. Eichman, W. J. Factored scales for the MMPI. Journal oj Clinical Psychology, 1962, 15, 363-395. Fillenbaum, G. G., & Pfeiffer, E. The Mini-Mult: A cautionary note. Journal oj Consulting and Clinical Psychology, 1976, 44, 698-703. Fiske, D. W., & Rice, L. Intra-individual response variability. Psychological Bulletin, 1955, 52, 217250. Gilberstadt, H., & Duker, J. A handbook for clinical and actuarial MMPI interpretation. Philadelphia, Pa.: Saunders, 196S. Goldberg, L. R. Diagnosticians vs. diagnostic signs: The diagnosis of psychosis vs. neurosis from the MMPI. Psychological Monographs, 1965, 79(9, Whole No. 602). Goldberg, L. R., & Jones, R. R. The reliability of reliability: The generality and correlates of intraindividual consistency in responses to structured personality inventories. Oregon Research Institute Monograph, 1969, 9, No. 2. Grayson, H. M. A psychological admissions testing program and manual. Los Angeles: Veterans Administration Center, Neuropsychiatric Hospital, 1951. Hase, H. D., & Goldberg, L. R. Comparative validity of different strategies of constructing personality inventory scales. Psychological Bulletin, 1967, 67, 231-248. Hedland, J. L., Won Cho, D., & Powell, B. J. Use of MMPI short forms with psychiatric patients. Journal of Consulting and Clinical Psychology, 1975, 43, 924. Hoffmann, N. G., & Butcher, J. N. Clinical limitations of three MMPI short forms. Journal of Consulting and Clinical Psychology, 1975, 43, 32-39. Jackson, D. N. The dynamics of structured personality tests: 1971. Psychological Review, 1971, 78, 239-248. Koss, M. P., & Butcher, J. N. A comparison of psychiatric patients' se-lf-report with other sources of clinical information. Journal of Research in Personality, 1973, 7, 225-236. Koss, M. P., Butcher, J. N., & Hoffmann, N. G. The MMPI critical items: How well do they work? Journal of Consulting and Clinical Psychology, 1976, 44, 921-928. Lykken, D. T. Statistical significance in psychological research. Psychological Bulletin, 1968, 70, 151-159. Marks, P. A., Sceman, W., & Haller, D. The actuarial use of the MMPI with adolescents and adults. Baltimore, Maryland: Williams & Wilkins, 1974. Mauger, P. A. The test-retest reliability of persons: An empirical investigation utilizing the Minnesota Multiphasic Personality Inventory and the Per-
628
JAMES N. BUTCHER AND AUKE TELLEGEN
sonality Research Form. (Doctoral dissertation, University of Minnesota, 1972.) Dissertation Abstracts International 1972, 33, 2806B. (University Microfilms No. 72-32, 328) Meehl, P. E. Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 1967, 34, 103-115. Overall, J. E., Hunter, S., & Butcher, J. N. Factor structure of the MMPI-168 in a psychiatric population. Journal of Consulting and Clinical Psychology, 1973, 41, 284-286. Payne, F. D., & Wiggins, J. S. MMPI profile type and the self-report of psychiatric patients. Journal of Abnormal Psychology, 1972, 79, 1-8. Pepper, L. J. The MMPI: Initial test predictors of retest changes. Unpublished doctoral dissertation, University of North Carolina, 1964. Schmitt, N., Coyle, B. W,, & Rauschcnberger, J. A Monte Carlo evaluation of three formula estimates of cross-validated multiple correlations. Psychological Bulletin, 1977, 84, 751-7S8. Schofield, W. MMPI response changes with certain therapies. Unpublished doctoral dissertation, University of Minnesota, 1948.
Stein, K. B. The TSC scales: The outcome of a cluster analysis of the 550 MMPI items. In P. McReynolds (Ed.), Advances in psychological assessment. Palo Alto, Calif.: Science & Behavior Books, 1968. Ullmann, L. P., & Wiggins, J. S. Endorsement frequency and the number of differentiating MMPI items to be expected by chance. Newsletter of Research in Psychology, 1962, 4, 29-35. Welsh, G. S. Factor dimensions A and R. In G. S. Welsh & W. G. Dahlstrom (Eds.), Basic readings on the MMPI in psychology and medicine. Minneapolis: University of Minnesota Press, 1956. Wiggins, J. Content dimensions in the MMPI. In J. N. Butcher (Ed.), MMPI: Research developments and clinical applications. New York: McGraw-Hill, 1969. Wiggins, J. S. Personality and prediction: Principles of personality assessment. Reading, Mass.: Addison-Wesley, 1973. Received September 26, 1977 •
Common Methodological Problems in MMPI Research James N. Butcher and Auke Tellegen University of Minnesota Research with the Minnesota Multiphasic Personality Inventory (MMPI) continues at a high rate. Unfortunately, too many articles submitted and even accepted for publication are methodologically weak. In this article we discuss some common methodological problems involving the use of the MMPI, encountered in the course of reviewing articles submitted for publication. A number of relevant issues are discussed, and some suggestions for improving research designs are made. Research with the Minnesota Multiphasic Personality Inventory (MMPI) has continued at a high level in recent years. As a self-report inventory that includes several measures of psychopathology, the MMPI, or some derivative of it, has become widely used as a descriptive instrument or a criterion measure in a vast array of clinical and research investigations. In 1972 Euros reported that over 200 books and articles on the MMPI are published annually. Dahlstrom, Welsh, and Dahlstrom (197S) cited over 6,000 references to the MMPI; Butcher and Pancheri (1976) reported over 600 recent references in cross-national MMPI research alone. Butcher and Owen (in press) recently reviewed and classified MMPI research for the past 5 years. As shown in Table 1, they found that over one fourth of the research was focused on two areas of "popular" investigation: alcohol and drug abuse, and crime and delinquency. Approximately 37% of the studies focused on the use of the MMPI to study nonpsychiatric populations—medical patients, parents, women, ethnic groups, college students, and the aged. The MMPI seems to have been widely accepted as a criterion measure of psychopathology by researchers who want to We wish to thank David Lykkcn for commenting on an earlier version of this article. Requests for reprints should be sent to James N. Butcher, Department of Psychology, Elliott Hall, University of Minnesota, 75 East River Road, Minneapolis, Minnesota S545S.
measure psychological problems in a variety of nonpsychiatric groups. But studies pointing to the need for revision or modification of the MMPI are scarce. This absence of revisionoriented research is impressive, since the MMPI is around 40 years old and can be expected to show some signs of aging. The only serious efforts to modify the MMPI have been directed at shortening it: About 12% of the published research in the past 5 years concerned the development or use of shorter versions of the instrument. There are many reasons for the extensive use of the MMPI: Its administration is relatively effortless; its scoring is objective; generally straightforward objective interpretive procedures are available; and its validity as a criterion measure is comparatively well founded. But some of the factors encouraging researchers to use a self-report personality measure such as the MMPI also result in the kinds of methodological flaws that this article may help to prevent. The MMPI is so easy to administer as a part of an ongoing study that researchers (even if they have little or no research background with the instrument) feel encouraged to include it. Furthermore, since good criteria in psychopathology or personality research are scarce, the MMPI scales are sometimes too readily adopted as expedient proximate criteria. (In some respects, one might almost consider the MMPI an "attractive nuisance" in the legal sense, to be approached with appropriate caution.) The remainder of this article will focus on
Copyright 1978 by the American Psychological Association, Inc. 0022-006X/78/4604-0620$00.75
620
METHODOLOGICAL PROBLEMS
some methodological problems that are often associated with research using the MMPI. We hope to provide some guidelines that could help avoid these difficulties, by providing minimal criteria. Some of these criteria emerged as a result of our experiences as reviewers of MMPI manuscripts that were submitted for publication in several psychological journals. It will not be possible to provide an exhaustive review of all pertinent issues nor to avoid seeming at times somewhat dogmatic. Yet it is not our intention to "dictate" which areas of investigation are important or which ones should be avoided, or to impose idiosyncratic preferences for certain research methods when several defensible ones are available. What Kind oj Instrument is the MMPI? The MMPI is often mistakenly considered to be an all-purpose personality assessment instrument that is sensitive to "normal range" personality attributes. Consequently, some researchers use the MMPI with groups for which a different instrument might be more appropriate. The standard clinical MMPI scales are measures of psychopathology, not general personality. The MMPI should not be made to do what it is not designed to do. Rather than use the MMPI inappropriately, it would be better to consider alternative or supplementary measures such as the California Psychological Inventory (CPI) or Personality Research Form, which focus on normal-range personality dimensions. What kind of MMPI measures can or should be used? The eight clinical scales and the three validity scales plus Mi and Si are the best established and most widely used MMPI scales.1 In view of the many problems that we have encountered in the computation of clinical scale scores, two suggestions are in order: 1. In research computations more understandable results will often be obtained if one does not use the K correction, but uses K as a separate indicator. This simply means that K'& validity as a "suppressor" is not assumed. Of course, appropriate multivariate analysis of one's data will permit determination whether K in fact does function as a suppressor in one's particular data set. The K
621
Table 1 Classification of Recent Minnesota Multiphasic Personality Inventory Research Category Alcohol and drug abuse: detection and treatment Short forms Use with other tests or rating scales Crime and delinquency Diagnostic considerations: rules, profiles, and code types Psychometric characteristics Medical patients: physical disorders and
symptoms Correlation with diverse criteria Women Treatment variables and therapy outcome Depression and suicide College students and adolescents Parents, couples, and families Drug therapy: choice and effect Race: cross-cultural, ethnic Automated interpretation Employment screening and job performance Anxiety and stress Brain damage Sleep Sexual deviation Aging
No. articles
96 66 62 56 SO 49
47 46 42 38 38 34 33 31 31 26 24 18 18 10 9 5
Note. Adapted from Butcher and Owen (in press).
score will often be found to be an additional measure of good adjustment rather than a measure of invalid variance. If non-K-corrected T scores are used, use the correct conversion tables. Occasionally manuscripts are received containing figures with non-/£-corrected profiles drawn according to incorrect norms. Do not use the raw score indications printed on the standard profile sheet unless your scores are actually K corrected! If you are plotting non-A"-corrected scores for Scales Hs, Pd, Pt, Sc, and Ma and plan to display these on the standard profile sheet, it is necessary to obtain the correct T-score elevation from a conversion table (e.g., in Dahlstrom, Welsh, & Dahlstrom, 1972) and draw the appropriate elevation from the T-score indica1
Researchers interested in obtaining access to the item response data of the original Minnesota standardization sample should contact W. Grant Dahlstrom, Department of Psychology, University of North Carolina, Chapel Hill, North Carolina 27S14.
622
JAMES N. BUTCHER AND AUKE TELLEGEN
tions on the side of the profile sheet. In spite of the above general recommendation about using non-^-corrected scores, it is sometimes useful for comparisons with other studies to do a parallel set of analyses using ^-corrected scores. 2. Use raw scores in research computations rather than T scores unless the use of T scores is specifically indicated, for example, in the analysis of code types or when combining data from both sexes. The eight clinical scales were developed primarily by means of empirical keying methods, using clinical diagnoses as criteria. Because of the primary reliance on external criteria rather than on internal structure, these scales are for the most part quite heterogeneous. Consequently, although these scales, particularly when used in combination, provide useful information as to the diagnostic status of a person (Dahlstrom et al., 1972; Overall, Note 1), they do not in any simple way reflect "what the patient is saying," that is, how the patient is describing herself/himselj through the medium of the MMPI items. This, too, is important information. Nowadays we recognize more clearly than in the heyday of empirical keying and "dust bowl empiricism" that important and clinically relevant direct relations exist between the manifest content of a person's self-description and clinical symptomatology and behavior in general (e.g., Hase & Goldberg, 1967; Jackson, 1971; Koss & Butcher, 1973; Payne & Wiggins, 1972). In addition, in clinical use the client/patient's reply to MMPI questions, impersonal as the process may seem, is still a communication that he/she might expect the psychologist to receive. We therefore need to take steps to make this possible. Some organized way of "coding" the respondent's MMPI "message" is clearly called for. Two approaches are now open: 1. use of content homogeneous scales, including factor or cluster scales (Block, 1965; Eichman, 1962; Overall, Hunter, & Butcher, 1973; Stein, 1968; Welsh, 1956; Wiggins, 1969) and 2. use of so-called "critical items" (Grayson, 1951; Koss & Butcher, 1973; Koss, Butcher, & Hoffmann, 1976).
Both approaches can serve a very useful role complementary to that of the clinical scales. Scale Proliferation The development of additional MMPI scales has been a rather popular pastime for psychologists. There are presently more MMPI scales than there are items on the inventory! Unfortunately, much of the MMPI scale development does not derive from a sound conceptual framework. Many scales have been constructed by contrasting different "samples of convenience" (often of heterogeneous makeup or with important characteristics unknown). Often these scales are not cross-validated, and more often than not their psychometric properties and interrelations with other scales are not reported. Many new scales, since they are, after all, derived from the same item pool, prove to be largely redundant alternative versions of existing scales, although sometimes of poorer quality. A researcher who is tempted to add to this plethora of scales would find his or her contribution to the MMPI literature better received and used if: 1. The investigator has very carefully considered the question of whether the MMPI item pool adequately covers the domain of content of the construct to be measured. If the answer is not affirmative, then the MMPI will obviously not serve the investigator's purpose very well, and the new scale will reflect more the characteristics of the MMPI item pool than the domain of the intended construct. We believe that this is in fact the case with some new MMPI scales. 2. The investigator can show the scale to be conceptually interesting. 3. The scale is developed and cross-validated on reasonably large, well-defined samples (whether it is an empirically keyed scale or one derived by internal consistency methods, e.g., factor analysis). 4. The internal structure of the scale is reported (internal consistency, possibly factor structure), and the relation to other MMPI scales is reported. These relations along with validity data would ideally reveal that the scale either is a superior alternative to an
METHODOLOGICAL
existing MMPI scale or is a measure of a disposition or state not measured by existing scales. S. In addition to publishing statistics on scale relationships, the author reports substantial external validational data. It is not sufficient to know that a scale correlates appreciably with a criterion, since, ideally, both convergent and discriminant validity information should be reported (Campbell & Fiske, 1959) for the new scale. It is also important to know how well a scale predicts behavior for the individual in various contexts. Thus some data reporting prediction success and failure in specific settings (e.g., in the form of hitmiss tables) are desirable. New MMPI scales, like many of the old ones, should not be assumed to measure the characteristics suggested by its name or by its author. The researcher should be aware of the track record of a particular scale before he or she gambles on it. Block (1965) provided an interesting illustrative example: However, ease of analysis should not mean casualness in regard to the scales chosen for study. On occasion, scales have been employed in circumstances which can only mean ignorance or naivete. These are harsh words but consider the following: several correlational studies in the response set domain have employed both the Prejudice (Pr) and Tolerance (To) scales of the MMPI. Both of these scales are due to Gough (1951; 1952). The Pr scale was developed and validated as an MMPI measure to correlate with the California Ethnocentrism-Fascism Scale (Adorno et al., 1950). Later, Gough decided to revise the Pr scale slightly and for entirely appropriate conceptual reasons took the occasion to relabel the scale as a measure of tolerance, reversing the direction of scoring of the items. In the To scale, 29 of the 30 items overlap with the Pr scale but scored oppositely. Since the sample of scales selected for study can determine the shape of the results obtained, simple considerations of availability must be bolstered by actual knowledge of candidate scales. To use both the Pr and the To scales in the same study and draw weighty conclusions from their opposite but identical factor loadings is informative only about the investigators, (p. 118)
The MMPI wears other clothes as well. Many scales in the personality research domain have been generated from the MMPI item pool or from constructs so close to those covered by the MMPI that the items are virtually the same. Often the new researcher
PROBLEMS
623
is unaware of the original relationships or similarities and then conducts an "empirical" study rediscovering the affinity of the tests in question. A few instruments that are wholly or in part derived from the MMPI or are closely related are the CPI (over 200 items are common), the Taylor Manifest Anxiety Scale, and Lanyon's Psychological Screening Inventory. MMPI Short Forms A large number of recent studies have used one of the several MMPI short forms. It should be noted that the only MMPI short form recognized by the test authors and the publishers consists of a reduced number of items (around 400) that include all of the items required to score the 3 validity and 10 standard scales. The items excluded are those at the end of the booklet that are not scored on the basic scales. Some short forms have been the object of a large amount of recent MMPI work and have been developed by varying methods and for different purposes (the Mini-Mult, Midi-Mult, the Faschingbauer Abbreviated MMPI, MMPI-168, etc.) Several recent studies have pointed out the limitations of MMPI short forms (Fillenbaum & Pfeiffer, 1976; Hedlund, Won Cho, & Powell, 1975; Hoffmann & Butcher, 1975). Although MMPI short forms may correlate significantly with the full MMPI, the resulting code-type congruence (hit rate) between the two forms is quite low (from 33% to 49%), too low to result in very similar individual clinical decisions. Although these studies indicate the need for caution in not accepting short forms at face value as adequate substitutions for full-length MMPI measures, many investigators continue to consider an MMPI short form as a near-equivalent set of scales. It is conceivable that some short forms could serve in a much more limited way as measures of global psychopathology. How to Find Relationships: Problems in Discovering and Appraising Relations Between the MMPI and Other Variables All MMPI research can be viewed as essentially correlational research, whether the
624
JAMES N. BUTCHER AND AUKE TELLEGEN
MMPI scales are the "dependent" or "independent" variables and whether we compute correlations explicitly or, for example, do t tests, comparing one group with another on certain MMPI measures (e.g., patients vs. normals), or determine hit rates obtained with certain MMPI-based diagnostic rules. Sometimes the strength of the relation investigated is immediately given by our statistics (e.g., correlation coefficient, hit rate), sometimes it is not (t test, F ratio). It is known but easily forgotten that some measure of strength of relation is always necessary if we are to determine the importance of our findings. Demonstrating statistical significance is necessary, but it is not enough. Not only should the magnitude of, for example, a correlation coefficient be considered but also its confidence interval. Yet one suspects that an inverse relation exists between the sample sizes used in published studies and the strength of the relations that are claimed to exist between certain variables. The reason, of course, is that a good way of "discovering" dramatically strong relationships in a set of data is to collect several measures, preferably those that can be expected to show some moderate interrelations, using a small sample. Chance alone will move a few of these expectable relations up into the "strikingly high" range (the "trade-off" being that one can expect to "lose" some of the other relationships by fluctuations in the opposite direction). In short, sample sizes should be large enough to permit conclusions about the strength of the relationships under consideration that go beyond the inference that the null hypothesis is false. The rejection of this null hypothesis is a trivial contribution, particularly if the observed relationship is in the (more or less) expected direction, because, as has been pointed out, we practically know the null hypothesis to be false—almost everything is somewhat related to everything (Lykken, 1968; Meehl, 1967). Example: Suppose one collects data on a (small) sample of 19 subjects and "discovers" a somewhat unexpected correlation of .50 between Scale 7 and a psychophysiological measure of "arousal," skin conductance level. The corresponding z value of .50 is .55 with a standard error of .25.
The .05 (two-tailed) confidence interval for this correlation, then, in 2 values, is .55 ± .49. In correlational values the corresponding range is between .06 and .78. The finding of a possibly very weak correlation, one that is only significantly greater than .06, we submit, is hardly a contribution (particularly not in this case, since the direction of the relation was more or less anticipated), in spite of its statistical significance and observed magnitude. Large sample sizes are essential to achieve less trivial results. Admittedly, if an observed relation is definitely not in the expected direction, and is definitely "significant" (even though the estimate of its strength still has a large margin of error), and involves a truly interesting relationship, then we might speak of a finding that deserves publication at this point to stimulate further research in what now appears to be a promising area. Nevertheless, it would still be better to follow up one's own findings with a replication permitting a more precise estimate and demonstrating a relationship of nontrivial strength. (After all, the more interesting our findings are to us, the less expected they must have been, and therefore the less reason we apparently have to assume their replicability without further test!) One fairly common practice is to calculate multiple correlations, often on a small sample, and to announce their statistical significance without reminding the reader that the magnitude of the obtained multiple correlations may be substantially inflated. Without cross-validation or an appropriate "shrinkage" estimate, this procedure can be grossly misleading. (See Schmitt, Coyle, & Rauschenberger, 1977, for a recent discussion of this topic.) Almost all MMPI research is multivariate (whether or not multivariate statistics are actually used). This, in combination with the frequent availability of "samples of convenience" (consisting, e.g., of individuals who have completed the MMPI as part of a screening procedure in some clinical or counseling setting), inevitably invites "look-see" or "shotgun" studies, guided only by the investigator's vague hope of finding something somewhere among the many possible relations that might be subjected to scrutiny. Particularly if the sample is small, chance findings
METHODOLOGICAL PROBLEMS
will often occur, and if one's sample of convenience is, in addition, poorly denned, then even nonchance findings may well prove not replicable in the absence of clear understanding of to what population the results are to be generalized. Unfortunately, a rather large number of manuscripts describing "one shot" studies using small and haphazardly collected samples continue to be submitted for publication—and to appear in print. Another undesirable practice is to boost sample sizes by lumping together subjects differing on some potentially important characteristics such as sex, age, race, and socioeconomic status (Carlson, 1971). Sometimes these variables may in fact prove not to be important for the relations of interest to the investigator. But this cannot be assumed. At the very least sex ought to be included among one's variables. However, this would still not allow for the possibility that relationships between the MMPI and other variables, for example, response to treatment or diagnostic status, might differ for the two sexes, for different age groups, and so on. If sample sizes permit, the simplest first step would be to subdivide one's sample into relatively homogeneous subgroups in respect to major differentiators (sex, age, and so forth) and to evaluate the relationships in the different subgroups for possible differences. Because of the MMPI "code type" tradition, it is not uncommon to analyze relationships by presenting MMPI correlates in the form of relevant base rates or means associated with each of several distinct code types. (The actuarial atlases of Marks, Seeman, & Haller, 1974, and Gilberstadt & Duker, 1965, are well-known examples.) One distinct potential advantage of this approach is that certain "configural" relations, reflecting interactions among MMPI scales in relation to criterion variables, will be used. For example, if correlates of Scale 7 differ depending on the subject's scores on additional scales, say 2, 4, or 8, then the use of different code types such as 2-7, 4-7, 2-7-8, and so on, permits one to capitalize on these interactions. But reliance on code types has disadvantages. One is that some information is inevitably lost when a profile is assigned to a code type. Another disadvantage is the small
625
number of classifiable subjects for many code types, even in rather extensive studies. These classification problems are often unnecessary, since many important relations between MMPI and external criteria are often not configural but essentially linear. That is, important relations may be adequately described by the linear relations existing in the total sample between one or more MMPI variables and other measures. One example is Goldberg's (1965) analysis of methods of distinguishing psychotic from neurotic patients on the basis of the MMPI. Goldberg showed that a simple linear combination of several clinical scales did just as well as the highly configural Meehl-Dahlstrom rules. Although the Goldberg analysis is one of the few well-documented examples, we have little doubt that (e.g., in the material from which Marks et al. 1974, and Gilberstadt & Duker, 1965, atlases have been derived) a number of linear relations between single or several MMPI scales and important nontest data are hidden, other than the psychotic versus neurotic distinction. One example is an apparent straightforward (linear) relationship between scores on Scale 4 and alcoholism in the Gilberstadt and Duker (1965) sample (pointed out by them). This example illustrates another disadvantage of exclusively relying on the "codetype approach": The method may exploit but does not reveal the nature of the relations, linear or nonlinear, that may exist in one's data. Goldberg's monograph is one example of how such relations can be pursued systematically, albeit in this case with negative results with respect to nonlinear relations. However, we believe that even nonlinear relations may emerge from systematic analyses provided the search is guided by a certain degree of theorizing permitting a focus on a smaller set of possible configurations, thus reducing the probability of drowning a few actual configural relations in a sea of unreplicable chance relationships. The reason is that without some theoretical constraints, the number of configural patterns that would have to be considered in an exhaustive search easily becomes extremely and unmanageably large, thus requiring unattainably large sample sizes to minimize the occurrences of chance patterns.
626
JAMES N. BUTCHER AND AUKE TELLEGEN
We find, then, that code and profile types provide convenient ways of assimilating the information contained in a set of MMPI scores. But in spite of the continuing advocacy (and plausibility) of typological classifications, acceptance of such classifications should be provisional rather than uncritical. We still do not know a great deal about how much and when type of membership would add to the information already contained in a (more parsimonious and psychometrically conservative) dimensional predictor, for example, an appropriate linear regression formula. Research on this question seems quite feasible and could give interesting results. Assessing Profile Change The measurement of change is methodologically complex, and the 'researcher engaged in such measurement should become aware of the difficulties involved (Cronbach & Furby, 1970; Fiske & Rice, 1955; Mauger, 1972; see especially Dahlstrom et al., 1972, chap. 7). We cannot review all of the relevant issues in the limited space of this article. However, it is important to keep in mind that a simple difference in MMPI mean scale scores on retest may not indicate that the individuals involved have necessarily changed as a result of some interposed treatment. Familiarity with and critical understanding of such concepts as "residual gain" are essential (e.g., Cronbach & Furby, 1970). To balance somewhat the note of pessimism in Cronbach and Furby's article, it should be pointed out that experimental designs containing comparison groups can be effective tools for understanding change. One substantive issue that the investigator should keep in mind is that MMPI measures appear to tap a mixture of changeable and stable characteristics. Some of the scales (e.g., D) appear to be reflective of "state" as well as "trait" characteristics and are sensitive to mood changes. Other scales focus on "biographical" factors (5* or Pd) and seem relatively more stable. What is the meaning of a particular MMPI scale change? If not a statistical artifact, does it reflect a change in affective state, a permanent change, a change in self-presentation? In addition, a change on a particular scale
is not necessarily best described in terms of the name of the scale in question. For example, a significant change on the Pd scale may be due primarily to changes on a subset of Pd items reflecting a negative mood state; content-homogeneous MMPI scales are certainly less potentially misleading in the interpretation of change than the more heterogeneous clinical scales. These problems should be addressed in any thoughtful discussion of MMPI data pertaining to change. As a general point, it is good to keep in mind that some scales may be useful for reflecting change, say, resulting from intervention, but other scales may be more useful for predicting response to treatment (without necessarily reflecting the effects of treatment). On the whole it may be true that the original method of developing the MMPI favored the selection and construction of relatively stable items and scales, respectively. None of the original work was directly focused on measuring change. Subsequent efforts at using MMPI items to detect changeability or to develop scales that would give clues to potential for change have not met with much success (Mauger, 1972; Pepper, 1964). The "stability" of the MMPI is shown by the fairly consistent finding that about 9,1% of the MMPI items are answered in the same direction on retest (Butcher & Gur, 1974; Goldberg & Jones, 1969; Schofield, 1948; Ullmann & Wiggins, 1962). Consequently, the number of items that can be expected to vary on retest is relatively small. Reporting Group Data What do we know from a group mean profile that summarizes a set of MMPI scores? Can we assume that if the group average is a 278" profile type that the established personality correlates for the 278" profile necessarily fit the group as a whole or some subset of individuals in the group? No, sometimes we fall into the trap of drawing such a conclusion. It may actually be the case that no individual in the group has the code type corresponding to the group mean score. Researchers who are not content to present merely the means and standard deviations of individual scales could report the percentages
METHODOLOGICAL PROBLEMS
of the different code or profile types occurring in their sample. In summary, the MMPI continues to be a widely used research instrument as well as a clinical assessment inventory. Reasons for its wide application include its easy administration, its objective scoring, the volume of data demonstrating its validity, and the automation of its interpretation. The MMPI is such an easily used research instrument that it is sometimes misapplied or the data obtained with it are incorrectly analyzed by researchers who are not familiar with some of its limitations or peculiarities. The MMPI can play and has played a useful role in significant research. The main prerequisites are the investigator's long-term and genuine concern with an important problem and her/his ability to draw on the store of substantial methodological knowledge and wisdom, accumulated in the field of personality assessment in general (e.g., Wiggins, 1973) and in the area of MMPI research in particular (e.g., Dahlstrom et al., 1972, 1975). Reference Note 1. Overall, J. E. Implementation of an actuarial diagnostic program in a clinical, setting. Paper presented at the llth annual symposium on recent developments in the use of the MMPI. Minneapolis, April 1976.
References Block, J. The challenge of response sets: Vnconfounding meaning, acquiescence and social desirability in the MMPI. New York: Appleton-CenturyCrofts, 196S. Euros, 0. Seventh mental measurements yearbook. Highland Park, N.J.: Gryphon Press, 1972. Butcher, J. N., & Gur, R. Hebrew translation of the MMPI: An assessment of translation adequacy and preliminary validation. Journal of Cross-cultural Psychology, 1974, 5, 220-227. Butcher, J. N., & Owen, P, L. Objective personality inventories. In B. B. Wolman (Ed.), Clinical diagnosis of mental disorders: A handbook. New York: Plenum Press, in press. Butcher, J. N., & Panchcri, P. Handbook oj crossnational MMPI research. Minneapolis: University of Minnesota Press, 1976, Carlson, R. Where is the person in personality research? Psychological Bulletin, 1971, 75, 203-219. Campbell, D. T., & Fiske, D. W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 1959, 56, 81-105.
627
Cronbach, L. J,, & Furby, L. How we should measure "change"—Or should we? Psychological Bulletin, 1970, 74, 68-80. Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. An MMPI handbook (Vol. 1). Minneapolis: University of Minnesota Press, 1972. Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. An MMPI handbook (Vol. 2). Minneapolis: University of Minnesota Press, 1975. Eichman, W. J. Factored scales for the MMPI. Journal oj Clinical Psychology, 1962, 15, 363-395. Fillenbaum, G. G., & Pfeiffer, E. The Mini-Mult: A cautionary note. Journal oj Consulting and Clinical Psychology, 1976, 44, 698-703. Fiske, D. W., & Rice, L. Intra-individual response variability. Psychological Bulletin, 1955, 52, 217250. Gilberstadt, H., & Duker, J. A handbook for clinical and actuarial MMPI interpretation. Philadelphia, Pa.: Saunders, 196S. Goldberg, L. R. Diagnosticians vs. diagnostic signs: The diagnosis of psychosis vs. neurosis from the MMPI. Psychological Monographs, 1965, 79(9, Whole No. 602). Goldberg, L. R., & Jones, R. R. The reliability of reliability: The generality and correlates of intraindividual consistency in responses to structured personality inventories. Oregon Research Institute Monograph, 1969, 9, No. 2. Grayson, H. M. A psychological admissions testing program and manual. Los Angeles: Veterans Administration Center, Neuropsychiatric Hospital, 1951. Hase, H. D., & Goldberg, L. R. Comparative validity of different strategies of constructing personality inventory scales. Psychological Bulletin, 1967, 67, 231-248. Hedland, J. L., Won Cho, D., & Powell, B. J. Use of MMPI short forms with psychiatric patients. Journal of Consulting and Clinical Psychology, 1975, 43, 924. Hoffmann, N. G., & Butcher, J. N. Clinical limitations of three MMPI short forms. Journal of Consulting and Clinical Psychology, 1975, 43, 32-39. Jackson, D. N. The dynamics of structured personality tests: 1971. Psychological Review, 1971, 78, 239-248. Koss, M. P., & Butcher, J. N. A comparison of psychiatric patients' se-lf-report with other sources of clinical information. Journal of Research in Personality, 1973, 7, 225-236. Koss, M. P., Butcher, J. N., & Hoffmann, N. G. The MMPI critical items: How well do they work? Journal of Consulting and Clinical Psychology, 1976, 44, 921-928. Lykken, D. T. Statistical significance in psychological research. Psychological Bulletin, 1968, 70, 151-159. Marks, P. A., Sceman, W., & Haller, D. The actuarial use of the MMPI with adolescents and adults. Baltimore, Maryland: Williams & Wilkins, 1974. Mauger, P. A. The test-retest reliability of persons: An empirical investigation utilizing the Minnesota Multiphasic Personality Inventory and the Per-
628
JAMES N. BUTCHER AND AUKE TELLEGEN
sonality Research Form. (Doctoral dissertation, University of Minnesota, 1972.) Dissertation Abstracts International 1972, 33, 2806B. (University Microfilms No. 72-32, 328) Meehl, P. E. Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 1967, 34, 103-115. Overall, J. E., Hunter, S., & Butcher, J. N. Factor structure of the MMPI-168 in a psychiatric population. Journal of Consulting and Clinical Psychology, 1973, 41, 284-286. Payne, F. D., & Wiggins, J. S. MMPI profile type and the self-report of psychiatric patients. Journal of Abnormal Psychology, 1972, 79, 1-8. Pepper, L. J. The MMPI: Initial test predictors of retest changes. Unpublished doctoral dissertation, University of North Carolina, 1964. Schmitt, N., Coyle, B. W,, & Rauschcnberger, J. A Monte Carlo evaluation of three formula estimates of cross-validated multiple correlations. Psychological Bulletin, 1977, 84, 751-7S8. Schofield, W. MMPI response changes with certain therapies. Unpublished doctoral dissertation, University of Minnesota, 1948.
Stein, K. B. The TSC scales: The outcome of a cluster analysis of the 550 MMPI items. In P. McReynolds (Ed.), Advances in psychological assessment. Palo Alto, Calif.: Science & Behavior Books, 1968. Ullmann, L. P., & Wiggins, J. S. Endorsement frequency and the number of differentiating MMPI items to be expected by chance. Newsletter of Research in Psychology, 1962, 4, 29-35. Welsh, G. S. Factor dimensions A and R. In G. S. Welsh & W. G. Dahlstrom (Eds.), Basic readings on the MMPI in psychology and medicine. Minneapolis: University of Minnesota Press, 1956. Wiggins, J. Content dimensions in the MMPI. In J. N. Butcher (Ed.), MMPI: Research developments and clinical applications. New York: McGraw-Hill, 1969. Wiggins, J. S. Personality and prediction: Principles of personality assessment. Reading, Mass.: Addison-Wesley, 1973. Received September 26, 1977 •
