Psychiatry Research,39:25-31
25
Elsevier
Cross-National Reliability, Concurrent Validity, and Stability of a Brief Method for Assessing Expressed Emotion Birgit Leeb, Kurt Hahlweg, Michael J. Goldstein, Elias Feinstein, Ulla Mueller, Matthias Dose, and Ana Magana-Amato Received January 4, 1991; revised version received July 17, 1991; accepted August IO, 1991. Abstract. The Five-Minute
Speech Sample (FMSS; Magana et al., 1986) is a brief method designed to assess the “expressed emotion” (EE) status of a respondent. The FMSS-EE rating is derived from statements made by a patient’s key relative during a 5-minute monologue, with codes similar to those used in the original EE rating system (e.g., criticism and emotional overinvolvement). Rating is done from the audiotape and takes about 15-30 minutes. The article reports on the results of a cross-national study carried out in Germany using the FMSS with a sample of 60 relatives of schizophrenic patients. Results indicated that (1) German investigators could readily learn the system and could achieve a high degree of interrater reliability; (2) the association with the original index of EE, the Camberwell Family Interview, was comparable to that found by Magana et al.; and (3) the method yields very stable data over a 4-5 week retest period.
Key Words. Schizophrenia, content Camberwell Family Interview.
analysis, speech sample, family studies,
Numerous studies have indicated that affective attitudes held by relatives toward a recently discharged schizophrenic patient are correlated with the probability of relapse over a 9- to 1Zmonth period following an index episode of the disorder (Brown et al., 1972; Vaughn and Leff, 1976~; Vaughn et al., 1984; Nuechterlein et al., 1986). While not all studies have found such a relationship (Dulz and Hand, 1986; Parker et al,, 1988), the preponderance of the evidence has been positive. The traditional method for assessing these affective attitudes is an instrument called the Camberwell Family Interview (CFI; Vaughn and Leff, 19763), which generally lasts from 1 to 1 ‘/z hours and requires considerable time to learn the system and to do each respondent’s rating. The construct rated has been termed expressed emotion (EE) and is based on the following criteria: (1) critical comments, which are counted; (2) hostility, which is rated on a 4-point scale; and (3) emotional overinvolvement (EOI), which is rated on a 5-point scale.
Birgit Leeb, DipL-Psych., Elias Feinstein, Dipl. Psych., Ulla Mueller, Ph.D., and Matthias Dose, M.D., all are Research Associates at the Max Planck Institute of Psychiatry, Munich, Germany. Michael J. Goldstein, Ph.D., is Professor of Psychology and Psychiatry, University of California, Los Angeles. Ana Magana-Amato is Research Assistant, Department of Psychology, University of California, Los Angeles. Kurt Hahlweg, Ph.D., is Professor of Psychology, Technical University Braunschweig, Braunschweig, Germany. (Reprints to Prof.Dr. Kurt Hahlweg, TU Braunschweig, Institut fiir Psychologie, Spielmannstrasse 12 a, W-3300 Braunschweig, Germany.) 0165-1781/91/$03.50
@ 1991 Elsevier Scientific
Publishers
Ireland
Ltd.
26 Although the CFI is a rich source of information as well as the basis for ratings of the high or low EE status of the respondent, the time taken to learn, administer, and score the instrument has limited its usefulness in both research and clinical practice. Therefore, Magana et al. (1986) proposed a brief method for assessing those affective attitudes that relied upon on a 5-minute speech sample (FMSS) modified from the Gottschalk and Gleser (1969) procedure with instructions developed by Gift et al. (1985). The FMSS task is aimed at identifying the relative’s attitudes and feelings as well as perceptions of the quality of the relative’s relationship with the patient. The 5-minute monologue is audiotaped and later analyzed, with both the verbal and the vocal aspects of the speech being taken into account. The following categories are used (Magana, 1986, p. 4): l
l
l
l
The quality of the initial statement rating is based on the first thought or idea expressed by the relative about the patient. The initial statement is rated as either positive, neutral, or negative. The quality of the relationship rating is based on statements that describe the relationship between the relative and the patient. The entire speech is taken into account before the relationship is rated as positive, neutral, or negative. A criticism is a comment indicating that the relative dislikes, resents, disapproves of, or is annoyed by the patient’s behavior or characteristics. Criticisms are assessed on the basis of content or tone, and a frequency count is taken over the entire speech. Emotional overinvolvement (EOI) is rated from statements which indicate that the relative is excessively involved with the patient. Five subcategories contribute to an EOI rating. Two of the five use frequency counts: (1) Statements of attitude (expression of strong positive feelings (love and devotion) toward the patient). (2) Positive remarks (excessive praise). The other three categories require ratings based on the entire speech sample. They are: (3) Reports of self-sacrificing/ overprotective behavior in the past. (4) Excessive detail (relative gives an excessive amount of irrelevant information about the relative’s past). (5) Emotional display (emotional breakdown, e.g., crying during the speech sample).
The criteria for determining EE status from the FMSS are the following: (1) High EE, critical. Any one of the following criteria must be met: (a) negative initial statement, (b) negative relationship rating, or (c) one criticism. (2) High EE, EOI. Any one of the following criteria must be met: (a) emotional display, (b) overprotective or self-sacrificing behavior, or (any two of the following) (c) excessive detail about the past, (d) statement of attitude, or (e) five positive remarks. (3) Low EE. This rating is assigned to respondents not fulfilling criteria (1) or (2). In the report by Magana et al. (1986), a close approximation to data from the CFI was obtained in both English-and Spanish-speaking relatives of schizophrenic patients; however, it was found that the FMSS yielded a lower rate of high EE relatives than the CFI. The purpose of the present study was three-faceted: To establish: (1) whether the FMSS scoring system could be used and rated with German-speaking relatives of schizophrenic patients; (2) whether the association with the concurrently
27 administered the measure
Phase
CFI was similar to that found was stable over time.
1: Training
and Reliability
with the U.S. samples;
and (3) whether
Procedures
Methods In 1986, Ana Magana came to Munich, Germany, and conducted, in English, a 15hour workshop on the administration and scoring of the FM.%-EE procedure using English tapes and an English manual. All trainees were bilingual. Written transcripts of each FMSS sample were also available. Nine Psychology graduates participated, of whom two had previously qualified as CFI-EE raters. At the end of the training, all raters independently rated a set of 10 tapes in English, which had been used to establish reliability in the American studies. Reliability estimates for each rater were computed for these tapes. After reliability was established, five of the raters trained on the English tapes, but who had no experience with the CFI, coded 10 audiotapes of German relatives who had participated in an ongoing project at the Max Planck Institute of Psychiatry in Munich on relapse prevention. Interrater agreement was established using one of the authors (B.L.) as criterion rater, as that person had 100% agreement on the American data set for the high vs. low EE categorization as well as the subcomponents that went into a rating of either high EE-critical or high EE-emotionally overinvolved.
Results All nine raters completed the original training course in English. Four achieved perfect agreement with the standard Magana ratings for the distinction between high and low FMSS-EE (K = 1.0); three achieved satisfactory agreement (K = 0.84); and two were marginal (K = 0.50). Thus, seven of nine German trainees could learn the system to a satisfactory level using an English language manual and rating Englishspeaking respondents. Next, we examined the agreement for five raters who had achieved satisfactory reliability on the English tapes when rating FMSS samples obtained from Germanspeaking relatives of schizophrenic patients. The agreement with the criterion rater (B.L.) was also quite satisfactory, as three raters achieved kappas of 0.80 for the high-low EE distinction, and for the remaining two, the kappas were 0.68 and 0.60. These figures indicated that the raters could use the system effectively in another linguistic and cultural context. These two analyses indicated that although trained in a second language, raters could use the system reliably. Reliability was a precondition for the next phase of this study which examined the association between FMSS-EE and CFI-EE ratings obtained concurrently.
Phase 2: Concurrent
Validity
of the FMSS-EE
Method
Method Sixty German relatives of 45 schizophrenic patients participated in this phase of All patients were diagnosed as suffering from schizophrenia on the basis Diagnostic Criteria (Spitzer et al., 1978). The mean age of the patients was 27 15-50). Twenty-two of the patients were female and 23 male. The 60 respondents
the research. of Research years (range fell into the
following categories: 26 mothers, 20 fathers, 12 spouses, 1 sister, and 1 grandmother. The FMSS was administered just before the CFI. The persons who administered the CFI (E.F. and U.M.) had been trained by Christine Vaughn and had also qualified as reliable CFI-EE raters (100% and 88% agreement, respectively) with the 10 English-language CFI tapes used as a standard part of the CFI training program. In addition, these two raters independently rated 10 of the German-speaking CFI tapes used in phase 2 of this study. Both raters were unaware of the FMSS-EE rating in those 10 cases. They achieved a 90% agreement on the high vs. low EE status of the relatives for those 10 German tapes, based on the standard scoring criteria for the CFI. The FMSS tapes were scored directly from the audiotape according to the manual (Magana et al., 1986) by one of the authors (B.L.), who had no knowledge of and did not participate in the CFI ratings.
Results Table 1 presents the association between the FMSS-EE and CFI-EE ratings for the 60 respondents. Below this are comparable data from the U.S. subjects as reported by Magana et al. (1986). As seen in Table 1, the degree of agreement is very similar in both groups: 73% overall agreement in the German group and 75% on the U.S. group. Both are highly significant by x2 test. In the German subjects, the sensitivity of the FMSS-EE rating was 80% and the specificity was 71%. The similarity between the two data sets is even more dramatic when expressed as percentages-FMSS-high EE, CFI-high EE: German subjects, 20%; U.S. subjects, 26%. FMSS-low EE, CFIhigh EE: German subjects, 22%; U.S. subjects, 20%. FMSS-high EE, CFI-low EE: German subjects, 5%; U.S. subjects, 5%. FMSS-low, CFI-low EE: German subjects, 53%; U.S. subjects, 48%. The German data set supports the earlier U.S. findings that subjects classified by the FMSS as high EE are almost always also classified as high EE by the CFI, but that about 20% of FMSS-defined low EE subjects are defined as high EE on the CFI. Table 1. Association between FMSS-EE and CFIEE ratings in German and U.S. Subjects German-speaking subjects (n = 60) CFI-EE
FMSS-EE
High
Low
High
12
Low
13
3 32
,y* 12.09, df = 1, p < 0.001 U.S. subjects (n = 114)’ CFI-EE
FMSS-EE
High
Low
High
30
Low
23
6 55
,y2=13.29,df=1,p
25
Elsevier
Cross-National Reliability, Concurrent Validity, and Stability of a Brief Method for Assessing Expressed Emotion Birgit Leeb, Kurt Hahlweg, Michael J. Goldstein, Elias Feinstein, Ulla Mueller, Matthias Dose, and Ana Magana-Amato Received January 4, 1991; revised version received July 17, 1991; accepted August IO, 1991. Abstract. The Five-Minute
Speech Sample (FMSS; Magana et al., 1986) is a brief method designed to assess the “expressed emotion” (EE) status of a respondent. The FMSS-EE rating is derived from statements made by a patient’s key relative during a 5-minute monologue, with codes similar to those used in the original EE rating system (e.g., criticism and emotional overinvolvement). Rating is done from the audiotape and takes about 15-30 minutes. The article reports on the results of a cross-national study carried out in Germany using the FMSS with a sample of 60 relatives of schizophrenic patients. Results indicated that (1) German investigators could readily learn the system and could achieve a high degree of interrater reliability; (2) the association with the original index of EE, the Camberwell Family Interview, was comparable to that found by Magana et al.; and (3) the method yields very stable data over a 4-5 week retest period.
Key Words. Schizophrenia, content Camberwell Family Interview.
analysis, speech sample, family studies,
Numerous studies have indicated that affective attitudes held by relatives toward a recently discharged schizophrenic patient are correlated with the probability of relapse over a 9- to 1Zmonth period following an index episode of the disorder (Brown et al., 1972; Vaughn and Leff, 1976~; Vaughn et al., 1984; Nuechterlein et al., 1986). While not all studies have found such a relationship (Dulz and Hand, 1986; Parker et al,, 1988), the preponderance of the evidence has been positive. The traditional method for assessing these affective attitudes is an instrument called the Camberwell Family Interview (CFI; Vaughn and Leff, 19763), which generally lasts from 1 to 1 ‘/z hours and requires considerable time to learn the system and to do each respondent’s rating. The construct rated has been termed expressed emotion (EE) and is based on the following criteria: (1) critical comments, which are counted; (2) hostility, which is rated on a 4-point scale; and (3) emotional overinvolvement (EOI), which is rated on a 5-point scale.
Birgit Leeb, DipL-Psych., Elias Feinstein, Dipl. Psych., Ulla Mueller, Ph.D., and Matthias Dose, M.D., all are Research Associates at the Max Planck Institute of Psychiatry, Munich, Germany. Michael J. Goldstein, Ph.D., is Professor of Psychology and Psychiatry, University of California, Los Angeles. Ana Magana-Amato is Research Assistant, Department of Psychology, University of California, Los Angeles. Kurt Hahlweg, Ph.D., is Professor of Psychology, Technical University Braunschweig, Braunschweig, Germany. (Reprints to Prof.Dr. Kurt Hahlweg, TU Braunschweig, Institut fiir Psychologie, Spielmannstrasse 12 a, W-3300 Braunschweig, Germany.) 0165-1781/91/$03.50
@ 1991 Elsevier Scientific
Publishers
Ireland
Ltd.
26 Although the CFI is a rich source of information as well as the basis for ratings of the high or low EE status of the respondent, the time taken to learn, administer, and score the instrument has limited its usefulness in both research and clinical practice. Therefore, Magana et al. (1986) proposed a brief method for assessing those affective attitudes that relied upon on a 5-minute speech sample (FMSS) modified from the Gottschalk and Gleser (1969) procedure with instructions developed by Gift et al. (1985). The FMSS task is aimed at identifying the relative’s attitudes and feelings as well as perceptions of the quality of the relative’s relationship with the patient. The 5-minute monologue is audiotaped and later analyzed, with both the verbal and the vocal aspects of the speech being taken into account. The following categories are used (Magana, 1986, p. 4): l
l
l
l
The quality of the initial statement rating is based on the first thought or idea expressed by the relative about the patient. The initial statement is rated as either positive, neutral, or negative. The quality of the relationship rating is based on statements that describe the relationship between the relative and the patient. The entire speech is taken into account before the relationship is rated as positive, neutral, or negative. A criticism is a comment indicating that the relative dislikes, resents, disapproves of, or is annoyed by the patient’s behavior or characteristics. Criticisms are assessed on the basis of content or tone, and a frequency count is taken over the entire speech. Emotional overinvolvement (EOI) is rated from statements which indicate that the relative is excessively involved with the patient. Five subcategories contribute to an EOI rating. Two of the five use frequency counts: (1) Statements of attitude (expression of strong positive feelings (love and devotion) toward the patient). (2) Positive remarks (excessive praise). The other three categories require ratings based on the entire speech sample. They are: (3) Reports of self-sacrificing/ overprotective behavior in the past. (4) Excessive detail (relative gives an excessive amount of irrelevant information about the relative’s past). (5) Emotional display (emotional breakdown, e.g., crying during the speech sample).
The criteria for determining EE status from the FMSS are the following: (1) High EE, critical. Any one of the following criteria must be met: (a) negative initial statement, (b) negative relationship rating, or (c) one criticism. (2) High EE, EOI. Any one of the following criteria must be met: (a) emotional display, (b) overprotective or self-sacrificing behavior, or (any two of the following) (c) excessive detail about the past, (d) statement of attitude, or (e) five positive remarks. (3) Low EE. This rating is assigned to respondents not fulfilling criteria (1) or (2). In the report by Magana et al. (1986), a close approximation to data from the CFI was obtained in both English-and Spanish-speaking relatives of schizophrenic patients; however, it was found that the FMSS yielded a lower rate of high EE relatives than the CFI. The purpose of the present study was three-faceted: To establish: (1) whether the FMSS scoring system could be used and rated with German-speaking relatives of schizophrenic patients; (2) whether the association with the concurrently
27 administered the measure
Phase
CFI was similar to that found was stable over time.
1: Training
and Reliability
with the U.S. samples;
and (3) whether
Procedures
Methods In 1986, Ana Magana came to Munich, Germany, and conducted, in English, a 15hour workshop on the administration and scoring of the FM.%-EE procedure using English tapes and an English manual. All trainees were bilingual. Written transcripts of each FMSS sample were also available. Nine Psychology graduates participated, of whom two had previously qualified as CFI-EE raters. At the end of the training, all raters independently rated a set of 10 tapes in English, which had been used to establish reliability in the American studies. Reliability estimates for each rater were computed for these tapes. After reliability was established, five of the raters trained on the English tapes, but who had no experience with the CFI, coded 10 audiotapes of German relatives who had participated in an ongoing project at the Max Planck Institute of Psychiatry in Munich on relapse prevention. Interrater agreement was established using one of the authors (B.L.) as criterion rater, as that person had 100% agreement on the American data set for the high vs. low EE categorization as well as the subcomponents that went into a rating of either high EE-critical or high EE-emotionally overinvolved.
Results All nine raters completed the original training course in English. Four achieved perfect agreement with the standard Magana ratings for the distinction between high and low FMSS-EE (K = 1.0); three achieved satisfactory agreement (K = 0.84); and two were marginal (K = 0.50). Thus, seven of nine German trainees could learn the system to a satisfactory level using an English language manual and rating Englishspeaking respondents. Next, we examined the agreement for five raters who had achieved satisfactory reliability on the English tapes when rating FMSS samples obtained from Germanspeaking relatives of schizophrenic patients. The agreement with the criterion rater (B.L.) was also quite satisfactory, as three raters achieved kappas of 0.80 for the high-low EE distinction, and for the remaining two, the kappas were 0.68 and 0.60. These figures indicated that the raters could use the system effectively in another linguistic and cultural context. These two analyses indicated that although trained in a second language, raters could use the system reliably. Reliability was a precondition for the next phase of this study which examined the association between FMSS-EE and CFI-EE ratings obtained concurrently.
Phase 2: Concurrent
Validity
of the FMSS-EE
Method
Method Sixty German relatives of 45 schizophrenic patients participated in this phase of All patients were diagnosed as suffering from schizophrenia on the basis Diagnostic Criteria (Spitzer et al., 1978). The mean age of the patients was 27 15-50). Twenty-two of the patients were female and 23 male. The 60 respondents
the research. of Research years (range fell into the
following categories: 26 mothers, 20 fathers, 12 spouses, 1 sister, and 1 grandmother. The FMSS was administered just before the CFI. The persons who administered the CFI (E.F. and U.M.) had been trained by Christine Vaughn and had also qualified as reliable CFI-EE raters (100% and 88% agreement, respectively) with the 10 English-language CFI tapes used as a standard part of the CFI training program. In addition, these two raters independently rated 10 of the German-speaking CFI tapes used in phase 2 of this study. Both raters were unaware of the FMSS-EE rating in those 10 cases. They achieved a 90% agreement on the high vs. low EE status of the relatives for those 10 German tapes, based on the standard scoring criteria for the CFI. The FMSS tapes were scored directly from the audiotape according to the manual (Magana et al., 1986) by one of the authors (B.L.), who had no knowledge of and did not participate in the CFI ratings.
Results Table 1 presents the association between the FMSS-EE and CFI-EE ratings for the 60 respondents. Below this are comparable data from the U.S. subjects as reported by Magana et al. (1986). As seen in Table 1, the degree of agreement is very similar in both groups: 73% overall agreement in the German group and 75% on the U.S. group. Both are highly significant by x2 test. In the German subjects, the sensitivity of the FMSS-EE rating was 80% and the specificity was 71%. The similarity between the two data sets is even more dramatic when expressed as percentages-FMSS-high EE, CFI-high EE: German subjects, 20%; U.S. subjects, 26%. FMSS-low EE, CFIhigh EE: German subjects, 22%; U.S. subjects, 20%. FMSS-high EE, CFI-low EE: German subjects, 5%; U.S. subjects, 5%. FMSS-low, CFI-low EE: German subjects, 53%; U.S. subjects, 48%. The German data set supports the earlier U.S. findings that subjects classified by the FMSS as high EE are almost always also classified as high EE by the CFI, but that about 20% of FMSS-defined low EE subjects are defined as high EE on the CFI. Table 1. Association between FMSS-EE and CFIEE ratings in German and U.S. Subjects German-speaking subjects (n = 60) CFI-EE
FMSS-EE
High
Low
High
12
Low
13
3 32
,y* 12.09, df = 1, p < 0.001 U.S. subjects (n = 114)’ CFI-EE
FMSS-EE
High
Low
High
30
Low
23
6 55
,y2=13.29,df=1,p
