Psychiatry Research 226 (2015) 186–191
Contents lists available at ScienceDirect
Psychiatry Research journal homepage: www.elsevier.com/locate/psychres
The public stigma of mental illness means a difference between you and me Patrick W. Corrigan n, Andrea B. Bink, J. Konadu Fokuo, Annie Schmidt Illinois Institute of Technology Department of Psychology 3424S. State Street, Chicago, IL 60616, USA
art ic l e i nf o
a b s t r a c t
Article history: Received 14 May 2014 Received in revised form 30 December 2014 Accepted 31 December 2014 Available online 9 January 2015
Social desirability can influence reports of stigma change in that subscribing to stigmatizing attitudes might pose a threat to personal beliefs of open-mindedness, while endorsing difference might not be as troubling. A measure is needed that assesses stigma change but is less susceptible to desirability effects. This study examined the psychometrics of various assessments of perceived difference from a person with mental illness. A total of 460 participants were recruited online using Amazon's Mechanical Turk. Four measures of difference, the Likert Scale of Difference, Semantic Differential: Similar-Different Scale, Semantic Differential: Mental Illness versus Other Illness scale, and Cause of Perceived Difference Scale were compared to measures of stereotypes, affirming attitudes, and care seeking. A vignette describing a person with mental illness anchored the Difference Scale and a measure of stereotype. Results showed that measures of difference yielded significantly higher endorsements than measures of stereotypes; the Semantic Differential Scale: Similar-Different was endorsed at a higher rate than other difference scales. Difference scores were positively related to stereotypes and inversely related to affirming attitudes. Difference was also found to influence empowerment separate from, and in addition to stereotype. These results suggest a new domain as an efficient and sensitive measure of stigma change. & 2015 Elsevier Ireland Ltd. All rights reserved.
Keywords: Attitude Stereotyping Prejudice Self-assessment Rehabilitation Empowerment Psychometrics
1. Introduction Public stigma has egregious effects on the lives of people with mental illness leading to significant barriers to the individual's pursuit of vocational, housing, and healthcare goals (Sartorius and Schulze, 2005; Callard et al., 2012). Many social scientists have described stigma as prejudice and discrimination; stereotypic beliefs that lead power groups – employers, landlords, and healthcare providers – to restrict opportunities of people labeled with mental illness (Link and Phelan, 2001). In addition, stigma might stop people with mental illness from seeking out care. Called label avoidance, people in distress avoid mental health treatment thereby escaping the stigmatizing label that accompanies it (Corrigan et al., 2014a). One model of prejudice and discrimination, based on attribution theory, has been widely tested to describe the stigmatizing experience of people with mental illness (Corrigan, 2000). The model rests on two empirically supported paths (Corrigan et al., 2003; Pingani et al., 2012; Roe et al., 2012): (1) beliefs that people are responsible for their mental illness lead to anger and an unwillingness to help.
n
Corresponding author. Tel.: þ 1 312 567 6751; fax: 1 312 567 6753. E-mail address: [email protected] (P.W. Corrigan).
http://dx.doi.org/10.1016/j.psychres.2014.12.047 0165-1781/& 2015 Elsevier Ireland Ltd. All rights reserved.
(2) beliefs that people with mental illness are dangerous lead to fear, desire to stay apart from this group, and calls for coercive treatment and institutionalization. A measure of this model, the Attribution Questionnaire (long and short forms), has been shown to be reliable, valid, and sensitive to the effects of stigma change programs (Corrigan et al., 2002; Brown, 2008; Pinto et al., 2012). Stigma, however, is more than endorsing disrespectful beliefs; labeled persons are placed into categories different from the majority resulting in separation of “us” from “them” (Link and Phelan, 2001). In some ways, this might be considered the “content-less” belief; there is no substantive attribution for separateness, only the assertion that people with mental illness are different from me. The Opinion about Mental Illness (OMI) scale, used for more than 50 years to document aspects of stigma, included an item on difference: “A heart patient has just one thing wrong with him while a mentally ill person is completely different from other patients” (Cohen and Struening, 1962). A subsequent review of the assessment literature on mental illness stigma uncovered 16 quantitative studies that included some measurement of cognitive separating (social labels imply difference between us and them), but separation was assessed only by one or two items per scale (Link et al., 2004). These studies did not include psychometrics on any assessment of difference per se. We conducted an additional review of the social science literature and
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
only found one paper measuring difference as a social construct; this however, was an assessment of the appreciation of human similarities and difference in general (Miville et al., 1999). Stigma measurement might be diminished by social desirability (Stier and Hinshaw, 2007; Corrigan and Shapiro, 2010); i.e., people underreport endorsement of stigmatizing beliefs in order to avoid perceptions of being bigoted and lacking open-mindedness. This is problematic when conducting outcome assessments of antistigma interventions. Floor effects on stigma measures that result from social desirability restrict the range of possible benefits after participating in anti-stigma programs. Viewing someone as different may be less threatening to personal beliefs of openmindedness. Hence, in addition to offering another way to understand the prejudice of mental illness, measures of “difference” may provide a more sensitive assessment of anti-stigma interventions. Stigmatizing beliefs and stereotypes have been assessed using varied psychometric strategies including Likert scales and semantic differentials with the object of difference being me, most other people, or people with other kinds of illness (Link et al., 2004; Corrigan and Shapiro, 2010). The purpose of this study is to test the psychometrics of various assessments of difference. We expect to show research participants are more likely to endorse items on these differentness scales than stigmatizing beliefs as measured on the Attribution Questionnaire. We also expect the differentness scales to be significantly associated with other measures of stigma.
187
pity, anger, help, danger, fear, avoidance, segregation, and coercion. Psychometrics on the AQ-9 suggest some problem with reliability and validity because of the pity item; pity might alternately be construed as a benefit (people who are pitied receive more help) or negative stereotype (pity may be perceived as degrading). Hence, an 8-item version of the AQ (AQ-8) was generated for these analyses representing mean response to all items except pity. Decreasing stereotypes and prejudice is not sufficient to erase the stigma; the public must also embrace affirming attitudes and behavior. Two scales have been developed and tested to evaluate the affirming constructs of recovery and empowerment (Corrigan et al., 2014b). The Recovery Scale (adapted from Corrigan et al., 2004) comprises three statements – e.g., People with mental illness are hopeful about their future. – to which participants respond with a 9-item agreement scale (9 ¼ strongly disagree). The Empowerment Scale (adapted from Rogers et al., 2010) also comprises three statements – e.g., People with mental illness are able to do things as well as most other people. – to which participants respond with a 9-item agreement scale (9 ¼ strongly disagree). Recovery and Empowerment scores were determined from averages of participant responses. Finally, in order to assess label avoidance, research participants completed the six item Care Seeking Scale. Using a 9-point agreement scale (9 ¼ strongly disagree), research participants reported willingness to seek help for anxiety or depression from traditional healthcare providers (e.g., primary care doctor, psychiatrist, counselor) or other helpers (clergy, friend/family, peer support). Although we hypothesize participant responses to Harry are based on his mental illness, we directly tested this in one last measure. The Cause of Perceived Difference Scale has research participants report why Harry was perceived “different from me” based on 10 descriptors taken directly from the vignette: e. g., Harry is 30 years old; Harry is single; Harry has schizophrenia; Sometimes Harry hears voices. Research participants responded to individual items using a 9-point agreement scale (9 ¼ agree very much). Note that the order of individual measures was randomly varied to avoid order effects. 2.1. Sample
2. Methods Stigma is often assessed by presenting a vignette of a person with serious mental illness. Our difference measures used a vignette that has been tested and validated in research on the Attribution Questionnaire (Corrigan et al., 2002): “Harry is a 30 year old single man with schizophrenia. Sometimes he hears voices and becomes upset. He lives alone in an apartment and works as a clerk at a large law firm. He has been hospitalized six times because of his illness.”
Several measures of difference were constructed reflecting methodologically varied perspectives of assessing psychological constructs like stigma. Scales were either Likert-format or semantic differential. Likert scales are interval measures from low to high on a construct; e.g., “Do you agree? Harry is dangerous.” In stigma research, the low to high distinction might evoke social desirability by artificially deflating endorsement of stereotypes. Semantic differentials require research participants to contrast polar ends of a construct: safe, dangerous. The low-high bias is less obvious, perhaps yielding diminished social desirability. Scales also differed in terms of referent: To whom was Harry compared to determine whether he was different: me, most other people, or people with other illnesses? Three measures were constructed based on these parameters. The Likert Scale of Difference followed Harry's vignette with eight items: four representing how different, unlike, comparable, and similar is Harry to me, and four items comparing Harry to most other people. Responses were made on 9-point agreement scales (9 ¼ agree very much). Three scores were determined by averaging responses for Difference from Me, Difference from Others, and Total Difference. The Semantic Differential: Similar-Different Scale comprised three items. In using the Semantic Differential, respondents choose their position vis-à-vis a psychological construct anchored by bipolar descriptors: e.g., unlike me – like me (Snider and Osgood, 1969). The closer a response is to one pole, the more the respondent endorses that pole. Research participants responded to 9-point semantic differentials where Harry was rated as similar or not similar to me, like or unlike me, and comparable or not comparable to me. A total score was determined by averaging responses across the three items. Finally, research participants were asked to rate which person with various illnesses was “most like me” using a different 9-point semantic differential; readers should note that choices made on this last scale concern a generic person and not Harry. Scales were anchored by five illnesses, which have been shown to be stigmatized in public response (Corrigan, 2014): mental illness, autism, Alzheimer's disease, alcoholism, and lung cancer. A Semantic Differential: MI versus Other Illness scale consisted of four items with mental illness and each of the four other illnesses at bipolar ends. A total score was determined by averaging responses between mental illness and the four other conditions. A short, 9-item, version of the Attribution Questionnaire (AQ-9) was administered to assess endorsement of stigmatizing stereotypes (Corrigan et al., 2014b). Research participants rate Harry on 9-point agreement scales (9 ¼ strongly agree) representing the stereotype constructs in our attribution model: responsibility,
Adults from across the United States were solicited to participate in this study using Mechanical Turk (MTurk) from February to March, 2014. MTurk, operated by Amazon, is a crowdsourcing internet marketplace that, among other things, is used to solicit participants for social science research. Data show more than 100,000 workers are registered with MTurk (Pontin, 2007). Research is mixed regarding the degree to which demographics of MTurk workers match the US population, though there is some consensus that MTurk samples work best for random population modeling (Paolacci et al., 2010; Ross et al., 2010; Buhrmester et al., 2011). A solicitation was posted on the MTurk Human Intelligence Tasks list requesting US workers to participate in a survey “examining knowledge and thoughts about mental health issues”. Consistent with our review of MTurk payments for similar social science projects, workers completing the task would be paid 25 cents. We were concerned about failing to meet recruitment goals after obtaining 98 participants, so the reimbursement rate was doubled to 50 cents. A total of 684 MTurk workers responded to the solicitation. One concern about online surveys is research participants who fail to fully attend to task. Our MTurk survey included validity questions meant to catch people in this group; e.g., “Please choose the number '8' for your answer below”. We also excluded people whose time on task was below the minimal cutoff to complete the survey competently or who took the survey more than once. After collecting 184 surveys, the research team learned the online survey was not compatible with some hand-held devices. We therefore excluded those surveys because it was unclear whether responses were tainted. As a result, 460 MTurk workers provided useable data. After being fully informed about the study and consenting to participate, survey respondents answered items about demographics. Table 1 summarizes their characteristics and shows that the sample, in some ways, paralleled the young and middle age American population. The sample was almost 35 years of age on average and 50% female. In terms of ethnicity, the sample was 82.8% European American but under represented African Americans (7.8%). More than 80% of the sample had some college experience with 45% having earned a degree. About 40% of the sample was employed full time, earning towards the low end of the scale (44.8% of the sample reported yearly income below $25,000). About 9% reported being LGBT. 2.2. Data analyses Since one of the primary goals of this study was to compare difference, stereotype, and affirming attitude scores, all Likert scales were set at 9 and averages determined as the index of interest. Moreover, direction of scales was set so that high scores represented stigmatizing responses: perceived difference, high stereotypes, low affirming attitudes, and low care seeking. Exploratory factor analyses (EFA) were completed to examine factor structure of the two scales specifically developed for this study: Cause of Perceived Difference and Care Seeking Scales. Internal consistency was reported for all measures as an index of reliability. Planned comparisons of scales measuring difference, stereotype, and affirming attitudes were examined using within subjects ANOVAs. Relationships among
188
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
Table 1 Summary of demographics of survey participants. Mean (S.D.) or frequencies Age Gender Race European/European American African/African American Asian/Asian American Native American Hawaiian/Pacific islander Other Not reported Ethnicity Latino/Latina Highest educational achievement Some high school High school diploma Associates degree Some college Bachelor's degree Graduate degree Employmenta Full time Part time None Annual income o 25,000 25,001–50,000 50,001–75,000 75,001–100,000 100,001–125,000 125,001–150,000 150,001–175,000 4 175,000 Sexual orientation LGB Straight Other
34.9 (12.3) 48.9% female 82.8% 7.8 6.3 1.1 0.9 0.4 0.7 4.3% 1.1% 13.0 10.7 29.6 33.3 12.6 43.9% 28.0 28.1 44.8% 31.3 15.0 6.3 1.5 0.2 0.2 0.7 8.9% 90.4 0.6
Note a Employment responses exceeded 100% because some participants reported part time and full time work.
measures of difference, stereotypes, affirming attitudes, and care seeking were examined with Pearson Product Moment Correlations. An important question is whether difference measures accounted for significant variance in affirming attitudes and care seeking after partialling out AQ-8 scores. Multiple regression analyses were conducted to address these questions.
3. Results Using principal component analyses and varimax rotation, the EFA of the Cause of Perceived Difference Scale yielded a two factor solution with Eigenvalues greater than 1.8. The first factor accounted for 47.4% of the variance and represented attribution of difference to illness descriptors (e.g., Harry has schizophrenia or sometimes Harry hears voices). The second factor (18.9% of variance) represented difference attributions based on demographics (Harry is a 30 year old single man). EFA of the Care Seeking Scale also yielded a two-factor solution (Eigenvalues greater than 1.1) with the first (46.8% of variance) representing intent to seek care from professionals (e.g., primary care physician, psychiatrist, or counselor) and the second (18.5% of variance) intent to seek care from others (clergy or self-help programs). Table 2 provides means and standard deviations of factor scores plus other scale scores. One of the primary hypotheses of this study was that research participants were more willing to admit that people with mental illness are different from them than endorse other stereotypes. Hence a within-subject ANOVA compared ratings on the three measures of differences (shaded in Table 2: Likert Scale of Total
Difference, Semantic Differential: Similar, Different, and Semantic Differential: MI versus Other Illnesses) and the 8 item Attribution Questionnaire (AQ-8) which represents stereotypes. Results were significant with a moderate effect size (F(3,457) ¼ 259.17; po 0.001; η2 ¼0.63). Post hoc contrasts showed the Attribution Scale score was lower (less stigmatizing) than each of the difference measures (p o0.001; η2 4 0.40). Another withinsubject ANOVA comparing scores from the three differentness measures yielded significant results (F(2,458) ¼165.23; p o0.001; η2 ¼0.42). Post hoc contrasts showed the three scores significantly differed from each other (p o0.001; η2 40.15) with the Semantic Differential: Similar-Different scale yielding the most stigmatizing scores. Low scores on the Recovery and Empowerment Scales also indicates stigma; e.g., people with mental illness do not get better and should not have power over their lives. Subsequent withinsubject ANOVAs showed the three difference scores were more likely to be endorsed than low recovery (F(3,457) ¼114.01; po 0.001; η2 ¼0.43) or low empowerment (F(3,457) ¼158.75; po 0.001; η2 ¼ 0.51). Post hoc tests showed two of three difference scores were endorsed significantly worse than low recovery (p o0.001; η2 40.13); all three difference scores were worse than low empowerment. To further understand the impact of difference scores, subfactor analyses of measures were conducted. Intrigued by the comparatively benign scores of the Semantic Differential: MI versus Other Illness scale, we examined how the MI differential compared to other conditions: lung cancer, autism, Alzheimer's disease, and alcoholism. Although results of a within-subject ANOVA were significant with a moderate effect size (F(4,456)¼46.25; po0.001; η2 ¼0.30), post hoc tests were contrary to what was expected. The MI versus Other Illness was significantly more benign than three other comparisons: other illnesses versus autism, Alzheimer's disease, and lung cancer (po0.001; η2 ¼ 0.15). We wondered if these unexpected effects were moderated by age, gender, sexual orientation, and race. Hence, we examined relationships between these demographics and the difference indicators: Likert Scale of Total Difference, Semantic Differential: Similar-Different Scale, and Semantic Differential: MI versus Other Illness. Age was significantly associated with all three (r ranged from 0.10–0.19, po0.05) with older participants more likely to endorse difference. Gender was significantly associated with only one difference measure: Semantic Differential: Similar-Different (r¼0.30, po0.001). Men were more likely to endorse difference than women. Difference scores were not found to vary significantly across sexual orientation (LGBT versus straight) or race (European American versus not European American). We also completed a within-subject ANOVA of the two subfactors of the Likert Scale of Difference Measure; nonsignificant trends (F(1,459) ¼ 3.56; p o0.10; η2 ¼0.01) suggested endorsement of the different-from-me scores was worse than the differencefrom-other score. The two subfactors of the Causes of Perceived Difference Scale were significantly different (F(1,459) ¼685.33; po 0.001; η2 ¼0.60) with research participants believing Harry's mental illness descriptors better accounted for perceived differences. In fact, Pearson Product Moment Correlations showed the mental illness descriptors were significantly associated with all three indices of difference (r ranged from 0.24–0.50, p o0.001) while demographic descriptors were weakly associated solely with the Semantic Differential: Similar-Different Scale (r ¼0.10, po 0.05). We sought to validate the difference scores by examining correlations with measures of other stereotypes, affirming attitudes, and care seeking; Table 3 summarizes these relationships. The top left section of the Table are relationships among the three difference measures which are all significant (po0.001). The relationship between Likert Scale of Total Difference Score and
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
189
Table 2 Mean and standard deviations of Difference Measures as well as measures of stigma, recovery, empowerment, and care seeking. Alphas for these measures are also provided. Measurement strategy
(S.D.)
Likert Scale of Total Difference Likert Scale of Difference from Me Likert Scale of Difference from Others Cause of Perceived Difference Scale: Demographic Descriptors Cause of Perceived Difference Scale: Mental Illness Descriptors Semantic Differential: Similar-Difference Scale Semantic Differential: MI-vs Other Illness Scale Autism versus Other Illness Scale Alzheimer's versus Other Illness Scale Lung Cancer versus Other Illness Scale Alcoholism versus Other Illness Scale Stigma on the 8 item Attribution Questionnaire Recovery Scale Empowerment Scale Care Seeking Scale: Professional Others (ministers, self-help)
Internal consistency
5.65 (1.80) 5.71 (2.07) 5.60 (1.78) 4.36 (2.50) 7.45 (1.75) 6.17 (2.01) 4.46 (1.33) 5.37 (1.14) 5.47 (0.97) 5.21 (1.14) 4.67 (1.32) 3.76 (1.15) 4.64 (1.50) 3.52 (2.36) 4.58 (2.02) 5.56 (2.09)
0.94 0.93 0.91 0.90 0.81 0.93 0.82 0.57 0.39 0.67 0.75 0.82 0.47 0.93 0.80 0.38
Table 3 Pearson Product Moment Correlations among Difference Scale Scores and with measures of stereotypes, recovery, empowerment, and care seeking. Like Scale Total Difference Likert Scale Total Difference SemDiff: SimilarDifference SemDiff: MI vs other illnesses AQ-8
SemDiff: SimilarDifference
SemDiff: MI vs other
Recovery Scale
Empowerment Scale
Care seeking: professional
Care seeking: others
0.78nnn
0.30nnn
0.19nnn
0.22nnn
0.11nn
–0.04
0.07
–0.08
0.32
nnn
0.15
n
0.07 0.43nnn
0.39nnn
0.22nnn
0.31nnn
0.17
nnn
nnn
nn
0.24
0.10
0.27nnn
0.13n
–0.08 –0.02
Note. SemDiff is Semantic Differential, AQ-8 is the 8-item Attribution Questionnaire. Note that Pearson r's for difference measures and both the Recovery Scale, Empowerment Scale and Care Seeking Scale are positive because higher scores on the three latter scales represent diminished endorsement of recovery and empowerment. n p o .05; nn p o.01; nnn p o.001.
Semantic Differential: Similar Difference yielded a moderate effect size (r2 ¼0.61). The AQ-8 was our index of stigmatizing stereotypes. The bottom row of the Table summarizes the relationship between the three difference scores and the AQ-8; coefficients for all were significant (p o0.001). The bottom row also showed positive significant relationships between the AQ-8 and the two measures of affirming attitudes: Recovery (r ¼0.31) and Empowerment (r ¼ 0.27). The AQ-8 was significantly associated with professional care seeking but not care seeking from others. In fact, the latter care seeking variable was not found to be associated with any of the difference measures. The Likert Scale of Total Difference and Semantic Differential: Similar-Different Scale were significantly associated with Recovery and Empowerment Scale scores. The Semantic-Differential: MI versus other illnesses was only associated significantly with the Empowerment Scale. The Care Seeking Scale professional subfactor was associated significantly with The Likert Scale of Total Difference and Semantic Differential: MI versus Other Illnesses scale. Note that while associations between AQ-8 and measures of affirming attitudes and care seeking were higher than the associations of these scales with the difference measures, t-tests of significance for correlation coefficients failed to show any of these differences as significant. Given that the measure of stereotypes (AQ-8) and some measures of difference were significantly associated with Recovery, Empowerment, and Care Seeking Scale scores, a final question was whether difference accounted for independent variance in recovery or empowerment after partialling out AQ-8. Results of
multiple regression analyses with Recovery, Empowerment, and Care Seeking Scores as dependent variables are summarized in Table 4. After entering AQ-8, none of the difference measures were found to predict significant independent variance in recovery. However, two of three difference measures – Likert Scale Total Difference Score, and Semantic Differential: MI versus other illnesses – accounted for significant additional variance in empowerment after partialling out AQ-8 (p o0.001). In addition, the partial correlation of Semantic Difference: Similar-Different was described by a non-significant trend (0.05 o po 0.10). For care seeking, none of the difference measures were found to predict significant independent variance after partialling out AQ-8 values. Effect size of all the regressions was low ranging from 0.07 to 0.10.
4. Discussion Perceived difference is hypothesized to be another way of conceptualizing the stigma of mental illness, which the public is more likely to endorse than measures of other stereotypes. Results supported this hypothesis with all three measures of difference yielding higher endorsements than items of the Attribution Questionnaire. The difference scale scores all had strong internal consistencies. Although participants endorsed stereotypes about Harry at a low level, they were quite willing to view him differently: altogether, from me, and from others. Effect sizes were well in the moderate range. Our data suggest Harry was viewed as
190
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
Table 4 Multiple regression analyses to determine whether the 8 item Attribution Questionnaire (AQ-8) and difference measures account for significant independent variance in Recovery and Empowerment Scale Scores. Dependent variable is Recovery Scale Score Model 1 AQ-8 Likert Scale Total Difference Score
beta 0.28 0.07
t 5.59*** 1.46 R2 ¼ 0.10
Model 2 AQ-8 Semantic Differential: Similar-Difference
0.29 0.03
6.09*** 0.65 R2 ¼0.09
Dependent variable is Empowerment Scale Score Model 1 AQ-8 Likert Scale Total Difference Score
beta 0.22 0.12
T 4.33*** 2.51** R2 ¼0.08
Model 2 AQ-8 Semantic Differential: Similar-Difference
0.24 0.08
4.87*** 1.66 ┤ R2 ¼0.07
Model 3 AQ-8 Semantic Differential: MI vs other illnesses
0.23 0.19
4.99*** 4.27*** R2 ¼0.10
Dependent variable is Care Seeking Scale Score Model 1 AQ-8 Likert Scale Total Difference Score
beta 0.10 0.07
T 1.92* 1.30 R2 ¼0.02
Model 2 AQ-8 Semantic Differential: MI vs other illnesses
0.11 0.07
2.36* 1.49 R2 ¼0.02
Note. AQ-8 is the *p o0.05;**po 0.01;***p o 0.001.
different because of his mental illness, and not demographic descriptors. Additional analyses unpacked aspects of the difference effect. The Semantic Differential: Similar-Different Scale yielded the greatest endorsements, significantly higher than the two other difference scores. This finding seems to support our assumption that the high-low range on a Likert scale accentuates social desirability effects – higher ratings of difference are perceived as more bigoted – suppressing scores on the Likert Scale of Total Difference. Comparison of the “difference from me” – “difference from others” scores on the Likert Scale yielded a non-significant trend. Although the difference was smaller than we expected, these results suggest viewing someone as different is a personal issue; comparing Harry to me yielded bigger effects. The semantic differential comparing mental illness to other conditions yielded the lowest endorsement of difference among the three measures. In part, this may have occurred because research participants compared a generic person with mental illness, rather than Harry, to other conditions. Perhaps loss of familiarity engendered by a vignette minimizes the stigma that occurs. Rather than some vague concept, the vignette labels Harry with schizophrenia reminding the reader this includes psychotic symptoms and hospitalization, thereby increasing the potency of stigma. Surprisingly, mental illness, along with alcoholism, was rated as more similar, and less stigmatizing to research participants than the other health conditions. Perhaps this occurred because vignette and questionnaires sensitized research participants to mental illness. Alternatively, the other conditions in the scale – lung cancer, Alzheimer's Disease, or autism – might seem less familiar and more rarified to research participants. Future research needs to examine these assumptions.
Difference scale scores were found to be significantly associated with other proxies of stigma. Significant correlations showed difference scores were positively associated with stereotypes on the AQ-8; research participants viewing people with mental illness as different were likely to endorse prejudice and discrimination towards that group. In addition, difference scores were most often inversely associated with affirming attitudes of recovery and empowerment. Perceived difference undermined beliefs that people with mental illness recover or should have personal power over their lives. In addition, high difference scores suggested research participants were less likely to seek professional care when in need. A final question examined in the study was whether relationships between difference scores and measures of affirming attitudes or care seeking overlapped with stereotypes assessed on the AQ-8 or accounted for unique variance. Results of multiple regression supported our question for empowerment. Measures of difference were significantly associated with the Empowerment Scale score after partialling out stereotypes on the AQ-8. In other words, difference perceptions influence the public's view of empowerment separate from, and in addition to stereotypes of the kind assessed on the AQ-8. There are limitations to these methods that need to be considered in future research. Despite providing ease in recruiting a large and seemingly diverse group of participants, MTurk did not yield a representative sample. In fact, the group of respondents in this study was over represented by research participants in lower income brackets. Future research may wish to use strategies to obtain data descriptive of populations. Still, absence of this kind of representativeness does not undermine findings of the study. Our goal was to test hypotheses about difference vis-à-vis other stereotypes and affirming attitudes. The MTurk group provided sufficient sample size to do this. Also, the correlations found here do not suggest order of effects; whether, for example, difference undermines care seeking, or if it is explained by some third variable. Future research needs to incorporate longitudinal strategies to better examine causal relationships. In fact, investigations should consider causal paths: if, for example, difference is primary, impacting stereotypic attitudes, which in turn undermines affirming attitudes and care seeking. In crafting the scales, we forgot to hold number of items equivalent across scales which may explain variation in scales. Future research may wish to consider the impact of this difference. Other research shows people who are relatively more familiar with mental illness through lived experience are less likely to endorse stigma (Corrigan et al., 2001). We did not assess survey participants for familiarity of mental illness or any of the other conditions. Perhaps people who have mental illness or close family members are less likely to endorse difference. Alternatively, perhaps difference is another aspect of self-stigma that undermines the self-esteem and self-efficacy of people with lived experience (Mittal et al., 2012). Subsequent studies need to incorporate lived experience as an important demographic of research participants. Assertions about differentness and social desirability would be stronger if a subsequent study included a separate measure of social desirability. In addition, future research should examine differentness in the broader landscape of social desirability biases. Other research has posed alternative ways to diminish social desirability including privacy in administration (Corrigan and Shapiro, 2010), alternative stimulus formats (Wolkenstein and Meyer, 2008), and implicit measures (assessed using reaction times; Monteith and Pettit, 2011). In the study described herein, privacy was maximized by assessing difference online and yet there were still significant differences between differentness and the other stigma measures. Wolkenstein and Meyer (2008) proposed a valuable approach to stigma assessment by introducing
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
attribution to some generalized other. Future research might combine the difference measures with alternative rating perspectives. Finally, future research might show difference is associated with implicit measures to a greater degree than the other measures of stigma. Differentness is presented here as a categorical variable; namely, that the person with mental illness is viewed in a qualitatively distinct group from the respondent. Some have argued that viewing symptoms and groupness on a continuum may be a way of diminishing stigma. Two recent studies, for example, showed continuum beliefs – we all have symptoms of mental illness in varying degrees – are inversely related to stigma (Schomerus et al., 2013; Wiesjahn et al., 2014). As one might expect, conditions like depression were viewed more on the continuum than schizophrenia (Schomerus et al., 2013). Finally, although differentness is framed as a negative view of people with mental illness, categorization per se does not mean stigma. For example, clinicians using the DSM are expressing difference but are not necessarily being stigmatizing. Future research needs an independent index to determine whether the difference scales represent negative perceptions. Social desirability continues to be a significant barrier to assessing the impact of anti-stigma programs on outcomes (Corrigan and Shapiro, 2010). Perhaps one of the greatest impacts of these findings is difference as a more sensitive measure of change. Future research should consider adding one of these indices of difference to its battery. Given that the 3-item Semantic Differential: SimilarDifferent seemed to be the most sensitive index of this construct, it is a relatively efficient way to assess impact on what may be a new domain. References Brown, S.A., 2008. Factors and measurement of mental illness stigma: a psychometric examination of the Attribution Questionnaire. Psychiatric Rehabilitation Journal 32, 89–94. http://dx.doi.org/10.2975/32.2.2008.89.94. Buhrmester, M., Kwang, T., Gosling, S.D., 2011. Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 3–5. http://dx.doi.org/10.1177/1745691610393980. Callard, F., Sartorius, N., Arboleda-Flórez, J., Bartlett, P., Helmchen, H., Stuart, H., Taborda, J., Thornicroft, G., 2012. Mental Illness, Discrimination and the Law: Fighting for Social Justice. John Wiley and Sons, Chichester, West Sussex, UK http://dx.doi.org/10.1002/9781119945352. Cohen, J., Struening, E.L., 1962. Opinions about mental illness in the personnel of two largemental hospitals. Journal of Abnormal and Social Psychology 64, 349–360. Corrigan, P.W., 2000. Mental health stigma as social attribution: implications for research methods and attitude change. Clinical Psychology: Science and Practice 7, 48–67. http://dx.doi.org/10.1093/clipsy.7.1.48. Corrigan, P.W., 2014. The Stigma of Disease and Disability: Understanding Causes and Overcoming Injustices. American Psychological Association, Washington DC, USA. Corrigan, P.W., Edwards, A.B., Green, A., Diwan, S.L., Penn, D., 2001. Prejudice, social distance, and familiarity with mental illness. Schizophrenia Bulletin 27, 219–225. Corrigan, P.W., Druss, B.G., Perlick, D., 2014a. The impact of mental illness stigma on seeking and participating in mental health care. Psychological Science in the Public Interest 15, 37–70. http://dx.doi.org/10.1177/1529100614531398. Corrigan, P.W., Markowitz, F.E., Watson, A., Rowan, D., Kubiak, M.A., 2003. An attribution model of public discrimination toward persons with mental illness. Journal of Health and Social Behavior 44, 162–179.
191
Corrigan, P.W., Powell, K.J., Michaels, P.J., 2014b. Brief battery for measurement of stigmatizing versus affirming attitudes about mental illness. Psychiatry Research 215, 466–470. http://dx.doi.org/10.1016/j.psychres.2013.12.006. Corrigan, P.W., Rowan, D., Green, A., Lundin, R., River, P., Uphoff-Wasowski, K., White, K., Kubiak, M.A., 2002. Challenging two mental illness stigmas: personal responsibility and dangerousness. Schizophrenia Bulletin 28, 293–309. Corrigan, P.W., Salzer, M., Ralph, R.O., Sangster, Y., Keck, L., 2004. Examining the factor structure of the Recovery Assessment Scale. Schizophrenia Bulletin 30, 1035–1041. Corrigan, P.W., Shapiro, J.R., 2010. Measuring the impact of programs that challenge the public stigma of mental illness. Clinical Psychology Review 30, 907–922. http://dx.doi.org/10.1016/j.cpr.2010.06.004. Link, B.G., Phelan, J.C., 2001. Conceptualizing stigma. Annual Review of Sociology 27, 363–385. http://dx.doi.org/10.1146/annurev.soc.27.1.363. Link, B.G., Yang, L.H., Phelan, J.C., Collins, P.Y., 2004. Measuring mental illness stigma. Schizophrenia Bulletin 30, 511–541. Mittal, D., Sullivan, G., Chekuri, L., Allee, E., Corrigan, P.W., 2012. Empirical studies of self-stigma reduction strategies: a critical review of the literature. Psychiatric Services 63, 974–981. Miville, M.L., Gelso, C.J., Pannu, R., Liu, W., Touradji, P., Holloway, P., Fuertes, J., 1999. Appreciating similarities and valuing differences: the Miville–Guzman Universality-Diversity Scale. Journal of Counseling Psychology 46, 291–307. Monteith, L.L., Pettit, J.W., 2011. Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal of Social and Clinical Psychology 30, 484–505. http://dx.doi.org/10.1521/jscp.2011.30.5.484. Paolacci, G., Chandler, J., Ipeirotis, P.G., 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 5, 411–419. Pingani, L., Forghieri, M., Ferrari, S., Ben-Zeev, D., Antoni, P., Mazzi, F., Palmieri, G., Rigatelli, M., Corrigan, P.W., 2012. Stigma and discrimination toward mental illness: translation and validation of the Italian version of the Attribution Questionnaire-27 (AQ-27-I). Social Psychiatry and Psychiatric Epidemiology 47, 993–999. http://dx.doi.org/10.1007/s00127-011-0407-3. Pinto, M.D., Hickman, R., Logsdon, M.C., Burant, C., 2012. Psychometric evaluation of the revised Attribution Questionnaire (r-AQ) to measure mental illness stigma in adolescents. Journal of Nursing Measurement 20, 47–58. Pontin, J., 2007, March 25. Artificial intelligence: with help from the humans. The New York Times. Retrieved from 〈http://www.nytimes.com/2007/03/25/busi ness/yourmoney/25Stream.html〉. Roe, D., Mashiach-Eizenberg, M., Corrigan, P., 2012. Confirmatory factor analysis of the brief version of the Recovery Assessment Scale. Journal of Nervous and Mental Disease 200, 847–851. http://dx.doi.org/10.1097/NMD.0b013e31826b 6d4a. Rogers, E.S., Ralph, R.O., Salzer, M.S., 2010. Validating the Empowerment Scale with a multisite sample of consumers of mental health services. Psychiatric Services 61, 933–936. http://dx.doi.org/10.1176/appi.ps.61.9.933. Ross, J., Irani, I., Silberman, M.S., Zaldivar, A., Tomlinson, B., 2010. Who are the crowdworkers? Shifting demographics in Amazon Mechanical Turk. Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems 2010 , http://dx.doi.org/10.1145/1753846.1753873 (Atlanta, GA). Sartorius, N., Schulze, H., 2005. Reducing the Stigma of Mental Illness: A Report from a Global Programme of the World Psychiatric Association. Cambridge University Press, New York, NY, USA. Schomerus, G., Matschinger, H., Angermeyer, M.C., 2013. Continuum beliefs and stigmatizing attitudes towards persons with schizophrenia, depression, and alcohol dependence. Psychiatry Research 209, 665–669. http://dx.doi.org/ 10.1016/j.psychres.2013.02.006. Snider, J.G., Osgood, C.E., 1969. Semantic Differential Technique: A Sourcebook. Aldine Publishing Co., Chicago, IL, USA. Stier, A., Hinshaw, S.P., 2007. Explicit and implicit stigma against individuals with mental illness. Australian Psychologist 42, 106–117. http://dx.doi.org/10.1080/ 00050060701280599. Wiesjahn, M., Brabban, A., Jung, E., Gebauer, U.B., Lincoln, T.M., 2014. Are continuum beliefs about psychotic symptoms associated with stereotypes about schizophrenia? Psychosis: Psychological Social and Integrative Approaches 6, 50–60. http://dx.doi.org/10.1080/17522439.2012.740068. Wolkenstein, L., Meyer, T.D., 2008. Attitudes of young people towards depression and mania. Psychology and Psychotherapy 81, 15–31.
Contents lists available at ScienceDirect
Psychiatry Research journal homepage: www.elsevier.com/locate/psychres
The public stigma of mental illness means a difference between you and me Patrick W. Corrigan n, Andrea B. Bink, J. Konadu Fokuo, Annie Schmidt Illinois Institute of Technology Department of Psychology 3424S. State Street, Chicago, IL 60616, USA
art ic l e i nf o
a b s t r a c t
Article history: Received 14 May 2014 Received in revised form 30 December 2014 Accepted 31 December 2014 Available online 9 January 2015
Social desirability can influence reports of stigma change in that subscribing to stigmatizing attitudes might pose a threat to personal beliefs of open-mindedness, while endorsing difference might not be as troubling. A measure is needed that assesses stigma change but is less susceptible to desirability effects. This study examined the psychometrics of various assessments of perceived difference from a person with mental illness. A total of 460 participants were recruited online using Amazon's Mechanical Turk. Four measures of difference, the Likert Scale of Difference, Semantic Differential: Similar-Different Scale, Semantic Differential: Mental Illness versus Other Illness scale, and Cause of Perceived Difference Scale were compared to measures of stereotypes, affirming attitudes, and care seeking. A vignette describing a person with mental illness anchored the Difference Scale and a measure of stereotype. Results showed that measures of difference yielded significantly higher endorsements than measures of stereotypes; the Semantic Differential Scale: Similar-Different was endorsed at a higher rate than other difference scales. Difference scores were positively related to stereotypes and inversely related to affirming attitudes. Difference was also found to influence empowerment separate from, and in addition to stereotype. These results suggest a new domain as an efficient and sensitive measure of stigma change. & 2015 Elsevier Ireland Ltd. All rights reserved.
Keywords: Attitude Stereotyping Prejudice Self-assessment Rehabilitation Empowerment Psychometrics
1. Introduction Public stigma has egregious effects on the lives of people with mental illness leading to significant barriers to the individual's pursuit of vocational, housing, and healthcare goals (Sartorius and Schulze, 2005; Callard et al., 2012). Many social scientists have described stigma as prejudice and discrimination; stereotypic beliefs that lead power groups – employers, landlords, and healthcare providers – to restrict opportunities of people labeled with mental illness (Link and Phelan, 2001). In addition, stigma might stop people with mental illness from seeking out care. Called label avoidance, people in distress avoid mental health treatment thereby escaping the stigmatizing label that accompanies it (Corrigan et al., 2014a). One model of prejudice and discrimination, based on attribution theory, has been widely tested to describe the stigmatizing experience of people with mental illness (Corrigan, 2000). The model rests on two empirically supported paths (Corrigan et al., 2003; Pingani et al., 2012; Roe et al., 2012): (1) beliefs that people are responsible for their mental illness lead to anger and an unwillingness to help.
n
Corresponding author. Tel.: þ 1 312 567 6751; fax: 1 312 567 6753. E-mail address: [email protected] (P.W. Corrigan).
http://dx.doi.org/10.1016/j.psychres.2014.12.047 0165-1781/& 2015 Elsevier Ireland Ltd. All rights reserved.
(2) beliefs that people with mental illness are dangerous lead to fear, desire to stay apart from this group, and calls for coercive treatment and institutionalization. A measure of this model, the Attribution Questionnaire (long and short forms), has been shown to be reliable, valid, and sensitive to the effects of stigma change programs (Corrigan et al., 2002; Brown, 2008; Pinto et al., 2012). Stigma, however, is more than endorsing disrespectful beliefs; labeled persons are placed into categories different from the majority resulting in separation of “us” from “them” (Link and Phelan, 2001). In some ways, this might be considered the “content-less” belief; there is no substantive attribution for separateness, only the assertion that people with mental illness are different from me. The Opinion about Mental Illness (OMI) scale, used for more than 50 years to document aspects of stigma, included an item on difference: “A heart patient has just one thing wrong with him while a mentally ill person is completely different from other patients” (Cohen and Struening, 1962). A subsequent review of the assessment literature on mental illness stigma uncovered 16 quantitative studies that included some measurement of cognitive separating (social labels imply difference between us and them), but separation was assessed only by one or two items per scale (Link et al., 2004). These studies did not include psychometrics on any assessment of difference per se. We conducted an additional review of the social science literature and
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
only found one paper measuring difference as a social construct; this however, was an assessment of the appreciation of human similarities and difference in general (Miville et al., 1999). Stigma measurement might be diminished by social desirability (Stier and Hinshaw, 2007; Corrigan and Shapiro, 2010); i.e., people underreport endorsement of stigmatizing beliefs in order to avoid perceptions of being bigoted and lacking open-mindedness. This is problematic when conducting outcome assessments of antistigma interventions. Floor effects on stigma measures that result from social desirability restrict the range of possible benefits after participating in anti-stigma programs. Viewing someone as different may be less threatening to personal beliefs of openmindedness. Hence, in addition to offering another way to understand the prejudice of mental illness, measures of “difference” may provide a more sensitive assessment of anti-stigma interventions. Stigmatizing beliefs and stereotypes have been assessed using varied psychometric strategies including Likert scales and semantic differentials with the object of difference being me, most other people, or people with other kinds of illness (Link et al., 2004; Corrigan and Shapiro, 2010). The purpose of this study is to test the psychometrics of various assessments of difference. We expect to show research participants are more likely to endorse items on these differentness scales than stigmatizing beliefs as measured on the Attribution Questionnaire. We also expect the differentness scales to be significantly associated with other measures of stigma.
187
pity, anger, help, danger, fear, avoidance, segregation, and coercion. Psychometrics on the AQ-9 suggest some problem with reliability and validity because of the pity item; pity might alternately be construed as a benefit (people who are pitied receive more help) or negative stereotype (pity may be perceived as degrading). Hence, an 8-item version of the AQ (AQ-8) was generated for these analyses representing mean response to all items except pity. Decreasing stereotypes and prejudice is not sufficient to erase the stigma; the public must also embrace affirming attitudes and behavior. Two scales have been developed and tested to evaluate the affirming constructs of recovery and empowerment (Corrigan et al., 2014b). The Recovery Scale (adapted from Corrigan et al., 2004) comprises three statements – e.g., People with mental illness are hopeful about their future. – to which participants respond with a 9-item agreement scale (9 ¼ strongly disagree). The Empowerment Scale (adapted from Rogers et al., 2010) also comprises three statements – e.g., People with mental illness are able to do things as well as most other people. – to which participants respond with a 9-item agreement scale (9 ¼ strongly disagree). Recovery and Empowerment scores were determined from averages of participant responses. Finally, in order to assess label avoidance, research participants completed the six item Care Seeking Scale. Using a 9-point agreement scale (9 ¼ strongly disagree), research participants reported willingness to seek help for anxiety or depression from traditional healthcare providers (e.g., primary care doctor, psychiatrist, counselor) or other helpers (clergy, friend/family, peer support). Although we hypothesize participant responses to Harry are based on his mental illness, we directly tested this in one last measure. The Cause of Perceived Difference Scale has research participants report why Harry was perceived “different from me” based on 10 descriptors taken directly from the vignette: e. g., Harry is 30 years old; Harry is single; Harry has schizophrenia; Sometimes Harry hears voices. Research participants responded to individual items using a 9-point agreement scale (9 ¼ agree very much). Note that the order of individual measures was randomly varied to avoid order effects. 2.1. Sample
2. Methods Stigma is often assessed by presenting a vignette of a person with serious mental illness. Our difference measures used a vignette that has been tested and validated in research on the Attribution Questionnaire (Corrigan et al., 2002): “Harry is a 30 year old single man with schizophrenia. Sometimes he hears voices and becomes upset. He lives alone in an apartment and works as a clerk at a large law firm. He has been hospitalized six times because of his illness.”
Several measures of difference were constructed reflecting methodologically varied perspectives of assessing psychological constructs like stigma. Scales were either Likert-format or semantic differential. Likert scales are interval measures from low to high on a construct; e.g., “Do you agree? Harry is dangerous.” In stigma research, the low to high distinction might evoke social desirability by artificially deflating endorsement of stereotypes. Semantic differentials require research participants to contrast polar ends of a construct: safe, dangerous. The low-high bias is less obvious, perhaps yielding diminished social desirability. Scales also differed in terms of referent: To whom was Harry compared to determine whether he was different: me, most other people, or people with other illnesses? Three measures were constructed based on these parameters. The Likert Scale of Difference followed Harry's vignette with eight items: four representing how different, unlike, comparable, and similar is Harry to me, and four items comparing Harry to most other people. Responses were made on 9-point agreement scales (9 ¼ agree very much). Three scores were determined by averaging responses for Difference from Me, Difference from Others, and Total Difference. The Semantic Differential: Similar-Different Scale comprised three items. In using the Semantic Differential, respondents choose their position vis-à-vis a psychological construct anchored by bipolar descriptors: e.g., unlike me – like me (Snider and Osgood, 1969). The closer a response is to one pole, the more the respondent endorses that pole. Research participants responded to 9-point semantic differentials where Harry was rated as similar or not similar to me, like or unlike me, and comparable or not comparable to me. A total score was determined by averaging responses across the three items. Finally, research participants were asked to rate which person with various illnesses was “most like me” using a different 9-point semantic differential; readers should note that choices made on this last scale concern a generic person and not Harry. Scales were anchored by five illnesses, which have been shown to be stigmatized in public response (Corrigan, 2014): mental illness, autism, Alzheimer's disease, alcoholism, and lung cancer. A Semantic Differential: MI versus Other Illness scale consisted of four items with mental illness and each of the four other illnesses at bipolar ends. A total score was determined by averaging responses between mental illness and the four other conditions. A short, 9-item, version of the Attribution Questionnaire (AQ-9) was administered to assess endorsement of stigmatizing stereotypes (Corrigan et al., 2014b). Research participants rate Harry on 9-point agreement scales (9 ¼ strongly agree) representing the stereotype constructs in our attribution model: responsibility,
Adults from across the United States were solicited to participate in this study using Mechanical Turk (MTurk) from February to March, 2014. MTurk, operated by Amazon, is a crowdsourcing internet marketplace that, among other things, is used to solicit participants for social science research. Data show more than 100,000 workers are registered with MTurk (Pontin, 2007). Research is mixed regarding the degree to which demographics of MTurk workers match the US population, though there is some consensus that MTurk samples work best for random population modeling (Paolacci et al., 2010; Ross et al., 2010; Buhrmester et al., 2011). A solicitation was posted on the MTurk Human Intelligence Tasks list requesting US workers to participate in a survey “examining knowledge and thoughts about mental health issues”. Consistent with our review of MTurk payments for similar social science projects, workers completing the task would be paid 25 cents. We were concerned about failing to meet recruitment goals after obtaining 98 participants, so the reimbursement rate was doubled to 50 cents. A total of 684 MTurk workers responded to the solicitation. One concern about online surveys is research participants who fail to fully attend to task. Our MTurk survey included validity questions meant to catch people in this group; e.g., “Please choose the number '8' for your answer below”. We also excluded people whose time on task was below the minimal cutoff to complete the survey competently or who took the survey more than once. After collecting 184 surveys, the research team learned the online survey was not compatible with some hand-held devices. We therefore excluded those surveys because it was unclear whether responses were tainted. As a result, 460 MTurk workers provided useable data. After being fully informed about the study and consenting to participate, survey respondents answered items about demographics. Table 1 summarizes their characteristics and shows that the sample, in some ways, paralleled the young and middle age American population. The sample was almost 35 years of age on average and 50% female. In terms of ethnicity, the sample was 82.8% European American but under represented African Americans (7.8%). More than 80% of the sample had some college experience with 45% having earned a degree. About 40% of the sample was employed full time, earning towards the low end of the scale (44.8% of the sample reported yearly income below $25,000). About 9% reported being LGBT. 2.2. Data analyses Since one of the primary goals of this study was to compare difference, stereotype, and affirming attitude scores, all Likert scales were set at 9 and averages determined as the index of interest. Moreover, direction of scales was set so that high scores represented stigmatizing responses: perceived difference, high stereotypes, low affirming attitudes, and low care seeking. Exploratory factor analyses (EFA) were completed to examine factor structure of the two scales specifically developed for this study: Cause of Perceived Difference and Care Seeking Scales. Internal consistency was reported for all measures as an index of reliability. Planned comparisons of scales measuring difference, stereotype, and affirming attitudes were examined using within subjects ANOVAs. Relationships among
188
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
Table 1 Summary of demographics of survey participants. Mean (S.D.) or frequencies Age Gender Race European/European American African/African American Asian/Asian American Native American Hawaiian/Pacific islander Other Not reported Ethnicity Latino/Latina Highest educational achievement Some high school High school diploma Associates degree Some college Bachelor's degree Graduate degree Employmenta Full time Part time None Annual income o 25,000 25,001–50,000 50,001–75,000 75,001–100,000 100,001–125,000 125,001–150,000 150,001–175,000 4 175,000 Sexual orientation LGB Straight Other
34.9 (12.3) 48.9% female 82.8% 7.8 6.3 1.1 0.9 0.4 0.7 4.3% 1.1% 13.0 10.7 29.6 33.3 12.6 43.9% 28.0 28.1 44.8% 31.3 15.0 6.3 1.5 0.2 0.2 0.7 8.9% 90.4 0.6
Note a Employment responses exceeded 100% because some participants reported part time and full time work.
measures of difference, stereotypes, affirming attitudes, and care seeking were examined with Pearson Product Moment Correlations. An important question is whether difference measures accounted for significant variance in affirming attitudes and care seeking after partialling out AQ-8 scores. Multiple regression analyses were conducted to address these questions.
3. Results Using principal component analyses and varimax rotation, the EFA of the Cause of Perceived Difference Scale yielded a two factor solution with Eigenvalues greater than 1.8. The first factor accounted for 47.4% of the variance and represented attribution of difference to illness descriptors (e.g., Harry has schizophrenia or sometimes Harry hears voices). The second factor (18.9% of variance) represented difference attributions based on demographics (Harry is a 30 year old single man). EFA of the Care Seeking Scale also yielded a two-factor solution (Eigenvalues greater than 1.1) with the first (46.8% of variance) representing intent to seek care from professionals (e.g., primary care physician, psychiatrist, or counselor) and the second (18.5% of variance) intent to seek care from others (clergy or self-help programs). Table 2 provides means and standard deviations of factor scores plus other scale scores. One of the primary hypotheses of this study was that research participants were more willing to admit that people with mental illness are different from them than endorse other stereotypes. Hence a within-subject ANOVA compared ratings on the three measures of differences (shaded in Table 2: Likert Scale of Total
Difference, Semantic Differential: Similar, Different, and Semantic Differential: MI versus Other Illnesses) and the 8 item Attribution Questionnaire (AQ-8) which represents stereotypes. Results were significant with a moderate effect size (F(3,457) ¼ 259.17; po 0.001; η2 ¼0.63). Post hoc contrasts showed the Attribution Scale score was lower (less stigmatizing) than each of the difference measures (p o0.001; η2 4 0.40). Another withinsubject ANOVA comparing scores from the three differentness measures yielded significant results (F(2,458) ¼165.23; p o0.001; η2 ¼0.42). Post hoc contrasts showed the three scores significantly differed from each other (p o0.001; η2 40.15) with the Semantic Differential: Similar-Different scale yielding the most stigmatizing scores. Low scores on the Recovery and Empowerment Scales also indicates stigma; e.g., people with mental illness do not get better and should not have power over their lives. Subsequent withinsubject ANOVAs showed the three difference scores were more likely to be endorsed than low recovery (F(3,457) ¼114.01; po 0.001; η2 ¼0.43) or low empowerment (F(3,457) ¼158.75; po 0.001; η2 ¼ 0.51). Post hoc tests showed two of three difference scores were endorsed significantly worse than low recovery (p o0.001; η2 40.13); all three difference scores were worse than low empowerment. To further understand the impact of difference scores, subfactor analyses of measures were conducted. Intrigued by the comparatively benign scores of the Semantic Differential: MI versus Other Illness scale, we examined how the MI differential compared to other conditions: lung cancer, autism, Alzheimer's disease, and alcoholism. Although results of a within-subject ANOVA were significant with a moderate effect size (F(4,456)¼46.25; po0.001; η2 ¼0.30), post hoc tests were contrary to what was expected. The MI versus Other Illness was significantly more benign than three other comparisons: other illnesses versus autism, Alzheimer's disease, and lung cancer (po0.001; η2 ¼ 0.15). We wondered if these unexpected effects were moderated by age, gender, sexual orientation, and race. Hence, we examined relationships between these demographics and the difference indicators: Likert Scale of Total Difference, Semantic Differential: Similar-Different Scale, and Semantic Differential: MI versus Other Illness. Age was significantly associated with all three (r ranged from 0.10–0.19, po0.05) with older participants more likely to endorse difference. Gender was significantly associated with only one difference measure: Semantic Differential: Similar-Different (r¼0.30, po0.001). Men were more likely to endorse difference than women. Difference scores were not found to vary significantly across sexual orientation (LGBT versus straight) or race (European American versus not European American). We also completed a within-subject ANOVA of the two subfactors of the Likert Scale of Difference Measure; nonsignificant trends (F(1,459) ¼ 3.56; p o0.10; η2 ¼0.01) suggested endorsement of the different-from-me scores was worse than the differencefrom-other score. The two subfactors of the Causes of Perceived Difference Scale were significantly different (F(1,459) ¼685.33; po 0.001; η2 ¼0.60) with research participants believing Harry's mental illness descriptors better accounted for perceived differences. In fact, Pearson Product Moment Correlations showed the mental illness descriptors were significantly associated with all three indices of difference (r ranged from 0.24–0.50, p o0.001) while demographic descriptors were weakly associated solely with the Semantic Differential: Similar-Different Scale (r ¼0.10, po 0.05). We sought to validate the difference scores by examining correlations with measures of other stereotypes, affirming attitudes, and care seeking; Table 3 summarizes these relationships. The top left section of the Table are relationships among the three difference measures which are all significant (po0.001). The relationship between Likert Scale of Total Difference Score and
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
189
Table 2 Mean and standard deviations of Difference Measures as well as measures of stigma, recovery, empowerment, and care seeking. Alphas for these measures are also provided. Measurement strategy
(S.D.)
Likert Scale of Total Difference Likert Scale of Difference from Me Likert Scale of Difference from Others Cause of Perceived Difference Scale: Demographic Descriptors Cause of Perceived Difference Scale: Mental Illness Descriptors Semantic Differential: Similar-Difference Scale Semantic Differential: MI-vs Other Illness Scale Autism versus Other Illness Scale Alzheimer's versus Other Illness Scale Lung Cancer versus Other Illness Scale Alcoholism versus Other Illness Scale Stigma on the 8 item Attribution Questionnaire Recovery Scale Empowerment Scale Care Seeking Scale: Professional Others (ministers, self-help)
Internal consistency
5.65 (1.80) 5.71 (2.07) 5.60 (1.78) 4.36 (2.50) 7.45 (1.75) 6.17 (2.01) 4.46 (1.33) 5.37 (1.14) 5.47 (0.97) 5.21 (1.14) 4.67 (1.32) 3.76 (1.15) 4.64 (1.50) 3.52 (2.36) 4.58 (2.02) 5.56 (2.09)
0.94 0.93 0.91 0.90 0.81 0.93 0.82 0.57 0.39 0.67 0.75 0.82 0.47 0.93 0.80 0.38
Table 3 Pearson Product Moment Correlations among Difference Scale Scores and with measures of stereotypes, recovery, empowerment, and care seeking. Like Scale Total Difference Likert Scale Total Difference SemDiff: SimilarDifference SemDiff: MI vs other illnesses AQ-8
SemDiff: SimilarDifference
SemDiff: MI vs other
Recovery Scale
Empowerment Scale
Care seeking: professional
Care seeking: others
0.78nnn
0.30nnn
0.19nnn
0.22nnn
0.11nn
–0.04
0.07
–0.08
0.32
nnn
0.15
n
0.07 0.43nnn
0.39nnn
0.22nnn
0.31nnn
0.17
nnn
nnn
nn
0.24
0.10
0.27nnn
0.13n
–0.08 –0.02
Note. SemDiff is Semantic Differential, AQ-8 is the 8-item Attribution Questionnaire. Note that Pearson r's for difference measures and both the Recovery Scale, Empowerment Scale and Care Seeking Scale are positive because higher scores on the three latter scales represent diminished endorsement of recovery and empowerment. n p o .05; nn p o.01; nnn p o.001.
Semantic Differential: Similar Difference yielded a moderate effect size (r2 ¼0.61). The AQ-8 was our index of stigmatizing stereotypes. The bottom row of the Table summarizes the relationship between the three difference scores and the AQ-8; coefficients for all were significant (p o0.001). The bottom row also showed positive significant relationships between the AQ-8 and the two measures of affirming attitudes: Recovery (r ¼0.31) and Empowerment (r ¼ 0.27). The AQ-8 was significantly associated with professional care seeking but not care seeking from others. In fact, the latter care seeking variable was not found to be associated with any of the difference measures. The Likert Scale of Total Difference and Semantic Differential: Similar-Different Scale were significantly associated with Recovery and Empowerment Scale scores. The Semantic-Differential: MI versus other illnesses was only associated significantly with the Empowerment Scale. The Care Seeking Scale professional subfactor was associated significantly with The Likert Scale of Total Difference and Semantic Differential: MI versus Other Illnesses scale. Note that while associations between AQ-8 and measures of affirming attitudes and care seeking were higher than the associations of these scales with the difference measures, t-tests of significance for correlation coefficients failed to show any of these differences as significant. Given that the measure of stereotypes (AQ-8) and some measures of difference were significantly associated with Recovery, Empowerment, and Care Seeking Scale scores, a final question was whether difference accounted for independent variance in recovery or empowerment after partialling out AQ-8. Results of
multiple regression analyses with Recovery, Empowerment, and Care Seeking Scores as dependent variables are summarized in Table 4. After entering AQ-8, none of the difference measures were found to predict significant independent variance in recovery. However, two of three difference measures – Likert Scale Total Difference Score, and Semantic Differential: MI versus other illnesses – accounted for significant additional variance in empowerment after partialling out AQ-8 (p o0.001). In addition, the partial correlation of Semantic Difference: Similar-Different was described by a non-significant trend (0.05 o po 0.10). For care seeking, none of the difference measures were found to predict significant independent variance after partialling out AQ-8 values. Effect size of all the regressions was low ranging from 0.07 to 0.10.
4. Discussion Perceived difference is hypothesized to be another way of conceptualizing the stigma of mental illness, which the public is more likely to endorse than measures of other stereotypes. Results supported this hypothesis with all three measures of difference yielding higher endorsements than items of the Attribution Questionnaire. The difference scale scores all had strong internal consistencies. Although participants endorsed stereotypes about Harry at a low level, they were quite willing to view him differently: altogether, from me, and from others. Effect sizes were well in the moderate range. Our data suggest Harry was viewed as
190
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
Table 4 Multiple regression analyses to determine whether the 8 item Attribution Questionnaire (AQ-8) and difference measures account for significant independent variance in Recovery and Empowerment Scale Scores. Dependent variable is Recovery Scale Score Model 1 AQ-8 Likert Scale Total Difference Score
beta 0.28 0.07
t 5.59*** 1.46 R2 ¼ 0.10
Model 2 AQ-8 Semantic Differential: Similar-Difference
0.29 0.03
6.09*** 0.65 R2 ¼0.09
Dependent variable is Empowerment Scale Score Model 1 AQ-8 Likert Scale Total Difference Score
beta 0.22 0.12
T 4.33*** 2.51** R2 ¼0.08
Model 2 AQ-8 Semantic Differential: Similar-Difference
0.24 0.08
4.87*** 1.66 ┤ R2 ¼0.07
Model 3 AQ-8 Semantic Differential: MI vs other illnesses
0.23 0.19
4.99*** 4.27*** R2 ¼0.10
Dependent variable is Care Seeking Scale Score Model 1 AQ-8 Likert Scale Total Difference Score
beta 0.10 0.07
T 1.92* 1.30 R2 ¼0.02
Model 2 AQ-8 Semantic Differential: MI vs other illnesses
0.11 0.07
2.36* 1.49 R2 ¼0.02
Note. AQ-8 is the *p o0.05;**po 0.01;***p o 0.001.
different because of his mental illness, and not demographic descriptors. Additional analyses unpacked aspects of the difference effect. The Semantic Differential: Similar-Different Scale yielded the greatest endorsements, significantly higher than the two other difference scores. This finding seems to support our assumption that the high-low range on a Likert scale accentuates social desirability effects – higher ratings of difference are perceived as more bigoted – suppressing scores on the Likert Scale of Total Difference. Comparison of the “difference from me” – “difference from others” scores on the Likert Scale yielded a non-significant trend. Although the difference was smaller than we expected, these results suggest viewing someone as different is a personal issue; comparing Harry to me yielded bigger effects. The semantic differential comparing mental illness to other conditions yielded the lowest endorsement of difference among the three measures. In part, this may have occurred because research participants compared a generic person with mental illness, rather than Harry, to other conditions. Perhaps loss of familiarity engendered by a vignette minimizes the stigma that occurs. Rather than some vague concept, the vignette labels Harry with schizophrenia reminding the reader this includes psychotic symptoms and hospitalization, thereby increasing the potency of stigma. Surprisingly, mental illness, along with alcoholism, was rated as more similar, and less stigmatizing to research participants than the other health conditions. Perhaps this occurred because vignette and questionnaires sensitized research participants to mental illness. Alternatively, the other conditions in the scale – lung cancer, Alzheimer's Disease, or autism – might seem less familiar and more rarified to research participants. Future research needs to examine these assumptions.
Difference scale scores were found to be significantly associated with other proxies of stigma. Significant correlations showed difference scores were positively associated with stereotypes on the AQ-8; research participants viewing people with mental illness as different were likely to endorse prejudice and discrimination towards that group. In addition, difference scores were most often inversely associated with affirming attitudes of recovery and empowerment. Perceived difference undermined beliefs that people with mental illness recover or should have personal power over their lives. In addition, high difference scores suggested research participants were less likely to seek professional care when in need. A final question examined in the study was whether relationships between difference scores and measures of affirming attitudes or care seeking overlapped with stereotypes assessed on the AQ-8 or accounted for unique variance. Results of multiple regression supported our question for empowerment. Measures of difference were significantly associated with the Empowerment Scale score after partialling out stereotypes on the AQ-8. In other words, difference perceptions influence the public's view of empowerment separate from, and in addition to stereotypes of the kind assessed on the AQ-8. There are limitations to these methods that need to be considered in future research. Despite providing ease in recruiting a large and seemingly diverse group of participants, MTurk did not yield a representative sample. In fact, the group of respondents in this study was over represented by research participants in lower income brackets. Future research may wish to use strategies to obtain data descriptive of populations. Still, absence of this kind of representativeness does not undermine findings of the study. Our goal was to test hypotheses about difference vis-à-vis other stereotypes and affirming attitudes. The MTurk group provided sufficient sample size to do this. Also, the correlations found here do not suggest order of effects; whether, for example, difference undermines care seeking, or if it is explained by some third variable. Future research needs to incorporate longitudinal strategies to better examine causal relationships. In fact, investigations should consider causal paths: if, for example, difference is primary, impacting stereotypic attitudes, which in turn undermines affirming attitudes and care seeking. In crafting the scales, we forgot to hold number of items equivalent across scales which may explain variation in scales. Future research may wish to consider the impact of this difference. Other research shows people who are relatively more familiar with mental illness through lived experience are less likely to endorse stigma (Corrigan et al., 2001). We did not assess survey participants for familiarity of mental illness or any of the other conditions. Perhaps people who have mental illness or close family members are less likely to endorse difference. Alternatively, perhaps difference is another aspect of self-stigma that undermines the self-esteem and self-efficacy of people with lived experience (Mittal et al., 2012). Subsequent studies need to incorporate lived experience as an important demographic of research participants. Assertions about differentness and social desirability would be stronger if a subsequent study included a separate measure of social desirability. In addition, future research should examine differentness in the broader landscape of social desirability biases. Other research has posed alternative ways to diminish social desirability including privacy in administration (Corrigan and Shapiro, 2010), alternative stimulus formats (Wolkenstein and Meyer, 2008), and implicit measures (assessed using reaction times; Monteith and Pettit, 2011). In the study described herein, privacy was maximized by assessing difference online and yet there were still significant differences between differentness and the other stigma measures. Wolkenstein and Meyer (2008) proposed a valuable approach to stigma assessment by introducing
P.W. Corrigan et al. / Psychiatry Research 226 (2015) 186–191
attribution to some generalized other. Future research might combine the difference measures with alternative rating perspectives. Finally, future research might show difference is associated with implicit measures to a greater degree than the other measures of stigma. Differentness is presented here as a categorical variable; namely, that the person with mental illness is viewed in a qualitatively distinct group from the respondent. Some have argued that viewing symptoms and groupness on a continuum may be a way of diminishing stigma. Two recent studies, for example, showed continuum beliefs – we all have symptoms of mental illness in varying degrees – are inversely related to stigma (Schomerus et al., 2013; Wiesjahn et al., 2014). As one might expect, conditions like depression were viewed more on the continuum than schizophrenia (Schomerus et al., 2013). Finally, although differentness is framed as a negative view of people with mental illness, categorization per se does not mean stigma. For example, clinicians using the DSM are expressing difference but are not necessarily being stigmatizing. Future research needs an independent index to determine whether the difference scales represent negative perceptions. Social desirability continues to be a significant barrier to assessing the impact of anti-stigma programs on outcomes (Corrigan and Shapiro, 2010). Perhaps one of the greatest impacts of these findings is difference as a more sensitive measure of change. Future research should consider adding one of these indices of difference to its battery. Given that the 3-item Semantic Differential: SimilarDifferent seemed to be the most sensitive index of this construct, it is a relatively efficient way to assess impact on what may be a new domain. References Brown, S.A., 2008. Factors and measurement of mental illness stigma: a psychometric examination of the Attribution Questionnaire. Psychiatric Rehabilitation Journal 32, 89–94. http://dx.doi.org/10.2975/32.2.2008.89.94. Buhrmester, M., Kwang, T., Gosling, S.D., 2011. Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 3–5. http://dx.doi.org/10.1177/1745691610393980. Callard, F., Sartorius, N., Arboleda-Flórez, J., Bartlett, P., Helmchen, H., Stuart, H., Taborda, J., Thornicroft, G., 2012. Mental Illness, Discrimination and the Law: Fighting for Social Justice. John Wiley and Sons, Chichester, West Sussex, UK http://dx.doi.org/10.1002/9781119945352. Cohen, J., Struening, E.L., 1962. Opinions about mental illness in the personnel of two largemental hospitals. Journal of Abnormal and Social Psychology 64, 349–360. Corrigan, P.W., 2000. Mental health stigma as social attribution: implications for research methods and attitude change. Clinical Psychology: Science and Practice 7, 48–67. http://dx.doi.org/10.1093/clipsy.7.1.48. Corrigan, P.W., 2014. The Stigma of Disease and Disability: Understanding Causes and Overcoming Injustices. American Psychological Association, Washington DC, USA. Corrigan, P.W., Edwards, A.B., Green, A., Diwan, S.L., Penn, D., 2001. Prejudice, social distance, and familiarity with mental illness. Schizophrenia Bulletin 27, 219–225. Corrigan, P.W., Druss, B.G., Perlick, D., 2014a. The impact of mental illness stigma on seeking and participating in mental health care. Psychological Science in the Public Interest 15, 37–70. http://dx.doi.org/10.1177/1529100614531398. Corrigan, P.W., Markowitz, F.E., Watson, A., Rowan, D., Kubiak, M.A., 2003. An attribution model of public discrimination toward persons with mental illness. Journal of Health and Social Behavior 44, 162–179.
191
Corrigan, P.W., Powell, K.J., Michaels, P.J., 2014b. Brief battery for measurement of stigmatizing versus affirming attitudes about mental illness. Psychiatry Research 215, 466–470. http://dx.doi.org/10.1016/j.psychres.2013.12.006. Corrigan, P.W., Rowan, D., Green, A., Lundin, R., River, P., Uphoff-Wasowski, K., White, K., Kubiak, M.A., 2002. Challenging two mental illness stigmas: personal responsibility and dangerousness. Schizophrenia Bulletin 28, 293–309. Corrigan, P.W., Salzer, M., Ralph, R.O., Sangster, Y., Keck, L., 2004. Examining the factor structure of the Recovery Assessment Scale. Schizophrenia Bulletin 30, 1035–1041. Corrigan, P.W., Shapiro, J.R., 2010. Measuring the impact of programs that challenge the public stigma of mental illness. Clinical Psychology Review 30, 907–922. http://dx.doi.org/10.1016/j.cpr.2010.06.004. Link, B.G., Phelan, J.C., 2001. Conceptualizing stigma. Annual Review of Sociology 27, 363–385. http://dx.doi.org/10.1146/annurev.soc.27.1.363. Link, B.G., Yang, L.H., Phelan, J.C., Collins, P.Y., 2004. Measuring mental illness stigma. Schizophrenia Bulletin 30, 511–541. Mittal, D., Sullivan, G., Chekuri, L., Allee, E., Corrigan, P.W., 2012. Empirical studies of self-stigma reduction strategies: a critical review of the literature. Psychiatric Services 63, 974–981. Miville, M.L., Gelso, C.J., Pannu, R., Liu, W., Touradji, P., Holloway, P., Fuertes, J., 1999. Appreciating similarities and valuing differences: the Miville–Guzman Universality-Diversity Scale. Journal of Counseling Psychology 46, 291–307. Monteith, L.L., Pettit, J.W., 2011. Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal of Social and Clinical Psychology 30, 484–505. http://dx.doi.org/10.1521/jscp.2011.30.5.484. Paolacci, G., Chandler, J., Ipeirotis, P.G., 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 5, 411–419. Pingani, L., Forghieri, M., Ferrari, S., Ben-Zeev, D., Antoni, P., Mazzi, F., Palmieri, G., Rigatelli, M., Corrigan, P.W., 2012. Stigma and discrimination toward mental illness: translation and validation of the Italian version of the Attribution Questionnaire-27 (AQ-27-I). Social Psychiatry and Psychiatric Epidemiology 47, 993–999. http://dx.doi.org/10.1007/s00127-011-0407-3. Pinto, M.D., Hickman, R., Logsdon, M.C., Burant, C., 2012. Psychometric evaluation of the revised Attribution Questionnaire (r-AQ) to measure mental illness stigma in adolescents. Journal of Nursing Measurement 20, 47–58. Pontin, J., 2007, March 25. Artificial intelligence: with help from the humans. The New York Times. Retrieved from 〈http://www.nytimes.com/2007/03/25/busi ness/yourmoney/25Stream.html〉. Roe, D., Mashiach-Eizenberg, M., Corrigan, P., 2012. Confirmatory factor analysis of the brief version of the Recovery Assessment Scale. Journal of Nervous and Mental Disease 200, 847–851. http://dx.doi.org/10.1097/NMD.0b013e31826b 6d4a. Rogers, E.S., Ralph, R.O., Salzer, M.S., 2010. Validating the Empowerment Scale with a multisite sample of consumers of mental health services. Psychiatric Services 61, 933–936. http://dx.doi.org/10.1176/appi.ps.61.9.933. Ross, J., Irani, I., Silberman, M.S., Zaldivar, A., Tomlinson, B., 2010. Who are the crowdworkers? Shifting demographics in Amazon Mechanical Turk. Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems 2010 , http://dx.doi.org/10.1145/1753846.1753873 (Atlanta, GA). Sartorius, N., Schulze, H., 2005. Reducing the Stigma of Mental Illness: A Report from a Global Programme of the World Psychiatric Association. Cambridge University Press, New York, NY, USA. Schomerus, G., Matschinger, H., Angermeyer, M.C., 2013. Continuum beliefs and stigmatizing attitudes towards persons with schizophrenia, depression, and alcohol dependence. Psychiatry Research 209, 665–669. http://dx.doi.org/ 10.1016/j.psychres.2013.02.006. Snider, J.G., Osgood, C.E., 1969. Semantic Differential Technique: A Sourcebook. Aldine Publishing Co., Chicago, IL, USA. Stier, A., Hinshaw, S.P., 2007. Explicit and implicit stigma against individuals with mental illness. Australian Psychologist 42, 106–117. http://dx.doi.org/10.1080/ 00050060701280599. Wiesjahn, M., Brabban, A., Jung, E., Gebauer, U.B., Lincoln, T.M., 2014. Are continuum beliefs about psychotic symptoms associated with stereotypes about schizophrenia? Psychosis: Psychological Social and Integrative Approaches 6, 50–60. http://dx.doi.org/10.1080/17522439.2012.740068. Wolkenstein, L., Meyer, T.D., 2008. Attitudes of young people towards depression and mania. Psychology and Psychotherapy 81, 15–31.
