Journal of Anxiety Disorders 28 (2014) 966–970

Contents lists available at ScienceDirect

Journal of Anxiety Disorders

Screen for Child Anxiety Related Emotional Disorders: Are subscale scores reliable? A bifactor model analysis Diogo Araújo DeSousa a,b,∗ , Murilo Ricardo Zibetti c , Clarissa Marceli Trentini c , Silvia Helena Koller b , Gisele Gus Manfro a,d , Giovanni Abrahão Salum a,d a Anxiety Disorders Outpatient Program for Child and Adolescent Psychiatry (PROTAIA), Hospital de Clínicas de Porto Alegre (HCPA), Federal University of Rio Grande do Sul (UFRGS), Brazil b Center for Psychological Studies on At-Risk Populations (CEP-Rua), Institute of Psychology, Federal University of Rio Grande do Sul (UFRGS), Brazil c Study Group in Psychopathology and Psychological Assessment, Institute of Psychology, Federal University of Rio Grande do Sul (UFRGS), Brazil d National Science and Technology Institute for Child and Adolescent Psychiatry (INPD), Hospital de Clínicas de Porto Alegre (HCPA), Brazil

a r t i c l e

i n f o

Article history: Received 23 March 2014 Received in revised form 18 October 2014 Accepted 21 October 2014 Available online 30 October 2014 Keywords: Screen for Child Anxiety Related Emotional Disorders Factor structure Bifactor model Anxiety Psychometrics

a b s t r a c t The aim of this study was to investigate the utility of creating and scoring subscales for the self-report version of the Screen for Child Anxiety Related Emotional Disorders (SCARED) by examining whether subscale scores provide reliable information after accounting for a general anxiety factor in a bifactor model analysis. A total of 2420 children aged 9–18 answered the SCARED in their schools. Results suggested adequate fit of the bifactor model. The SCARED score variance was hardly influenced by the specific domains after controlling for the common variance in the general factor. The explained common variance (ECV) for the general factor was large (63.96%). After accounting for the general total score (ωh = .83), subscale scores provided very little reliable information (ωh ranged from .005 to .04). Practitioners that use the SCARED should be careful when scoring and interpreting the instrument subscales since there is more common variance to them than specific variance. © 2014 Elsevier Ltd. All rights reserved.

As described in the Diagnostic and Statistical Manual of Mental Disorders (DSM 5; American Psychiatric Association, 2013), the group of anxiety disorders includes different subtypes of mental illnesses such as separation, social, and generalized anxiety disorders. On one hand, all disorders in this group share some core symptoms that include subjective, physiological, and behavioral symptoms of fear and anxiety. On the other hand, each anxiety disorder presents specific symptoms related to the phobic or anxiogenic objects/situations (Beesdo, Knappe, & Pine, 2009; Craske et al., 2009; Salum, DeSousa, Rosário, Pine, & Manfro, 2013). Such conceptualization of mental disorders in childhood creates a very difficult task for those aiming to assess and quantify anxiety symptoms using self-report questionnaires and scales. They are challenged to balance the degree of conceptual breadth of the measurement instrument, deciding whether a broad or a narrow construct is

∗ Corresponding author at: 2600 Ramiro Barcelos, Room 104, 90035-003 Porto Alegre, RS, Brazil. Tel.: +55 51 33085150; fax: +55 51 32410074. E-mail addresses: [email protected] (D.A. DeSousa), [email protected] (M.R. Zibetti), [email protected] (C.M. Trentini), [email protected] (S.H. Koller), [email protected] (G.G. Manfro), [email protected] (G.A. Salum). http://dx.doi.org/10.1016/j.janxdis.2014.10.002 0887-6185/© 2014 Elsevier Ltd. All rights reserved.

going to be assessed when choosing the items to be included in the instrument. One of the most used scales to measure anxiety symptoms in childhood is the Screen for Child Anxiety Related Emotional Disorders (SCARED; Birmaher et al., 1997, 1999). The SCARED is a self-report measure of pediatric anxiety symptoms frequency/intensity. The scale was conceptualized as having five factors: panic/somatic; generalized anxiety; social phobia; separation anxiety; and school phobia. The answers to the SCARED, similarly to other multidimensional measures of pediatric anxiety, are often psychometrically analyzed calculating five subscale scores and, at the same time, a summed up total score of anxiety level. Therefore researchers are interested in measuring multiple aspects of the anxiety trait, but at the same time, keenly interested in scaling individuals on the general trait that underlies the diverse aspects of the construct. This approach can lead to interpretation ambiguity since it does not separate the specific contributions of each domain or subscale (i.e., each specific anxiety disorder) from the common variance shared by all of them (i.e., the general construct of anxiety) (Chen, Hayes, Carver, Laurenceau, & Zhang, 2012). After accounting for the general factor it is unclear if the item set is going to allow a precise scale of individuals on separate dimensions.

D.A. DeSousa et al. / Journal of Anxiety Disorders 28 (2014) 966–970

To address this specific question, the measurement of latent psychological constructs at different levels of the construct hierarchy using bifactor model analyses has been reported in recent studies. For instance, bifactor models have been used to investigate the multifaceted constructs of depression (Brouwer, Meijer, & Zevalkink, 2013), sensitivity to anxiety (Ebesutani, McLeish, Luberto, Young, & Maack, 2013), intelligence (Gignac & Watkins, 2013), and personality (Chen et al., 2012). In a bifactor model, the commonality of all scale items is shared by the general factor and, after that, the contribution of each cluster of items to the distinct specific factors is calculated (Chen et al., 2012; Reise, 2012). This approach also allows bifactor models to check whether the specific factors are maintained as distinct after controlling for the shared variance among them that is captured by the general factor (Chen et al., 2012; Reise, 2012). Since the group of anxiety disorders (a) share some core symptoms of fear and anxiety and (b) present high rates of comorbidity between the specific disorders, but also (c) present specificities from each of the disorders it is theoretically plausible to consider a bifactor model for interpreting the assessment of anxiety disorders symptomatology through self-report questionnaire scores. A similar approach was used in an evaluation of the Revised Child Anxiety and Depression Scale (RCDAS; Ebesutani et al., 2012) investigating the suitability of an exploratory bifactor model for the 37 specific anxiety symptoms in the scale. Nonetheless to our knowledge no studies have tested the adequacy of a bifactor model to the factor structure of the Screen for Child Anxiety Related Emotional Disorders (SCARED). The objective of this study was therefore to test the utility of creating and scoring subscales for the SCARED as a measure of anxiety disorders symptoms. We aimed to investigate if, after accounting for the shared variance between all the items, SCARED subscales are still reliable to inform about distinct dimensions of anxiety. 1. Method 1.1. Participants and procedures A total of 2457 children participated in the study in public schools in the city of Porto Alegre, capital of the state of Rio Grande do Sul, the southernmost state in Brazil. Of these, 37 protocols (1.5%) were excluded due to presenting missing data (current analyses were performed listwise). Therefore our analytic sample was composed of 2420 children aged 9–18 years (M = 13.74; SD = 2.34), 53.3% girls, participated in the study Prior to the study, both students and their parents received written information and parents were required to provide written informed dissent. Parents who did not give permission for their child to participate were asked to return a signed dissent form. Written informed consent was obtained for all the participating schools and the study design was reviewed and approved by the ethics committee of Hospital de Clínicas de Porto Alegre (protocol number 08-017). The self-report version of the SCARED was administered to all students that agreed to participate. All participants were asked to complete the questionnaire in their schools. This study is part of a larger cross-sectional project named “Multidimensional Evaluation and Treatment of Anxiety in Children and Adolescents – the PROTAIA project” (Salum et al., 2011). For more detailed descriptions about sample procedures and design, see Salum et al. (2011). 1.2. Instrument The SCARED (Birmaher et al., 1997, 1999) is a 41-item measure of pediatric anxiety symptoms divided into five factors: panic/somatic (SOM; 13 items; e.g., “when I get frightened, my heart beats fast”);

967

generalized anxiety (GAD; 9 items; e.g., “I am a worrier”); social phobia or social anxiety (SoAD; 7 items; e.g., “I feel shy with people I don’t know well”); separation anxiety (SeAD; 8 items; e.g., “I worry that something bad might happen to my parents”); and school phobia (SCH; 4 items; e.g., “I worry about going to school”). In this study we used the self-report version of the instrument. Children score each item in a 3-point scale (0 = not true or hardly ever true; 1 = sometimes true; 2 = true or often true). Total scores range from 0 to 82, with higher scores reflecting higher anxiety levels. A recent meta-analysis evaluated the cross-cultural psychometric properties of the SCARED and found that the instrument has robust psychometric properties and can be used in different countries as a screening measure of pediatric anxiety symptoms (Hale, Crocetti, Raaijmakers, & Meeus, 2011). In Brazil, the SCARED has been translated to Brazilian–Portuguese and presented good psychometric properties (DeSousa, Salum, Isolan, & Manfro, 2013; Isolan, Salum, Osowski, Amaro, & Manfro, 2011). To our knowledge there are no other psychometric studies that have evaluated the factor structure of the SCARED using a bifactor model approach. 1.3. Data analysis Confirmatory Factor Analysis (CFA) was used to evaluate the latent factor structure of the anxiety symptoms measured by the SCARED. We compared the fit of three different models: (1) one factor; (2) five correlated factors; and (3) a bifactor model with a general factor and five group factors based on the five correlated factors model. In the previous study of Isolan et al. (2011), the factor structure of the Brazilian version of the SCARED was tested using the estimation method of Unweighted Least Squares (ULS). Although the ULS accounts for a lack of normality in data, it does not account for the categorical nature of the SCARED items. Therefore in this study we conducted the CFA using the Weighted Least Squares Means and Variance Adjusted (WLSMV) estimation method in the Mplus software version 7.11. We calculated the Comparative Fit Index (CFI), Tucker–Lewis Index (TLI), and Root Mean Square Error of Approximation (RMSEA) with 90% Confidence Interval fit indices. Values of the CFI and TLI higher than .90 represent an acceptable fit, and higher than .95 represent a good fit. Values of the RMSEA lower than .08 represent an acceptable fit, and lower than .05 represent a good fit (Hu & Bentler, 1999). In regard to the factor loadings of the items, the factor loadings of the general factor in the bifactor model were compared with the loadings of the one factor model. If the general factor presents considerably lower factor loadings than the one factor, it is evidenced that the variance in the item responses is influenced by the group factors and therefore the data cannot be interpreted as unidimensional. Furthermore the factor loadings of the group factors in the bifactor model were compared with the loadings of the five correlated factors model. The differences between these loadings indicate the degree to which the variance in the item responses in the correlated factors model remains specific in the bifactor model after accounting for the shared variance through the general factor. Finally, the factor loadings of the general factor were compared with the loadings of the group factors within the bifactor model. The differences between these loadings demonstrate the degree to which each item reflects the general factor or its specific group factor in the bifactor model (Brouwer et al., 2013). We calculated the explained common variance (ECV) percentage attributable to each factor in the five correlated factors model and in the bifactor model. The ECV of each factor is the ratio of the sum of the squared factor loadings for that factor (i.e., the variance explained by the factor) by the sum of all squared factor loadings (i.e., the variance explained by the whole model). In the bifactor model, we calculated the general factor ECV and the group factors ECVs (Reise, 2012).

968

D.A. DeSousa et al. / Journal of Anxiety Disorders 28 (2014) 966–970

Table 1 Fit indices for the Screen for Child Anxiety Related Emotional Disorders (SCARED) models tested by means of Confirmatory Factor Analysis.

One factor model Five correlated factors model Bifactor model

␹2

df

CFI

TLI

RMSEA [90% CI]

7223.22 4123.25 2902.54

779 769 738

.847 .920 .949

.839 .915 .943

.058 [.057–.060] .042 [.041–.044] .035 [.033–.036]

Note. CFI, Comparative Fit Index; TLI, Tucker–Lewis Index; RMSEA [90% CI], Root Mean Square Error of Approximation with 90% Confidence Interval.

We also calculated ω coefficients to assess the reliability of the instrument scores. For the one factor model and the five correlated factors model we calculated ω coefficients. For the bifactor model we calculated both ω and ω-hierarchical (ωh ) coefficients, since in the bifactor model the item responses are assumed to influenced by both the general anxiety factor and the specific anxiety group factors. Values of ω and ωh coefficients may vary between 0 and 1, with higher scores indicating greater reliability: a value of 1 indicates that the instrument’s sum score measures the target construct with perfect accuracy. Comparing these reliability coefficients of the general factor and the specific factors provides information on how reliable the specific factor sum scores are (Brouwer et al., 2013; Reise, 2012). 2. Results The CFA results are depicted in Table 1. The one factor model presented acceptable RMSEA, but the CFI and TLI indices were not acceptable. Both the five correlated factors model and the bifactor model presented good RMSEA, and acceptable CFI and TLI. The bifactor model demonstrated the best fit indices among the models tested. Chi-square difference tests revealed that the bifactor model had improved fit as compared to the one-factor model (2 = 3183.35, df = 41, p < .001) and as compared to the five correlated factors model (2 = 899.38, df = 31, p < .001). To investigate if gender and age variation of the participants could have affected our results we conducted post hoc multigroup CFA (MCFA) to examine factor invariance of the five correlated factors model and the bifactor model between genders (boys and girls) and age groups (children aged 9–13 and adolescents aged 14–18). In each MCFA we tested: (1) an unconstrained model to assess configural invariance, i.e., whether the scale configuration (number of factors and items per factor) was acceptable for both groups; (2) a constrained model to assess metric invariance by constraining the factor loadings to be equal across groups; and (3) a constrained model to assess scalar invariance by constraining the factor loadings and the intercepts/thresholds to be equal across groups. The CFI difference test (CFI) evaluated measurement invariance comparing the configural unconstrained model to the metric constrained model and the metric constrained model to the scalar constrained model. A CFI equal to or lower than .01 indicates factorial invariance for the evaluated parameter (Brown, 2006). We found both models to be invariant across genders (CFIs ≤ .002 for the bifactor model and CFIs ≤ .003 for the five correlated factors model) and across age groups (CFIs < .001 for the bifactor model and CFIs ≤ .001 for the five correlated factors model) and therefore it is unlikely that gender and age specificities affected our primary CFA results (further multigroup results data available upon request). Table 2 depicts the factor loadings of the items in the models tested. The differences in the factor loadings of the one factor model and the loadings of the general factor in the bifactor model were overall very low (M = .027; SD = .036). This suggests that the item variance is not mainly influenced by the group factors, i.e., the majority of the item variance was due to a general factor. The factor loadings in the five correlated factors model were high (M = .62; SD = .11), ranging from .33 to .77, with only one item (item

8) presenting a loading below .40. In contrast, the factor loadings of the group factors in the bifactor model were considerably lower (M = .34; SD = .20), ranging from −.09 to .77, with 26 of the 41 items presenting loadings below .40. This suggests that after controlling for the shared variance through the general factor, the variance in the item responses in the group factors of the bifactor model did not remain as specific as in the five correlated model for most of the items. Within the bifactor model, the factor loadings were higher on the general factor than on the group factors for 32 of the 41 items (Table 2). For instance, all items in the panic/somatic subscale had higher loadings on the general factor than on the SOM group factor. This result suggests that most of the items reflect more of the general factor than of its specific group factor in the bifactor model. Nevertheless, some items presented higher loadings on the group factors than on the general factor (Table 2). In the GAD subscale, items 21, 33 and 35 (“I worry about things working out for me”; “I worry about what is going to happen in the future”; “I worry about how well I do things”). In the SoAD subscale, items 26, 32, and 41 (“I feel shy with people I don’t know well”; “It is hard for me to talk with people I don’t know well”; “I am shy”). In the SeAD subscale, items 13 and 25 (“I am afraid to be alone in the house”; “I worry about sleeping alone”). In the SCH subscale, item 36 (“I am scared to go to school”). This result suggests that some items in the SCARED scale do inform about specific anxiety dimensions after accounting for the covariance among all items. As can be seen in Table 2, in the five correlated factors model, the explained common variance (ECV) of the factors ranged from 36.62% (SOM) to 9.08% (SCH). In the bifactor model, 63.96% of the common variance was attributable to the general factor, and the five group factors altogether accounted for the remaining 36.04% of the variance. When the ECV for the general factor in a bifactor model is large (ECV > .60), the estimates of the factor loadings for a unidimensional model are close to the general factor loadings in the bifactor model (Brouwer et al., 2013). Furthermore the reliability coefficient of the total score in the bifactor model was ωh = .83, while the reliability coefficients of the subscale scores after controlling for the general factor were very low, ranging from .005 (SCH) to .04 (SOM). The total score therefore predominantly reflects one common source even when considering multidimensional data (Reise, 2012). 3. Discussion In this study we examined the structural validity and reliability of the SCARED total score and SCARED subscale scores using CFAs and a bifactor model approach. Our results support the use of the SCARED total score and discourage the use of SCARED subscale scores given that most of the reliability attributed to the SCARED subscales was due to the common general component which is present in all subscale scores. Therefore once we control for the general dimension we could not assess individuals in the subdimensions reliably. When empirical findings from a bifactor model indicate that subscale scores are highly related to the general construct, it is advisable to rely more on the total score to interpret the instrument variance (Brouwer et al., 2013). We found, for instance, that

Table 2 Factor loadings (and standard errors), explained common variance, and reliability coefficients for the Screen for Child Anxiety Related Emotional Disorders (SCARED) models. Item

Model One factor

Five correlated factors SOM

.58 (.021) .65 (.021) .50 (.019) .69 (.019) .59 (.019) .65 (.015) .62 (.016) .55 (.020) .67 (.020) .72 (.017) .71 (.016) .61 (.027) .70 (.019) .44 (.019) .59 (.016) .41 (.021) .54 (.018) .62 (.016) .57 (.018) .53 (.018) .34 (.022) .60 (.016) .32 (.021) .53 (.018) .52 (.018) .60 (.016) .51 (.018) .51 (.019) .38 (.020) .52 (.023) .28 (.023) .53 (.022) .58 (.017) .58 (.017) .58 (.019) .40 (.020) .57 (.018) .42 (.020) .37 (.026) .53 (.021) .54 (.030)

ECV (%) ω ωh

.94

Bifactor model SoAD

SeAD

SCH

.62 (.022) .69 (.022) .53 (.021) .73 (.019) .63 (.019) .70 (.016) .66 (.016) .58 (.020) .71 (.020) .77 (.016) .76 (.016) .65 (.028) .75 (.019) .51 (.021) .67 (.018) .47 (.023) .63 (.018) .71 (.016) .65 (.019) .61 (.019) .40 (.023) .68 (.018) .41 (.025) .67 (.020) .67 (.018) .77 (.016) .65 (.019) .64 (.021) .51 (.021) .60 (.025) .33 (.026) .62 (.023) .67 (.018) .68 (.019) .68 (.020) .46 (.022) .67 (.020) .55 (.026) .48 (.032) .68 (.025) .70 (.033) 36.62 .92

19.85 .83

16.81 .82

17.64 .81

9.08 .70

G

gSOM .53 (.025) .56 (.025) .55 (.020) .65 (.022) .58 (.020) .64 (.017) .61 (.018) .52 (.022) .66 (.022) .64 (.022) .70 (.018) .53 (.031) .63 (.023) .40 (.022) .61 (.018) .38 (.023) .47 (.021) .60 (.018) .55 (.020) .47 (.021) .26 (.024) .59 (.018) .30 (.023) .51 (.020) .44 (.021) .51 (.019) .48 (.020) .49 (.021) .29 (.022) .51 (.026) .28 (.025) .48 (.026) .60 (.018) .61 (.017) .55 (.022) .40 (.022) .59 (.019) .44 (.021) .38 (.027) .53 (.023) .52 (.033)

63.96 .95 .83

gGAD

gSoAD

gSeAD

gSCH

.41 (.034) .53 (.032) −.09 (.039) .36 (.035) .24 (.034) .23 (.030) .22 (.031) .29 (.035) .20 (.037) .54 (.031) .24 (.032) .47 (.044) .46 (.033)

D.A. DeSousa et al. / Journal of Anxiety Disorders 28 (2014) 966–970

1 6 9 12 15 18 19 22 24 27 30 34 38 5 7 14 21 23 28 33 35 37 3 10 26 32 39 40 41 4 8 13 16 20 25 29 31 2 11 17 36

GAD

.36 (.027) .05 (.027) .31 (.029) .57 (.025) .33 (.025) .29 (.028) .50 (.024) .55 (.026) .25 (.025) .25 (.026) .32 (.025) .57 (.022) .67 (.020) .38 (.023) .29 (.027) .58 (.022) .38 (.037) .16 (.036) .66 (.041) .05 (.033) .03 (.032) .62 (.038) .20 (.033) .13 (.033) .10 (.042) .13 (.051) .48 (.102) .77 (.160) 9.37 .73 .04

7.36 .68 .02

8.53 .69 .02

5.96 .51 .01

4.82 .49