Consumer satisfaction with the Child and Adolescent Mental Health Service and its association with treatment outcome: A 3–4-year follow-up study CATHRINE SOLBERG, BO LARSSON, THOMAS JOZEFIAK

Solberg C, Larsson B, Jozefiak T. Consumer satisfaction with the Child and Adolescent Mental Health Service and its association with treatment outcome: A 3–4-year follow-up study. Nord J Psychiatry 2015;69:224–232. Background: Consumer satisfaction studies with the Child and Adolescent Mental Health Service (CAMHS) have mainly assessed evaluations in a short-term follow-up perspective. Adolescent reports with CAMHS have not been included nationally. Aims: The purposes of this study were to explore adolescent and parental satisfaction with the CAMHS in a 3–4-year follow-up perspective, and to examine the relationships between reported consumer satisfaction and clinical parameters such as reason for adolescent referral, emotional/behavioral symptoms and treatment outcome. Methods: Of 190 adolescent–parent pairs in a sample of CAMHS outpatients, 120 completed a Consumer Satisfaction Questionnaire. Parents assessed adolescent emotional/behavior problems both at baseline and at follow-up by completing the Child Behavior Checklist (CBCL). Correlations were examined between adolescent and parental evaluations. The relationships between service satisfaction and symptom load at baseline and follow-up and treatment outcome at follow-up were explored. Results: Overall, adolescents and parents were satisfied with the services received from the CAMHS. The correlations between adolescent and parent consumer satisfaction ratings were low to moderate. Consumer satisfaction was significantly and negatively correlated with symptom load on the CBCL Total Problems scores at baseline, but not at follow-up. There was no difference in satisfaction levels between those who improved after treatment and those who did not. Conclusions: Given the differences in informant ratings of consumer satisfaction, it is important to include both adolescent and parental perceptions in evaluations of CAMHS services and treatment outcomes. Consumer satisfaction should serve as a supplement to established standardized outcome measures. • Adolescence, Consumer satisfaction, Long-term follow-up, Mental health services, Treatment outcome. Cathrine Solberg, Karolinerveien 2a, 7021 Trondheim, Norway, E-mail: [email protected]; Accepted 29 September 2014.

C

onsumer satisfaction is a central aspect in evaluations of mental health services. It has received increased attention, as consumers’ autonomy and accountability on personal health become increasingly important political issues. Within the Child and Adolescent Mental Health Service (CAMHS), cross-sectional and follow-up surveys of consumer satisfaction have been conducted primarily in short-term perspectives (1–5). The consumers are generally satisfied with the services received (2, 4–13). Although parents may represent young children when evaluating consumer satisfaction with the CAMHS, as

© 2014 Informa Healthcare

the child grows into adolescence, his/her opinions become increasingly important and critical to include (4, 5, 9, 12, 14–17). Given that most studies have found only minimal to moderate relationships between adolescent and parental reports of satisfaction (4, 5, 9, 12, 14, 15, 17), it is important to include both informant sources and to further assess the discrepancy between them in order to further improve adolescent services (12, 16, 17). It is suggestive that consumers described as “satisfied” also would be those who experienced positive treatment outcomes. However, previous research has only shown minimal relationships between these aspects (2, 4, 5). DOI: 10.3109/08039488.2014.971869

CONSUMER SATISFACTION WITH CAMHS

Treatment outcome, as measured by symptom improvement and/or social function, was related to consumer satisfaction in several (2–5) but not all studies (1). While CAMHS satisfaction levels have correlated negatively with age (5, 9, 14, 18), others have reported no correlation with age (7, 8, 15), race/ethnicity (7–9), or gender of patient (5, 7–9, 18) or parent (15). Consumer satisfaction has been found inversely related to adolescent symptom load (7, 15), and adolescents with disruptive/externalized behaviors have reported less satisfaction than others (5, 18). Overall, the results of international studies have been inconsistent regarding the relationships between consumer satisfaction and mental health symptoms in adolescents (1–5, 7, 15, 18). It is likely that consumers do not fully acknowledge or achieve improvements from psychiatric treatment in a short-time perspective. Attained positive treatment outcomes may also be diluted or reversed over time. Many adolescents referred to the CAMHS suffer from chronic psychiatric disorders that keep them in contact with the services over extended periods. Therefore, various aspects of adolescent consumer satisfaction with CAMHS need to be further explored in both short-term and long-term perspectives. Although parental views of service satisfaction have been included (11, 18, 19), adolescent opinions on service received have not previously been investigated in Norway. To the best of our knowledge, this is the first longterm follow-up study of consumer satisfaction with the CAMHS including both adolescent and parental perspectives in relation to important clinical factors such as referral reasons, emotional/behavioral symptom load and treatment outcome.

Aims The overall aims of the study were to explore both adolescents’ and parents’ views of satisfaction with the CAMHS in a 3–4-year perspective, and to examine the relationships between their reported consumer satisfaction and clinical parameters such as referral reason, adolescent emotional/behavioral symptom load and treatment outcome.

The specific aims were to examine 1) Adolescent and parental evaluations of services received from the CAMHS in a long-term perspective of 3–4 years after their first consultation, and the correlations between their reports; 2) Associations between adolescent- and parent-reported consumer satisfaction and adolescent symptom load as reflected by Total Problems score on the Child Behavior Checklist (CBCL); 3) Differences in adolescent- and/or parent-reported consumer satisfaction levels by referral reason, here cateNORD J PSYCHIATRY·VOL 69 NO 3·2015

gorized into emotionally vs. behaviorally related problems; 4) The power of Total Problems scores on the CBCL at baseline to predict levels of adolescent and parental consumer satisfaction 3–4 years later; 5) Relationships between positive and negative treatment outcomes and adolescent versus parental consumer satisfaction.

Material and methods Participants For an earlier study (20), adolescents aged 8–15.5 years, referred for the first time and with at least two visits to outpatient CAMHS clinics in three geographical sites in Central Norway between July 2003 and December 2005, were consecutively asked to participate. Exclusion criterion was that neither the parent nor the patient had necessary competence in the Norwegian language (n ⫽ 11). Parents of 82 (16.4%) of the 501 eligible adolescents did not give informed consent, and clinical staff did not follow our research protocol for 74 adolescents (14.8%). The final sample of the earlier study thus consisted of 345 adolescents, a response rate of 68.9% (Fig. 1). There were no significant differences between participants and non-participants in terms of living conditions (with one or both biological parents), of problems described in the physician referral, or psychosocial functioning as measured by the Global Assessment of Psychosocial Disability (GAPD) scale (see below) (mean⫾ standard deviation (s) ⫽ 2.9 ⫾ 1.2 and 3.0 ⫾ 1.2, respectively). For more details, see Jozefiak et al. (20). The present study sample included two rural outpatient clinics, comprising 190 of the 345 patients included in the earlier study (Fig. 1). Due to administrative changes, geographic relocation and reorganization the city site adolescents were not eligible. There was no significant difference between urban and rural adolescents with regard to levels of psychosocial functioning. However, rural patients (mean⫾s ⫽ 45.2 ⫾ 24.3) had significantly [t(324) ⫽ 3.39, P ⫽ 0.001] lower Total problems scores on the CBCL than urban patients (mean⫾s ⫽ 54.7 ⫾ 25.9). The present study was performed 3–4 years (mean⫾s ⫽ 3.36 ⫾ 0.68 years, range ⫽ 2.4–5.2 years) after the adolescents’ first contact with the clinic, in which 120 adolescents and their parents participated (Fig. 1). The age range of the adolescents was 11–20.5 years (Table 1). There were no significant differences in the distributions of gender and age at baseline between participants and non-participants in the follow-up study. Neither did we observe differences in CBCL Total Problems at baseline regarding gender in the follow-up study. The Consumer Satisfaction Questionnaires (CSQ) were not completed for 47 adolescents and 26 parents (Fig. 1). In attrition analyses, completers (mean⫾s ⫽ 15.9 ⫾ 1.8

225

C SOLBERG ET AL. Eligible for earlier study n=82 did not give informed consent

N=501

n=74 staff did not follow protocol

referred to three outpatient sites of the CAHMS/St. Olav’s University Hospital

Included for earlier study N=345 (Response rate 68.9%) at three outpatient sites of the CAHMS/St. Olav’s university hospital

Rural 1

Rural 2

City

n=66

n=124

n=155 not included in present study

Eligible forpresent study N=190

n=70 adolescents did not give informed consent

(Baseline)

n=120 adolescents and their parents participated at 3-4-years follow-up and completed questionnaires

n=47 adolescents did not complete all items on CSQ n=26 parents did not complete all items on CSQ n=28 parents did not complete a valid CBCL at baseline or follow-up

CSQ available for n=73 adolescents CSQ available for n=94 parents CBCL was available for n=92 parents (who had completed CBCL at baseline)

Fig. 1. Patient flow chart and attrition for the earlier study (boxes with grey border) (20) and present study (boxes with black border).

years) were significantly [t(111) ⫽ –9.6, P ⬍ 0.001] older at follow-up than non-completers (mean⫾s ⫽ 12.8 ⫾ 1.3 years). There were no significant differences with regard to CBLC Total Problems scores at follow-up.

Procedures All adolescents and parents invited to the follow-up study were contacted by telephone and given an information letter with questionnaires to complete.

226

Measures SOCIO-DEMOGRAPHICS, REFERRAL REASON AND PSYCHOSOCIAL FUNCTIONING

Information on adolescent age, gender, residence and physician referral reason, was obtained through the electronic medical record system for patients (Table 1). Parents’ highest education level was used as a measure of socioeconomic status (SES) rated on a standard sevenpoint scale (21). Information about the patients “Living NORD J PSYCHIATRY·VOL 69 NO 3·2015

CONSUMER SATISFACTION WITH CAMHS

Table 1. Socio-demographic variables at the 3–4-year follow-up. Gender Boys Girls Mean age Norwegian language at home Parent educational status Did not finish lower secondary school (⬍ 9 years) Finished lower secondary school (9 years) Did not finish upper secondary school (10–11 years) Finished upper secondary school (12 years) Upper secondary school ⫹ 1 additional year Higher education of 1–4 years Higher education of more than 4 years Referral reason Emotionally related problems Behaviorally related problems Other problems

66 (55%) 54 (45%) 14.9 years (range 11–20.5) 119 (99.2%) None 5 (4.2%) 37 (30.8%) 20 (16.7%) 15 (12.5%) 23 (19.2%) 13 (10.8%) 38 (31.7%) 59 (49.2%) 23 (19.2%)

conditions” was obtained from the medical records, including seven categories: 1) lives alternately with biological mother and father; 2) with one biological parent and his/her partner; 3) another family; 4) in foster home; 5) in institution; 6) with one biological parent; and 7) with two biological parents. Only the last two categories were used in our analysis. The physician who referred patients to CAMHS chose one main referral reason on a standard list for each patient. For the purpose of this study, “suicidal, inhibited behavior, anxiety and depression” were categorized as emotionally related problems, while “autism, psychosis, absence from school, behavior problems, hyperactivity and learning disability” were categorized as behaviorally related problems. The Global Assessment of Psychosocial Disability (GAPD) scale scale was used to assess psychosocial functioning in the adolescent (0 ⫽ “superior/good social functioning”, to 8 ⫽ “profound and pervasive social disability”) (22). Specialists in child psychiatry or clinical child psychology scored GAPD information as a standard procedure. The GAPD has been found sufficiently reliable for clinical practice regarding inter-rater reliability (intraclass correlations, ICC values between 0.77 and 0.90) (23), while another study showed lower ICC values of 0.54 (24). However, psychometric information for this measure is still scarce in the literature (25). THE CHILD BEHAVIOR CHECKLIST 6–18 (CBCL) The CBCL Total Problems scale consists of 118 items scored on 0–2 scale; 0 ⫽ “Not True”; 1 ⫽ “Somewhat or Sometimes True”; 2 ⫽ “Very True or Often True”, with a total score range of 0–236. Parents reported on the adoNORD J PSYCHIATRY·VOL 69 NO 3·2015

lescent’s emotional and behavioral problems over the preceding 6 months. The Norwegian translation of the CBCL has shown satisfactory reliability and validity (26, 27). Norwegian normative data on the CBCL are available for the geographic areas in which the study was conducted (27). THE CONSUMER SATISFACTION QUESTIONNAIRE (CSQ) The CSQ was developed by two of the authors (TJ and BL) in 2003 for the CAMHS. We searched the literature (1, 5, 7, 28–30) before deciding on the following content areas: relationships with health personnel, information and autonomy, perceived treatment outcome and global satisfaction with the service. The CSQ consists of nine items (Table 2). Items 1–8 are rated on a 4-point scale (1 ⫽ “strongly agree”, to 4 ⫽ “strongly disagree”), while item 9 is scored on a 5-point scale (1 ⫽ “not satisfied”, to 5 ⫽ “very satisfied”) (Table 2). Later research (10, 11, 13, 17–19) has given support to the relevance of our selection of content areas. Item 7 was administered to parents only. The eight corresponding items for the parent and adolescent versions were summarized into a Total Consumer Satisfaction score (range ⫽ 0–25). When used in the analysis, items 1, 3, 5 and 8 were reversed, so that higher values on the total scale represented a higher degree of consumer satisfaction. Internal consistency for the Total Consumer Satisfaction scale in the present sample was good (alpha ⫽ 0.84 for adolescents and 0.92 for parents). A confirmatory factor analysis (CFA) of the data in the present sample tested a one-factor solution. When item 3 (“I/we received adequate information”) correlated with item 4 (“I/we did not have enough influence on the treatment plan”), item 4 with 6 (“The clinic did not help with the youth–parent relationship”) and item 5 (“The clinic helped me/us”) with 8 (“I/we can recommend the clinic to others”), we found a comparative fit index (CFI) of 0.995; and a Tucker–Lewis index (TLI) of 0.991, both representing a good model fit to the parents’ reports. However, the root mean square error of approximation (RMSEA) was only acceptable (0.086), and the chi-square test of model fit was significant (P ⫽ 0.025). While adolescents’ reports showed CFI ⫽ 0.956 and TLI ⫽ 0.928, which is satisfactory, the RMSEA was unsatisfactory (0.129) and the chisquare test of model fit was significant (P ⫽ 0.001). Therefore, an exploratory factor analysis (EFA) was conducted, which showed a very good model fit for a twofactor solution (CFI ⫽ 0.996, TLI ⫽ 0.991 and RMSEA ⫽ 0.046). Since factor 2 only included the three negative statements of the CSQ (items 2 (“The clinicians did not have enough time for me/us”), 4 and 6), we controlled the analysis for a negative statement method measurement error and could exclude such a possibility. On the other hand, item 4 (“I/we did not have enough influence on the treatment plan”) showed a very high

227

C SOLBERG ET AL.

Table 2. Adolescent and parental evaluations of Child and Adolescent Mental Health Service on the Consumer Satisfaction Questionnaire (CSQ) by item. Item (1) The clinicians showed concern and understanding of my/our situation (2) The clinicians did not have enough time for me/us (3) I/we received adequate information (4) I/we did not have enough influence on the treatment plan (5) The clinic helped me/us (6) The clinic did not help with the youth–parent relationship (7) The clinic did not cooperate well with other professionals involved with my child (8) I/we can recommend the clinic to others (9) Are you satisfied with the help received from the clinic? Total Consumer Satisfaction Score Positive Statements of the Total Satisfaction Score Negative Statements of the Total Satisfaction Score

Informant

n

Mean

Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents Adolescents Parents

77 106 78 105 77 103 77 105 78 105 76 104

1.74 1.69 3.37 2.90 2.03 1.99 3.05 3.03 1.95 1.99 3.03 2.93

106 75 105 80 105 73 98 74

2.91 1.87 1.69 3.75 3.81 17.59 17.43 11.15

75

6.45

s

Median

0.616 2.00 0.575 2.00 0.584 3.00 0.827 3.00 0.778 2.00 0.664 2.00 0.793 3.00 0.686 3.00 0.820 2.00 0.826 2.00 0.864 3.00 0.754 3.00 Not applicable 0.775 3.00 0.777 2.00 0.670 2.00 1.119 4.00 0.921 4.00 4.43 18.00 4.93 17.00 3.26 12.00 Not applicable 1.75 6.00 Not applicable

Range 1–4* 1–4 2–4 1–4 1–4 1–4 1–4 1–4 1–4 1–4 1–4 1–4 1–4 1–4 1–4 1–5† 1–5 8–25‡ 1–25 1–16§ 2–9||

s, standard deviation. *1 ⫽ strongly agree; 4 ⫽ strongly disagree for items (1) to (8). †1 ⫽ not satisfied; 5 ⫽ very satisfied for item (9). ‡Possible range: 0–25 (Total Consumer Satisfaction Score). §Possible range: 0–16 (Positive Statements of the Total Satisfaction Score). ||Possible range: 0–9 (Negative Statements of the Total Satisfaction Score).

(⬎ 1.000) geomin rotated loading in the EFA with a negative residual variance. Thus, the final structure of the adolescent CSQ is not yet established, yielding either a one- or two-factor solution. We have chosen to use the Total score of the adolescent CSQ in our present analyses, because we wanted to compare adolescent with parent reports. For adolescent we also report descriptive statistics of the two sub-scores (two factors) found in the EFA. (More detailed results of our analyses are available from the second author). Test–retest reliability is yet not established for the CSQ.

Statistics Statistical analyses were conducted using SPSS version 19 and Mplus version 7 (Muthén, 1998–2012). The factor analyses were estimated with the weighted least square parameter estimator due to the categorical nature of the items. With respect to CFI and TFI, values above 0.95 are considered indications of good fit, while the corresponding level for RMSEA, is equal to or below 0.06 (31). We excluded CBCL forms with more than eight items missing in accordance with scoring guidelines (32). Missing values of CBCL items varied from 0 to 0.9%

228

and were substituted with “0” in accordance with scoring guidelines (32). We only included complete CSQs. Internal homogeneity was examined using Cronbach’s alpha. Differences between group means were compared using Student’s t-test. Correlation analyses were conducted using the Pearson product-moment and Spearman’s (rho) correlation coefficients. Correlations were interpreted according to Cohen (33) where correlation ⱖ 0.1 is small, ⱖ 0.3 is moderate and ⱖ 0.5 is large. We conducted two multivariate linear regression analyses using satisfaction scores as dependent variables. CBCL Total Problems scores at baseline and at the 3–4year follow-up were included as independent variables in the first block. The second block controlled for the effects of adolescent age, gender and SES. A significance level of P ⬍ 0.05 was used. To identify adolescents who improved from baseline to follow-up on the CBCL Total Problems score, we calculated a clinically significant individual change index for each adolescent (34–37). Adolescents were assigned to treatment outcome groups based on whether changes in their CBCL Total Problems scores fulfilled the following criteria: (1) the magnitude of change from baseNORD J PSYCHIATRY·VOL 69 NO 3·2015

CONSUMER SATISFACTION WITH CAMHS

line to follow-up was statistically reliable, and (2) the score was in the clinical range at baseline and in the range of normal functioning at follow-up. With regard to statistical reliability, we used the Edwards–Nunnally method (35), in which a confidence interval consisting of two standard errors of measurement was centered around each adolescent’s estimated true baseline CBCL Total Problems score. Each follow-up score was categorized relative to this interval. The statistical reliability of change index was calculated using the following formula: XL⬎ or ⬍[rxx(XA–mean)⫹ mean]⫹ 2s(1–rxx)1/2, where XL ⫽ individual’s raw CBCL Total Problems score at follow-up, rxx ⫽ test–retest reliability, XA ⫽ individual’s raw CBCL Total Problems score at baseline, and mean ⫽ mean of CBCL Total Problems scores in the normative schoolchildren sample in the same age group, which was 14.2 (27). To calculate whether the CBCL Total Problems scores were in the range of normal functioning, we used the 90th percentile (31 points) (27) as the cut-off point in defining the clinical range on the normative CBCL Total Problems score, in line with recommendations by the originator (32). Following Bastiaansen et al. (37), we constructed four categories to describe different patterns of individual change in CBCL Total Problems scores: “Recovered” (statistically reliable positive change fulfilling criteria 1 and 2, n ⫽ 31 [25.8%]; “Improved” (statistically reliable positive change fulfilling criterion 1 but not 2, n ⫽ 17 [14.2%]; “Unchanged” (fulfilled neither criteria 1 nor 2, n ⫽ 8 [6.7%]; and “Deteriorated” (statistically reliable negative change with criterion 1 fulfilled in a negative direction and criterion 2 not fulfilled, n ⫽ 12 [10.0%]). The “Recovered” and “Improved” subgroups were included into a “Positive outcome group”, and the “Unchanged” and “Deteriorated” subgroups were included into a “Negative outcome group”. Thirty-five adolescents (29.2%) were excluded because their CBCL Total Problems scores were in the normal range at baseline.

Ethics Adolescents aged 12 years and older and all parents provided written informed consent for study participation. The Norwegian Regional Medical Ethics Committee approved the study.

P ⫽ 0.004), “I/we received adequate information” (rho ⫽ 0.36, P ⫽ 0.003), “The clinic helped me/us” (rho ⫽ 0.40, P ⫽ 0.001), and “Are you satisfied with the help received from the clinic?” (rho ⫽ 0.29, P ⫽ 0.02). We found no correlation for the remaining items, which were “I/we did not have enough influence on the treatment plan” (rho ⫽ 0.16, P ⫽ 0.20), “The clinic did not help with the youth–parent relationship” (rho ⫽ 0.19, P ⫽ 0.14), and “I/we can recommend the clinic to others” (rho ⫽ 0.17, P ⫽ 0.18). The correlation between adolescent and parental report was moderate on the Total Consumer Satisfaction score (r ⫽ 0.40, P ⫽ 0.002). Parents of adolescents primarily referred because of emotionally related problems (mean⫾s ⫽ 19.1 ⫾ 4.8) were significantly [t(74) ⫽ –2.38, P ⫽ 0.020] more satisfied with the CAMHS than those of adolescents referred because of behaviorally related problems (mean⫾s ⫽ 16.4 ⫾ 5.1). However, there was no difference between these groups with regard to adolescent reports. CBCL Total Problems scores at baseline showed a moderate negative correlation with Total Consumer Satisfaction scores as reported by adolescents (r ⫽ –0.30, P ⫽ 0.009), and a moderate negative correlation with parental report (r ⫽ ⫺ 0.42, P ⬍ 0.001) (Table 3). The CBCL Total Problems scores at the 3–4-year follow-up did not correlate to adolescent (r ⫽ ⫺ 0.17, P ⫽ 0.10) or parental (r ⫽ ⫺ 0.15, P ⫽ 0.07) Total Consumer Satisfaction scores (Table 3). The results of multivariate linear regression analyses showed that the CBCL Total Problems score at baseline emerged as a significant predictor of the Total Consumer Satisfaction scores as reported by adolescents at the 3–4-year follow-up (P ⫽ 0.036), explaining 10% of the variance. After controlling for the effects of gender, age, and SES as potential confounders, only a trend (P ⫽ 0.052) remained (Table 4). For parent reports, the CBCL Total Problems score significantly (P ⬍ 0.001) predicted and explained 17% of the total variance (Table 4). We found no significant differences between adolescent and parental reports on the Total Consumer Satisfaction Scale when comparing adolescents with a positive outcome and those with a negative outcome at the 3–4year follow-up. Nor was any difference found in this regard when comparing the groups “Recovered” and “Deteriorated”.

Results Most adolescents and parents reported high satisfaction levels with the CAMHS on the CSQ (Table 2). Spearman correlations between adolescent and parental responses on each item showed significant and positive low to moderate correlations for the following items: “The clinicians showed concern and understanding of my/our situation” (rho ⫽ 0.35, P ⫽ 0.004), “The clinicians did not have enough time for me/us” (rho ⫽ 0.35, NORD J PSYCHIATRY·VOL 69 NO 3·2015

Discussion Adolescent and parental evaluations of consumer satisfaction This study examined adolescent and parental evaluations of consumer satisfaction with outpatient CAMHS in Central Norway in a 3–4-year follow-up perspective. In accordance with previous research performed in shorter time perspectives (2, 4–13), adolescents and parents

229

230

– – ⫺ 0.20P ⫽ 0.06; ⴚ 0.17P ⫽ 0.05 – ⫺ 0.07P ⫽ 0.30; ⴚ 0.13P ⫽ 0.11 0.36P ⫽ 0.002; 0.11P ⫽ 0.15 – ⫺ 0.18P ⫽ 0.09; ⴚ 0.05P ⫽ 0.32 ⫺ 0.17P ⫽ 0.10; ⴚ 0.03P ⫽ 0.38 ⫺ 0.09P ⫽ 0.24; ⴚ 0.20P ⫽ 0.03 Total Consumer Satisfaction Score CBCL at baseline CBCL at follow-up Gender (1 ⫽ f, 2 ⫽ m) Age SES

– ⫺ 0.30P ⫽ 0.009; ⴚ 0.42P ⬍ 0.001 ⫺ 0.17P ⫽ 0.10; ⴚ 0.15P ⫽ 0.07 ⫺ 0.13P ⫽ 0.16; ⴚ 0.12P ⫽ 0.12 0.25P ⫽ 0.02; ⴚ 0.02P ⫽ 0.42 ⫺ 0.26P ⫽ 0.02; ⴚ 0.01P ⫽ 0.45

– 0.29P ⫽ 0.01; 0.29P ⫽ 0.003 ⫺ 0.12P ⫽ 0.17; ⴚ 0.01P ⫽ 0.48 ⫺ 0.21P ⫽ 0.06; 0.04P ⫽ 0.34 0.01P ⫽ 0.48; ⴚ 0.09P ⫽ 0.20

SES Age Gender CBCL at follow-up CBCL at baseline Total Consumer Satisfaction score

Table 3. Pearson correlations between adolescent consumer satisfaction scores, Child Behavior Checklist (CBCL) Total Problems Scores at baseline and the 3–4-year follow-up by gender, age and socio-economic status (SES) (adolescents, n=61; parents, n=90).

C SOLBERG ET AL.

expressed satisfaction with services received, and only small to medium associations between adolescent and parental ratings of consumer satisfaction with the CAMHS were found.

Associations between adolescent and parental evaluations A number of studies have emphasized that adolescents’ opinions are important to consider in order to further improve the quality of CAMHS (4, 5, 9, 12, 14–17). Our findings of low to moderate associations between youth and parent reports of consumer satisfaction support these conclusions, suggesting that parents’ opinions are not representative of adolescents’ opinions. With consumer satisfaction becoming an increasingly important clinical and health political matter, we argue adolescent views to be an important informant source, in particular with regard to evaluation of their received mental health services.

Associations between referral reasons and consumer satisfaction Adolescents with behaviorally related problems and their parents have been found less satisfied with CAMHS than other problem groups (5, 18). Our findings show parents of adolescents with behaviorally related problems to be less satisfied than parents of adolescents with emotionally related problems; however, we found no differences regarding the adolescent satisfaction levels. This discrepancy supports the need for assessment of both adolescent and parental opinions when evaluating the quality of CAMHS, as well as the need for further exploring reasons behind perceived less quality of care among parents of adolescents with behaviorally related problems.

Prediction of consumer satisfaction by total problem load at baseline The adolescents’ Total Problem levels on the CBCL at baseline predicted satisfaction 3–4 years later in which less troubled adolescents and their parents showed higher satisfaction with CAMHS. Symptom load has previously been shown negatively correlated to consumer satisfaction (7, 15). The methods of our study differs from previous studies in that we assessed consumer satisfaction levels and symptom load at follow-up 3–4 years after the baseline assessment, thereby reducing the risk of transient treatment outcomes influencing our findings.

Relationship between treatment outcome and consumer satisfaction Although several studies have investigated consumer satisfaction with the CAMHS in short-term perspectives (1–5), few have investigated this aspect in relationship to treatment outcome over an extended period. One study NORD J PSYCHIATRY·VOL 69 NO 3·2015

CONSUMER SATISFACTION WITH CAMHS

Table 4. Results of linear regression analysis with consumer satisfaction scores as dependent variable. Adolescents (n ⫽ 61)

Step 1 Constant CBCL Total Problem Score at baseline CBCL Total Problem Score at follow-up Step 2 Constant CBCL Total Problem Score at baseline CBCL Total Problem Score at follow-up Gender Age at first consultation SES measured by the parent with highest education

Parents (n ⫽ 90)

B

CI

Beta

B

CI

Beta

20.12 ⫺ 0.05 ⫺ 0.01

17.81–22.44 ⫺ 0.09 to 0.00 ⫺ 0.05 to 0.03

⫺ 0.28 P ⫽ 0.04 ⫺ 0.08 P ⫽ 0.52

21.37 ⫺ 0.08 ⫺ 0.01

19.22–23.52 ⫺ 0.12 to ⫺ 0.04 ⫺ 0.05 to 0.03

⫺ 0.40P ⬍ 0.001 ⫺ 0.04P ⫽ 0.72

19.27 ⫺ 0.04 ⫺ 0.02 ⫺ 0.84 0.38 ⫺ 0.58

5.44 ⫺ 0.085 to 0.00 ⫺ 0.05 to 0.02 ⫺ 3.04 to 1.34 ⫺ 0.33 to 1.08 ⫺ 1.35 to 0.18

⫺ 0.26 ⫺ 0.10 ⫺ 0.10 0.14 ⫺ 0.20

23.67 ⫺ 0.08 ⫺ 0.01 ⫺ 1.23 0.04 ⫺ 0.15

15.23–32.11 ⫺ 0.13 to ⫺ 0.04 ⫺ 0.05 to 0.03 ⫺ 3.23 to 0.78 ⫺ 0.49 to 0.56 ⫺ 0.86 to 0.56

⫺ 0.41P ⬍ 0.001 ⫺ 0.05P ⫽ 0.62 ⫺ 0.12P ⫽ 0.23 0.01P ⫽ 0.90 ⫺ 0.04P ⫽ 0.67

P ⫽ 0.052 P ⫽ 0.42 P ⫽ 0.45 P ⫽ 0.29 P ⫽ 0.13

CI, 95% confidence interval; CBCL, Child Behavior Checklist; SES, socio-economic status. Step 1: R² ⫽ 0.10 for adolescent and R² ⫽ 0.17 for parents; Step 2: Step 2: ΔR²⫽ 0.10 for adolcents and ΔR² ⫽ 0.02 for parents.

found a positive relationship between the two aspects (38); however, this was a cross-sectional study. In a cross-sectional design, it is likely that satisfied consumers have a stronger “subjective feeling of positive treatment outcome”. Previous longitudinal research have been performed in time frames of 6 months to 2 years without being able to strongly relate consumer satisfaction to treatment outcome (1, 2, 4, 5). To the best of our knowledge, this is the first study performed with an extended follow-up perspective of 3–4 years. Our findings showed treatment outcome to be unrelated to adolescent and parental satisfaction with the CAMHS, indicating that assessment of consumer satisfaction should complement rather than replace established measures when assessing the quality of CAMHS.

a satisfactory model fit to the data. We suggest that the psychometric properties of the CSQ should be tested further in larger clinical samples. While the generalizability of the findings of this study is hampered by attrition, we found no significant differences with regard to the levels of emotional and behavioral problems between participants and non-participants at follow-up, also when controlling for gender. While participants could be more satisfied with the CAMHS than non-participants, here leading to overestimation of satisfaction levels, we did not have any means of assessing whether such a bias exists in our material. Lastly, we obtained the GAPD scores from the medical records, and no information on inter-rater reliability was obtained.

Limitations Only parents, and not adolescent or clinicians, assessed emotional/behavioral problems among adolescents on the standardized CBCL questionnaire. The available resources in this study did not permit the addition of questionnaires or interviews. We were therefore unable to rule out possible relationships between changes in clinician or adolescent reports of symptoms/problems and levels of consumer satisfaction. Our categorizing of referral reasons into emotionally and behaviorally related referral reasons are limited to the present study and does not represent a validated and well-established method. The reported psychometric analyses of the CSQ are limited to the present relatively small sample size. While a one-factor structure of the CSQ parent version was confirmed, further factor analyses with larger sample sizes are needed to establish the final factor solution (one- or two factor solution) of the adolescent version. Test–retest reliability data do not exist since the CSQ was used only at follow-up. However, internal reliability of the measure was good, and our factor analyses showed NORD J PSYCHIATRY·VOL 69 NO 3·2015

Conclusions We recommend that evaluations of consumer satisfaction of the CAMHS should include both adolescent and parental perspectives. The addition of systematic use of consumer satisfaction aspects may help to improve the quality of CAMHS. However, such information cannot substitute established measures when assessing treatment outcome in CAMHS. Acknowledgements—The Norwegian Council of Mental Health/EXTRA funded the study. Thanks to all youths and parents who participated in the study, and to all personnel at the department of Child and Adolescent Psychiatry, St. Olav’s Hospital, Trondheim University Hospital, Norway. CS wrote the first draft of the manuscript. BL and TJ planned the study and developed the Consumer Satisfaction Questionnaire. CS and TJ performed the analyses. All authors have contributed to the final manuscript.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

231

C SOLBERG ET AL.

References 1. Lambert W, Salzer MS, Bickman L. Clinical outcome, consumer satisfaction, and ad hoc ratings of improvement in children’s mental health. J Consult Clin Psychol 1998;66:270–9. 2. Garland AF, Aarons GA, Hawley KM, Hough RL. Relationship of youth satisfaction with mental health services changes in symptoms and functioning. Psychiatr Serv 2003;54:1544–6. 3. Liao Q, Sukumar B, editors. Relationship among service use, outcomes, and caregiver satisfaction. 17th Annual Conference Proceedings—A System of Care for Children’s Mental Health: Expanding the Research Base; 2005; Research and Training Center for Children’s Mental Health—Tampa, FL. http://rtckids.fmhi.usf. edu/rtcconference/proceedings/17thproceedings/chapter01.pdf. 4. Garland AF, Haine RA, Boxmeyer CL. Determinates of youth and parent satisfaction in usual care psychotherapy. Eval Program Plann 2007;30:45–54. 5. Turchik JA, Karpenko V, Ogles BM, Demireva P, Probst DR. Parent and adolescent satisfaction with mental health services: Does it relate to youth diagnosis, age, gender, or treatment outcome? Community Ment Health J 2010;46:282–8. 6. Garland AE, Besinger BA. Adolescents’ perceptions of outpatient mental health services. J Child Fam Stud 1996;5:335–75. 7. Garland AF, Aarons GA, Saltzman MD, Kruse MI. Correlates of adolescents’ satisfaction with mental health services. Ment Health Serv Res 2000;2:127–39. 8. Martin JS, Petr CG, Kapp SA. Consumer satisfaction with Children’s Mental Health Services. Child Adolesc Soc Work J 2003;20:211–26. 9. Copeland VC, Koeske G, Greeno CG. Child and mother client satisfaction questionnaire scores regarding mental health services: Race, age, and gender correlates. Res Soc Work Pract 2004;14:434–42. 10. Barber AJ, Tischler VA, Healy E. Consumer satisfaction and child behaviour problems in Child and Adolescent Mental Health Services. J Child Health Care 2006;10:9–21. 11. Bjertnaes OA, Garratt A, Helgeland J, Holmboe O, Dahle KA, Hanssen-Bauer K, et al. Foresattes vurdering av barne- og ungdomspsykiatriske poliklinikker [Parent assessment of outpatient Child and Adolescent Mental Health Services]. Tidsskr Nor Laegeforen 2008;128:1041–5. 12. Athay MM, Bickman L. Development and psychometric evaluation of the Youth and Caregiver Service Satisfaction Scale. Adm Policy Ment Health 2012;39:71–7. 13. Shafer A, Temple J. Factor Structure of the Mental Health Statistics Improvement Program (MHSIP) Family and Youth Satisfaction surveys. J Behav Health Serv Res 2013;40:306–16. 14. Stüntzner-Gibson D, Koren PE, DeChillo N. The Youth Satisfaction Questionnaire: What kids think of services. Fam Soc 1995;76:616–24. 15. Godley SH, Fiedler EM, Funk RR. Consumer satisfaction of parents and their children with child/adolescent mental health services. Eval Program Plann 1998;21:31–45. 16. Biering P. Child and adolescent experience of and satisfaction with psychiatric care: A critical review of the research literature. J Psychiatr Ment Health Nurs 2010;17:65–72. 17. Aarons GA, Covert J, Skriner LC, Green A, Marto D, Garland AF, et al. The eye of the beholder: Youths and parents differ on what matters in mental health services. Adm Policy Ment Health 2010;37:459–67. 18. Bjorngaard JH, Wessel Andersson H, Osborg Ose S, Hanssen-Bauer K. User satisfaction with Child and Adolescent Mental Health Services: Impact of the service unit level. Soc Psychiatry Psychiatr Epidemiol 2008;43:635–41. 19. Holmboe O, Iversen HH, Hanssen-Bauer K. Determinants of parents’ experiences with outpatient Child and Adolescent Mental Health Services. Int J Ment Health Syst 2011;5:22. 20. Jozefiak T, Larsson B, Wichstrom L, Wallander J, Mattejat F. Quality of life as reported by children and parents: A comparison between students and child psychiatric outpatients. Health Qual Life Outcomes 2010;8:136. 21. Hollingshead AdB. Two factor Index of Social Position. New Haven, CT: Hollingshead; 1957.

232

22. World Health Organization. Multiaxial classification of child and adolescent psychiatric disorders: The ICD-10 classification of mental and behavioural disorders in children and adolescents. Cambridge: Cambridge University Press; 1996. 23. Dyrborg J, Warborg Larsen F, Nielsen S, Byman J, Buhl Nielsen B, Gautrè-Delay F. The Children’s Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice—Substance and reliability as judged by intraclass correlations. Eur Child Adolesc Psychiatry 2000;9: 195–201. 24. Hanssen-Bauer K, Gowers S, Aalen O, Bilenberg N, Brann P, Garralda E, et al. Cross-national reliability of clinician-rated outcome measures in Child and Adolescent Mental Health Services. Adm Policy Ment Health 2007;34:513–8. 25. Schorre B, Vandvik I. Global assessment of psychosocial functioning in child and adolescent psychiatry. Eur Child Adolesc Psychiatry 2004;13:273–86. 26. Novik TS. Validity of the Child Behaviour Checklist in a Norwegian sample. Eur Child Adolesc Psychiatry 1999;8:247–54. 27. Jozefiak T, Larsson B, Wichstrøm L, Rimehaug T. Competence and emotional/behavioural problems in 7–16-year-old Norwegian school children as reported by parents. Nord J Psychiatry 2011;66:311–9. 28. Bradley EJ, Clark BS. Patients’ characteristics and consumer satisfaction on an inpatient child psychiatric unit. Can J Psychiatry 1993;38:175–80. 29. Mattejat F, Remschmidt H. Evaluation von Therapien mit psychisch kranken Kindern und Jugendlichen: Entwicklung und Überprüfung eines Fragebogens zur Beurteilung der Behandlung (FBB) [Evaluation of therapies with psychiatric disturbed children and adolescents: Development and examination of a therapy evaluation questionnaire (TEQ)]. Z Klin Psychol 1993;22:192–233. 30. Kaplan S, Busner J, Chibnall J, Kang G. Consumer satisfaction at a child and adolescent state psychiatric hospital. Psychiatr Serv 2001;52:202–6. 31. Yu CY. Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Los Angeles, CA: University of California, Graduate School of Education and Information Studies; 2002. 32. Achenbach TM, Rescorla L. An integrated system of multiinformant assesment—School-age forms and profiles. Washington, DC: Library of Congress: 2001. 33. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum Associates; 1988. 34. Jacobson NS, Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12–9. 35. Speer DC. Clinically significant change: Jacobson and Truax (1991) revisited. J Consult Clin Psychol 1992;60:402–8. 36. Jacobson NS, Roberts LJ, Berns SB, McGlinchey JB. Methods for defining and determining the clinical significance of treatment effects: Description, application, and alternatives. J Consult Clin Psychol 1999;67:300–7. 37. Bastiaansen D, Koot HM, Ferdinand RF. Psychopathology in children: Improvement of quality of life without psychiatric symptom reduction? Eur Child Adolesc Psychiatry 2005;14: 364–70. 38. Day C, Michelson D, Hassan I. Child and adolescent service experience (ChASE): Measuring service quality and therapeutic process. Br J Clin Psychol 2011;50:452–64. Cathrine Solberg, M.D., Faculty of Medicine, Norwegian University of Science and Technology, Trondheim, Norway. Bo Larsson, M.D., Regional Centre for Child and Youth Mental Health and Child Welfare, Faculty of Medicine, Norwegian University of Science and Technology, Trondheim, Norway. Thomas Jozefiak, M.D., Regional Centre for Child and Youth Mental Health and Child Welfare, Faculty of Medicine, Norwegian University of Science and Technology, Trondheim, and Department of Child and Adolescent Psychiatry, St. Olav’s Hospital, Trondheim University Hospital, Norway.

NORD J PSYCHIATRY·VOL 69 NO 3·2015

Copyright of Nordic Journal of Psychiatry is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.