Community Ment Health J (2014) 50:402–408 DOI 10.1007/s10597-013-9667-6
BRIEF REPORT
Quality Assessment of Mental Health Care by People with Severe Mental Disorders: A Participatory Research Project Angelo Barbato • Alessia Bajoni • Filippo Rapisarda • Vito D’Anza • Luigi Fabrizio De Luca • Cristiana Inglese • Sonia Iapichino • Fabrizio Mauriello • Barbara D’Avanzo
Received: 19 July 2012 / Accepted: 30 November 2013 / Published online: 8 December 2013 Ó Springer Science+Business Media New York 2013
Abstract This study assessed the perceived quality of care by consumers with severe mental disorders. A questionnaire investigating service quality was developed by a consumer focus group and filled by 204 consumers. In five areas the negative evaluations exceeded or closely approximated the positive ones: choice of professionals, waiting times, information about illness and medications. All five do not refer to the outcomes of care, but to the concept of responsiveness. The results confirmed that people with severe mental disorders can give value judgments on various aspects of care. However, even in a service strongly oriented towards community care, the consumers’ needs in sensitive areas concerning choices, respect and autonomy are not met. The application of the concept of responsiveness to quality improvement may help services to meet consumers’ expectations. Keywords Quality of mental health care Community mental health services Participatory research Severe mental illness
A. Barbato (&) A. Bajoni F. Rapisarda B. D’Avanzo Laboratory of Epidemiology and Social Psychiatry, IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Via La Masa 19, 20156 Milan, Italy e-mail:
[email protected] A. Barbato IRIS Postgraduate School of Psychotherapy, Milan, Italy V. D’Anza L. F. De Luca C. Inglese S. Iapichino Department of Mental Health of Pistoia, Pistoia, Italy F. Mauriello Rosaspina Self-Help Association, Pistoia, Italy
123
Introduction Demands for accountability and ethical imperatives have been leading to the recognition of assessment of consumers’ perceptions of healthcare as an important step in quality improvement. In mental health user involvement in service planning, delivery and evaluation has been prompted by early evidence of patients’ dissatisfaction with services (Rogers and Pilgrim 1993). By contrast, consistent findings show a positive correlation between patients’ perceptions that their needs are met, therapeutic alliance and care outcome (Roth and Crane Ross 2002). A large body of research is available on various aspects of consumers’ perspectives: attitudes, satisfaction, preferences, wishes and expectations about care (Noble et al. 2001; Marshall et al. 2007). However, there is limited material on their involvement in quality evaluation and improvement (Weinstein 2006). Although in most cases a focus has been placed on service satisfaction, a need to move beyond satisfaction to assess consumers’ judgment about service models, values, organization and performances has been recently stressed (Groenewegen et al. 2005; Hopkins et al. 2009). Consumer satisfaction is often not related to well defined standards, it is not reliable in evaluating healthcare performance and it is especially exposed to the conformity bias (Avis et al. 1997; Haggerty 2010). Direct assessment of consumers’ care perception has been defined as a core element of new generation of performance measures (Smith and Ganju 2006). A number of features of mental health care make consumer involvement in quality assessment an important and challenging endeavor. First, the power imbalance between consumers and providers is higher than in other healthcare sectors (Laugharne and Priebe 2006). Second, there are discrepant perspectives by consumers, providers and other
Community Ment Health J (2014) 50:402–408
stakeholders on some key aspects of service quality (Shumway et al. 2003; Mason et al. 2004). Lack of concordance has been found between patients and psychiatrists for treatment goals in psychoses (Bridges et al. 2013) and a disagreement between consumers and relatives is apparent in many surveys, with relatives assigning higher importance to intensive interventions, compulsory admission and medications (De Haan et al. 2001). Third, people with mental disorders are often socially marginalized and experience social exclusion (Jorm and Oh 2009). Last, most of them are considered unable, because of lack of insight and impaired reality testing, to provide meaningful feedback about their care experience (Buckley et al. 2007). Such challenges are especially relevant for people with most severe illness, often showing high needs and requiring long-term care and rehabilitation. However, a number of studies show that they can be involved in service evaluation and experience their involvement as an aspect of an empowerment process (Simpson and House 2002). It has been suggested that user involvement has been growing to a greater extent where service provision has been shifted from an institutional towards a communitybased model (Thronicroft and Tansella 2005). Although this stands to reason, it is not necessarily true. In Italy, despite a radical change towards community care following the closure of mental hospitals in the late 70 s (Barbato 1998), user participation, with few exceptions, lagged behind until few years ago. Recent growth of interest in this issue led first to the development of instruments to assess users’ satisfaction (Ruggeri and Dall’Agnola 1993), and later to the realization of a number of studies mainly focused on satisfaction and needs assessment (Gigantesco et al. 2002; Ruggeri et al. 2007). To expand the knowledge on this area of research, we present here the results of a study, implemented in an Italian mental health service strongly community oriented, in which the views of consumers with severe mental disorders were gathered within the framework of a quality improvement initiative. The aim was to obtain consumers’ opinions about service organization and treatment aspects they deem meaningful, to produce indications for the implementation of changes.
Methods In 2009 the Department of Mental of Health of Pistoia funded a quality assurance project based on evaluation of service performance and organization by users with severe mental disorders. The project endorsed the basic elements of both Participatory Research and Continuous Quality Improvement models. Participatory Research has been defined as a
403
systematic investigation conducted by researchers in collaboration with those affected by the issue being studied, for purposes of education, action or social change (Cornwell and Jewkes 1995). The goal of participatory research is to benefit the partners in the research process, while developing valid knowledge. Bringing together consumer participation and quality improvement entails a systematic cyclical method of planning, taking action, observing, evaluating and critical reflection prior to planning the next cycle (Shojania and Grimshaw 2005). Consistently with this approach, the phases of the project were: participated discussion between users, providers and researchers; assessment tool development by means of users focus group; users engagement in definition of data collection procedures and data gathering; report back of results to service and users, participated discussion of the results and of the changes to be introduced by the service organization; implementation of actions to improve services, return to gathering of users’ view to evaluate changes impact on service quality. This paper presents and discusses the results of the first phase, i.e. the evaluation of service quality by consumers. We describe here the process of indicators identification and tool development, the survey conduction and findings. The discussion of the results among consumers and staff, the definition of aspects most in need of change and the implementation of quality improvement, together with the findings of the survey following the improvement actions will be presented subsequently. Setting The project was conducted in a public agency providing mental health care to the province of Pistoia, in Tuscany, including 22 municipalities with a population of 292.000. It is characterized by a stable population with lower unemployment rates and higher income than average in Italy. Foreign migrants represented 9 % of the population. In 2009 the service included a network of integrated facilities: two Community Mental Health Centers operating 12 h a day 7 days a week, with teams providing outpatient care, rehabilitation, counseling and psychotherapy, home care and emergency interventions; Two General Hospital Inpatient Units for acute admissions, with a total of 15 beds; 2 Day Centers; Two fully staffed Residential Facilities with 16 places; Two Group Homes with 15 places; a Supported Housing Program. The multidisciplinary team included 59 nurses, 37 support workers, 27 rehabilitation counselors, 26 psychiatrists, five psychologists, two social workers. The Rosaspina consumer association worked in close collaboration with the service, running self-help groups and providing peer support.
123
404
In 2009 the service had a caseload of 4,471 patients, of whom 717 were first-time users. Access to care, as elsewhere in Italy, was free of charge, except a small co-payment for drug prescriptions. No private facilities were available in the area. The service was targeted mainly to severe mental disorders and adopted a treatment model centered on teamwork, continuity of care across facilities, network interventions and psychosocial rehabilitation. An individually tailored community support module, called ‘Social Inclusion Program’, encompassing a variety of interventions aimed at facilitating social integration in community activities, was available for most disabled people.
Community Ment Health J (2014) 50:402–408
mental health services since 10 years or more. 52 % had a diagnosis of schizophrenic disorder, 30 % affective disorders, 18 % personality disorders. 16 % were in residential facilities. Concerning the patterns of care in last year, 28 % had one or more hospital admission, 61 % were delivered rehabilitation interventions in community settings on outpatient basis, 35 % received residential care, 73 % home visits, 12 % family interventions, 25 % individual counseling, 9 % psychotherapy, 25 % participated in self-help groups. 90 % were on antipsychotic maintenance medication, 36 % were taking antidepressants and 51 % mood stabilizers. 27 % were prescribed more than one antipsychotic drug.
Participants Questionnaire Development All consumers satisfying the following criteria were asked to participate: age 18–60; ICD-10 diagnosis of affective disorder (F30–F39) or personality disorder (F60–F69) or schizophrenia and other psychotic disorder (F20-F29); high use of the service, defined by satisfying in last year at least one of the following criteria: (a) either one compulsory admission or two voluntary admissions to acute inpatient care, (b) admission to residential care or group home or supported housing program, (c) Day Center attendance, (d) any home visit or intervention, (e) enrollment in individual social inclusion program. These criteria were aimed to identify people with severe mental illness and high needs, included in the ‘Social Inclusion Program’. Like most public services in Italy, the service was strongly oriented to meet the needs of this high need population. The survey and the quality improvement process wanted to address the opinions of the population included in the ‘Social Inclusion Program’, because the implementation of the program had represented a strong commitment for the service, and the subjects included should be considered particularly competent about strengths and weaknesses of the service. The service also deemed that the implementation of individually tailored support for effective social integration had to be integrated with empowering practices in the service, also encompassing quality evaluation from the point of view of the users. The application of these criteria identified 290 subjects. Of these subjects, three died, three were in jail, 12 were out of area, 17 were impossible to trace, 15 were unable to answer because of unsuitable clinical conditions and 36 (12 %) refused. Therefore, 204 (70 % of the eligible population) accepted to participate. There were no significant differences in socio-demographic and clinical characteristics between the total sample and the respondent sample. Sample characteristics were as follows: 54 % male, 57 % aged between 41 and 60, 78 % unemployed, 12 % living alone and 57 % with parents, 70 % in contact with
123
A review of quality evaluation tools identified 21 questionnaires published between 1982 and 2009 concerning measurement of consumer satisfaction or consumer-driven quality assessment. From this corpus, a first draft of a questionnaire was developed and submitted for discussion to a focus group comprising participants in self-help groups held by the Rosaspina Association. Focus groups have been used as an appropriate means to develop a quality assessment tool in a shared fashion, where users’ experience can be valued (Frankel and Hourigan 2004). The draft addressed the following areas: relationships with staff, treatment efficacy, physical environment, information on service organization and available treatments; information on diagnosis and treatment plan; involvement in treatment decisions; visit waiting times, home visits. The draft was discussed and modified in four focus group meetings involving 14 users. Most of them were men (86 %), the age range was 29–53 years. 71 % did not have a job and 57 % lived with their family of origin. More than half of them had been in contact with the service for more than 10 years. All group members had a very active role in definition of the areas of interest, wording of questions and communication styles. They suggested a self-administered and anonymous questionnaire, asking for the replies to be held securely and confidentially. Moreover, they underlined the importance of job searching and support, and the scope for a more satisfying negotiation in drug prescription. They also proposed to investigate relationships with each type of professional, treatment adverse effects, staff attitudes during acute hospital admission and reception procedures in the mental health center. Users also prompted a more direct and friendly style in questions wording (i.e., ‘‘Do you get on well with your social worker?’’) and the use of Likerttype scales in the answers, associated with smiles. Role
Community Ment Health J (2014) 50:402–408
playing techniques were used by the researchers who attended the groups to facilitate user participation and exchange. The final version of the questionnaire comprised 11 yes/ no questions aimed at investigating whether the responder had experienced an aspect of healthcare in the last year (i.e. ‘‘Have you been admitted to the Inpatient Psychiatric Unit ?’’ or ‘‘Is a psychologist included in your treatment plan ?’’) and a 34-item Likert scale in which the respondent was asked for a judgment about quality in various areas, encompassing five possible answers accompanied by a smile: ranging from ‘very positive’, ‘positive’, ‘uncertain’, ‘negative’ and ‘very negative’. The items investigated relations with the professionals (behavior, accessibility, competence); interventions received (helpfulness and adverse effects), environment and facilities, shared decision-making, organization aspects such as waiting list, waiting time at the phone, home visits, help in emergency. Respondents were asked to give a judgment only on items related to the their specific healthcare experience. Since the focus was on consumers’ subjective view, no objective quality indicators were collected. Statistical Analyses and Reliability Frequencies were computed for each category of sociodemographic and clinical variables. Quality was assessed by computing percentages of responders who expressed a positive versus a negative evaluation for each item. To rule out the bias towards a positive evaluation often found in consumer surveys, the ‘very negative’ and ‘negative’ values were collapsed together into a single ‘negative’ category and only the ‘very positive’ were considered as an indicator of good quality. The decision was taken because a number of authors suggested that only extremely positive judgments should be considered as an indicator of optimal care, while more weight should be given to even partially negative evaluations (Collins and O’Cathain 2003). Reliability was assessed by measuring the degree of agreement between two measurements in a subsample of 20 randomly selected subjects retested after 2 weeks. Cohen’s k statistics were computed for yes/no questions investigating healthcare experiences, while weighted k was used for Likert quality items (Cohen 1968). We assumed that k values between 0.8 and 1.0 indicate an almost perfect agreement, between 0.6 and 0.79 substantial, between 0.4 and 0.59 moderate, between 0.2 and 0.39 low, and between 0 and 0.19 very low. Items with a score under 0.4 were considered unreliable and then excluded from analysis (Landis and Koch 1977). Two out of 11 yes/no questions about the healthcare experience showed an unsatisfactory reliability. This concerned the inclusion of nurses and support workers in the
405 Table 1 Perceived quality Item
Respondents N
Quality evaluation Positive (%)
Negative (%) 10
Relationship with psychiatrist
190
56.8
Relationship with rehabilitation counselor
111
52.3
7.2
Relationship with nurse
145
62.8
5.5
96
58.3
7.3
67
4.7
Relationship with support worker Psychiatrist’s skills
191
Rehabilitation counselor’s skills
110
60
5.5
Nurse’s skills
142
58.5
3.5
Social worker’s skills
91
58.2
7.7
Support worker’s skills
97
54.6
4.1
Medication helpfulness Medication side effects
190 185
42.6 37.4
8.9 26.8
Self-help group helpfulness
67
35.8
26.9
Vocational rehabilitation helpfulness
91
50.5
17.6
Social inclusion program helpfulness
93
48.4
15.1
Mental Health Centre environment
193
51.8
9.8
Hospital Inpatient Unit environment
172
37.2
18 15.4
Day Care Centre environment
149
40.9
Staff support during admission
112
45.5
9.8
Information about illness
189
36.5
29.1
Information about drugs
188
36.7
26.1
Choice of professionals
187
21.4
56.7
Reception at Mental Health Center
193
47.7
16.1
Waiting time for meeting a professional
188
19.1
50
Waiting time on the phone
188
28.7
34.6
Help in emergency
191
56
13.6
treatment team, probably because the respondents were unable to identify a difference between the two professions. Scores below 0.4 were found in eight items (23 %) of the Likert scale, four related to psychological interventions and others related to relationship with social worker, information about service programs, consideration of users’ opinion about treatments and home visits.
Results Six users were hired to deliver the questionnaire to the identified subjects, helping them to complete it if necessary, and collect the filled questionnaires. The survey
123
406
lasted three months and was conducted in three phases: first in day centers and residential facilities, then in the community mental health centers and clinics, and finally at the patients’ home. The users collecting the data worked in a completely autonomous fashion in day centers and residential facilities, whereas they received some logistic support to contact the sample subjects in other settings. In general, professionals involvement was kept minimal, to avoid in the interviewed the tendency towards conformity and overrating of positive answers (Bowling 2005). Percentages of positive and negative evaluations in all reliable items are shown in Table 1. Positive judgments outnumbered the negative ones in almost all items, with significant differences across areas. In items on relationships with professionals and professional skills, figures of positive evaluation ranged between 52 and 67 %. Favorable opinions were over 50 % on vocational rehabilitation, other treatment modalities attracted less than 50 % of positive evaluations. Relatively high proportion of negative ratings between 15 and 26 % were given to most psychosocial interventions, including self-help groups. Environmental quality was judged better in outpatient than in inpatient services. Lowest positive ratings were found for waiting times and choice of professionals, information about drugs and illness. Negative opinions peaked for choice of professional and were above 25 % for waiting times, information about illness and drugs, self-help groups and medication side effects.
Discussion The findings showed five areas as most critical: choice of professionals, waiting times for a meeting and on the phone, information about illness and, to a lesser extent, information on medications. Treatment helpfulness was judged as limited, but attitudes toward professionals was good. Our results confirm that people with severe mental disorders are able to give value judgments on various aspects of mental healthcare, provided they receive technical and organizational support. The questionnaire developed through the collaboration between researchers and consumers showed satisfactory psychometric properties and a good overall reliability. The low reliability of answers related to the inclusion of nurses and support workers in the treatment team reflects the consumers’ unclear perception of their professional roles, making it difficult to tell the two providers apart. The low
123
Community Ment Health J (2014) 50:402–408
reliability of items concerning the psychological interventions might be due to the low frequency of such treatments. The decision to entrust the focus group members the task of distributing the questionnaires may have fostered the reduction of refusal rate to 12 %, which is very low for a self-administered questionnaire (McColl et al. 2001). The respondents represented the vast majority of people with severe mental disorders treated by the department, therefore the results can be considered as very relevant for the quality improvement process. It has been repeatedly observed that in mental health most surveys aimed at eliciting consumer opinion consistently report high levels of positive evaluations. This has been related to a variety of factors, such as the social desirability bias, the lack of involvement of consumers in instrument development and data collection, the fear to disclose their true feelings or opinions due to the power imbalance with staff (Linhorst and Eckert 2002). This is especially true for severely ill people, as shown by the US Consumer Assessment of Behavioral Health Survey, in which the responses to the questionnaire designed for individuals with serious mental illnesses showed very low rates around 10 % of negative evaluations in almost all items (Eisen et al. 2001). We tried to minimize the trend towards positive answers by involving consumers in all research phases and avoiding the staff participation in distribution and collection of questionnaires, as suggested by previous research (Clark et al. 1999). However, to some extent, this problem remained and should therefore be considered in the interpretation of results. Although the responses were skewed towards the positive end of the scale, we focus on the items with a relevant proportion of low quality ratings, not only to take into account the conformity bias, but also because areas with critical judgments warrant more attention to improve service quality. Special attention should be paid to the areas in which the negative evaluations exceed or closely approximate the positive ones: choice of professionals, waiting times for a meeting and on the phone, information about illness and information on medications. All five do not refer to the outcomes of care, but to the concept of responsiveness, developed by World Health Organization to analyze the health systems performance and recently applied to consumers’ expectations in mental health (Bramesfeld et al. 2007). Responsiveness refers to nonhealth aspects of care, related to the desire of people to be treated in a respectful manner with due consideration of their rights. It consists of eight domains: confidentiality, autonomy, communication, dignity, social support, amenities, attention, choice.
Community Ment Health J (2014) 50:402–408
In a recent study of mental health consumers focus groups (Bramesfeld et al. 2007), choice has been defined as the situation in which providers can be chosen freely, communication as information about treatment and procedures in comprehensible manner, and attention as possibility of talking to a professional at short notice. Such definitions are very consistent with the items in our questionnaire on choice, waiting times and information about illness and drugs. In our survey the consumers identified these sensitive issues as areas of low quality requiring changes to make the service more responsive to their needs. Further observations can be made on judgments about treatments: relatively few consumers consider medications as unhelpful, but many negative opinions are expressed towards their side effects, which is in agreement with a balanced view of the role of antipsychotic medications. The proportion of negative ratings between 15 and 26 % on psychosocial interventions can be surprising, especially for self-help groups. This requires a careful consideration of the ways in which psychosocial interventions are planned, delivered and included in overall treatment plan. The most valued intervention in this area is vocational rehabilitation, clearly indicating that work is a priority for many consumers. It is possible that self-help groups are valued by a subgroup of consumers only, and therefore a number of people felt themselves pushed to participate in a disturbing way. The comparison with the results of other surveys is not easy, because this study was tailored to the needs of a specific consumer group. However, it should be noted that the area that attracted the highest rate of critical evaluations, i.e. the choice of the clinician, has been seldom investigated in most consumer opinion surveys. In the already mentioned US Consumer Assessment of Behavioral Health Survey an item addressing this issue was included in the questionnaire designed for less severe mental disorders only. However, it resulted ‘‘not applicable’’ for almost 70 % of the respondents (Eisen et al. 2001). By contrast, the need for information has been often addressed and it has been constantly identified as an area to be improved according to many consumers (Eisen et al. 2001; Ruggeri et al. 2007; Gigantesco et al. 2002; Howard et al. 2003; Cleary et al. 2003). In summary, this survey provided meaningful feedbacks about care experiences of a large representative sample of consumers with very severe mental disorders, showing that even in a service strongly oriented towards community care, the consumers’ needs in sensitive areas concerning choices, respect and autonomy are not met. The application of the concept of responsiveness to quality assurance is an important step towards the improvement of the service performance in accordance with consumers’ wishes and expectations. The next step will be to assess to what extent
407
the consumer feedback is taken into account for quality improvement actions.
References Avis, M., Bond, M., & Arthur, A. (1997). Questioning patient satisfaction: An empirical investigation in two outpatient clinics. Social Science and Medicine, 44, 88–92. Barbato, A. (1998). Psychiatry in transition: Outcomes of mental health policy shift in Italy. Australian and New Zealand Journal of Psychiatry, 32, 673–679. Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27, 281–291. Bramesfeld, A., Kippel, U., Seidel, G., Schwartz, F. W., & Dierks, M. L. (2007). How do patients expect the mental health service system to act ? Testing the WHO responsiveness concept for its appropriateness in mental health care. Social Science and Medicine, 65, 880–889. Bridges, J. F. P., Slawik, L., Schmeding, A., Reimer, J., Naber, D., & Kuhnigk, O. (2013). A test of concordance between patient and psychiatrist valuations of multiple treatment goals for schizophrenia. Health Expectations, 16, 164–176. Buckley, P. F., Wirshing, D. A., Bhushan, P., Pierre, J. M., Resnick, S. A., & Wirshing, W. C. (2007). Lack of insight in schizophrenia: Impact on treatment adherence. CNS Drugs, 21, 129–141. Clark, C. C., Scott, E. A., Boydell, K. M., & Goering, P. (1999). Effects of client interviewers on client-reported satisfaction with mental health services. Psychiatric Services, 50, 961–963. Cleary, M., Horsfall, J., & Hunt, G. E. (2003). Consumer feedback on nursing care and discharge planning. Journal of Advanced Nursing, 42, 269–277. Cohen, J. (1968). Weighted kappa. Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. Collins, K., & O’Cathain, A. (2003). The continuum of patient satisfaction—from satisfied to very satisfied. Social Science and Medicine, 57, 2465–2470. Cornwall, A., & Jewkes, R. (1995). What is participatory research? Social Science and Medicine, 41, 1667–1676. De Haan, L., Van Raaij, B., Van den Berg, R., Jager, M., Houweling, P., Stockmann, M., et al. (2001). Preferences for treatment during a first psychotic episode. European Psychiatry, 16, 83–89. Eisen, S. V., Shaul, J. A., Leff, H. S., Stringfellow, V., Clarridge, B. R., & Cleary, P. D. (2001). Toward a national consumer survey: Evaluation of the CABHS and MHSIP Instruments. Journal of Behavioral Health Services and Research, 28, 347–369. Frankel, R. M., & Hourigan, N. T. (2004). Thirty-five voices in search of an author: What focus groups reveal about patients’ experiences in managed care settings. Communication and Medicine, 1, 45–58. Gigantesco, A., Picardi, A., Chiaia, E., Balbi, A., & Morosini, P. (2002). Patients’ and relatives’ satisfaction with psychiatric services in a large catchment area in Rome. European Psychiatry, 17, 139–147. Groenewegen, P. P., Kerssens, J. J., Sixma, H. J., Van der Eijk, I., & Boerma, W. G. W. (2005). What is important in evaluating health care quality? An international comparison of user views. BMC Health Services Research, 5, 16.
123
408 Haggerty, J. L. (2010). Are measures of patient satisfaction hopelessly flawed? British Medical Journal, 341, 790–791. Hopkins, J. E., Loeb, S. J., & Fick, D. M. (2009). Beyond satisfaction, what service users expect of inpatient mental health care: A literature review. Journal of Psychiatric and Mental Health Nursing, 16, 927–937. Howard, P. B., El-Mallakh, P., Rayens, M. K., & Clark, J. J. (2003). Consumer perspectives on quality of inpatient mental health services. Archives of Psychiatric Nursing, 17, 205–217. Jorm, A., & Oh, E. (2009). Desire for social distance from people with mental disorders. Australian and New Zealand Journal of Psychiatry, 43, 183–200. Landis, J., & Koch, J. J. (1977). The measurement of observed agreement for categorical data. Biometrics, 33, 159–174. Laugharne, R., & Priebe, S. (2006). Trust, choice and power in mental health. A literature review. Social Psychiatry and Psychiatric Epidemiology, 41, 843–852. Linhorst, D. M., & Eckert, A. (2002). Involving people with severe mental illness in evaluation and performance improvement. Evaluation and the Health Professions, 25, 284–301. Marshall, S. L., Crowe, T. P., Oades, L. G., Dean, F. F., & Kavanagh, D. J. (2007). A review of consumer involvement in evaluations of case management: Consistency with a recovery paradigm. Psychiatric Services, 58, 396–401. Mason, K., Olmos-Gallo, A. O., Bacon, D., McQuilken, M., Henley, A., & Fisher, S. (2004). Exploring the consumers’ and providers’ perspective on service quality in community mental health care. Community Mental Health Journal, 40, 33–46. McColl, E., Jacoby, A., Thomas, L., Soutter, J., Bamford, C., Steen, N., et al. (2001). Design and use of questionnaires: A review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment, 5, 1–256. Noble, L. M., Douglas, B. C., & Newman, S. P. (2001). What do patients expect of psychiatric services? A systematic and critical review of empirical studies. Social Science and Medicine, 52, 985–998. Rogers, A., & Pilgrim, D. (1993). Service users’ views of psychiatric treatments. Sociology of Health & Illness, 15, 612–631.
123
Community Ment Health J (2014) 50:402–408 Roth, D., & Crane-Ross, D. (2002). Impact of services met needs, and service empowerment on consumer outcomes. Mental Health Services Research, 4, 43–56. Ruggeri, M., & Dall’Agnola, R. (1993). The development and use of the Verona expectations for care scale (VECS) and the Verona service satisfaction scale (VSSS) for measuring expectations and satisfaction with community-based psychiatric services to patients, relatives and professionals. Psychological Medicine, 23, 511–523. Ruggeri, M., Lasalvia, G., Salvi, G., Cristofalo, D., Bonetto, C., & Tansella, M. (2007). Application and usefulness of routine measurement of patients’ satisfaction with community-based mental health care. Acta Psychiatrica Scandinavica, 116(suppl. 437), 53–65. Shojania, K. G., & Grimshaw, J. M. (2005). Evidence-based quality improvement: The state of the science. Health Affairs, 24, 138–150. Shumway, M., Saunders, T., Shern, D., Pines, E., Downs, A., Burbine, T., et al. (2003). Preferences for schizophrenia treatment outcomes among public policy makers, consumers, families, and providers. Psychiatric Services, 54, 1124–1128. Simpson, E. L., & House, A. O. (2002). Involving users in the delivery and evaluation of mental health services: Systematic review. British Medical Journal, 325, 1265–1268. Smith, M. E., & Ganju, V. (2006). The MHSIP mental health quality report: The next generation of performance measures. In R. W. Manderscheid & J. T. Berry (Eds.), Mental health, United States, 2004. DHHS Pub No. (SMA)-06-4195. Rockville, MD: Substance Abuse and Mental Health Service Administration. Thornicroft, G., & Tansella, M. (2005). Growing recognition of the importance of service user involvement in mental health service planning and evaluation. Epidemiologia e Psichiatria Sociale, 14, 1–3. Weinstein, J. (2006). Involving mental health service users in quality assurance. Health Expectations, 9, 98–109.