520794 research-article2014
APY0010.1177/1039856214520794Australasian PsychiatryHolmes et al.
AP
Psychiatric services
Developing performance indicators for homeless outreach psychiatric services
Australasian Psychiatry 2014, Vol 22(2) 160–164 © The Royal Australian and New Zealand College of Psychiatrists 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1039856214520794 apy.sagepub.com
Alexander Holmes Associate Professor and Psychiatrist, Department of Psychiatry,) University of Melbourne, Melbourne, VIC, Australia
Gail Bradley Senior Clinical Psychologist, Inner West Area Mental Health Service, Melbourne, VIC, Australia Trevor Carslie Social Worker, Homeless Psychiatric Outreach Service, Inner West Area Mental Health Service, Melbourne, VIC, Australia
Kate Lhuede Senior Occupational Therapist, Homeless Psychiatric Outreach Service, Inner West Area Mental Health Service, Melbourne, VIC, Australia
Abstract Objective: Performance indicators (PIs) aim to improve services by measuring key activities in a way that allows comparison over time, between services and against benchmarks. This paper describes the development and implementation of Homeless Psychiatric Service PIs and explores their potential benefits and limitations. Method: We collected descripton of quality service from key stakeholders. We identified eight key parameters, from which PIs were developed and tested over a 12-month period. Results: The use of the PIs led to increased awareness of the practice being measured. PIs were used to stimulate practice changes. They played a positive role in team dynamics and were useful in clarifying team aims and identity. The main challenge to their use was the burden of data collection and analysis. Conclusion: Homeless service PIs can assist in determining how well the programs are performing in activities that are relevant to clients and non clinical services for the homeless. With the movement of homeless clients away from inner urban areas, homeless performance measures may aid teams to develop the capacity to work effectively with homeless clients. Keywords: benchmarking, homeless, homeless services, performance indicators, program planning, quality
P
erformance indicators (PIs) aim to improve services by measuring key activities in a way that allows comparison over time, between services and against benchmarks. Within Australian public mental health, the focus to date has been on PIs applicable over a wide range of services,1 yet the need to develop program-specific measures has been clearly articulated.2 Program-specific indicators measure activities that are more closely related to the key aims and objectives of the program, and have the advantage of promoting quality in their more specialised activities. Homeless Outreach Psychiatric Services is one example of where specific measures may have advantages. This paper describes the development and implementation of homeless psychiatric service PIs and explores their potential benefits and limitations. The Second National Mental Health Plan3 stipulated the development of performance indicators for Australian 160
public mental health services. These were part of the ‘information infrastructure’ designed to encourage good practice, inform about outcomes and value for money, and contribute to state and national data.4 In broad terms, these PIs are used to determine the degree that services are effective, responsive, continuous, appropriate, accessible, capable, efficient, safe and sustainable.4 PIs need to be observable, understandable, valid, relevant and efficient.5 An example of the current national indicator is ’28-day readmissions to acute mental health units’, for which the threshold set at 15%.6
Corresponding author: Alex Holmes, Department of Psychiatry, University of Melbourne, Royal Melbourne Hospital, Grattan Street, Parkville, Melbourne, VIC 3050, Australia. Email: [email protected]
Holmes et al.
PIs were also developed for the Child and Adolescent Mental Health Services (CAMS)7 and ConsultationLiaison (C-L) psychiatry services.8 Both assess ‘time to initial assessment’, a measure of responsiveness, which is important when providing mental health expertise to general health services. The CAMS PIs also measure dropout rate (continuity) and activity (hours of contact), highlighting the challenges of engagement and the value of the therapeutic alliance in this population. In C-L, PIs measure successful communication, in keeping with the consultation and collaboration at the core of C-L practice. Use of C-L PIs over a year can lead to improved communication,9 at the cost of regular data collection and analysis. No single method has been identified by which PIs are developed. A common approach is to first develop a battery of measures representing all the functions of a service, as identified by professionals, carers, patients and affiliated organisations.10,11 For example, Shield et al.,11 working in primary care, identified a potential of 334 indicators. These were rated for validity by representative panels, with 26% achieving consensus across all groups. This method is comprehensive and promotes a balance of perspectives, but requires the capacity to engage all stakeholders. An alternative technique is for expert groups to formulate indicators with reference to their experience and the literature.2 Once a PI is defined, irrespective of the method used, a benchmark is established. This is informed by analysis of the literature, information about ‘current reasonable practice’ and the benefits of a particular activity. For example, the threshold for ‘28-day readmissions’ was set using length-of-stay data and the principle that unplanned early admission reflects a breakdown of community care.6 Finally, there is a limit on the number of PIs that can be applied in any one setting, because their utility is known to decrease as their number and complexity increase.12
Developing performance measures for homeless psychiatric services The PIs for the homeless were developed within an inner urban homeless mental health service,13 using both stakeholder and expert data. Stakeholders (clients, carers, accommodation and mental health services) required different methods for collecting information. Client and carer needs were determined by reviewing the documented response to the question, ‘What are your current needs?’ for the approximately 100 new cases undergoing needs assessment during the previous year. Stakeholders in accommodation services were asked to complete a written survey, which asked: ‘What are the most important aspects of a homeless mental health service?’ We sent the survey to the 25 services within the catchment area: with the aid of telephone follow-up, all were able to provide information. The Homeless Mental Health Service perspective was derived
Table 1. Needs that may be addressed by a homeless mental health service, as identified by different stakeholders Stakeholders
Themes
Clients Carers Accommodation services Homeless Psychiatric Services
Autonomy Improved housing Improved finances Effective treatment Communication Management of mental illness Expertise and advice Responsiveness Outreach Collaboration Communication Support Engagement of disengaged homeless persons Developing a pathway out of homelessness Assertive outreach Responsiveness and accessibility Continuity of care Shared care and collaborative casework
with reference to the literature13–15 and face-to-face communication with the limited number of staff working in specialised programs state-wide. The raw data was subjected to a simple thematic inductive analysis. Initially, the themes were identified and coded. These themes were then taken back to each case, in order to determine if other themes were identified that were not covered thematically. This process allowed for confidence that, aside from individual or idiosyncratic needs, no major needs were excluded. The final themes are listed in Table 1. PIs were developed to address the themes identified (Table 2). Not all themes were fully covered. A measure of improved finances was not included, as the factors influencing this were seen to extend well beyond the influence of the clinical service. Agency support was conceptually similar to ‘responsiveness and accessibility’, and subsumed under that theme. Two PIs required the use of scales. For the ‘assessment of improved engagement’, we used a 5-point scale (1 = refuses to engage, 2 = poor engagement, 3 = some engagement, 4 = moderate engagement and 5 = actively engaged). If the Life Skills Profile is being collected as an outcome measure, then item 19 there can be used as an
161
Australasian Psychiatry 22(2)
Table 2. Performance indicators for Homeless Psychiatric Services Name
Themes
Definition
Threshold
1
Contact Success
Outreach
70%
2
Timeliness
Responsiveness
3
Engaged in active treatment Shared care and collaborative casework
Mental illness management Collaboration, communication, shared care Communication
N patients seen, as a percentage of the n patients referred N patients seen within 72 hours, as a percentage of n patients seen N patients engaged in active treatment, as percentage of n patients seen N patients with active collaborative case plan with homeless agency, as a percentage of n patients engaged in active treatment N patients for who there was regular contact with family/carers, as a percentage of n patients for whom family/carers are identified N patients with improved engagement at discharge, as a percentage of n patients discharged N patients discharged to, and engaged with, appropriate mental health service; as a percentage of n patients discharged N patients with improved accommodation type at discharge, as compared with admission, as a percentage of n patients discharged
4 5
Communication with family/carers
6
Improved engagement at Engagement discharge Successful discharge Continuity of care planning
7 8
Improved accommodation at discharge
Improved housing
50% 50% 50% 50% 50% 50% 50%
N or n: number
alternative16 (Does this person co-operate with health services? Always = 4, Usually = 3, Rarely = 2 and Never =1). Assessment of an improved accommodation requires ‘ranking’ by accommodation type. The hierarchy used was developed in reference to the definition of homeless used by the Australian Bureau of Statistics17 (1 = shelterless, 2 = emergency accommodation, 3 = single room without amenities, 4 = transitional accommodation and 5 = stable and secure housing). The benchmarks for the PIs were set with reference to what was known about the current performance, aspirations for improvement and a judgement as to the limits of effectiveness. For example, prior to commencement the number of referred patients whom were assessed was < 50%. At the same time, the capacity to engage the disengaged was identified by three stakeholders: it is a key component of the Homeless Outreach Psychiatric Service model.13 Setting the threshold at 70% reflected an expectation of improvement, balanced with the practical difficulty of finding homeless persons.
Implementing performance measures We collected data over a calendar year, for all of the cases referred to a homeless psychiatric service.13 This service covers the central business district and inner northern suburbs of Melbourne, Australia, comprising a popula-
162
tion of approximately 180,000 persons. The catchment included two male homeless shelters containing 120 beds. During the year, no substantial changes occurred within mental health nor accommodation services. During the initial implementation phase, concern was raised by participating clinicians regarding the burden of the collection and recording of data, and the complexity of interpreting numerous PIs. In response, cognisant of the principle limiting the number of PIs, we removed the two that were proving the most difficult to collect (communication with family and collaborative casework). Data was collected at entry and at discharge from the service. When data was incomplete or unclear, neither the numerator nor the denominator were entered. Numerators and denominators for each PI were calculated for each month, and then entered into a preformatted Excel spread sheet, which generated a 12-month summary and graph (Figure 1). We presented and discussed the data at monthly team meetings, where we devised and implemented strategies related to improving specific activities. Overall, the activities recorded by the PIs occurred at levels similar to established thresholds (Table 3). When changes occurred, they emerged over the course of the year. In the first months, their primary impact was through an increased awareness of the practice being measured. Subsequently, sub-optimal performance relative to the
Proportion engaged at discharge (%)
Holmes et al. 100 90 80 70 60 50 40 30 20 10 0 Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Month of the year
Figure 1. Homeless Psychiatric Service Performance Indicator: Improved engagement at discharge.
Table 3. Mean percentage and range of individual PIs over a 12-month period Description
Threshold
Yearly mean, %
Range (over 1 year)
Contact success Timeliness Engaged in active treatment Improved engagement at discharge Successful discharge planning Improved accommodation at discharge
70% 50% 50% 50% 50% 50%
85 50 58 50 54 45
70 – 100 14 – 100 25 – 100 17 – 100 33 – 100 17 – 100
PI: Performance indicator
arbitrary threshold (timeliness, active treatment or improved discharge accommodation) drew greater attention, leading to both analysis and practice change. For example, exploration of the low levels of engagement at discharge revealed the impact of a precipitous unplanned exit from an emergency accommodation. These exits were instigated either by the client or the service, in response to particular behaviours. The team action was to become an active participant in accommodation planning (coordinated by the housing worker) and to develop management plans, to deal with the behaviours associated with eviction. In time, fewer clients left the homeless setting without follow-up care being arranged. Similar improvements occurred in accommodation at discharge and successful discharge planning.
practices, having much in common with those applied in generic mental health services, place specific emphasis on access, engagement, continuity and interagency collaboration. When incorporated into a routine team process, homeless PIs can lead to practice change and improved care.
Discussion
The PIs had a range of effects on the various stakeholders. In the client domain, accommodation at discharge improved. Although the complex interplay between housing type, stability and mobility, and mental illness precludes a comprehensive analysis of all potential mechanisms for this finding, one is worth describing. When information about individuals with a mental illness is communicated to housing workers it changes expectations and improves engagement, resulting in greater tolerance and more appropriate discharge planning. Within the team, the PIs had a positive role in team dynamics. The monitoring of activities defined as ‘most important’ was seen as more meaningful and demonstrating gains for clients that enhanced team agency and identity. This support of group process is particularly useful when working with homeless clients, whom can be difficult and time consuming to work with, and about whom pessimism and frustration are commonly expressed.
We designed these PIs to measure activities identified as important by those involved in the care of the homeless mentally ill, including the clients themselves. These
The major drawback of using PIs is the burden of collection, analysis and use of data within cycles of quality improvement. A quantification of this effort was not
Not all PIs improved over the year. Where early levels were high (contact success), there was little scope for improvement. For others, for example ‘engaged in active treatment’, initial improvements were not sustained. One interpretation of this finding is that initial improvements relating to an increased focus on a function, but not supported by effective change in method, will not be lasting.
163
Australasian Psychiatry 22(2)
possible in our project, due to the disparate elements involved, and the overlap of development and implementation. Data collection can be integrated into clinical documentation, and review incorporated into routine team meetings, but the analysis and formatting does require time and commitment. This activity is only likely to occur if it is perceived as useful. For this reason, it may be that these PIs are used by some services over a defined period of time. The methods we used to develop the PIs have limitations. Face-to-face interviews have a greater capacity to explore responses than written questions, but they require a level of access and collaboration that is not easily found in homeless settings. Not all perspectives were equally represented. Only one need that was specifically identified by clients (improved accommodation) was included. The request for autonomy by clients was difficult to address, as for some it more accurately represented a desire to be ‘left alone’; however, working with the client’s autonomy is implicit to having a collaborative relationship and improvement in that domain may be seen to reflect this. The removal of the PIs relating to family and collaboration also limited the scope of representation. Homeless PIs may be useful in developing homeless functions in less-specialised community teams. Originally, the homeless teams were established to enhance access to care, largely in inner urban settings.13 The gentrification of the inner city, housing pressures and urban sprawl have all contributed to a migration of those with accommodation instability to the middle and outer suburbs. Periods of homelessness are common in patients attending community clinics.18 The capacity to access and work with homeless people may be required for many urban mental health services, practices which may be measured using these PIs.
Conclusion Homeless service PIs can determine how well programs are performing in activities that are relevant to clients and service providers within settings where there are homeless people. Using them can improve care and facilitate team morale. These measures may also be useful to nonspecialised teams, when working with homeless clients. Acknowledgements Thanks to Lisa Gibbs, of the University of Melbourne School of Population and Global Health, for her advice and expertise in qualitative research methods.
164
Disclosure The authors report no conflict of interest. The authors alone are responsible for the content and writing of the paper.
References 1. Australian Institute of Health and Welfare. A set of performance indicators across the health and aged care system. Canberra: Australian Institute of Health and Welfare, 2008. 2. National Mental Health Working Group Information Strategy Committee Performance Indicator Drafting Group. Key performance indicators for Australian Public Mental Health Services. Report, Discussion paper no. 6, 2005. Canberra: Australian Government Department of Health and Aging. 3. Australian Health Ministers. Second national mental health plan. Mental Health Branch, Commonwealth Department of Health and Family Services, July 1998. 4. National Health Performance Committee. National health performance framework. Report to the Australian Health Ministers, 2001. Brisbane: Queensland Health. 5. Poertner J and Rapp CA. Designing social work management information systems: The case for performance guidance systems. In: Patti RJ, Poertner J and Rapp CA (eds) Managing for service effectiveness. New York: Haworth Press, 1998, pp. 177–190. 6. Callaly T, Hyland M, Trauer T, et al. Readmission to an acute psychiatric unit within 28 days of discharge: Identifying those at risk. Aust Health Rev 34: 282–285. 7. Birleson P, Brann P and Smith A. Using program theory to develop key performance indicators for child and adolescent mental health services. Aust Health Rev 2001; 24: 10–21. 8. Holmes AC, Judd FK, Lloyd JH, et al. The development of clinical indicators for a consultation-liaison service. Aust NZ J Psychiatry 2000; 34: 496–503. 9. Holmes AC, Judd FK, Yeatman R, et al. A 12-month follow up of the implementation of clinical indicators in a consultation-liaison service. Aust NZ J Psychiatry 2001; 35: 236–239. 10. O’Brien AP, O’Brien AJ, Hardy DJ, et al. The New Zealand development and trial of mental health nursing clinical indicators: A bicultural study. Int J Nurs Stud 2003; 40: 853–861. 11. Shield T, Campbell S, Rogers A, et al. Quality indicators for primary care mental health services. Qual Saf Health Care 2003; 12: 100–106. 12. Addington D, Kyle T, Desai S, et al. Facilitators and barriers to implementing quality measurement in primary mental health care: Systematic review. Can Fam Physician 56: 1322–1331. 13. Holmes A, Hodge M, Newton R, et al. Development of an inner urban homeless mental health service. Australas Psychiatry 2005; 13: 64–67. 14. Catholic Social Services. Summary report: Access to psychiatric care. Twelve months on. Report, 1995. Victoria: Catholic Social Services. 15. Human Rights and Equal Opportunity Commission. Human rights and mental illness. Report of the National Inquiry into the Human Rights of People with Mental Illness, 1993. 16. Rosen A, Hadzi-Pavlovic D and Parker G. The Life Skills Profile: A measure assessing function and disability in schizophrenia. Schizophr Bull 1989; 15: 325–337. 17. Australian Bureau of Statistics. Information paper - A statistical definition of homlessness. Report, 2012. Canberra: Australian Bureau of Statistics. 18. Holmes AC, Hodge MA, Bradley G, et al. Accommodation history and continuity of care in patients with psychosis. Aust NZ J Psychiatry 2005; 39: 175–179.
Copyright of Australasian Psychiatry is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
APY0010.1177/1039856214520794Australasian PsychiatryHolmes et al.
AP
Psychiatric services
Developing performance indicators for homeless outreach psychiatric services
Australasian Psychiatry 2014, Vol 22(2) 160–164 © The Royal Australian and New Zealand College of Psychiatrists 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1039856214520794 apy.sagepub.com
Alexander Holmes Associate Professor and Psychiatrist, Department of Psychiatry,) University of Melbourne, Melbourne, VIC, Australia
Gail Bradley Senior Clinical Psychologist, Inner West Area Mental Health Service, Melbourne, VIC, Australia Trevor Carslie Social Worker, Homeless Psychiatric Outreach Service, Inner West Area Mental Health Service, Melbourne, VIC, Australia
Kate Lhuede Senior Occupational Therapist, Homeless Psychiatric Outreach Service, Inner West Area Mental Health Service, Melbourne, VIC, Australia
Abstract Objective: Performance indicators (PIs) aim to improve services by measuring key activities in a way that allows comparison over time, between services and against benchmarks. This paper describes the development and implementation of Homeless Psychiatric Service PIs and explores their potential benefits and limitations. Method: We collected descripton of quality service from key stakeholders. We identified eight key parameters, from which PIs were developed and tested over a 12-month period. Results: The use of the PIs led to increased awareness of the practice being measured. PIs were used to stimulate practice changes. They played a positive role in team dynamics and were useful in clarifying team aims and identity. The main challenge to their use was the burden of data collection and analysis. Conclusion: Homeless service PIs can assist in determining how well the programs are performing in activities that are relevant to clients and non clinical services for the homeless. With the movement of homeless clients away from inner urban areas, homeless performance measures may aid teams to develop the capacity to work effectively with homeless clients. Keywords: benchmarking, homeless, homeless services, performance indicators, program planning, quality
P
erformance indicators (PIs) aim to improve services by measuring key activities in a way that allows comparison over time, between services and against benchmarks. Within Australian public mental health, the focus to date has been on PIs applicable over a wide range of services,1 yet the need to develop program-specific measures has been clearly articulated.2 Program-specific indicators measure activities that are more closely related to the key aims and objectives of the program, and have the advantage of promoting quality in their more specialised activities. Homeless Outreach Psychiatric Services is one example of where specific measures may have advantages. This paper describes the development and implementation of homeless psychiatric service PIs and explores their potential benefits and limitations. The Second National Mental Health Plan3 stipulated the development of performance indicators for Australian 160
public mental health services. These were part of the ‘information infrastructure’ designed to encourage good practice, inform about outcomes and value for money, and contribute to state and national data.4 In broad terms, these PIs are used to determine the degree that services are effective, responsive, continuous, appropriate, accessible, capable, efficient, safe and sustainable.4 PIs need to be observable, understandable, valid, relevant and efficient.5 An example of the current national indicator is ’28-day readmissions to acute mental health units’, for which the threshold set at 15%.6
Corresponding author: Alex Holmes, Department of Psychiatry, University of Melbourne, Royal Melbourne Hospital, Grattan Street, Parkville, Melbourne, VIC 3050, Australia. Email: [email protected]
Holmes et al.
PIs were also developed for the Child and Adolescent Mental Health Services (CAMS)7 and ConsultationLiaison (C-L) psychiatry services.8 Both assess ‘time to initial assessment’, a measure of responsiveness, which is important when providing mental health expertise to general health services. The CAMS PIs also measure dropout rate (continuity) and activity (hours of contact), highlighting the challenges of engagement and the value of the therapeutic alliance in this population. In C-L, PIs measure successful communication, in keeping with the consultation and collaboration at the core of C-L practice. Use of C-L PIs over a year can lead to improved communication,9 at the cost of regular data collection and analysis. No single method has been identified by which PIs are developed. A common approach is to first develop a battery of measures representing all the functions of a service, as identified by professionals, carers, patients and affiliated organisations.10,11 For example, Shield et al.,11 working in primary care, identified a potential of 334 indicators. These were rated for validity by representative panels, with 26% achieving consensus across all groups. This method is comprehensive and promotes a balance of perspectives, but requires the capacity to engage all stakeholders. An alternative technique is for expert groups to formulate indicators with reference to their experience and the literature.2 Once a PI is defined, irrespective of the method used, a benchmark is established. This is informed by analysis of the literature, information about ‘current reasonable practice’ and the benefits of a particular activity. For example, the threshold for ‘28-day readmissions’ was set using length-of-stay data and the principle that unplanned early admission reflects a breakdown of community care.6 Finally, there is a limit on the number of PIs that can be applied in any one setting, because their utility is known to decrease as their number and complexity increase.12
Developing performance measures for homeless psychiatric services The PIs for the homeless were developed within an inner urban homeless mental health service,13 using both stakeholder and expert data. Stakeholders (clients, carers, accommodation and mental health services) required different methods for collecting information. Client and carer needs were determined by reviewing the documented response to the question, ‘What are your current needs?’ for the approximately 100 new cases undergoing needs assessment during the previous year. Stakeholders in accommodation services were asked to complete a written survey, which asked: ‘What are the most important aspects of a homeless mental health service?’ We sent the survey to the 25 services within the catchment area: with the aid of telephone follow-up, all were able to provide information. The Homeless Mental Health Service perspective was derived
Table 1. Needs that may be addressed by a homeless mental health service, as identified by different stakeholders Stakeholders
Themes
Clients Carers Accommodation services Homeless Psychiatric Services
Autonomy Improved housing Improved finances Effective treatment Communication Management of mental illness Expertise and advice Responsiveness Outreach Collaboration Communication Support Engagement of disengaged homeless persons Developing a pathway out of homelessness Assertive outreach Responsiveness and accessibility Continuity of care Shared care and collaborative casework
with reference to the literature13–15 and face-to-face communication with the limited number of staff working in specialised programs state-wide. The raw data was subjected to a simple thematic inductive analysis. Initially, the themes were identified and coded. These themes were then taken back to each case, in order to determine if other themes were identified that were not covered thematically. This process allowed for confidence that, aside from individual or idiosyncratic needs, no major needs were excluded. The final themes are listed in Table 1. PIs were developed to address the themes identified (Table 2). Not all themes were fully covered. A measure of improved finances was not included, as the factors influencing this were seen to extend well beyond the influence of the clinical service. Agency support was conceptually similar to ‘responsiveness and accessibility’, and subsumed under that theme. Two PIs required the use of scales. For the ‘assessment of improved engagement’, we used a 5-point scale (1 = refuses to engage, 2 = poor engagement, 3 = some engagement, 4 = moderate engagement and 5 = actively engaged). If the Life Skills Profile is being collected as an outcome measure, then item 19 there can be used as an
161
Australasian Psychiatry 22(2)
Table 2. Performance indicators for Homeless Psychiatric Services Name
Themes
Definition
Threshold
1
Contact Success
Outreach
70%
2
Timeliness
Responsiveness
3
Engaged in active treatment Shared care and collaborative casework
Mental illness management Collaboration, communication, shared care Communication
N patients seen, as a percentage of the n patients referred N patients seen within 72 hours, as a percentage of n patients seen N patients engaged in active treatment, as percentage of n patients seen N patients with active collaborative case plan with homeless agency, as a percentage of n patients engaged in active treatment N patients for who there was regular contact with family/carers, as a percentage of n patients for whom family/carers are identified N patients with improved engagement at discharge, as a percentage of n patients discharged N patients discharged to, and engaged with, appropriate mental health service; as a percentage of n patients discharged N patients with improved accommodation type at discharge, as compared with admission, as a percentage of n patients discharged
4 5
Communication with family/carers
6
Improved engagement at Engagement discharge Successful discharge Continuity of care planning
7 8
Improved accommodation at discharge
Improved housing
50% 50% 50% 50% 50% 50% 50%
N or n: number
alternative16 (Does this person co-operate with health services? Always = 4, Usually = 3, Rarely = 2 and Never =1). Assessment of an improved accommodation requires ‘ranking’ by accommodation type. The hierarchy used was developed in reference to the definition of homeless used by the Australian Bureau of Statistics17 (1 = shelterless, 2 = emergency accommodation, 3 = single room without amenities, 4 = transitional accommodation and 5 = stable and secure housing). The benchmarks for the PIs were set with reference to what was known about the current performance, aspirations for improvement and a judgement as to the limits of effectiveness. For example, prior to commencement the number of referred patients whom were assessed was < 50%. At the same time, the capacity to engage the disengaged was identified by three stakeholders: it is a key component of the Homeless Outreach Psychiatric Service model.13 Setting the threshold at 70% reflected an expectation of improvement, balanced with the practical difficulty of finding homeless persons.
Implementing performance measures We collected data over a calendar year, for all of the cases referred to a homeless psychiatric service.13 This service covers the central business district and inner northern suburbs of Melbourne, Australia, comprising a popula-
162
tion of approximately 180,000 persons. The catchment included two male homeless shelters containing 120 beds. During the year, no substantial changes occurred within mental health nor accommodation services. During the initial implementation phase, concern was raised by participating clinicians regarding the burden of the collection and recording of data, and the complexity of interpreting numerous PIs. In response, cognisant of the principle limiting the number of PIs, we removed the two that were proving the most difficult to collect (communication with family and collaborative casework). Data was collected at entry and at discharge from the service. When data was incomplete or unclear, neither the numerator nor the denominator were entered. Numerators and denominators for each PI were calculated for each month, and then entered into a preformatted Excel spread sheet, which generated a 12-month summary and graph (Figure 1). We presented and discussed the data at monthly team meetings, where we devised and implemented strategies related to improving specific activities. Overall, the activities recorded by the PIs occurred at levels similar to established thresholds (Table 3). When changes occurred, they emerged over the course of the year. In the first months, their primary impact was through an increased awareness of the practice being measured. Subsequently, sub-optimal performance relative to the
Proportion engaged at discharge (%)
Holmes et al. 100 90 80 70 60 50 40 30 20 10 0 Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Month of the year
Figure 1. Homeless Psychiatric Service Performance Indicator: Improved engagement at discharge.
Table 3. Mean percentage and range of individual PIs over a 12-month period Description
Threshold
Yearly mean, %
Range (over 1 year)
Contact success Timeliness Engaged in active treatment Improved engagement at discharge Successful discharge planning Improved accommodation at discharge
70% 50% 50% 50% 50% 50%
85 50 58 50 54 45
70 – 100 14 – 100 25 – 100 17 – 100 33 – 100 17 – 100
PI: Performance indicator
arbitrary threshold (timeliness, active treatment or improved discharge accommodation) drew greater attention, leading to both analysis and practice change. For example, exploration of the low levels of engagement at discharge revealed the impact of a precipitous unplanned exit from an emergency accommodation. These exits were instigated either by the client or the service, in response to particular behaviours. The team action was to become an active participant in accommodation planning (coordinated by the housing worker) and to develop management plans, to deal with the behaviours associated with eviction. In time, fewer clients left the homeless setting without follow-up care being arranged. Similar improvements occurred in accommodation at discharge and successful discharge planning.
practices, having much in common with those applied in generic mental health services, place specific emphasis on access, engagement, continuity and interagency collaboration. When incorporated into a routine team process, homeless PIs can lead to practice change and improved care.
Discussion
The PIs had a range of effects on the various stakeholders. In the client domain, accommodation at discharge improved. Although the complex interplay between housing type, stability and mobility, and mental illness precludes a comprehensive analysis of all potential mechanisms for this finding, one is worth describing. When information about individuals with a mental illness is communicated to housing workers it changes expectations and improves engagement, resulting in greater tolerance and more appropriate discharge planning. Within the team, the PIs had a positive role in team dynamics. The monitoring of activities defined as ‘most important’ was seen as more meaningful and demonstrating gains for clients that enhanced team agency and identity. This support of group process is particularly useful when working with homeless clients, whom can be difficult and time consuming to work with, and about whom pessimism and frustration are commonly expressed.
We designed these PIs to measure activities identified as important by those involved in the care of the homeless mentally ill, including the clients themselves. These
The major drawback of using PIs is the burden of collection, analysis and use of data within cycles of quality improvement. A quantification of this effort was not
Not all PIs improved over the year. Where early levels were high (contact success), there was little scope for improvement. For others, for example ‘engaged in active treatment’, initial improvements were not sustained. One interpretation of this finding is that initial improvements relating to an increased focus on a function, but not supported by effective change in method, will not be lasting.
163
Australasian Psychiatry 22(2)
possible in our project, due to the disparate elements involved, and the overlap of development and implementation. Data collection can be integrated into clinical documentation, and review incorporated into routine team meetings, but the analysis and formatting does require time and commitment. This activity is only likely to occur if it is perceived as useful. For this reason, it may be that these PIs are used by some services over a defined period of time. The methods we used to develop the PIs have limitations. Face-to-face interviews have a greater capacity to explore responses than written questions, but they require a level of access and collaboration that is not easily found in homeless settings. Not all perspectives were equally represented. Only one need that was specifically identified by clients (improved accommodation) was included. The request for autonomy by clients was difficult to address, as for some it more accurately represented a desire to be ‘left alone’; however, working with the client’s autonomy is implicit to having a collaborative relationship and improvement in that domain may be seen to reflect this. The removal of the PIs relating to family and collaboration also limited the scope of representation. Homeless PIs may be useful in developing homeless functions in less-specialised community teams. Originally, the homeless teams were established to enhance access to care, largely in inner urban settings.13 The gentrification of the inner city, housing pressures and urban sprawl have all contributed to a migration of those with accommodation instability to the middle and outer suburbs. Periods of homelessness are common in patients attending community clinics.18 The capacity to access and work with homeless people may be required for many urban mental health services, practices which may be measured using these PIs.
Conclusion Homeless service PIs can determine how well programs are performing in activities that are relevant to clients and service providers within settings where there are homeless people. Using them can improve care and facilitate team morale. These measures may also be useful to nonspecialised teams, when working with homeless clients. Acknowledgements Thanks to Lisa Gibbs, of the University of Melbourne School of Population and Global Health, for her advice and expertise in qualitative research methods.
164
Disclosure The authors report no conflict of interest. The authors alone are responsible for the content and writing of the paper.
References 1. Australian Institute of Health and Welfare. A set of performance indicators across the health and aged care system. Canberra: Australian Institute of Health and Welfare, 2008. 2. National Mental Health Working Group Information Strategy Committee Performance Indicator Drafting Group. Key performance indicators for Australian Public Mental Health Services. Report, Discussion paper no. 6, 2005. Canberra: Australian Government Department of Health and Aging. 3. Australian Health Ministers. Second national mental health plan. Mental Health Branch, Commonwealth Department of Health and Family Services, July 1998. 4. National Health Performance Committee. National health performance framework. Report to the Australian Health Ministers, 2001. Brisbane: Queensland Health. 5. Poertner J and Rapp CA. Designing social work management information systems: The case for performance guidance systems. In: Patti RJ, Poertner J and Rapp CA (eds) Managing for service effectiveness. New York: Haworth Press, 1998, pp. 177–190. 6. Callaly T, Hyland M, Trauer T, et al. Readmission to an acute psychiatric unit within 28 days of discharge: Identifying those at risk. Aust Health Rev 34: 282–285. 7. Birleson P, Brann P and Smith A. Using program theory to develop key performance indicators for child and adolescent mental health services. Aust Health Rev 2001; 24: 10–21. 8. Holmes AC, Judd FK, Lloyd JH, et al. The development of clinical indicators for a consultation-liaison service. Aust NZ J Psychiatry 2000; 34: 496–503. 9. Holmes AC, Judd FK, Yeatman R, et al. A 12-month follow up of the implementation of clinical indicators in a consultation-liaison service. Aust NZ J Psychiatry 2001; 35: 236–239. 10. O’Brien AP, O’Brien AJ, Hardy DJ, et al. The New Zealand development and trial of mental health nursing clinical indicators: A bicultural study. Int J Nurs Stud 2003; 40: 853–861. 11. Shield T, Campbell S, Rogers A, et al. Quality indicators for primary care mental health services. Qual Saf Health Care 2003; 12: 100–106. 12. Addington D, Kyle T, Desai S, et al. Facilitators and barriers to implementing quality measurement in primary mental health care: Systematic review. Can Fam Physician 56: 1322–1331. 13. Holmes A, Hodge M, Newton R, et al. Development of an inner urban homeless mental health service. Australas Psychiatry 2005; 13: 64–67. 14. Catholic Social Services. Summary report: Access to psychiatric care. Twelve months on. Report, 1995. Victoria: Catholic Social Services. 15. Human Rights and Equal Opportunity Commission. Human rights and mental illness. Report of the National Inquiry into the Human Rights of People with Mental Illness, 1993. 16. Rosen A, Hadzi-Pavlovic D and Parker G. The Life Skills Profile: A measure assessing function and disability in schizophrenia. Schizophr Bull 1989; 15: 325–337. 17. Australian Bureau of Statistics. Information paper - A statistical definition of homlessness. Report, 2012. Canberra: Australian Bureau of Statistics. 18. Holmes AC, Hodge MA, Bradley G, et al. Accommodation history and continuity of care in patients with psychosis. Aust NZ J Psychiatry 2005; 39: 175–179.
Copyright of Australasian Psychiatry is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
