International Review of Psychiatry, August 2015; 27(4): 276–285

Outcome measurement in New Zealand


Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Te Pou o Te Whakaaro Nui, Hamilton, New Zealand Abstract This paper provides a detailed description and critique of the development of routine outcome measurement (ROM) within New Zealand’s mental health and addiction services. The paper will include a brief description of the New Zealand setting and the events that led to routine outcome measures, demographic and diagnostic characteristics of the population in New Zealand, characteristics of the New Zealand mental health and addictions services, a description of the outcome measures and rationale for their selection, the information collection protocol for the outcome measures, outcome data completion rates and aggregated outcome reporting uses, barriers and facilitators to the adoption of routine outcome measures in New Zealand, and current status and next steps.

The New Zealand setting and background to routine outcome measures New Zealand is a small multi-ethnic country with a population of approximately 4.2 million. The country has a long tradition of publicly funded health services. The government’s annual funding to the health sector is around NZ$15.6 billion (Ryall, 2014). Mental health funding is delivered via public service district health boards (DHBs), non-governmental organizations (NGOs) and a heavily subsidized primary mental health sector. Over the past 15 years a number of national mental health strategies have driven New Zealand’s vision to measure outcomes in order to improve mental health and addiction services for service-users, and to better utilize resources. The historical background to ROMs in New Zealand will be briefly discussed in this section. An initial report, commissioned in 1994 by the Ministry of Health (Peters, 1994) examined literature related to outcome indicators in mental health. In 1997 the Health Funding Authority established the Mental Health Research and Development Strategy (MHR&DS) (Ministry of Health, 1997), administered by the Health Research Council of New Zealand. The MHR&DS strategy focused on four priority areas: • epidemiology – to measure the incidence and prevalence of different mental health problems in the New Zealand population

• outcomes – to develop and assess measures of mental health outcomes, • casemix – to develop and assess a casemix classification system to inform planning, purchasing and delivery of mental health and addiction services in New Zealand, • quality and best practice. Support for the strategy came from the sector funder, the Ministry of Health, as well as the Mental Health Commission. A number of key documents (Ministry of Health, 1991, 1994) underpin the ongoing development of mental health and addiction outcome measures in New Zealand. The Mental Health Commission’s Blueprint I (Mental Health Commission, 1998) makes reference to mental health outcome measures, and in particular their role in measuring effectiveness and the allocation of funding. It states: Services cannot know if they are operating effectively unless they can systematically measure improvements in the health of people who use the services. In the absence of measuring outcomes it is difficult to determine whether funds are being spent in the most effective way. (p. 6) In 2000 New Zealand introduced the Mental Health Information National Collection (MHINC) which collected activity data from mental health services. MHINC did not include outcomes information, even

Correspondence: Dr Mark Smith, Clinical Lead, Te Pou o Te Whakaaro Nui, PO Box 307, Waikato Mail Centre, Hamilton, New Zealand 3204. Tel: ⫹ 64 7 857 1278. E-mail: [email protected] (Received 9 November 2014 ; accepted 23 February 2015) ISSN 0954–0261 print/ISSN 1369–1627 online © 2015 Institute of Psychiatry DOI: 10.3109/09540261.2015.1023783

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Outcome measurement in New Zealand though New Zealand had a strong interest in outcomes for a number of years prior to the introduction of the MHINC (Mellsop & Wilson, 2006). A publication by Hazelton and Farrell (1998) shows how professional groups, in this case nurses, were starting to take outcome measurement more seriously. In 2002 the New Zealand Classification and Outcome Study (CAOS) (Gaines et al., 2003) involved research in a number of DHBs which subsequently attempted to develop a national casemix classification system and implement the routine use of selected measures in those services. Much of this work was influenced by Australian research (Andrews et al., 1994; Buckingham et al., 1998) in which there was particular emphasis on the use of the Health of the Nation Outcome Scales (HoNOS) (Wing et al., 1998). In 2004 a critical decision was made to develop a national strategy for information (Ministry of Health, 2005). This strategy identified the need to move beyond collecting data to actually using the information collected. In 2005 The Mental Health Standard Measures of Recovery Initiative or MHSMART was launched by the Ministry of Health. MHSMART outlined a long-term vision and commitment to an outcomes culture in New Zealand through the development of five outcome areas where ROMs could be introduced (see the New Zealand outcome measures and rationale for choice section below for details). These would be implemented as mandated measures in a phased manner. In 2006, following the introduction of MHSMART, Te Pou took up the role of national implementation of outcome measurement in New Zealand. Te Pou is a national mental health workforce centre which provides training, resources and support to the workforce serving adults in the mental health, disability and addiction sector. A notable feature of the New Zealand approach to ROMs is the strategic connection with a broader workforce development methodology. New Zealand has a number of national mental health workforce and development centres; they are: • Te Pou – adult mental health, disability and addiction services (includes Matua Raḵi, focusing on addictions), • Te Rau Matatini – Māori mental health and disability, • The Werry Centre – infant, child and adolescent mental health, • Le Va – mental health, disability and addiction services for Pacific peoples. The goal of these centres is to ensure the workforce has the capacity and capability to deliver on the


mental health and addiction strategy. Te Pou’s role in workforce development is specifically related to outcome measurement working with the other workforce centres. In 2008 the Programme for the Integration of Mental Health Data (PRIMHD) (Ministry of Health, 2013) was introduced as a national mental health and addiction database. This brought together both the activity data collected under MHINC and the outcomes data collected by MHSMART (Ministry of Health, 2014b). Data collected by clinicians from each DHB is submitted to the PRIMHD database. The data set consists of activity data, unique identifier, legal status, diagnosis, referral and outcome data. NGOs in New Zealand are also required to collect activity data; however, they are not currently required to collect any outcome data which feeds into the national collection. Since 2010 New Zealand has been making routine outcome reports available to all 20 DHBs. These reports have been used for a variety of purposes, including benchmarking and workforce development. The PRIMHD national database allows the Ministry of Health to report against data at national, service and team levels. It has proven to be a highly successful system and along with ROMs has laid the foundations for the development of the Key Performance Indicator (KPI) project. The KPI benchmarking system was implemented in 2009. The system was based on work carried out by Counties Manukau District Health Board (Counties Manukau DHB, 2007). The initial KPI focus was on adult inpatient units; however, this has now grown to KPIs for children and adolescents as well as forensic services. There is currently also greater emphasis on community services. The first KPI in the adult inpatient stream is the change in HoNOS scores from inpatient admission to discharge. This KPI work is coordinated by the Northern Regional Alliance (formerly Northern DHB Support Agency) which is a provider-led initiative. DHBs and NGOs report on nationally comparable indicators of service performance in order to bring about quality and performance improvement across the sector (Northern DHB Support Agency, 2014). The combination of a national database for mental health and addictions, KPI benchmarking, and outcome measurement has provided a simple but effective data set to increase the use of information within New Zealand’s mental health and addiction services. This information is used by the workforce centres for development aimed at specific sector groups and forms the basis of New Zealand’s outcome framework.


M. Smith & S. Baxendine

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Demographics and diagnostic profile of New Zealand The most recent New Zealand census (Statistics New Zealand, 2013) identified the four main ethnic groups of New Zealand’s population as European 74%, Māori 15%, Asian 12% and Pacific peoples 7%. (People were able to identify with more than one ethnic group and therefore percentages do not add to 100.) Other ethnic groups with smaller percentages make up the remainder. While present-day New Zealand is often considered a multicultural, multi-ethnic society, there is a strong emphasis on working towards biculturalism; that of Pākehā (New Zealanders of European descent) and Māori. Māori are the indigenous people of New Zealand who in 1840 signed a treaty (the Treaty of Waitangi) with the British crown laying the foundation for a contemporary New Zealand society (Ministry for Culture and Heritage, 2015). The previously mentioned Mental Health Research and Development Strategy (Ministry of Health, 1997) indicated epidemiology was an important focus of attention. The New Zealand Mental Health Survey, Te Rau Hinengaro (Oakley Brown et al., 2006) addressed the issue of epidemiology. The survey implied prevalence of mental disorder in New Zealand is common: 46.6% of the population are predicted to meet criteria for a disorder at some time in their lives, with 39.5% having already done so and 20.7% having a disorder in the past 12 months. (Te Rau Hinengaro) (Oakley Browne et al., 2006 p. xix) The report notes it is more common in some groups of the New Zealand population than others, acknowledging that while all groups had access to mental healthcare, Māori and Pacific people had poorer access. Te Rou Hinengaro also identified co-morbidity of mental disorder, physical disorder and drug and alcohol abuse as growing problems.

New Zealand mental health and addiction services Most mental health and addiction services in New Zealand are delivered through 20 DHBs. Additionally, NGOs and primary health organizations (PHOs) play a significant role in the delivery of mental health and addiction services in New Zealand. Approximately 30% of New Zealand’s mental health and addiction spend goes towards NGO-based services (Mellsop & Smith, 2010). In 2012 New Zealand introduced a new mental health and addiction strategic plan called Rising to the Challenge (Ministry of Health, 2012). This

plan builds on work in Blueprint II (Mental Health Commission, 2012) to show how services need to work across the spectrum of care and how important measurement and indicators are to that work. The PRIMHD database, KPIs and ROMs are seen as central to measurement and performance accountability in the mental health and addiction sector.

New Zealand outcome measures and rationale for choice Five areas were identified in the original MHSMART for inclusion in ROMs in New Zealand’s mental health and addiction services. These were symptomatology, a Māori specific measure, an alcohol and drug use measure, a service-user rated measure, and a measure of functioning. Symptomatology The HoNOS was chosen as the preferred measure and later added its various forms, such as for children (HoNOSCA), older adults (HoNOS65⫹), forensic (HoNOS-secure) and intellectually disabled (HoNOS-LD). The symptomatology domain was chosen as the first domain to have mandated ROMs in the MHSMART. The HoNOS family of measures were chosen because they are easy to use, acceptable to clinicians, psychometrically valid (Pirkis et al., 2005; Te Pou o Te Whakaaro Nui, 2012b) and already widely used. The Ministry of Health has set a goal of 80% collection rate, in both inpatient and community settings for these instruments, by 1 July 2015. Māori specific As no alternative was available internationally, New Zealand decided to develop its own culturally specific instrument. Hua Oranga (Kingi & Durie, 2000) is informed by the Te Whare Tapa Wha Māori concept of mental health (Durie, 1994). This model captures the physical, mental, social and spiritual aspects of a person’s life in a holistic manner. The Hua Oranga measure has been developed and piloted in a number of Māori mental health services nationwide (McClintock et al., 2013). Alcohol and drug After an extensive literature review New Zealand decided to develop its own measure of alcohol and drug measurement. This was because none of the available options was thought to meet New Zealand’s specific needs. Initially this measure was known

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Outcome measurement in New Zealand as the Alcohol and Drug Outcome Project Tool (ADOPT) and preliminary validation work was completed by Deering et al. (2009). The measure has subsequently come to be known as the Alcohol and Drug Outcome Measure (ADOM). The measure has three sections, frequency of use, psychosocial impact, and recovery. The ADOM is a self-rated measure for service-users in community addiction services and is meant to be completed collaboratively between service-users and clinician. The measure has been found to be psychometrically sound with good validity, reliability, sensitivity to change, and is generally seen as acceptable and feasible for use. ADOM has been mandated for national collection from 1 July 2015.

Service-user rated Significant work, including a literature review, ascertained that New Zealand should also develop its own service-user rated measure (Gordon et al., 2004). None of the existing measures were considered to offer a suitable cultural match to the New Zealand context. Twelve domains were identified for the measure and then worked into a comprehensive instrument known as Tāku Reo, Tāku Mauri Ora, my voice, my life (Gordon et al., 2009). Unfortunately the psychometric properties of the measure have not been established satisfactorily for aggregated use; however, the instrument remains available for individual use. There is currently no decision on a mandated service-user rated outcome measurement instrument for use in mental health services in New Zealand. Given that service-user rated ROMs comprise one of the five identified areas in MHSMART it is inevitable that a service-user rated measure will be introduced to the mental health sector. However, existing mandated measures should be fully embedded first to ensure we do not overburden the sector.

Functioning Mellsop and Smith (2010) point out that deciding on a measure for functioning in New Zealand has proved both difficult and contentious. Work involved trying to identify which measure or measures could be used to assess for functionality in the mental health sector. The ‘Personal and social performance’ (PSP) measure and the ‘Global assessment of functioning’ (GAF) measure were both favoured over the ‘Life skills profile’ (LSP) in the New Zealand context. However, no decision has been taken on the routine introduction of a mandated functionality measure, though some NGOs voluntarily use the Camberwell Assessment of Needs (CAN).


Sector response to the ROMs chosen Resistance by some clinicians to the use of the symptomatology measures has decreased over time. Interestingly, as Trauer (2010) shows, the attitude of most service-users and family members to the collection of outcome measures has been positive and affirming. New Zealand continues to adopt a phased approach to the introduction of mandated ROMs and it is anticipated that a service-user rated measure will also be mandated in time. New Zealand implemented a national real-time reporting system for service-users of mental health services to provide feedback on their experiences of care. This system is still in its infancy but shows real promise as a vehicle for obtaining feedback from service-users (Malatest International, 2014). In the absence of a mandated service-user rated measure this information provides a useful addition to the data collected. With the exception of ADOM there are no mandated outcome measures for either NGOs or the primary mental health sector in New Zealand. The only policy directive in this area around outcome measurement is that services use valid and reliable measures and anecdotal evidence suggests there is more routine collection of outcome measures in NGOs than first thought. The World Health Organization Quality of Life (WHO-QOL) measure, the Client Directed and Outcome Informed (CDOI) measures and the CAN are all widely used in New Zealand by NGOs, and there is a growing interest in recovery measures for both services and individuals. Whether any of these measures will be mandated in the NGO context is yet to be determined. New Zealand took a unique approach to ROMs, emphasizing the concept of standardised assessments (Mellsop & Smith, 2010), rather than simply referring to outcome measures. A commitment to the routine use of standardised and quantified assessment allows status comparisons over time and can be used as a measure of progress or outcome. The concept of assessment is one familiar to clinicians and considered more clinically acceptable than that of outcome measurement, even though they do refer to the same concept. Standardised assessments have helped clinicians to see the clinical utility of the HoNOS family of measures, through emphasis in HoNOS training on the service-user, clinician conversation and the use of ROMs in the multidisciplinary team. More work in the area of clinical utility is required. Especially in relation to the way clinicians actually use ROMs in their practice.

Information collection protocol New Zealand developed an information collection protocol (ICP) in 2006 (Te Pou o Te Whakaaro Nui,


M. Smith & S. Baxendine

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

2006b); this was last updated in 2012 (Te Pou o Te Whakaaro Nui, 2012a). The ICP indicates when outcome measures will be collected. Fig. 1 provides an overview of New Zealand’s ICP. All five of the HoNOS family measures use the same ICP; these are based around four collection occasions, namely admission, review, discharge, and assessment only. The ADOM will follow a similar ICP (Matua Raḵi, 2014) to Fig. 1. The main differences are that following an admission collection, a review collection after 6 weeks has been added to the protocol. This is then followed by a review after 12 weeks and

then reviews every subsequent 12 weeks until a discharge collection.

Data collection rates and aggregated reporting uses The New Zealand PRIMHD collects information from the HoNOS family of measures. Table 1 shows the percentage of national collections for inpatient and community mental health settings for a 3-month period, January to March, in each of six successive

Referral to MHS

Has an assessment occurred?


No outcome collection required


Following assessment, will the person be admitted to DHB Mental Health Services?


An ‘Assessment Only’ collection is required within one week of assessment

RFC: 01 Assessment Only Yes

Complete a ‘New episode’ collection RFC: 02 New referral if person new to MHS

Person being transferred to care of a MHS team in a different setting (i.e. inpatient community)

RFC: 03 From other treatment setting if person being transferred from care of a MHS team in a different setting (i.e. community inpatient)

Complete a ‘Review’ collection at least every 3 months until end of episode RFC: 05 3 Month review *Note RFC: 06 Review other can be completed at any stage to capture clinical significance

Episode of mental healthcare

Complete an ‘End of episode’ collection when care in current setting ends RFC: 07 No further care if person is discharged out of DHB MHS RFC: 08 To other treatment setting if person is being transferred to care of a MHS team in a different setting (i.e. community inpatient) *Note Other ‘End of episode’ reasons may be used – Lost to care, deceased, brief episode of care, episode end or other

Fig. 1. Mental Health Outcomes Information Collection Protocol flow chart. Adapted from Te Pou o Te Whakaaro Nui (2012a) p. 20.

Outcome measurement in New Zealand Table 1. Percentage of service-users seen with at least one outcome collection by setting: New Zealand. Period January–March January–March January–March January–March January–March January–March



60.3 64.6 79.6 82.3 85.0 85.6

35.3 41.6 45.7 52.9 55.8 60.0

2009 2010 2011 2012 2013 2014

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Unique clients seen ⫽ the number of unique clients who had ⱖ 1 face-to-face activity in the reporting period. Clients with ⱖ 1 outcome collection occasion ⫽ count of unique clients seen who had ⱖ 1 collection occasion record. Source: Ministry of Health, PRIMHD, 16 October 2014 (using codes from Ministry of Health 2014a).

years. This shows a general trend in collections over the time period. It demonstrates the trend is towards increased collections. Table 2 provides the percentage for various collection occasions in inpatient and community settings. Admission and discharge collections constitute the majority of collections for the three age-related HoNOS measures in inpatient settings. For HoNOS-LD and HoNOS-secure, in an inpatient setting it is review collections which constitute the majority. This presumably indicates longer-term care in these clinical areas. In community settings assessment-only reasons for collection are present and HoNOS-secure has the highest number of these, indicating initial screening of service-users in prisons and elsewhere. However, the most notable difference is the larger percentage of review collections in the community setting, generally indicating longer-term community care. By routinely collecting outcomes data New Zealand is in a position to report regularly on outcomes information nationally, for all 20 DHBs and some NGOs and at a team level for each of the mandated measures. Every 3 months Te Pou, the national agency working with this data, provides DHBs the following


outcome reports for each outcome measure. Tables and graphs that are provided for each of the outcome measures are: 1. Collection completion and validation • Percentage of service-users with at least one collection during the period. • Percentage of admission and discharge collections completed. • Percentage of invalid collections. 2. Outcomes – changes in service-user status information • Average total score. • Average number of clinically significant items (see below for definitions of clinical significance). • Average number of clinically significant items by ethnic group. • Percentage of collections in clinical range of each item. • Index of severity (see below for definition of index of severity). 3. Other measures of service activity • Index of severity by team. • Collections with ‘No’ items in the clinical range. • Focus of care categories (focus of care refers to a retrospective rating of the focus of clinical care during the last episode, which are, acute, functional gain, intensive extended, maintenance or assessment only). • Total score at review by focus of care. Clinically significant items are those items with a score of 2 or more for all the HoNOS family of measure tools, except HoNOS-secure security items. For HoNOS-secure security items clinical significance is constituted by a score of 1 or more. Clinically

Table 2. Valid outcome collections by reason for collection, outcome tool and setting: New Zealand, April 2013 to March 2014. Reason for collection Setting Inpatient



Assessment only (%)

Admission (%)

Review (%)

Discharge (%)

9 8 10 10 17

47 52 47 12 21 17 28 22 17 17

11 12 10 82 61 53 37 42 45 30

42 36 43 6 18 21 27 26 28 36

Total number 20,998 1,610 2,247 115 715 105,325 38,374 16,417 946 2,082

Valid collections have ⱕ 2 missing items for HoNOS, HoNOS65⫹, HoNOSCA and HoNOS-LD. HoNOS-secure collections have ⱕ 1 missing item for both the 12 items and secure items. Source: Ministry of Health, PRIMHD, 8 October 2014 (based on codes from change to Ministry of Health 2014a).

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.


M. Smith & S. Baxendine

significant items have been found to be more relevant to the sector than total score. Any items identified as clinically significant should have a plan associated with that item. This is the way the assessment of care using a standardised assessment is connected with planning. An index of severity in New Zealand was further developed from Parabiaghi (2005). HoNOS groups data into severe (at least two items ⱖ 3), moderate (one item ⱖ 3), mild (at least one item ⬎ 1 and all items ⬍ 3) and sub-clinical (all items ⬍ 2) for the first 10 items. This measure is liked by the sector because it is easily understood and provides a quick gauge on whether services are fulfilling their core role of working with people with high clinical needs.

Barriers and facilitators to collecting ROMs in New Zealand The main barriers and issues impeding implementation of outcome measures are the converse of the success strategies identified. They are summarized as strong leadership, long-term cultural change, workforce training and development, and information systems and access. Strong leadership New Zealand has found that strong leadership is vital to gain acceptance of the need to collect and use outcome measures. Strong leaders promote a clear vision for an outcomes-focused culture that is communicated regularly to confirm direction and to build the confidence and support of all stakeholders (Fonagy et al., 2004). Outcomes champions at all levels drive the use of outcomes. In New Zealand considerable work has gone into developing these champions. When MHSMART was implemented in 2005 every DHB in New Zealand appointed a site coordinator to help drive the collection and use of clinical outcomes (Fonagy et al., 2004; Pirkis et al., 2005). However, it has proven hard to maintain outcome champions, particularly the commitment by every DHB to have dedicated full-time positions. New Zealand has adopted a nationally driven and coordinated approach to implementing outcome measures. This has meant there has been a need for a national coordination lead body. Currently this role is supplied by Te Pou, though other workforce centres have a role within their areas of focus. Such a centralised outcomes support structure maintains balance between national consistency and local requirements. The lead body also identifies and promotes good ideas and ‘shining examples’ nationally (Fonagy et al., 2004; Pirkis et al., 2005).

Long-term cultural change It is recognized that building the collection and use of outcomes information in New Zealand involves a process of change. There is a general recognition that this will take time to accomplish by changing attitudes and behaviours and building a commitment to developing an outcomes-focused culture (Fonagy et al., 2004; Jacobs, 2009). As outcomes measurement becomes part of a continuous quality improvement framework, outcome measures should be embedded into the ‘business as usual’ process of services; this in turn will minimize the demands of data collection (Fonagy et al., 2004; Jacobs, 2009). The pacing and phasing of implementation is critical to keeping all stakeholders engaged; overloading the sector with compliance requirements will be counterproductive. The long-term cultural change towards measurement and accountability of performance is developmental and needs to follow a clear strategic pathway (Ministry of Health, 2012).While it must be acknowledged considerable cultural change has already occurred in New Zealand, much work still remains to be done. In regard to the NGO sector, Tobias (2010) indicates there have been, and are, many obstacles to the implementation of outcome measures in NGOs. Many NGOs do see the benefit of outcome measurement and share a motivation for recovery-oriented, service-user focused approaches. Te Pou’s 2006 survey of NGOs using routine outcome measures found only 9% of NGO services used a routine outcome measure (Te Pou o Te Whakaaro Nui, 2006a). Workforce training and development New Zealand has emphasized the importance of workforce development in the widest sense, rather than focusing on outcome measurement in isolation. Outcome training and refresher training as part of workforce development requires an ongoing commitment, appropriate resourcing, and consolidation of training development at both the national and service level. It also requires the dissemination of training resources and training updates (Fonagy et al., 2004; Pirkis et al., 2005). New Zealand developed a ‘train-the-trainer’ model for outcome trainers; this has had mixed success. We might have expected to see higher compliance rates (see Table 1) if the training model had been completely successful. What the compliance rates do tell us is that there is a general recognition of the need to collect these measures which is supported by the fact that the rates have increased over time. Engagement of all stakeholders is required to make workforce development successful. This means

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Outcome measurement in New Zealand services need to see beyond the need to merely collect these measures, to their clinical utility. It is important that stakeholders understand the benefits of collection and use for service-users their families, clinicians, and managers (Smith, 2011; Stewart, 2008; Te Pou o Te Whakaaro Nui, 2009). The issue of clinical utility is one that has been central to the implementation of outcome measurement in New Zealand. Information use workshops for services show how outcome information can be used by individual clinicians and by multidisciplinary teams, but we need to acknowledge this is still a work in progress. New Zealand has emphasized the development of recovery-focused mental health services as the underpinning philosophy of mental health and addiction services (Mental Health Commission, 1998). While definitions of recovery remain somewhat unclear, there is a general acceptance that we need outcome measures to assess whether recovery is actually occurring (Fonagy et al., 2004; Jacobs, 2009; Pirkis et al., 2005; Slade, 2009). IT structures, systems, and access to information Having sufficiently developed IT structures and systems is a key enabler to successful implementation of outcomes measurement in the New Zealand context. Moreover, this may well be one of the strongest factors in both improving collection rates and data quality (Fonagy et al., 2004; Pirkis et al., 2005). Feedback on outcome measures must have maximum clinical and management utility and occur quickly. Appropriate technology should allow ‘rapid turnaround’ such as the use of dashboards and apps. Feedback systems need to be user-friendly and provide easy access to the data for all stakeholders. Easy access keeps stakeholder engagement and benefits high. It is important that the success and utility of measures is demonstrated early and often, for example demonstrating how measures can be used at multiple levels (Te Pou o Te Whakaaro Nui, 2009). The fact that different DHBs in New Zealand have different information systems has not assisted in the development of a national implementation strategy for outcome measurement. A recent example of this is the HoNOS Tool™©. This app is available on both the iOS and Android platforms. Te Pou developed the tool to assist clinicians with the collection and use of outcomes information. A benefit of the app is that it provides aggregated case-load information not available elsewhere. It was hoped, irrespective of which system services used, clinicians would view the app as a smarter, more mobile way to work. However, uptake of the app has been slower than anticipated. Partly because the app is not integrated into DHB


IT systems, which means clinicians have to rate collections twice if they choose to use it. Further work is planned to see if and how integration can be achieved. While New Zealand provides aggregated feedback to services on their outcome measures, it is nowhere near to real time to be of significant value to clinicians. Data that informs reports in the datacollection rates section above has a lag of weeks or months between collection and the ability to report the data. Additionally, it is also difficult for clinicians without advanced data skills to access information in the national database themselves. It may be unrealistic to expect clinicians to routinely feed back outcome information to their service-users when those clinicians themselves do not have aggregated outcome information fed to them in a timely, understandable and easily accessible manner. Conclusion: next steps and lessons learnt New Zealand has been on a long journey with regard to outcome measurement in its mental health and addiction services. This journey started over 15 years ago and still has some way to run. New Zealand has adopted a nationally led approach to routine outcome measurement with a strong emphasis on workforce development and the use of standardised assessments, implemented in a phased manner. New Zealand needs to ‘hard wire’ the collection of existing outcome measures into services before adding other measures to prevent overburdening the sector with outcome measurement compliance. There is a need to embed the usefulness of data collected from outcome measures, in particular with clinicians, where most of the resistance to their introduction has occurred. Te Pou and the Ministry of Health acknowledge that outcome reports provided to the sector need to be more accessible and user friendly, and ultimately available in real time. There is also a need for ongoing consolidation of the measures New Zealand is currently collecting, with the possibility of adding other measures to the mix over the next few years. The introduction of PRIMHD and the ongoing development of the KPI benchmarking system have been positive additions for the sector, reinforcing outcome measurement at every level. The issues of leadership, workforce development, long-term cultural change, IT systems and access to information will continue to be significant factors in the New Zealand outcome measurement scene. However, perhaps most central of all is that New Zealand has learnt that introducing outcome measurement is not something which can be achieved in isolation from other information and workforce developments.


M. Smith & S. Baxendine


Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Assistance in completing this article was gratefully received from Barry Welsh, Principal Advisor, Ministry of Health, New Zealand; Richard Woodcock, Manager, National Service and Information Development Team; Cara Thomas, Clinical Lead; Sima Clarke, Administrator; Keri Opai, Māori Strategic Advisor, all from Te Pou, New Zealand. Declaration of interest: The authors are both employed in the service and information development team at Te Pou o Te Whakaaro Nui. Te Pou is a leading mental health workforce centre in New Zealand. Sandra Baxendine is an Information analyst. Apart from being employed by Te Pou the authors report no other conflicts of interest. The authors alone are responsible for the content and writing of the paper.

References Andrews, G., Peters, L., & Teesson, M. (1994). The Measurement of Consumer Outcome in Mental Health: A Report to the National Mental Health Information Strategy Committee. Sydney, Australia: Sydney Clinical Research Unit for Anxiety Disorders. Buckingham, W., Burgess, P., Solomon, S., Pirkis, J., & Eager, K. (1998). Developing a Casemix Classification for Mental Health Services. Canberra, Australia: Department of Health and Aging. Counties Manukau DHB. (2007). Report on the Key Performance Indicator Framework for New Zealand Mental Health and Addiction Services. Auckland, NZ: Counties Manukau District Health Board. Deering, D., Robinson, G., Wheeler, A., Pulford, J., Frampton, C., Dunbar, L., & Black. S. (2009). PreliminaryWork TowardsValidating a Draft Outcome Measure for Use in the Alcohol and Drug Sector. Auckland, NZ: Te Pou o Whakaaro Nui. Durie, M. (1994). Whaiora: M ori Health Development. Auckland, NZ: Oxford University Press. Fonagy, P., Mathews, R., & Pilling, S. (2004). The Mental Health Outcomes Measurement Initiative: Report from the Chair of the Outcomes Reference Group. London, UK: National Collaborating Centre for Mental Health. Gaines, P., Bower, A., Buckingham, W., Eager, K., Burgess, P., & Green, J. (2003). New Zealand Mental Health Classification and Outcome Study: Final Report. Auckland, NZ: Health Research Council of New Zealand. Gordon, S., Ellis, P., Haggerty, C., Pere, L., Platz, G., & McLaren, K. (2004). Preliminary Work Towards the Development of a SelfAssessed Measure of Consumer Outcome. Auckland, NZ: Health Research Council of New Zealand. Gordon, S., Ellis, P., Haggerty, C., O’Connor, S., Pere, L., Foster, C., … Walkey, F. (2009). Taku Reo Taku Mauri Ora: My Voice My Life. Auckland, NZ: Te Pou o Te Whakaaro Nui. Hazelton, M., & Farrell, G.A. (1998). Evaluating the Outcomes of Mental Health Care: An Introduction. Sydney, Australia: Australian and New Zealand College of Mental Health Nurses. Jacobs, R. (2009). Investigating Patient Outcome Measures in Mental Health. (CHE Research Paper). University of York, Centre for Health Economics. Kingi, T.K., & Durie, M. (2000). Hua Oranga: A M ori Measure of Mental Health Outcome. Palmerston North, NZ: Te Pumanawa Hauora School of M ori Studies, Massey University.

Malatest International. (2014). Independent Evaluation of the Health and Disability Commission’s Electronic Real Time Feedback System. Wellington, NZ: Malatest International. Matua Raki (2014). Alcohol and Drug Outcome Measure (ADOM) Guide for Addiction Clinicians. Version 3. Wellington, NZ: Te Pou o Whakaaro Nui Matua Raki. (2014). ADOM Systems Information. Version 4. Auckland, NZ: Te Pou o Whakaaro Nui. McClintock, K., Sokratov, A., Mellsop, G., Kingi, T.K. (2013). Hua Oranga: Service utility pilot of a mental health outcome measurement for an indigenous population. International Indigenous Policy Journal, 4(3), 1–14. Mellsop, G., & Smith, M. (2010). Outcome measures in New Zealand. In T. Trauer (Ed.), Outcome Measurement in Mental Health Theory and Practice (pp. 26–31). New York: Cambridge University Press. Mellsop, G., & Wilson, J. (2006). Outcome measures in mental health services. Humpty Dumpty is alive and well. Australasian Psychiatry, 14, 137–140. Mental Health Commission. (1998). Blueprint for Mental Health Services for New Zealand: How Things Need to be Done. Wellington, NZ: Mental Health Commission. Mental Health Commission. (2012). Blueprint II Improving Mental Health and Wellbeing for all New Zealanders. How Things Need to be. Wellington, NZ: Mental Health Commission. Ministry for Culture and Heritage. (2015). New Zealand History, Read the Treaty. Retrieved from politics/treaty/read-the-treaty/english-text Ministry of Health. (1991). Moving Forward: The National Mental Health Plan for More and Better Services. Wellington, NZ: Ministry of Health. Ministry of Health. (1994). Looking Forward: Strategic Directions of the Mental Health Services. Wellington, NZ: Ministry of Health. Ministry of Health. (1997). Mental Health Research and Development Strategy. Wellington, NZ: Ministry of Health. Ministry of Health. (2005). National Mental Health Information Strategy. Wellington, NZ: Ministry of Health. Ministry of Health. (2012). Rising to the Challenge. The Mental Health and Addiction Development Plan 2012–17. Wellington, NZ: Ministry of Health. Ministry of Health. (2014a). PRIMHD Data Process Standard – HISO 10023.3. Version 3.1 Wellington, NZ: Ministry of Health. Ministry of Health. (2014b). PRIMHD Mental Health Data. Retrieved from national-collections-and-surveys/collections/primhd-mentalhealth-data Northern DHB Support Agency. (2014). Key Performance Indicators for the New Zealand Mental Health and Addiction Sector. Retrieved from MentalHealth/KPIFramework.aspx Oakley Browne, M., Wells, J.E., Scott, K.M. (Eds). (2006). Te Rau Hinengaro: The New Zealand Mental Health Survey. Wellington, NZ: Ministry of Health. Parabiaghi, A., Barbato, A., D’Avanzo, B., Erlisher, A., Lora, A. (2005). Assessing reliable and clinically significant change on Health of the Nation Outcome Scales: Method for displaying longitudinal data. Australian and New Zealand Journal of Psychiatry, 39, 719–725. Peters, J. (1994). Performance and Outcome Indicators in Mental Health Service: A Review of Literature. Wellington, NZ: Ministry of Health. Pirkis, J., Burgess, P., Coombs, T., Clarke, A., Jones-Ellis, D., & Dickson, R. (2005). Routine measurement of outcomes in Australia’s public sector mental health services. Australia and New Zealand Health Policy, 2, 8. Ryall, T. (2014). Health budget increases to a record $15.6b. Press release retrieved from health-budget-increases-record-156b

Int Rev Psychiatry Downloaded from by RMIT University on 08/19/15 For personal use only.

Outcome measurement in New Zealand Slade, M. (2009). Personal Recovery and Mental Illness: A Guide for Mental Health Professionals. New York, USA: Cambridge University Press. Smith, M. (2011). Case study: Developing clinical utility for mental health outcome information. A New Zealand perspective. In A. Cashin & R. Cook (Eds), Evidence-Based Practice in Nursing Informatics (pp 163–165). Hershey, PA: Medical information Science Reference (an imprint of IGI Global). Statistics New Zealand. (2013). New Zealand Census 2013. Wellington, NZ: Statistics New Zealand. Stewart, M. (2008). Making the HoNOS(CA) clinically useful: A strategy for making HoNOS, HoNOSCA and HoNOS65 ⫹ useful to the clinical team. Australian and New Zealand Journal of Psychiatry, 42, A5. Te Pou o Te Whakaaro Nui. (2006a). NgOIT 2005 Landscape Survey. Auckland, NZ: Te Pou o Te Whakaaro Nui. Te Pou o Te Whakaaro Nui. (2006b). NZ Mental Health Standard Measures of Assessment and Recovery (MH-SMART) Initiative – Information Collection Protocol. Auckland, NZ: Te Pou o Te Whakaaro Nui.


Te Pou o Te Whakaaro Nui. (2009). From Data to Information: Data Use Guidelines for Standard Measures Collected in the New Zealand Mental Health System. Auckland, NZ: Te Pou o Te Whakaaro Nui. Te Pou o Te Whakaaro Nui. (2012a). Information Collection Protocol. Auckland, NZ: Te Pou o Te Whakaaro Nui. Te Pou o Te Whakaaro Nui. (2012b). Technical Review of the Psychometric Properties of the HoNOS Family of Measures. Auckland, NZ: Te Pou o Te Whakaaro Nui. Tobias, G. (2010). Mental health outcome measurement in non-governmental organisations. In T. Trauer (Ed.), Outcome Measurement in Mental Health Theory and Practice (pp. 164–172). New York: Cambridge University Press. Trauer, T. (2010). Stakeholder perspectives in outcome measurement in outcome measurement. In T. Trauer (Ed.) Mental Health: Theory and Practice (pp. 196–205). New York: Cambridge University Press. Wing, J.K., Beevor, A.S., Curtis, R.H., Park, S.B., Hadden, S., & Burns, A. (1998). Health of the Nation Outcome Scales research and development. British Journal of Psychiatry, 172, 11–18.

Outcome measurement in New Zealand.

This paper provides a detailed description and critique of the development of routine outcome measurement (ROM) within New Zealand's mental health and...
314KB Sizes 0 Downloads 8 Views