Evaluation and Program Planning 45 (2014) 29–41

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

An evaluation of an Australian initiative designed to improve interdisciplinary collaboration in primary mental health care Justine Fletcher a,*, Kylie King a, Jo Christo a, Anna Machlin a, Bridget Bassilios a, Grant Blashki b, Chris Gibbs c, Angela Nicholas c, Jane Pirkis a a b c

Centre for Health Policy, Programs and Economics, Melbourne School of Population Health, University of Melbourne, Victoria 3010, Australia Nossal Institute for Global Health, University of Melbourne, Victoria 3010, Australia Mental Health Professionals Network, PO Box 203, Flinders Lane, Victoria 8009, Australia

A R T I C L E I N F O

A B S T R A C T

Article history: Received 20 December 2011 Received in revised form 23 February 2014 Accepted 1 March 2014 Available online 11 March 2014

This paper reports on a multi-component evaluation of Australia’s Mental Health Professionals Network (MHPN). MHPN aims to improve consumer outcomes by fostering a collaborative clinical approach to primary mental health care. MHPN has promoted interdisciplinary communication and networking through activity in three inter-related areas: interdisciplinary workshops supported by education and training materials; fostering ongoing, self-sustained interdisciplinary clinical networks; and a website, web portal (MHPN Online) and a toll-free telephone information line. The evaluation showed that MHPN’s workshops were highly successful; almost 1200 workshops were attended by 11,930 individuals from a range of mental health professions. Participants from 81% of these workshops have gone on to join ongoing, interdisciplinary networks of local providers, and MHPN is now supporting these networks in a range of innovative ways to encourage them to become self-sustaining and to improve collaborative care practices. ß 2014 Elsevier Ltd. All rights reserved.

Keywords: Interdisciplinary network Mental health Collaborative care Primary care

1. Introduction Twenty percent of the Australian population will experience mental ill health each year (ABS, 2007), and mental disorders account for 24.9% of the disability burden in Australia (Begg et al., 2007). This high prevalence of mental disorders is echoed in other high income countries: the Centre for Disease Control and Prevention estimates that 25% of US adults have a mental illness each year (CDC, 2011), and the Mental Health Foundation estimates a similar prevalence rate for the UK (Mental Health Foundation, 2007). Research has established the benefit of including psychological interventions in primary care treatment of common mental disorders (Gilbody, Bower, Fletcher, Richards, & Sutton, 2006), and primary mental health care is an essential component of mental health care provision in Australia. General practitioners, in

* Corresponding author. Tel.: +61 38344 0663; fax: +61 39348 1174. E-mail addresses: justine.fl[email protected] (J. Fletcher), [email protected] (K. King), [email protected] (J. Christo), [email protected] (A. Machlin), [email protected] (B. Bassilios), [email protected] (G. Blashki), [email protected] (C. Gibbs), [email protected] (A. Nicholas), [email protected] (J. Pirkis). http://dx.doi.org/10.1016/j.evalprogplan.2014.03.002 0149-7189/ß 2014 Elsevier Ltd. All rights reserved.

particular, frequently provide an entry point into mental health care for high prevalence mental health disorders, creating the potential for early detection of mental disorders and appropriate referral pathways for ongoing care. Relevant to the provision of mental health services within the primary care sector is the growing evidence that collaborative mental health care further enhances treatment and is best practice (D’Amour, Ferrada-Videla, San Martin Rodriguez, & Beaulieu, 2005; Rosenberg & Hickie, 2009). Collaborative mental health care involves not only the input of a number of different mental health care professionals in the care of a consumer, but also involves these professionals communicating and working together in collective action oriented towards a common goal (D’Amour et al., 2005). A recent review of 119 papers found strong evidence for the link between collaborative activities, positive clinical service delivery and economic outcomes (Fuller et al., 2011). Similarly, other researchers have found positive consumer impacts (van Orden, Hoffman, Haffmans, Spinhoven, & Hoencamp, 2009; Zwarenstein, Reeves, & Perrier, 2005). Since the early 1990s, Australian National Mental Health Plans have emphasised the importance of joint planning, coordination of services and the development of links between providers across and within sectors in the delivery of mental health services. The

30

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

2009 Australian National Health and Hospitals Reform Commission emphasised that collaboration between support services was essential for recovery and individual self-determination, and many recent initiatives have consequently focused on linking primary care, specialist care and community-based services (Commonwealth of Australia, 2009). Despite this, the primary mental health care system has historically created a landscape wherein many professionals were operating in private practice settings in different locations and often working in professional ‘silos’ within their scope of practice (Davies Powell et al., 2006; McDonald, Powell Davies, & Fort Harris, 2009). Collaborative relationships do not happen instantly nor without considerable effort from all parties (Cook, 2005; Craven & Bland, 2006). Often co-ordinated leadership and external support are needed (Barker, Bosco, & Oandasan, 2005). In order to overcome barriers to collaboration within primary and community mental health systems, coordination of interactions between organisations and professionals needs to be reviewed (McDonald et al., 2009). In late 2006, Australia’s public universal health insurance system (Medicare) introduced a primary mental health scheme, The Better Access to Psychiatrists, Psychologists and General Practitioners, known as ‘Better Access’. The Better Access initiative is one of 18 Australian Government initiatives introduced under the Council of Australian Governments (COAG) National Action Plan on Mental Health 2006–2011 (Australian Government, 2006). Better Access was introduced in response to low treatment rates for mental disorders, and aims to improve outcomes for people with such disorders by encouraging a team-based approach to their mental health care. The general practitioner (GP) is the first contact for people accessing primary mental health care; the GP may then treat the consumer him/herself or may refer the patient to other mental health care providers (Henderson et al., 2000). Better Access allows consumers referred to certain mental health professionals (eligible psychologists, social workers, mental health nurses and occupational therapists operating on a fee-for-service basis in private rooms) to claim a rebate (a set amount paid back to the consumer by the Australian Government) for their psychological care (Australian Government, 2006). Providers who work in the primary mental health care sector generally operate in private practices on a fee-for-service basis. Prior to the introduction of Better Access the fee was paid by the consumer and in some circumstances the consumer’s private health insurance. Since the introduction of Better Access, this fee is paid by some combination of Medicare, the consumer, and, in some cases, the consumer’s private health insurance. The Australian Government’s Department of Health and Ageing (DoHA) has carriage of the Better Access programme and has recognised that the programme creates potential for improved interdisciplinary collaboration. At minimum, GPs and mental health professionals are required to collaborate through the provision of the GP’s written referral to the mental health professional, a mandatory written review and final report to the GP from the mental health professional, and a mandatory GP review of the patient after six sessions with the mental health professional. These minimal requirements for collaboration between the GP and treating clinician within the Better Access programme provide an ideal basis to foster more comprehensive collaborative interdisciplinary mental health care. Furthermore, in order for the GP to feel confident in referring their patients for quality mental health care, it is likely that they will want to meet and understand the expertise of those professionals to whom they are referring. Despite this unique opportunity to build collaborative care in the primary mental health care sector, there is little information available regarding systematic attempts to foster collaboration between health professionals, and we still have limited understanding of the complexity of relationships between professionals

(D’Amour & Oandasan, 2005). Some research describes attempts to improve interdisciplinary collaboration within or between organisations (e.g., Holleman, Bray, Davis, & Holleman, 2004; Kiesely, Duerden, Shaddick, & Jayabarathan, 2006; Michael, Howard, & Cox, 2008), but none that we are aware of describe any attempts to improve interdisciplinary collaboration in a primary care setting at a national level. The Mental Health Professionals Network (MHPN), a national initiative in Australia, undertook this monumental task. MHPN was funded by the Australian Government to bring together different primary care mental health professionals with the aim of fostering interdisciplinary networking, collaboration and ultimately improved consumer outcomes. MHPN was an innovative programme, as interdisciplinary teams are not common in Australian primary health care and collaborative care is made difficult by the boundaries between professionals and within health services (McDonald et al., 2009). This paper describes the evaluation of MHPN. The evaluation of MHPN, also funded by the Australian Government, sought to determine whether MHPN had been successful in fostering interdisciplinary networking and collaboration, and to provide formative feedback to MHPN (D’Amour & Oandasan, 2005). The overall purpose of MHPN is to support the development of sustainable, interdisciplinary collaboration in the primary mental health care sector. MHPN has been responsible for promoting interdisciplinary collaboration and networking between GPs, psychologists, social workers, occupational therapists, mental health nurses, paediatricians and psychiatrists. It has done this through activity in three inter-related areas: facilitating interdisciplinary workshops supported by education and training materials; fostering ongoing, self-sustained interdisciplinary clinical networks; and hosting a website, web portal (MHPN Online) and a toll-free telephone information line which supported the programme. The underlying premise for MHPN’s initial activities was that facilitated, interdisciplinary workshops would reinforce the importance of interdisciplinary collaboration and would enable relationships to develop between local providers. In turn, this would encourage participants to form ongoing networks comprising providers from a mix of disciplines. Support for both the workshops and the networks arising from them (e.g., via the website, MHPN Online and phone line) would therefore assist the networks to become self-sustaining. MHPN’s efforts have appropriately been conducted in interconnected phases, which means that some of the above areas have received more attention to date than others:  The initial establishment phase involved MHPN putting in place required personnel, governance mechanisms, infrastructure and resources across all three activity areas.  In the delivery phase, MHPN placed considerable emphasis on rolling out workshops nationally. This involved a major push to recruit local mental health professional facilitators. Mental health professionals were invited to attend a workshop in their local area, and those operating in private practice were paid for their first attendance. Workshops usually involved facilitated introductions, a meal, a discussion of a case study of a client with a mental disorder, and a discussion of the possibility of generating an ongoing local network. MHPN aimed to conduct 1200 workshops nationwide before the end of June 2010 (30% in rural areas as required by the contract between MHPN and the Australian Government). The aim for each workshop was to have 20 registrations in order to find a balance between those registering but not attending and people attending without registering, whilst maintaining the ‘small group’ feel. Composition-wise, the attendees were to include at least three different types of mental health professions, and at least four GPs. The

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

professions included were GPs, psychiatrists, paediatricians, psychologists, social workers, occupational therapists, mental health nurses and others where relevant.  Towards the end of the evaluation period, MHPN moved into its sustainability phase. The focus of this phase was on generating ongoing networks of interdisciplinary mental health professionals from the workshop attendees with the aim of achieving improved interdisciplinary collaboration. Network co-ordinators guided the direction of the network and attended to logistical tasks associated with organising meetings. MHPN-appointed Network Sustainability Project Officers provided administrative support and guidance to the network co-ordinators. MHPN also made $500AUD funding available annually to each network to assist with costs such as venue hire, catering and payment of guest speakers. MHPN aimed for 70% of its workshops to result in the formation of ongoing, interdisciplinary, clinical networks that would meet regularly to discuss aspects of collaborative mental health care.  Ultimately, MHPN is striving to reach a long-term phase. This phase will be characterised by improved collaborative care and better client outcomes in the primary mental health sector. The Centre for Health Policy, Programmes and Economics (CHPPE) at the University of Melbourne was contracted by MHPN following a competitive tender process to undertake an independent evaluation of its activities from July 2009 to June 2010. This paper describes the evaluation in terms of its methods, findings

31

and implications for the future directions of MHPN. The full evaluation report is available from the authors (Fletcher et al., 2010a). The current paper will discuss the achievement of some of these objectives in relation to workshop delivery, network formation, and sustainability and development of an online presence and community. 2. Method Consistent with contemporary evaluation theory and practice, an objectives-based evaluation was conducted (Owen, 1999). CHPPE worked closely with MHPN to articulate a programme logic that would underpin the evaluation. This involved clarifying a hierarchy of objectives that reflected activities and desired outcomes. In general terms, the lowest level objectives related to the establishment phase, the intermediate level objectives related to the delivery phase, and the higher level objectives related to the sustainability and long-term phases. Fig. 1 shows the hierarchy of objectives. Because MHPN was only moving into its sustainability phase at the time the evaluation ended, the evaluation was largely limited to an examination of the objectives in the establishment and delivery phases and to a small extent the sustainability phase. Ethical approval for each stage of the evaluation was obtained from the University of Melbourne’s research ethics committee. The evaluation participants, data sources, procedures and data analysis techniques are outlined below.

Fig. 1. Hierarchy of objectives and evaluation components.

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

32

2.1. Participants Three groups participated in the evaluation: MHPN facilitators, MHPN workshop participants and MHPN employees who played a role in network sustainability. Across all surveys the majority of participants were females, with psychologists making up the largest professional group. The distribution across states, and metropolitan and rural geographic areas was reflective of the distribution of mental health professionals in these areas (see Inline Supplementary Tables S1–S7 for further details). Inline Supplementary Tables S1–S7 can be found online at http://dx.doi.org/10.1016/j.evalprogplan.2014.03.002. 2.2. Data sources A range of primary and secondary data sources informed the evaluation. A single evaluation question was posed in relation to each objective: ‘Was the given objective achieved?’ Multiple evaluation components and data sources were used to answer this question for each objective (Ovretveit, 1998; Patton, 1990) and are illustrated in Tables 1 and 2. As shown in Table 1, seven surveys (the sustainability and web survey were administered at the one

time point) were conducted. As illustrated in Table 2, secondary data collected by MHPN concerning participation in workshops and networks, as well as use of technical resources were used to address the extent to which evaluation objectives were met. 2.3. Procedure From December 2009 to July 2010, participants for each survey were recruited via email on the basis of their stage of involvement in MHPN (e.g., invited to participate in the pre-workshop survey prior to attending their first workshop). All surveys were administered online so that participants could complete them at a time most convenient to them (note that the pre-workshop survey was also completed in paper and pencil format). As illustrated in Table 1, survey response rates varied between 18 and 63%. One 90-min semi-structured sustainability focus group was held in May 2010 with MHPN Network Sustainability Project Officers and Senior Project Officers (n = 9, response rate = 89%). This focus group sought information on MHPN employees’ experiences of working with mental health professionals in establishing and maintaining ongoing networks, and their views about what enabled or inhibited this process. The questions

Table 1 Description of primary survey data collected during evaluation. Data source

Number of questions

Tool description

Data collection notes

Number of participants

Response rate

Mental health professionals’ pre-workshop survey

23

Invited to participate subsequent to enrolment in first workshop

1696

38%

Mental health professionals’ post-workshop survey

22

Subsequent to participation in any workshop, recruited via

2369

33%

Mental health professionals’ 14-week follow-up survey

14

18%

28

331

63%

Facilitators’ in-depth survey

15

Surveys were completed between April 2010 and July 2010 at an interval of 14 weeks following workshop attendance The survey was emailed to workshop facilitators after each workshop. The survey data were collected between February 2010 and July 2010 Facilitators who responded to an invitation at any time during their involvement with MHPN were sent the survey

245 (as of July 2010)

Facilitators’ post-workshop survey

Survey collected demographic and professional/employment information, as well as information about expectations regarding workshop participation and current practices with respect to interdisciplinary collaboration and networking (e.g., contact with other mental health professionals and activities such as referrals, meetings and informal conversations) This survey rated how well the workshops’ objectives were met, the relevance and usefulness of the workshop to participants, professionals’ intention to participate in ongoing networks, the facilitation of the workshop, and the materials used in the workshop Questions mirrored those asked in the preworkshop survey with respect to interdisciplinary collaboration and networking, thus enabling changes in these activities to be identified in analysis This survey sought information from facilitators on how they felt the given workshop were received by participants, plans for ongoing network arrangements, suggestions for future workshops, and the quality of MHPN resources and support

190

28%

Sustainability and website survey

12 (sustainability); 6 (website)

All mental health professionals who attended workshops were sent an email invitation to participate in July 2010

1543

20%

MHPN Online survey

19

All 353 registered portal users were sent an email inviting them to participate in July 2010

73

21%

This survey was designed to collect information about facilitator’s perceptions of the effectiveness of MHPN in generating sustainable ongoing networks and improvements in collaborative mental health care The survey sought information about the MHPN website and participants’ level of desire for interdisciplinary networking, their preferred method for networking, and the support they would like from MHPN to achieve their networking goals This survey gathered information about the frequency of website access, as well as the reasons for access and perceptions of the content and structure of the website

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

33

Table 2 Secondary data provided by MHPN utilised in the evaluation. Data source

Data description

MHPN documentation

Relevant documentation (e.g., MHPN’s governance charter, latest annual report, organisational charts, meeting timetables, communication and marketing materials, participants’ manual, facilitators’ kit and network co-ordinators’ kit) was provided to, and reviewed by, the evaluation team The workshop calendar dataset included data about each workshop’s date and location, and information about facilitators’ professions The attendance list dataset also provided information on workshops by mental health professionals. This dataset included additional information on people who registered for a given workshop (e.g., their date of registration and whether the registration resulted in an attendance, a cancellation, or a no show) or ‘walked-in’ without registering. The attendance list dataset provided a slight undercount of the number of participants attending overall in comparison to registration because of delays or failures on the part of facilitators to provide the relevant data, despite follow-up prompts The workshop master list data is collated directly from the previous datasets and is updated by MHPN Project Officers from their communication with workshop facilitators. This dataset corrects the undercount in the MHPN attendance list, which presents what happened on the day. This dataset includes information about each workshop’s status (i.e., whether the group has agreed to meet again and if so, the progress it has made towards doing so and whether the facilitator has agreed to remain involved in the network) The network master list is gathered by MHPN project teams from the network co-ordinators. It includes data on the number of workshops represented in ongoing networks, the number of times networks have met, the profession of the co-ordinator and the ongoing involvement of the facilitator MHPN surveys network co-ordinators regarding the activities and composition of their network (e.g., the number of mental health professionals that were invited and that attended network meetings, and the types of professionals attending networks). It should be noted that MHPN data is incomplete, given that it is only available for networks that voluntarily provide it to MHPN. Networks that do not access the $500 payment offered to groups to facilitate ongoing networks have no responsibility to provide data. In addition, there are time lags in data receipt, as not all information is forwarded immediately following meetings A summary report from five network co-ordinator feedback forums conducted by MHPN in March 2010 was made available to the evaluation team. This report held information about the key themes that emerged from the forums in relation to the enablers and barriers in co-ordinating networks. Forums were held in Queensland, Victoria, South Australia, Western Australia and New South Wales. Forty-nine (of a total of 94 invited) co-ordinators representing a broad cross section of professions took part MHPN provided the evaluation team with a spreadsheet detailing the number of hits per month on each of the available web pages as collated by Google Analytics and other tracking software MHPN Online registration data were provided to the evaluation team by MHPN. These online data are collected on a daily basis from the website

MHPN workshop calendar MHPN workshop attendance list

MHPN workshop master list

MHPN network master list

MHPN network attendance list

National network co-ordinator feedback forums

Web portal data from MHPN MHPN Online registration data

included, for example, ‘‘what are the encountered barriers and enablers of establishing a network?’’ and ‘‘Describe the best example of an ongoing network you are involved with?’’ It was conducted by the evaluation team and transcribed verbatim. 2.4. Data analyses Quantitative data analysis was carried out using Stata version 11 and SPSS version 19. For each survey, basic frequencies were calculated to determine patterns of responses. No significance testing of these patterns was performed as the group numbers were quite small in some instances (e.g., professionals in smaller rural locations) and it was decided to take a uniform approach to analysis (i.e., not introduce significance tests in some instances and not others). Logistic regression analysis was performed to understand the predictors of network formation. To assess changes in knowledge and practice, mental health professionals’ responses to the pre-workshop survey were matched with their follow-up survey responses. Each participant had a unique identification code that was used to identify participation in the pre-workshop survey and to time their invitation to participate in the follow-up survey. This unique code was used to match the data files in SPSS. Qualitative analysis of focus group data was undertaken using template analysis (King, 1998). Template analysis involved identifying a set of key themes and producing a template to organise these themes into a coded hierarchy. This approach allowed the flexibility of using themes developed a priori and others developed during analysis. 3. Results The results of the evaluation relate to the specific objectives outlined in the programme logic displayed in Fig. 1 and are presented here according to the area with which they relate (Fig. 2).

3.1. Workshop achievements - area A: initial interdisciplinary collaborative workshops 3.1.1. Facilitator recruitment MHPN recruited workshop facilitators from a wide range of mental health professions through a variety of avenues, most commonly telephone and email communication. The original communication aimed to pique the interest of potential facilitators and emphasised how MHPN would support (e.g., with a manual and other materials, and through ongoing contact with an MHPN Project Officer) and reward them (e.g., through payment and professional development points). It also outlined the skills needed for good facilitation and running workshops. According to the workshop calendar, 748 facilitators were recruited and between them they facilitated 1162 workshops. The majority (515 or 69%) facilitated only one workshop, a further 142 (19%) facilitated two workshops, and the remainder facilitated more than two workshops. 3.1.2. Workshop participant recruitment The majority of potential workshop participants were identified through the membership lists of the participating partner organisations. MHPN and the relevant professional groups primarily sent workshop invitations to mental health professionals via Interested individuals then registered for a workshop by either faxing or emailing their details to MHPN project officers and, once established, through an online registration system. In total there were 1132 workshops. This number of workshops is lower than the number reported above in relation to facilitation derived from the workshop calendar (1132 compared with 1162) because this analysis relied on data from the attendance list database provided by workshop co-ordinators, who did not all provide the required attendance data. Overall, there were 19,926 expressions of interest in the workshops from mental health professionals. Of the 19,926 expressions of interest, 14,994 (75%)

34

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

Fig. 2. Achievement of objectives in the hierarchy.

resulted in participants attending workshops. The residual 4925 (25%) expressions of interest resulted in ‘failures to attend’, cancellations and ‘no shows’. The pattern of non-attendance was relatively consistent across professional groups. Of the 14,994 attendances accounted for by 11,930 unique individuals, 60% of these were psychologists and 25% were GPs. MHPN had a contractual target to deliver 1200 workshops, 30% of which were to take place in rural areas. Consistent with this target, 31% of the 1162 workshops conducted by MHPN between February 2009 and June 2010 occurred in rural areas (Australian Bureau of Statistics, 2005). Each workshop was to have a maximum of 20 expressions of interest by mental health professionals. Across workshops, the mean number of expressions of interest was 17.6 (sd = 5.6) and the average number of attendances was 13 (sd = 4.7). Each workshop was to have at least three different types of mental health professions and at least four GPs attending. Ninety-two per cent of the workshops achieved the former target and 42% achieved the latter. 3.1.3. Participants’ and facilitators’ experiences of workshops In the post-workshop survey participants were asked about their satisfaction with various elements of the workshops (including the facilitation and the materials). On a scale of 1–10, where 1 was ‘poor’ and 10 was ‘excellent’, participants gave facilitators mean ratings of between 8.2 and 8.8 (sd = 1.5–1.8) across the following six key areas: group management, knowledge, respect for all professions, time keeping, equity of input, and clarity

of instruction. Participants were also asked about their satisfaction with workshop materials in terms of their relevance, complexity and the discussion questions provided. Workshop materials were also rated positively, although not as positively as facilitation, with mean ratings ranging from 7 to 8 (sd = 1.9–2.1) on the same 10point scale. These patterns of ratings were fairly consistent across professional groups (see Inline Supplementary Tables S8 and S9 for further details). Inline Supplementary Tables S8 and S9 can be found online at http://dx.doi.org/10.1016/j.evalprogplan.2014.03.002. When asked in the pre-workshop survey about their reasons for attending workshops, 70% of participants indicated that they were keen to meet local mental health professionals. When asked about their satisfaction in the post-workshop survey with the mix of professionals attending the workshop on a threepoint scale of 1 ‘not at all’ to 3 ‘very much’, over half of all participants were ‘very much’ satisfied, and a further 40% were ‘a little’ satisfied. Again, these patterns showed only minor variability across professional groups (see Inline Supplementary Table S10 for further details). Inline Supplementary Table S10 can be found online at http:// dx.doi.org/10.1016/j.evalprogplan.2014.03.002. The facilitators’ post-workshop survey and the facilitators’ indepth survey both asked facilitators to rate their satisfaction with various resources and forms of support. Both questions used a scale of 1 ‘very poor’ to 5 ‘excellent’. Facilitators were positive about the support they received from MHPN in general and the Project Officers in particular, providing a mean satisfaction rating of 4.6 (sd = .55) on

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

the post-workshop survey and 4.3 (sd = .88) in the in-depth survey. Facilitators felt that their role was well explained (pre-workshop mean satisfaction = 4.3; sd = .72 and in-depth survey mean = 4.3; sd = .81), that the resources were valuable (means = 4.0, sd = .94 and 3.9, sd = .99), and that the structure of the workshop sessions was well organised (means = 4.1, sd = .92 and 4.3, sd = .92). 3.1.4. Mental health professionals’ perceptions of workshops: achievement of learning objectives The mental health professionals’ post-workshop survey gauged the extent to which participants’ learning objectives were met by the workshops, respondents indicated the objectives were ‘not met’, ‘partially met’ or ‘entirely met’. Between 90% and 95% of participants had their needs partially or entirely met with respect to recognising the expertise of other mental health professionals, identifying referral pathways to other local mental health professionals, identifying opportunities for ongoing professional development and mutual support with other mental health professionals, and the participants’ individual learning needs. The survey also explored participants’ views with respect to the relevance and usefulness of the workshops in terms of their practice and their networking opportunities. Again, the response was positive, with over 95% of participants indicating that the workshops were ‘partially’ or ‘entirely relevant’ and ‘partially’ or ‘entirely useful’. In addition, the survey examined the extent to which participants felt their knowledge of other professionals’ contribution to mental health care had increased as a result of attending the workshops. Over 90% of participants indicated that their knowledge had increased ‘a little’ or ‘very much’. The findings were relatively consistent across professional groups; where there were differences, they tended to be with occupational therapists and psychiatrists, who were somewhat less positive than other professional groups about the degree to which the workshops met their learning objectives (see Inline Supplementary Table S10 for further details). 3.2. Network formation – area B: ongoing, self-sustained interdisciplinary clinical networks The workshops appeared to generate participants’ interest in being part of an ongoing network, as assessed by several indicators. Almost all participants (98.8%) who responded to the postworkshop survey felt that interdisciplinary networking was ‘important’ or ‘very important’ when rated on a three-point scale of 1 ‘not important’ to 3 ‘very important’, and over half (52.9%) felt that the workshops had ‘very much’ increased their desire to engage in collaborative mental health care when measured on a scale of 1 ‘not at all’ to 3 ‘very much’. Using the same scale, a similar proportion agreed that the workshops had ‘very much’ assisted in creating ongoing local interdisciplinary network activity. When post-workshop survey respondents were asked more explicitly whether they wanted to participate in an ongoing interdisciplinary network, over 70% indicated that they did and almost all of the remainder responded with ‘maybe’. Psychiatrists were less likely than other professional groups to show interest in ongoing networking (see Inline Supplementary Material Table 10 for further details). Mental health professionals who completed the sustainability survey were also asked about their intentions with respect to ongoing networks. When asked to indicate how much they wanted to be part of an ongoing network (from 1 ‘not at all’ to 5 ‘very much’), one third of respondents ticked ‘5’ and a further third ticked ‘4’. Those who endorsed the remaining responses (n = 531) were asked to give the primary reason for their uncertainty about wanting to be part of an ongoing network. The majority (61%) said that they had not yet found a network that they would like to be

35

part of in the long term. Thirty-four percent said that it was too much effort, and 5% said it was because they already engaged in interdisciplinary networking. Additional information on participants’ likelihood of developing and maintaining ongoing networks came from the facilitators’ post-workshop survey. Through this survey, facilitators provided feedback about whether the formation of ongoing networks was encouraged by the workshops. Overall, the survey results indicated that the workshops provided a moderate level of encouragement, with the most common outcome being continued meetings. Facilitators themselves enthusiastically encouraged the workshop participants to continue to meet and discussed the benefits of ongoing networking. Further evidence of the extent to which the workshops generated participants’ interest in ongoing networks came from the sustainability focus group. Network Sustainability Project Officers and Senior Project Officers who participated in this focus group echoed the above comments about interest in ongoing networks, noting that they detected a strong interest in ongoing networks from mental health professionals. However, they commented that mental health professionals’ desire to engage in networks was strongly influenced by the characteristics of their local environment and by their professional grouping. For example, one participant commented that ‘‘It certainly varies from location to location, depending on existing structures within that area, what the landscape looks like, the way that they already work together if they are already working together. . .’’ In addition, they described mental health professionals in rural areas and/or those who are new to private practice as being the most motivated to engage in interdisciplinary networking. 3.2.1. Establishment of network structures and processes Early on, there was an expectation about creating sustainable networks, but no specified model as to how these networks would be developed and maintained and no dedicated resources to support them. As MHPN evolved, the need for greater focus on sustainability was acknowledged and a system of support for sustainable networks was developed at all levels, from governance through to MHPN staff. This took considerable time and resources to conceptualise and operationalise. In early 2010, MHPN worked to review and refine processes to support the establishment of networks after it became clear that there were increasing demands from network groups for administrative support, leadership and strategic planning. It was not until the beginning of July 2010, when MHPN shifted its emphasis from delivering workshops to fostering networks, that full MHPN resources were then able to be directed into sustainability. At this time, staff members were dedicated to assisting with the development and continuation of ongoing networks. Their role was to build relationships with, and to support, network co-ordinators, with the ultimate aim of building viable sustainable networks. 3.2.2. Barriers and enablers to network participation and coordination Using combined data from MHPN’s workshop master list dataset, workshop calendar dataset and workshop attendance datasets, Table 3 shows the findings from a logistic regression that considered the relationship between a number of workshoprelated variables and the likelihood of members of a workshop forming a network. It includes data from 899 workshops for which data on all variables was available. Initially, each variable was considered in isolation. The unadjusted odd ratios (ORs) shown in Table 3, suggested that three factors predicted network formation: workshop members agreeing to meet again, facilitator agreement to continue involvement as a network co-ordinator, and locality,

36

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

Table 3 Logistic regression analysis of workshop-related predictors of ongoing network formation. Network formed

Location of workshop Facilitator to continue involvement as network coordinator Agreed to meet again Number of GPs in workshop Number of different professional groups in workshop

Metro. Rural No Yes No Yes 0–2 3 1–2 3

No

Yes

83 26 60 49 26 83 45 64 7 102

523 267 181 609 2 788 318 472 57 733

Unadj. OR (95% CI)

P

Fully Adj. OR (95% CI)

P

1.00 1.63 1.00 4.12 1.00 123.42 1.00 1.04 1.00 0.88

0.039

1.00 1.24 1.00 2.60 1.00 74.66 1.00 0.91 1.00 0.98

0.416

(1.02–2.59) 0.000 (2.73–6.22) 0.000 (28.78–529.27) 0.837 (0.70–1.57) 0.763 (0.39–1.99)

(0.74–2.06) 0.000 (1.64–4.14) 0.000 (17.00–327.78) 0.693 (0.57–1.45) 0.968 (0.41–2.36)

Source: MHPN master list, workshop calendar and workshop attendance dataset. Note: The unadjusted odds ratio represents the results of a series of logistic regressions were each variable was entered individually. The adjusted odds ratio represents the result of a logistic regression model where all the variables were entered at the same time. The likelihood ratio test indicated that the overall adjusted model was statistically significant (x2 = 119.07; df = 5; P < 0.0001).

with rural workshops more likely to lead to networks. Subsequently, each variable was considered in the context of all other variables in the model. The adjusted ORs presented in Table 3 suggest that the intention of workshop members to meet again following the initial workshop continued to show statistically significant associations with network formation (and ongoing involvement of the workshop facilitator as a network coordinator). However, when considered with the other variables, rural workshops were no more likely than urban workshops to generate networks. The results from the logistic regression were echoed in the sustainability focus group discussions, but with some nuances. Staff members who took part in this focus group highlighted the importance of having a strong co-ordinator and a clear purpose for the network. They also noted that the dynamics of the workshop group influenced their desire to continue to meet. The sustainability survey asked mental health professionals to consider what kind of support from MHPN might be required for the establishment of ongoing networks. Specifically, it asked them to rank a series of potential support options in order of importance. Overwhelmingly, respondents thought that MHPN should ‘Provide financial incentives’ – 85% of respondents listed this as their first choice and 10% as their second. This prioritisation was consistent across professional groups (see Inline Supplementary Table S11 for further details). Inline Supplementary Table S11 can be found online at http:// dx.doi.org/10.1016/j.evalprogplan.2014.03.002. Sustainability focus group participants spoke about the key role of MHPN in contributing to network success. Many of the participants reported that without the involvement of MHPN, the networks would flounder. Participants reported that networks often sought guidance from MHPN regarding network meeting purpose, format and content, and that provision of this guidance was a key enabler of network success. Participants spoke of the capacity of MHPN to assist networks with administrative tasks (e.g., communicating with network members, organising meeting venues, managing meeting invitations and acceptances); many participants saw this as a key role of MHPN in enabling networks. One participant commented this was a key part of their role: ‘‘Support the networks as much as possible with those small administrative tasks that we can do early on . . . and be able to hand them over a nice neat package of their network, they know how to organise the meeting, they know where to meet, they know how to meet, all of those things, give them that so then they’ve got time to think about those other more important issues like what they are going to discuss, what are their common purposes . . .’’ Participants also viewed the practical resources provided by MHPN as enablers. In particular, they discussed the potential of

MHPN Online (a members only web-portal that provides resources and further networking opportunities), commenting on its ability to assist professionals to become more aware of networks and to assist members who might be unable or unwilling to attend network meetings. Having identified the above enablers, participants cited a range of barriers faced by MHPN in providing optimal support to networks. The first of these was staffing levels, which they saw as limiting their capacity to support emerging networks. The second was MHPN’s own lack of clarity about how best to support networks; participants spoke of their frustration in not being able to give networks clear guidance and instruction regarding what would make a successful network. The third perceived barrier was their inability to provide networks with assurances about the longevity of MHPN, which they felt jeopardised potential network likelihood of engagement. 3.3. Development of online resources and community – area C: website and web portal 3.3.1. The MHPN website The MHPN website was launched in February 2009 with the primary objectives of enabling facilitators to register an expression of interest to facilitate a workshop, and mental health professionals to register for workshops. The website was also designed to provide information regarding workshops (e.g., dates and times, eligibility criteria). Its development has been an evolving process, and it has undergone modifications in line with the activities of MHPN. It now has three main functions: to direct mental health professionals to local networks, to provide a gateway to MHPN Online, and to provide resources to support network meetings. In December 2009, functionality was added to the website to enable mental health professionals to register for initial workshops online. In February 2010, an online payment system was launched that allowed participants to register their details online in order to be paid for their attendance. This reduced the reliance on more labour-intensive methods for registration and payment, such-as paper-based forms. The website survey indicated that awareness of the website was sub-optimal, particularly among some professional groups. Overall, 20% of respondents did not know it existed; the figure was higher than average for psychiatrists (at 35%), GPs (at 25%) and ‘other’ professionals (at 30%). These findings were corroborated by the preworkshop, post-workshop and follow-up surveys of mental health professionals which indicated that between 21% and 39% of mental health professionals had never accessed the website (possibly because their reception staff submitted online registrations on their

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

behalf). Having said this, there are indications that these figures are declining over time (Fletcher et al., 2010b). Those who did access the website, however, viewed it positively. According to the website survey, those who used it tended to do so multiple times (43% did so between two and five times). They most commonly accessed it to find and register for a workshop and were positive about features relating to this activity, like its online registration functionality. When they were asked to rate various aspects of the website (e.g., its user-friendliness, ease of navigation, flow of information, aesthetics and relevance, and the extent to which it fulfilled their expectations), the average rating ranged from 3.2 to 3.3 (sd = 0.7–0.8) on a scale of 1–5, where 1 was ‘very poor’, 3 was ‘good’ and 5 was ‘excellent’. This indicated that, on average, respondents thought it was ‘good’. These patterns were largely consistent across professional groups (see Inline Supplementary Table S12 for further details). Inline Supplementary Table S12 can be found online at http:// dx.doi.org/10.1016/j.evalprogplan.2014.03.002. 3.3.2. Creation of an online community and virtual network: MHPN online In May 2010, the members-only web portal, MHPN Online, was launched. This was advertised in direct communications to workshop and network participants as well as in publications of professional groups. MHPN Online has several different functions aimed at supporting ongoing networking and interdisciplinary collaboration. The MHPN Online survey was designed to assess mental health professionals’ uptake of MHPN Online, and to examine enablers and barriers to its use. According to the survey, 33% of those who had accessed MHPN Online had done so once, 53% had done so 2 to 5 times, 11% had done so 6 to 10 times, and 1% had done so more than 10 times. Two thirds (67%) had first learnt about MHPN Online through an email from MHPN, and a further 14% had done so through attending an MHPN workshop. Smaller proportions had learnt about MHPN Online from work colleagues (7%), network members (1%), a notice in a professional publication (3%), or some other source (7%). Mental health professionals were asked to indicate on a scale of 1 to 5 (where 1 was ‘very poor’, 3 was ‘good’ and 5 was ‘excellent’), the extent to which they liked the idea of networking online. On average, the professionals gave it a rating of 3.9 (sd = 1), indicating that they ‘liked’ the idea. Most commonly, survey respondents hoped that MHPN Online would help them stay in contact with local mental health professionals for the purposes of consultation and referral of consumers. MHPN Online survey respondents indicated that on a scale of 1 ‘not met’, 2 ‘partially met’, 3 ‘mostly met’ to 4 ‘completely met’, on average, these expectations were partially met. Mental health professionals who had used MHPN Online were asked to rate it in terms of key aspects such as ease of navigation and presentation of information, as well as sections of the web portal

37

such as search functions and the mailbox tool. All responses were on a five-point scale where 1 was ‘very poor’ and 5 ‘excellent’. Overall, respondents rated each surveyed aspect and function between 2.8 and 3.2 (sd = 0.7–1), indicating that they viewed them as ‘good’. The MHPN Online survey also explored the ways in which the web portal was assisting mental health professionals to participate in local interdisciplinary networks. Respondents indicated that MHPN Online had fostered participation in a variety of ways, most commonly by helping them to learn about other professionals in their area (60%), but also by expanding their networks (53%). MHPN Online had assisted in practical ways too, by enabling participants to organise events and network meetings (40%), and enabling them to RSVP to such events (47%). 3.4. Long-term changes in interdisciplinary collaboration MHPN’s workshop master list dataset recorded the progress of workshops towards meeting as ongoing networks, using definitions described in Table 3. Workshops are deemed to have generated ongoing networks when some attendees have at least agreed to meet and have identified a network co-ordinator, and the relevant Network Sustainability Project Officer is confident, on the basis of his or her communication with the co-ordinator, that the workshop attendees will continue to the point of meeting. Workshops that are beyond this stage are also deemed to have generated ongoing networks. Table 4 shows that, by this definition, 81.2% of workshops identified on MHPN’s workshop master list dataset generated ongoing networks at a national level. A slightly lower proportion of workshops progressed to a network in metropolitan areas (79.0%) than in rural areas where 86.4% of workshops progressed to being a network. In all cases, they exceed the target of 70% articulated as one of MHPN’s goals. By July 2010, 705 networks had been generated from 938 workshops. These networks represented all states and territories and both metropolitan and rural areas (see Table 5). The number of networks is lower than the number of workshops because members of more than one workshop have often joined together to form a network. Two hundred and fifty three of the 705 networks (36%) have met at least once, with the maximum number of recorded meetings being nine. Data from MHPN’s network master list dataset shows that in 48% of cases psychologists have become network co-ordinators. In about half of all networks (54%) the original group facilitator has taken on the role of network co-ordinator, and in the remaining cases the co-ordinator has usually been another group member who has volunteered. According to staff members who took part in the sustainability focus group, there is recognition that network co-ordination places a significant imposition on already-busy mental health professionals, so it has often occurred under models designed to maximise involvement and minimise the burden for

Table 4 Progress of workshops (n = 1156) towards meeting as ongoing networks. Urban

National Freq. Network not in place

Network in place

Group has agreed not to meet Progress of group is unknown and follow up is required Group has agreed to meet but has no identified coordinator Sub-total Group has agreed to meet and has an identified coordinator, but no further plans Group has clear plans to meet Group has met at least once subsequent to initial workshop Sub-total Missing Total

Source: MHPN workshop master list and MHPN workshop calendar.

%

Freq.

Rural %

Freq.

Missing %

Freq.

%

68 60 89 217 280

5.9 5.2 7.7 18.8 24.2

58 47 62 167 166

7.3 5.9 7.8 21.0 20.9

10 11 27 48 110

2.8 3.1 7.6 13.5 31.1

0 2 0 2 4

0.0 25.0 0.0 25.0 50.0

262 396 938 1 1156

22.7 34.3 81.2 0.1 100.0

166 295 627 0 794

20.9 37.2 79.0 0 100.0

95 101 306 0 354

26.8 28.5 86.4 0 100.0

1 0 5 1 8

12.5 0.0 62.5 12.5 100.0

38

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

Table 5 Networks known to MHPN as at 31 July 2010, by location. Freq.

%

State/Territory New South Wales Victoria Queensland Western Australia South Australia Tasmania Northern Territory Australian Capital Territory Missing

211 213 149 61 36 18 8 7 2

29.9 30.2 21.1 8.7 5.1 2.6 1.1 1.0 0.3

Geographic classification Metropolitan Rural Missing Total

441 248 16 705

62.6 35.1 2.3 100.0

Source: MHPN network master list.

individuals, with joint co-ordinators and rotating rosters of coordinators being common. Mental health professionals who completed the sustainability survey identified funding for time spent co-ordinating and administrative support as key incentives for individuals to act as co-ordinators. By the end of July 2010, MHPN had sent 5015 invitations to mental health professionals to attend network meetings, according to the MHPN network attendance list. These invitations yielded 1587 attendances. In absolute terms, psychologists represented the greatest number of both invitees and attendees, followed by GPs. However, social workers and mental health nurses were, relatively, the most likely to take up an invitation to attend a network meeting. Paediatricians and GPs were the least likely to do so. Network attendees who completed the sustainability survey have found all elements of network meetings valuable, but had particularly appreciated opportunities for informal networking and learning about the availability and expertise of local mental health professionals. These early indicators suggest that MHPN has made significant advances in terms of establishing local networks of mental health professionals. MHPN staff members who participated in the sustainability focus group perceived a strong interest in ongoing networks on the part of mental health professionals, and this was supported by data from mental health professionals themselves, via the sustainability survey. When asked to rate on a scale of 1–5 (with 1 being ‘not at all’ and 5 being ‘very much’) their desire to be part of an ongoing network, survey respondents’ overall mean score was 3.8 (sd = 1). There was some variation by professional group, however, with psychologists, social workers and occupational therapists showing the highest level of interest and psychiatrists and GPs showing the lowest level (see Inline Supplementary Table S13 for further details). Those who are not so interested in being part of ongoing network most commonly attribute their reticence to not having yet found a network that they would like to be part of in the long term. Other common reasons were the degree of effort required and a lack of time. Inline Supplementary Table S13 can be found online at http:// dx.doi.org/10.1016/j.evalprogplan.2014.03.002. The mental health professionals’ pre-workshop and 14-week follow-up survey provided data on the extent to which MHPN had achieved the objective of changes in mental health professional’s knowledge and practice being attributable to their participation in MHPN workshops and ongoing networks.1 Both surveys asked respondents to indicate whether they were aware of providers 1 It should be noted, however, that although 1696 mental health professionals responded to the pre-workshop survey, only 245 responded to the 14-week followup survey, and data from both surveys was only available for 142 individuals.

from other professional groups to whom they would confidently refer consumers. After attending the workshops and taking advantage of early ongoing networking opportunities, greater percentages of providers indicated that they would refer to other professional groups, and would engage in interdisciplinary collaboration. In some cases, these rates rose by half. For example, the percentage of mental health professionals participating in interdisciplinary meetings rose from 32% before the workshops to 46% after the workshops, and the percentage of mental health professionals taking part in interdisciplinary lunches/recreational networking increased from 19% to 28%. The above observed changes in knowledge and practice were paralleled by increased levels of satisfaction with networking. Whereas 32% of mental health professionals were ‘not at all satisfied’ with their level of networking prior to the workshops, only 21% were ‘not at all satisfied’ after their conclusion. Conversely, before the workshops 64% were ‘moderately’ or ‘extremely satisfied’, and after, 77% were. Mental health professionals who took part in the 14-week follow-up survey were asked to what extent they would attribute any increases in their networking activities to their involvement with MHPN. Fifty three per cent indicated that MHPN was ‘moderately’ responsible for these increases, and 8% indicated that it was ‘extremely’ responsible. These patterns were relatively consistent across provider groups (see Inline Supplementary Table S14 for further details). Inline Supplementary Table S14 can be found online at http:// dx.doi.org/10.1016/j.evalprogplan.2014.03.002. 4. Discussion MHPN successfully developed and ran an ambitious series of initial workshops which reached a significant number of mental health professionals from a range of disciplines who were working across Australia in both metropolitan and rural locations. The workshops were received positively by participants and generated considerable interest in the formation of ongoing, self-sustained, interdisciplinary clinical networks. MHPN established structures and processes to foster these networks, and developed and delivered resources to assist them in their establishment phase. MHPN also developed and maintained a public website to market and manage the workshops, and MHPN Online to support withinnetwork communication and collaboration. Because MHPN was only moving into its sustainability phase at the time the evaluation ended, its remaining intermediate-level objectives were only partially within the scope of the evaluation. Nonetheless, there are early signs that MHPN is making inroads in terms of achieving these objectives. It has begun to provide coordination and support for emerging networks, including virtual support via MHPN Online. Although the networks are in their early developmental stages there are indicators that a reasonable proportion of them will continue to evolve and grow. There are also indicators that participation in the workshops and membership of these emerging networks are leading to some improvements in mental health professionals’ interdisciplinary knowledge and collaborative practice. Assessment of the achievement of the highest level objectives in the hierarchy (relating to MHPN’s purpose and overarching aims) was beyond the scope of the evaluation. It was not possible to assess whether collaborative care practices have changed in primary mental health care, nor whether client outcomes have improved. Such cultural and systemic change is difficult to measure, although there might be possibilities for doing so in the future by using existing provider-based and client-based data collections as baseline information and repeating these data collections to examine change (e.g., using the Urbis quantitative

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

survey of providers’ practices, which was conducted prior to the development of MHPN, as a baseline measure (Urbis, 2008)). 4.1. Understanding MHPN’s achievements It is worth considering why the workshops and networks have been so well received. One of the reasons probably relates to MHPN’s systematic approach. MHPN’s graded, flexible and supportive approach that involved conducting the three inter-locking areas of activity over four distinct phases has worked well. Identifying barriers and enablers and modifying elements of the project accordingly were also strengths. In their systematic review of diffusion of innovations in service organisations, Greenhalg, Robert, Macfarlane, Bate, and Kyriakidou, (2004) discuss the concept of ‘system readiness for innovation’ which directly highlights the circumstances under which MHPN was introduced. Specifically, there was ‘tension for change’ in the primary mental health care sector at the time MHPN began, in particular, there was an increasing recognition of the potential benefits (both for providers and consumers) of interdisciplinary collaboration but there were few formal avenues in Australia through which such collaboration was being encouraged. Other system changes which occurred at the same time (e.g. Medicare reimbursement for selected services by psychologists, social workers and occupational therapists under the Better Access programme) meant that new providers were emerging in far greater numbers in the primary mental health sector, thus creating circumstance for ‘Innovation-system fit’. In this context ‘Innovation-system fit’ relates to the individuals recognising that to provide optimal care to consumers, they needed to have good working relationships with others from whom they might receive referrals and/or with whom they might provide shared care. At a more basic level, they needed to understand with how these other professionals operate. The same imperative existed for medical providers (e.g., GPs and psychiatrists) although arguably the Medicare changes had less of an impact for them because their provision of mental health care services already attracted a rebate. As such, it could be said that MHPN was well timed to capitalise on these recent changes to primary mental health care in Australia, and hence helped to provide a receptive context for change (Greenhalgh et al., 2004). The initiative was delivered at a time when many mental health professionals in primary care were eager and ready for opportunities for interdisciplinary networking. 4.2. Strengths and limitations of the evaluation The current evaluation had a number of strengths, not least of which was the fact that it was largely developed alongside the MHPN project in collaboration with MHPN. It drew on both routinely collected data and purpose-designed data sources for the evaluation. This recourse to different data sources and methodologies allowed for triangulation of findings and engendered confidence in the conclusions that could be drawn from them (Ovretveit, 1998; Patton, 1990). The size and scope of the evaluation was also a major strength. However, some of the data sources were more reliable than others. For example, some of the routinely collected data relied on systems that ‘went down’ on occasion, resulting in periods where data were missing. Similarly, some of the purpose-designed surveys had sub-optimal response rates (e.g., the mental health professionals’ 14-week follow-up survey). In some of these cases, certain biases may have been introduced (e.g., if those who chose to participate had particularly positive or negative views). In addition, some data were arguably collected too early, before an appropriate establishment period for the relevant activity had elapsed (e.g., the MHPN Online survey); this was unavoidable in the context of meeting the contracted evaluation deadline. It

39

should also be noted that the various data elements came from mental health professionals who were interested in collaborative care and networking; the evaluation findings therefore cannot be generalised to all mental health professionals. 4.3. Where to from here? To date MHPN has only had the opportunity to begin the process of driving interdisciplinary collaboration within this framework. It has achieved its project deliverables within the relatively short allotted timeframe; however, because of the complexities of creating ongoing interdisciplinary networks, a longer period is necessary to allow these networks to develop and flourish. Networks are complex, evolving entities and are not yet fully understood. Many have not yet met, and it is likely that their membership may be quite fluid until their purpose and approach are more clearly defined. There are many workshop attendees who have yet to be convinced about the benefits of networking; they are not actively opposed to it but have, so far, not found a network to which they feel that they belong. Generally, even those networks that have met have not yet had time to establish themselves as fully functional entities. MHPN’s role in supporting these networks at the various stages in their evolution over the coming twelve months and beyond is likely to be crucial to their success. In addition, it will be important for MHPN to monitor workshops from which no networks have emerged to date, in order to ascertain their potential. We still have limited understanding of the complexity of relationships between professionals (D’Amour et al., 2005), but it does appear that those who work in the primary mental health care setting in Australia see the value in collaborative care and are receptive to take part in interdisciplinary activities. MHPN will need to set priorities, recognising that these priorities may change as more becomes known about the way networks operate and the relationships on which they are based. 4.4. Lessons learned In terms of establishing collaborative mental health care the lessons learned include:  A clear vision and plan to establish interdisciplinary collaboration is vital to creating momentum in developing interdisciplinary networks and motivating providers to participate in networks on an ongoing basis.  Ongoing support and leadership, such as that provided by MHPN, is needed to further create and support opportunities for collaborative mental health care.  Mental health professionals’ interest in engaging in ongoing networks was influenced by their local environment as well as their professional group, with those in rural areas and newer to private practice more engaged than those in urban areas and more established professionals. The lessons learned from a programme evaluation perspective include:  Maintaining a strong working relationship with the MHPN evaluation committee through regular meetings and contact with the evaluation manager ensured that the evaluation methods and materials were always relevant in the face of MHPN’s need to be flexible in the delivery of the project.  This strong working relationship further enhanced the project, as the evaluation data also provided MHPN with access to feedback, which was utilised to enable the project to be adapted to better meet the stated objectives.

40

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41

 In an ideal world, designing the data to be captured for both MHPN and the evaluation of MHPN would have been done in consultation before any workshops were started, and thus data that was required for administrative and evaluative purposes could have been collected in a more streamlined way.  Maintaining participant engagement across many evaluation components (surveys and focus groups) can be challenging, as reflected by lower response rates to later evaluation components and thus the use of some kind of incentives to maintain participants could be useful, for example providing vouchers upon completion of all evaluation components.

5. Conclusions MHPN has successfully undertaken an ambitious project designed to promote interdisciplinary networking and collaborative mental health care. Few comparable initiatives have been conducted in mental health (Craven & Bland, 2006), and none has been undertaken in a primary care setting at a national level. The Australian primary mental health care setting was one wherein previously little interdisciplinary collaborative work was being undertaken. The changes to the government funding of mental health professionals through the Better Access programme, coupled with the funding of MHPN to foster interdisciplinary collaboration is an example of the muchneeded systems and tools, and co-ordinated leadership and support that are necessary to overcome the barriers to collaboration in Australian primary care (Barker et al., 2005; McDonald et al., 2009). Acknowledgements We would like to thank the Mental Health Professionals Network and the participants in this evaluation. We would also like to thank the Commonwealth Department of Health and Ageing for funding the evaluation. References Australian Bureau of Statistics. (2005). Australian standard geographical classification. Canberra: Australian Bureau of Statistics. Australian Bureau of Statistics. (2007). National survey of mental health and wellbeing: Summary of results. Canberra: Australian Bureau of Statistics. Australian Government. (2006). Better access to mental health care. Canberra: Commonwealth of Australia. Barker, K., Bosco, C., & Oandasan, I. (2005). Factors in implementing interprofessional education and collaborative practice initiatives: Findings from key informant interviews. Journal of Interprofessional Care, 2005(Suppl. 1), 166–176. Begg, S., Vos, T., Barker, B., Stevenson, C., Stanley, L., & Lopez, A. D. (2007). The burden of disease and injury in Australia 2003 PHE 82. Canberra: Australian Institue of Health and Welfare. CDC. (2011). Mental Illness Surveillance Among Adults in the United States. MMWR60(Suppl.). Available at http://www.cdc.gov/mmwr/preview/mmwrhtml/ su6003a1.htm?s_cid=su6003a1_w. Commonwealth of Australia. (2009). A healthier future for all Australians: Final report of the national health and hospitals reform comission June 2009. Canberra, Australia: Commonwealth of Australia. Cook, D. (2005). Models of interprofessional learning in Cananda. Journal of Interprofessional Care, 19(Suppl. 1), 107–115. Craven, M., & Bland, R. (2006). Better practices in collaborative mental health care: An analysis of the evidence base. Canadian Journal of Psychiatry, 51, 8–72. D’Amour, D., Ferrada-Videla, M., San Martin Rodriguez, L., & Beaulieu, M. (2005). The conceptual basis for interprofessional collaboration: Core concepts and theoretical frameworks. Journal of Interprofessional Care, 2005(Suppl. 1), 116–131. D’Amour, D., & Oandasan, I. (2005). Interprofessionality as the field of interprofessional practice and interprofessional education: An emerging concept. Journal of Interprofessional Care, 2005(Suppl. 1), 8–20. Davies Powell, G., Harris, M., Perkins, D., Roland, M., Williams, A., Larsen, K., et al. (2006). Coordination of care within primary health care and with other sectors: A systematic review. Australian Primary Health Care Research Institute. Fletcher, J., Machlin, A., Christo, J., King, K., Bassilios, B., Blashki, G., et al. (2010a). The independent evaluation of the mental health professionals network: Final report. Melbourne: University of Melbourne.

Fletcher, J., Machlin, A., Christo, J., King, K., Bassilios, B., Blashki, G., et al. (2010b). The independent evaluation of the mental health professionals’ network: Technical resources report. Melbourne: University of Melbourne. Fuller, J. D., Perkins, D., Parker, S., Holdsworth, L., Kelly, B., Roberts, R., et al. (2011). Effectiveness of services linkages in primary mental health care: A narrative review part 1. Health Services Research, 11, 72. Gilbody, S., Bower, P., Fletcher, J., Richards, D., & Sutton, A. (2006). Collaborative care for depression: A cumulative meta-analysis and review of longer-term outcomes. Archives of International Medicine, 166, 2314–2321. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion on innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629. Henderson, S., Andrews, G., & Hall, W. (2000). Australia’s mental health: An overview of the general population survey. Australia New Zealand Journal of Psychiatry, 34, 197–205. Holleman, W. L., Bray, J. H., Davis, L., & Holleman, M. C. (2004). Innovative ways to address the mental health and medical needs of marginalized patients: Collaborations between family physicians, family therapists, and family psychologists. Annual Journal of Orthopsychiatry, 74, 242–252. Kiesely, S., Duerden, D., Shaddick, S., & Jayabarathan, A. (2006). Collaboration between primary care and psychiatric services. Canadian Family Physician, 52, 876–877. King, N. (1998). Template analysis. In G. Symon & C. Cassell (Eds.), Qualitative methods and analysis in organizational research. London: Sage Publications. McDonald, J., Powell Davies, G., & Fort Harris, M. (2009). Interorganisational and interprofessional partnership approaches to achieve more coordinated and integrated primary and community health services: The Australian experience. Australian Journal of Primary Health, 15, 262–269. Mental Health Foundation. (2007). The fundamental facts: The latest facts and figures on mental health. United Kingdom: Mental health foundation. Michael, L. T., Howard, C., & Cox, R. P. (2008). Collaborative intervention: A model for coordinated treatment of mental health issues within a gound combat unit. Military Medicine, 173, 339–348. Ovretveit, J. (1998). Evaluating health interventions. Buckingham: Open University Press. Owen, J. M. (1999). Program evaluation: Forms and approaches. Sydney: Allen and Unwin. Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury park: Sage. Rosenberg, S., Hickie, I., & Mendoza, J. (2009). National mental health reform: Less talk, more action. Medical Journal of Australia, 190, 193–195. Urbis. (2008). Environmental scan component of the mental health professionals’ association multidiscipinary training resource program: Final report. Melbourne: The Royal Australian and New Zealand College of Psychiatrists. van Orden, M., Hoffman, T., Haffmans, J., Spinhoven, P., & Hoencamp, E. (2009). Collaborative mental health care versus care as usual in a primary care setting: A randomized controlled trial. Psychiatric Services, 60(1), 74–79. Zwarenstein, M., Reeves, S., & Perrier, L. (2005). Effectiveness of pre-licensure interprofessional education and post-licensure collaborative interventions. Journal of Interprofessional Care, 2005(Suppl. 1), 148–165. Justine Fletcher Justine is a Research Fellow in the Centre for Health Policy Programmes and Economics and psychologist. She works with the mental health team on various research projects related to national mental health initiatives, publishing in peer-reviewed journals and contributing to commissioned reports. Justine has run her own private practice and worked in various other clinical settings. Kylie King Kylie King is a Research Fellow. She is currently a part of the evaluation team for the Access to Allied Psychological Services component of the Australian Better Outcomes in Mental Health Care Programme. Kylie King is also a Psychologist with a background in Health Psychology. She has experience working in the public and private sector in direct service provision, secondary consultation, research, health promotion and community education. Jo Christo Jo is a Registered Nurse with a keen interest in mental health. She has worked in the Centre for Health Policy Programmes and Economics as part of the mental health team. Contributing to research projects regarding national mental health initiatives and writing commissioned reports and peer-reviewed articles. Anna Machlin Anna is a Research Fellow in the Centre for Health Policy, Programmes and Economics at the University of Melbourne. Anna currently works on a series of projects looking at the impact of media reporting of suicide on actual suicidal behaviour as part of the AHNRC capacity building grant The Australian Health News Research Collaboration. In the past Anna has worked on various evaluations of national mental health programmes. Prior to joining the Centre, Anna was employed by the Cancer Council Victoria where she worked on a range of projects, involving extensive data management and analysis as well as reporting to a range of government, international and not-for-profit organisations. Bridget Bassilios Bridget Bassilios is a Research Fellow in the Centre for Health Policy, Programmes and Economics at the University of Melbourne; she has worked on a number of evaluations of mental health programmes. She is involved in qualitative and quantitative data collection, analysis and reporting. And has contributed to commissioned reports and peer-reviewed journal articles. She has supervised Master of Public Health research projects in mental health. She is also an independently

J. Fletcher et al. / Evaluation and Program Planning 45 (2014) 29–41 practising psychologist specialising in depression and anxiety. She has had previous experience working in mental health across various public and private sector clinical settings. Grant Blashki Grant is a Senior Research Fellow in the Nossal Institute for Global Health at the University of Melbourne, and a practising GP with a particular interest in mental health care. He is also an Honorary Senior Lecturer at the Institute of Psychiatry, Kings College London. He has published widely in the area of primary mental health care, has convened two conferences in general practice psychiatry. Chris Gibbs As MHPN’s CEO, Chris oversees all of the activities of the Mental Health Professionals Network. Chris has held significant leadership positions in the public health sector. He has undertaken roles at Melbourne Health and within both the Victorian Public Service and the community services sector. For 8 years, Chris was the Director of North Western Mental Health, one of the largest publicly funded providers of mental health services in Australia with a clinical workforce of 1300. Chris is currently Chair of VATMI Industries, a commercial business service employing 420

41

staff – 350 of whom have a disability. Chris commenced as CEO of MHPN in August 2008. Angela Nicholas Angela is a psychologist and has worked for the Australian Psychological Society and the Mental Health Professionals Network. She has experience in the health and mental health fields conducting research and education. Angela’s current focus is on professional writing. Jane Pirkis Jane Pirkis is the Director of the Centre for Health Policy, Programmes and Economics in the Melbourne School of Population Health at the University of Melbourne. Jane has conducted numerous evaluations of national mental health programmes and policy developments. She was the lead author on reviews of the National Youth Suicide Prevention Strategy and the National Suicide Prevention Strategy, was contracted to co-write the National Mental Health Plan (2003–2008) and the Fourth National Mental Health Plan, and was involved in stakeholder consultations that formed part of the evaluation of the First National Mental Health Plan.