Evaluation and Program Planning 52 (2015) 50–60

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Evaluation of a youth agency’s supervision practices: A mixed-method approach Julie Gosselin *, Sophie-Claire Valiquette-Tessier, Marie-Pier Vandette, Elisa Romano School of Psychology, University of Ottawa, Canada

A R T I C L E I N F O

A B S T R A C T

Article history: Received 10 September 2014 Received in revised form 9 February 2015 Accepted 12 March 2015 Available online 8 April 2015

This research presents the findings from an evaluation and organizational development initiative that was requested by a Canadian youth agency working in a large urban setting. A team of four researchers affiliated with the Center for Research on Educational and Community Services (CRECS) at the University of Ottawa conducted the evaluation. The purpose of the evaluation was to identify the supervision needs and challenges of coordinators and front line staff, assess the efficiency of the current supervision practices, and evaluate the supervisors’ and supervisees’ satisfaction with these current practices. A literature review was performed to help provide a clear definition of ‘supervision’ and the different professional roles it encompasses. Additionally, research evidence pertaining both to what contributes to supervision efficacy and supervisor competency was reviewed to distill the most robust findings in the existing literature. The lines of evidence consisted of a document and file review, an online employee survey, group discussions (i.e. focus groups), and interviews with key informants. The results of the evaluation helped the research team formulate recommendations to the agency for the development of enhanced supervision practices across its various service areas. ß 2015 Elsevier Ltd. All rights reserved.

Keywords: Youth agency Evaluation Supervision Organizational development

1. Introduction Clinical supervision is an activity/intervention provided by a more senior member to more junior members of that same organization. It can be a planned or unplanned activity that occurs one-on-one or in a group format. This relationship is evaluative, extends over time, and has the simultaneous purpose of enhancing professional functioning and supporting the well-being of the more junior members, while monitoring the quality of services/ care offered to the client (Falender & Shafranske, 2004). In addition, supervision can be more clinical or administrative in its focus. Clinical supervision more commonly focuses on ensuring accountability for effective services through providing professional development for practitioners, while administrative supervision is more oftentimes viewed as primarily hierarchical and focuses on practitioner performance reviews (Bogo, Patterson, Tufford, & King, 2011). In many helping professions, clinical supervision is the main vehicle for professional development, which includes the development of knowledge and skills as well as a sense of professional

* Corresponding author at: School of Psychology, University of Ottawa, 136 JeanJacques Lussier, #4013, Ottawa, ON, Canada K1N 6N5. Tel.: +1 613 562 5800x2208. E-mail address: [email protected] (J. Gosselin). http://dx.doi.org/10.1016/j.evalprogplan.2015.03.010 0149-7189/ß 2015 Elsevier Ltd. All rights reserved.

identity. It is through the interaction with their supervisor, and sometimes the observation of supervisors at work, that supervisees gain the necessary knowledge and skills to accomplish their work effectively. In fact, most professionals will spend a large portion of their post-secondary training in supervision. While a majority of professionals agree that clinical supervision is essential for professional development to occur, relatively few studies have been accomplished to help understand which evidence-based theoretical models predict the development of professional competencies, how the different aspects of clinical supervision contribute to professional development, how diversity issues may affect clinical supervision processes, or how to become a competent clinical supervisor (Bernard, 2005; Goodyear & Bernard, 1998; Goodyear, Bunch, & Claiborn, 2005; Kilminster & Jolly, 2000; Watkins, 1995). In addition, while supervision was common practice in most agencies and publicly funded programs in mental health, recent reviews suggests that the practice of clinical supervision has declined over the last decades (Giddings, Cleveland, & Smith, 2006; Hoge, Migdole, Farkas, Ponce, & Hunnicutt, 2011). The reality of the current work context in many agencies requires clinical supervisors to supervise professionals and staff from different fields of work, where different models of supervision coexist. This means that there are fewer resources assigned to the practice of clinical

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

supervision, while its practice has become more complex over time (Bogo et al., 2011; Hoge et al., 2011). For example, the shift from facility-based to community-based services has been linked to increasing caseloads, increasingly complex presentations of clients, and increasing autonomy in service provision (Hoge et al., 2011). At the same time, research indicates that effective supervision can affect job retention and turnover, increase job satisfaction and promote quality of client care (Bogo et al., 2011; Lambert et al., 2003; Mor Barak, Travis, Pyun, & Xie, 2009; Shoenwald, Sheidow, & Chapman, 2009). In this context, some experts have suggested the development of an interprofessional clinical supervision model for mental health and addictions services, to face the challenges related to the contemporary work context by finding common elements to the practice of supervision within and across disciplines (Bogo et al., 2011; D’Amour, FerradaVidela, Rodriguez, & Beaulieu, 2005). The fields of medicine, social work, and counseling have produced the most studies related to clinical supervision, and other fields such as clinical psychology have been heavily influenced by this body of research (Kilminster & Jolly, 2000). That being said, many limitations plague the majority of research studies in this area: it is common for studies not to differentiate between didactic and experiential training in predicting the development of professional competencies; very few studies have used randomized-controlled trials; and there is an overrepresentation of the use of self-reported measures as dependent variables (Goodyear & Bernard, 1998). For example, a systematic review of the research published on clinical supervision between 2000 and 2005 indicated that the vast majority of studies used subjective measures (i.e. participant’s perception or satisfaction) to explain what is related to effective clinical supervision (Goodyear et al., 2005). This is a problematic practice, given the well-known discrepancy that exist between self-assessment and observation of professional competency, especially in those with lower observable competency (for a review, see Davis et al., 2006). 1.1. A competency-based approach for clinical supervision There has been marked progression toward a perspective focused on the development and maintenance of professional competencies in many helping professions. This perspective has led to the development of guidelines for training programs, such as those developed in the domain of clinical psychology (Fouad et al., 2009; Hatcher & Lassiter, 2007). These authors proposed a roadmap for training supervisees that is oriented toward (1) a developmental approach aimed to increase expertise in the different areas of professional competencies, (2) a particular focus on the development of metacognition (the capacity to think reflectively about one’s own knowledge) and metacompetence (the capacity to think reflectively about one’s own competencies), and (3) the development of a supervisee’s ability to work collaboratively to take on challenges in the workplace. Competency-based clinical supervision is ‘an approach that explicitly identifies the knowledge, skills and values assembled to form each clinical competency and develop learning strategies and evaluation procedures to meet criterion-referenced competence standards in keeping with evidence-based practices and the requirements of the local setting’ (Falender & Shafranske, 2004, p. 233). It is consistent with a professional development approach that focuses on training for professionalism as a foundation for competent practice. Specifically, using competency benchmarks anchored in a developmental perspective helps supervisees develop an integrated professional identity that is based on the standards in their field of practice (Elman, Illfelder-Kaye, & Robiner, 2005). This is accomplished through the identification of competencies that supervisees must possess to be effective, and

51

the operationalization of their acquisition is demonstrated in the sequence of training (Canadian Interprofessional Health Collaborative, 2010; Fouad et al., 2009). This represents a shift from a more traditional perspective on supervision, which has often been associated with a relative absence of training and weaker professional regulation. Finally, the competency approach is also consistent with an interprofessional model of clinical supervision, which tend to focus on supportive, clinician-focused, contentoriented supervision offered by supervisors who possess the knowledge, skills and attitudes associated with competency in the area in which the service is delivered (Bogo et al., 2011). 1.2. Research on clinical supervision: what makes supervision effective? Given the many challenges facing this area of research, Proctor (1994) initially suggested the three following recommendations to orient future studies: develop a clear definition of what is meant by ‘clinical supervision’; pay particular attention to what distinguishes poor from good supervision; and adopt a developmental perspective with a special emphasis on challenges related to multidisciplinary work. Heeding these suggestions, Kilminster and Jolly (2000) reported in their own literature review methodological problems in studies that focus on clinical supervision in different healthcare fields, while also painting a descriptive profile of the various definitions used in this area of research, the emerging theoretical models used, and the numerous variables that could be linked to clinical supervision efficacy. A systematic review published five years later (Goodyear et al., 2005) identified a total of 49 publications on this topic between 2000 and 2005. The authors concluded that research in this area continues to develop at a very slow pace and that there is an overrepresentation of theoretical and/or conceptual articles published without any empirical support. A new systematic review conducted by the principal author of the current study (Gosselin, Barker, Kogan, Pomerleau, & Pitre-D’Iorio, in press) echoes the same findings. Despite methodological flaws, the majority of researchers have agreed on several factors judged to be related to good supervision by its recipients (Falender & Shafranske, 2004). According to participants (usually doctoral students in counseling or clinical psychology), a good supervisor establishes rules and expectations at the outset. The good supervisor creates a safe, non-judgmental, supportive work environment where supervision is focused on the here and now. The good supervisor provides structure but also encourages supervisees to reflect on their therapeutic work and supports them in developing their professional identity. The good supervisor lets the patient/client’s story unfold and invites supervisees to reflect on their reaction to their patient/client, to ask themselves how they want to broach topics of importance to their clinical work, and to develop insight into aspects they have ignored. Finally, the good supervisor gives concrete support when necessary to help supervisees develop knowledge and/or skills or to help them resolve resistance or impasses (Falender & Shafranske, 2004; Kilminster & Jolly, 2000). In return, supervisors also benefit from a variety of supervisee characteristics, which can make the supervision process more successful. These include openness to experience and feedback, coming prepared for supervision, capacity to listen to and implement supervisor suggestions and supervisee’s capacity to self-reflect (Falender & Shafranske, 2012). Supervisors and supervisees who do not adhere to these guidelines risk the development of a non-productive work environment where, according to some authors, there could be a negative impact on the professional development of supervisees and the quality of care delivered to patients/clients. Such negative impacts may include personality conflicts, loss of confidence,

52

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

development of a defensive attitude, decrease in learning, and loss of trust in the supervision process (Falender & Shafranske, 2004; Watkins, 1995). A recent systematic review (Wheeler & Richards, 2007) indicates that, while the quality of the evidence is variable, supervision appears to consistently demonstrate some positive impact on the supervisee as well as on client outcomes, when the latter are measured. For example, a recent study showed that supervisor focus on adherence to treatment principles predicted greater therapist adherence, and that two further aspects of supervision (adherence to the structure and process of supervision, and focus on clinician development) predicted change in client behavior (Schoenwald et al., 2009). Lastly, a meta-analysis interested in the impact of giving supervision feedback to clinicians about their client’s progress demonstrated a more significant positive impact on client outcomes for those who showed a poor initial response to treatment (Lambert et al., 2003). One of the most important factor for an effective supervision is the relationship between the supervisor and supervisee (Kilminster & Jolly, 2000). Indeed, Heppner and Handley (1982) found that trainees perceived supervisors as more expert, attractive, and trustworthy when they felt that they were engaged in the supervision process. A positive experience for the supervisee can also be related to similarity in theoretical orientation and interpretative style with their supervisor (Kennard, Stewart, & Gluck, 1987). More recent research found that supervisees were more comfortable to disclose when their supervisor was someone they had chosen (Webb & Wheeler, 1998). Finally, two characteristics appear to contribute to a better supervisory alliance: the degree of involvement and collaboration within the supervisory style (Bernard & Goodyear, 2013). 1.3. Research on clinical supervision: how can someone become an effective supervisor? Given that the quality of the clinical supervision offered is dependent on the skills and competencies of the supervisor, research has also been interested in analyzing the professional development of clinical supervisors. When we examine the research that has been accomplished on training provided to clinical supervisors, three trends emerge. These trends can be considered hypotheses that merit further empirical investigation. First, it appears that the type or format of training matters. Specifically, training that combines both didactic and experiential components and where the practical training involves the supervision of supervision appears to generate a more significant impact on the supervisor’s professional development and practice (Baker, Exum, & Tyler, 2002; Borders, 1996; Gosselin et al., in press; Haley, 2002; Lyon, Heppler, Leavitt, & Fisher, 2008; McMahon & Simons, 2004; Vidlak, 2002). Second, it appears that the key number of training experiences, that is the minimum amount of training that is necessary to produce any discernible change in professional practice, averages between two to three consecutive training opportunities. In other words, when compared to individuals who have not received supervision training, those who have completed three supervision training experiences (i.e. workshops, courses, supervision-of-supervision practice) have a higher sense of efficacy in their practice, while those with two supervision training experiences report adopting a less dogmatic position and being less critical and more supportive in their role as supervisor (Stevens, Goodyear, & Robertson, 1998). Third, supervisors who have completed more consecutive hours of clinical supervision training or who have had the opportunity to be supervised in their supervision for a longer period of time report a higher sense of efficacy in their practice (Lyon et al., 2008; Vidlak, 2002). Training that is offered as a short intensive workshop does not yield a significant impact on professional development, even

when assessed with controlled trials (Kavanagh et al., 2008). Finally, there is some preliminary evidence suggesting that providing training on clinical supervision to both supervisors and supervisees concurrently may have a more important impact on the quality of clinical supervision processes, than providing training to supervisors alone (Kavanagh et al., 2008). 2. Mixed methods approach to evaluation and organizational development 2.1. Purpose and scope The youth agency evaluated was founded in 1960. It is a multiservice non-profit agency that provides services in French and English in employment, community and housing, mental health, youth engagement, and youth justice. It is committed to providing a safe, non-judgmental, and accessible environment where individuals 12 years and older can pursue their life goals and be encouraged in making informed decisions. Their mandate is to work in partnership to develop, deliver, and advocate for better conditions, services, and opportunities for youth and families. A Board of Directors governs the organization and is composed of professionals whose role is to articulate the mission and the strategic planning of the organization, to define its goals, and to measure and report on their performance. The agency employs over three hundred professionals who work at one or more of their twenty locations delivering the programs and services that support at-risk youth and their families. The agency also collaborates with several agencies and organizations for service delivery, local food banks, colleges, hospitals, universities, and others across the region they serve. As such, the agency has taken a leadership role in the community, interacting with approximately 2500–3000 individuals on a monthly basis. In fact, the agency has been recognized for the past fifty years as having expertise in supporting youth, especially in the areas of diversity, employment, health, homelessness, housing, mental health, youth engagement, and youth justice. Among the professionals working within the agency are coordinators and front line staff. On one hand, the coordinator is responsible for the overall functioning and operation of the designated program. This involves activities such as supporting the effective recruitment, orientation, supervision, evaluation and training of unit staff and student placements, ensuring completion of annual employee performance reviews and work plans for assigned staff, overseeing and ensuring the smooth day to day operations of the unit including the delegation of and coordination of essential daily activities, contributing to program evaluation and service development including the identification of trends with program area, ensuring all administrative tasks are completed on time and in accordance with unit and policies, including preparation of schedules, shift summaries, discrepancy reports, submission of payroll, approvals of time off requests and data entry, etc. On the other hand, front line staff provides direct services to clients. This entails for instance participating in admission/discharge planning, and implementing case management. The evaluation was conducted in early 2014 to assess current supervision practices within three sectors at the agency (e.g. community services, mental health, and youth justice) to define enhanced, appropriate, and effective supervision practices for the agency’s multi-disciplinary work environment, to define structural options for successful integration of these practices into professional activities at the agency, and to develop an implementation plan for the practices that takes a phased concept approach, in order to move the agency toward a competencybased model of clinical supervision. To accomplish these

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

objectives, a participatory evaluation approach was used, following a consultation with the agency’s steering committee. This approach was chosen because it appeared to be the one most suitable, given the key organizational values of the agency, as well as the need to develop a nuanced understanding of the various work contexts at play. Several groups of stakeholders participated in the evaluation process, such as senior management and directors, coordinators, and front line staff. 2.2. Issues and evaluation questions The evaluation addressed the following questions. These questions were first developed by the authors, and were modified based on feedback received from the agency steering committee. The authors developed the questions based on the goals set by the agency for this evaluation. 2.2.1. Needs/challenges/strengths What is supervision in the context of employees’ work at the agency? What aspects of work activities are considered supervision? What are the coordinators/front line staff’s needs in terms of supervision?What are the challenges met by coordinators/front line staff in terms of supervision? 2.2.2. Performance To what extent have the supervision activities/practices at the agency impacted the work accomplished with clients? To what extent have the supervision activities/practices at the agency impacted employee wellness/empowerment at work? 2.2.3. Satisfaction with current supervision practices Thinking of the agency clients’ needs, are these supervision activities/practices frequent enough? Thinking of the agency clients’ needs, are these supervision activities/practices timely enough? Thinking of the agency clients’ needs, are these supervision activities/practices structured and managed effectively? Thinking of the agency clients’ needs, are these supervision activities/practices focused on addressing the core topics/issues that matter to your work? Thinking of the agency clients’ needs, are these current supervision activities/practices of good quality?

53

dictionary and profiles. Additional documents were provided to the evaluation team during the data collection, including handout packages on strengths-based approaches to supervision and information on collaborative practice groups. 3.2. Online staff survey An online survey was developed and made available via FluidSurvey for staff members (front line staff and coordinators) to complete. The survey was developed by the authors and modified following the feedback received from the steering committee. The authors developed the questions based on the agency inquiry about its members’ satisfaction concerning the actual supervision practices. Since all coordinators and front line staff could not take part in the focus groups, this tool was created to allow a maximum number of participants to provide their opinion on supervision practices at the agency. The response rate was approximately 25% (51 respondents), which is considered conservative but resembles the rates obtained by other authors conducting online surveys (Ballantyne, 2003; Dommeyer, Baum, Hanna, & Chapman, 2004; Nair, Wayland, & Soediro, 2005; Nulty, 2008; Ogier, 2005). The survey contained 14 multiple-choice questions and 1 open-ended question. Examples of multiple choices questions are: (1) Considering that you take part in different supervision activities, please indicate the percentage of planned and unplanned supervision you received during a month; (2) On a scale of 0–10 (0 = strongly disagree, 5 = neither agree nor disagree, 10 = strongly agree) please rate each of the following aspect concerning the current supervision activities/ practices. 3.3. Focus groups

2.2.4. Enhancement of the agency supervision activities What is necessary/desirable for enhanced supervision activities/practices to be more effective at the agency?

A total of six focus groups took place. There were three focus groups with coordinators as well as three focus groups with front line staff. In order to attain an accurate representation, coordinators and front line staff from different programs within each participating service area were included in these group discussions. Here is the distribution per sector: (1) Mental Health: six coordinators and ten front line staff; (2) Youth Justice: four coordinators and eight front line staff; and (3) Community Services: seven coordinators and seven front line staff. Semistructured questions guided the focus group meetings (e.g. ‘to what extent have the supervision activities/practices impacted the work you do with your clients?’, ‘what is necessary/desirable for enhanced supervision activities/practices to be useful for front line staff?’).

3. Methods

3.4. Interviews with key informants

A mixed-method approach was used, because it offers a broader landscape and more nuances for interpretation of data. It allows for subtle details to be outlined through qualitative methods, while the use of quantitative methods permits the identification of group trends. As a result, the choice of a mixed-method approach helps to triangulate the information obtained in the data collection process. Indeed, the following evaluation findings, conclusions and recommendations for organizational development are supported through at least three different sources of information.

Semi-structured interviews were conducted with the Directors and Program Managers of each participating service area as well as with the Director of Human Resources, the Associate Executive Director, and the Executive Director of the agency. Interview questions aimed to obtain information regarding employee needs in terms of supervision, challenges met by employees in terms of supervision, and what they perceived as the main strengths of their current supervision activities/practices. 3.5. Interview and focus group analytical strategy

3.1. Document and file review A file review of the following agency documents was conducted: the agency evaluation framework; the agency logic model; organizational chart; job descriptions; performance evaluation form; supervision form; and behavioral competency

Audio-recordings of group discussions and interviews were analyzed following Thomas’ (2006) inductive approach, an analytical strategy adapted for the needs of program evaluation. Such a strategy has three purposes: (1) to condense extensive raw data into a summary format; (2) to link the data with the research

54

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

objectives; and (3) to produce reliable and valid findings by following a systematic set of procedures. As is common practice, only themes discussed by more than one participant were retained for presentation in Section 4. However, the evaluation team noted a high level of consistency of key findings across levels of respondents (i.e., front line staff, coordinators, and directors) and across service areas. This was confirmed through post hoc analyses of the coding completed using NVivo 10 (QSR International, 2012). Data analysis was completed in three stages. First, all audiorecordings were transcribed verbatim and downloaded into a single database in NVivo 10 and then analyzed using an inductive approach. Second, two members of the team organized the information by evaluation question. Information that was not relevant to the question (e.g., chatting among respondents, anecdotes) was disregarded. Third, the same team members identified text segments relevant to the research questions and summarized the information by assigning a label (word or concise sentence) to the specific text segment. These labels are called codes. The codes were gathered in a codebook and organized in clusters forming themes. Only the themes discussed by more than one participant are presented in order to focus on more common experiences and views (Thomas, 2006). 4. Results Results are presented below and organized by core evaluation questions. 4.1. What is supervision in the context of the agency employees’ work? Results for this evaluation question were often overlapping, revealing a blurred representation of supervision throughout the agency. Hence, it is possible that the explanations of what a certain type of supervision entailed also related to another type of supervision. In fact, while coordinators and directors as a group mentioned all the following types of supervision, many of these types were not reported by front line staff, suggesting that there may not be a strong consensus across agency employees when it comes to what supervision activities they engage in on a regular basis. In total, fifteen supervision activities were noted by researchers, including weekly/bi-weekly/monthly staff meetings, individual supervision sessions, performance appraisals, administrative supervision meetings, on-the-job training, peer-to-peer supervision, informal supervision conversations, shift change meetings, on-call emergency supervision after hours, email exchanges/phone calls, review of clinical case files, consultation with community experts, reflective practice groups, and audiorecordings of session with agency clients. 4.2. What are the current supervision needs of the agency employees? Most of the front line staff members and coordinators interviewed noted that they do not currently receive enough supervision. Specifically, they wanted it to occur more regularly, probably on a weekly or monthly basis and as needed. Respondents generally agreed that they wanted more feedback, more direction, more strategies to improve their work with their clients, and more knowledge about clinical interventions that have been proven to be effective for their clients’ needs. Some front line staff members reported that some coordinators are mostly available on their cell phone, which is not always ideal, but can also provide more flexibility for ‘as needed support’. They stressed the necessity for the agency to keep in mind that supervision needs vary across departments, but that everybody needs support.

A number of specific supervision needs were identified by directors, coordinators, and staff. These were grouped into four main categories. First, staff members indicated that additional concrete supervision resources were necessary to meet current supervision needs. Respondents reported that, because coordinators were responsible for both administrative and clinical supervision of front line staff, it usually resulted in administrative supervision (i.e., performance appraisals, review of case files) being given priority to the expense of clinical supervision (i.e., individual supervision sessions, on-the-job training). Respondents proposed different ways that additional resources could be used to help support clinical supervision activities. For example, funds could be used to create an assistant director position to be in charge of the administrative tasks and billing issues, leaving a clinical director responsible for clinical supervision. Another related suggestion involved the delegation of some administrative tasks to a front line staff member so that coordinators have time to focus on clinical supervision responsibilities. Additional funds could also help acquire additional resources for staff members to engage in selfdirected professional development, or to hire additional staff so that workload issues could become more manageable within each team. In fact, many staff members felt that this would help free up much needed time for more supervision to occur. Second, respondents noted the need for improvement in the organizational culture relating to supervision competencies/ practices. Front line staff members reported that they were usually not included during committee discussion around potential solutions to supervision issues. They also expressed concern that those who are more vocal may receive greater attention, although they do not always represent the majority of staff members’ opinions. Staff members suggested that the agency should prioritize and value the local knowledge the agency already possesses. For example, staff members proposed that senior staff could be encouraged to mentor less experienced employees. Staff members also expressed a desire for more peer-to-peer supervision to occur in a formal way. Additionally, staff members indicated that supervision practices could be improved by having a more consistent approach across coordinators and sharing a common language in regards to supervision and performance appraisals. Respondents surmised that this would help clarify expectations and help to orient part-time staff members who may not see coordinators frequently. Third, respondents discussed improving communication more broadly within units and across programs. For example, front line staff members suggested that it might be helpful if coordinators had access to the client record management system and learned how to use it to improve their supervision of staff. At the same time, coordinators expressed a desire to share supervision strategies between coordinators to learn about best practices within the agency. Fourth, a majority of staff members also expressed a need for more training to occur on a regular basis, to help them build and maintain professional competence as well as a sense of belonging to their team. Finally, staff members talked about introducing benchmarking around what is needed in terms of clinical supervision, having a clearer definition for this activity, and distinguishing it from the other activities/responsibilities of their supervisor. 4.3. What are the challenges met by coordinators and front line staff in terms of supervision? Directors, coordinators, and front line staff all highlighted similar challenges, suggesting that there is a good understanding throughout the agency of the challenges met by coordinators and front line staff in their work. Specifically, seven challenges were

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

identified during focus group meetings and key informant interviews: informal and sporadic supervision, poor staffing ratios, overload of responsibilities, lack of guidance and training on how to supervise effectively, lack of positive reinforcement in daily work activities, and an absence of a common definition of ‘supervision’. Respondents noted that the unstable nature of the agency clientele makes it difficult to conform to a pre-established plan. Supervisors have many supervisees to monitor, making it laborious to schedule a common time for supervision to occur, as front line staff have limited periods that they can be ‘‘off the floor’’. Furthermore, supervisors expressed that it is a real challenge to know all of their staff, especially the part-time employees, the emergency staff, and those who only work the night shifts. It is not uncommon for a supervisor to not have seen some of these employees following the hiring for months at a time. Hence it becomes challenging to assess their work and, as a result, very little valuable feedback is provided. Additionally, it was reported that because of funding shortages for outside consultation and other types of support and training, the workload has become heavier for supervisors. Many staff members reported being unable to keep up with the high demands, leading to a leave of absence, which added up to the already existing stress of coordinators who have to find replacements for them. Coordinators also noted that they were tasked with supervision responsibilities for which they had not received training and may not feel adequately equipped to offer. On the other hand, front line staff noted that supervisors infrequently provided reinforcement for the work accomplished within each service area. In fact, it was their sentiment that service areas or programs primarily receive attention from the direction when problems arise. Respondents also reported that there was little common ground regarding the conceptualization of supervision within the organization, making it arduous to evaluate and enhance supervision-related activities. Their conceptualization of supervision within their own service area often showed an absence of common language, which may reflect differences in expectations and/or perceptions of what supervision looks like in their day-to-day activities. 4.4. To what extent have supervision activities impacted the work front line staff members do with the agency clients? In general, respondents believed that current supervision activities impacted the work they do with their clients, both in positive and negative ways. Front line staff reported that supervision gives them the confidence they need if they are uncertain about what to do in a given situation. For example, some front line staff work with high-risk youths and benefit from being able to consult as needed, especially in case of an emergency (i.e. on-call system). At the same time, some other front line staff reported that they do not believe that the supervision they receive currently impacted their work with youths. These respondents indicated that often their coordinator was not aware of the whole case, so they could only give general guidance to them. Others reported that supervision would have a greater impact if it occurred more frequently. Finally, staff members recognized that when they were able to receive supervision in a timely manner, it did generally help them feel better about their work, reduce their stress, and improve their sense of efficacy. However, they noted that supervision did not occur frequently enough to have a consistent impact on their work with the agency clients. 4.5. How have the supervision practices impacted employee wellness and empowerment? Two opposite trends emerge from the focus groups and key informant interviews. First, some respondents reported that

55

supervision practices were supportive in stress management. Several front line staff reported that coordinators were generally available when needed, especially in crisis situations. These informal conversations helped front line staff feel more confident when accomplishing their work which, according to them, also translated into improved well-being (e.g., fewer sick days). In other words, the supervisor’s availability was conducive to front line staff wellness and empowerment because there was shared accountability and collaboration. Several front line staff members also described supervision activities as a space to thrive through the respectful and trusting environment built between supervisor and supervisee. These supervisees felt that their supervisor recognized their potential and was supportive in their interactions. On the other hand, the majority of front line staff interviewed reported that there were instances where they found current supervision practices stressful. They identified that these instances were related to formal sit-down individual supervision activities. When these activities took place, most of these supervisees did not feel comfortable with their supervisor and avoided openly expressing themselves. Several respondents noted that, in individual meetings with supervisors, the emphasis tended to be placed on negative and critical feedback about their work. Supervisees also expressed that their knowledge and opinions were not solicited by their supervisor. Finally, respondents noted that these meetings were often rushed and that it felt like there was generally not enough time for supervision. They described these exchanges as reflecting a ‘Let’s just get this over with’ mindset, which may contribute to an already-tense supervision context. 4.6. Satisfaction with current supervision activities/practices In total, fifty employees participated in the online survey (seven coordinators, 43 front-line staff). Most of the respondents worked full-time (84%), and the proportion from each service area was similar to their relative importance within the agency overall staff. Generally speaking, full-time staff reported receiving a wider variety of supervision activities than part-time staff. Table 1 presents the aggregated results combining all respondents’ feedback about the agency’s current supervision practices. Both full-time and part-time employees reported receiving more unplanned supervision than planned supervision overall, which is consistent with the previously reported results concerning the lack of regularity in supervision activities across service areas. With regard to satisfaction, results of the survey for both groups also converged in the same direction. Part-time and full-time staff mean scores showed that employees were generally moderately satisfied with the current supervision practices. However, results also show wide variance in responses, with most questions yielding large standard deviation scores. While the results were mostly situated at the mid-point of the satisfaction scale, indicating that employees were not generally dissatisfied with current supervision practices, these variations in responses indicate that staff, whether full-time or part-time, also report very different experiences relating to supervision activities. In particular, questions relating to the frequency and timely nature of supervision practices tended to score lower and to present with larger standard deviations from the mean score. The higher scores related to the quality of supervision interventions, which denotes that when supervision does occur, it is considered to be generally useful and effective. Focus group participants noted a number of strengths that contribute to the efficacy of current supervision practices. In fact, there appeared to be a general consensus that, regardless of the current needs for improvement to supervision practices, agency employees reported a generally good level of success in their work,

56

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

Table 1 Satisfaction with agency supervision practices/activities. On a scale of 0–10 (0 = strongly disagree, 5 = neither agree nor disagree, 10 = strongly agree) please rate each of the following aspect concerning the agency’s current supervision activities/practices

Full time staff Mean score (s.d)

Part time staff Mean score (s.d)

All staff combined Mean score (s.d)

The supervision I receive helps me be timely (i.e. occurs when I need it) The supervision I receive helps me be effective in my work The supervision I receive fosters a safe and collaborative relationship with my supervisor The supervision I receive is supportive of my skills and competencies The supervision I receive offers guidance that helps me in my day-to-day tasks The supervision I receive helps with my professional development (i.e. developing skills/competencies necessary to do my work more effectively in the future) The supervision I receive is consistent The supervision I receive is anchored in best practices The supervision I receive is anchored in a competent understanding of my professional and ethical responsibilities and those of my supervisor The supervision I receive provides useful evaluative feedback Thinking of your clients’ needs, are the supervision activities/practices frequent enough? Thinking of your clients’ needs, are the supervision activities/practices timely enough? Thinking of your clients’ needs, are the supervision activities/practices structured and managed effectively (i.e. supervision time is used effectively)? Thinking of your clients’ needs, are the supervision activities/practices focused on addressing the core topics/issues that matter to your work? Thinking of your clients’ needs, are the supervision activities/practices of good quality?

4.07 (3.36)

5.50 (3.60)

4.50 (3.45)

5.21 (3.17)

6.25 (3.41)

5.60 (3.23)

4.85 (3.29)

5.25 (3.69)

4.97 (3.37)

5.57 (2.94)

5.83 (3.80)

5.65 (3.18)

4.75 (3.15)

5.81 (3.45)

5.05 (3.23)

4.7 (3.21)

5.08 (3.57)

4.80 (3.28)

3.89 (3.25) 4.57 (3.10)

4.16 (3.71) 6.0 (4.0)

3.97 (3.35) 5.02 (3.42)

4.57 (3.08)

6.6 (3.83)

5.05 (3.35)

4.35 (3.06)

5.08 (3.60)

4.65 (3.19)

3.81 (3.07)

4.0 (3.61)

3.87 (3.20)

3.84 (3.05)

4.50 (3.84)

4.05 (3.28)

4.11 (3.22)

4.41 (3.55)

4.20 (3.28)

4.74 (2.90)

5.4 (3.34)

4.90 (3.01)

4.70 (2.95)

5.83 (3.51)

5.0 (3.13)

with positive client outcomes. Respondents reported that these positive results were attributable to three main organizational strengths. First, respondents reported a strong belief in the organizational competence to effectively connect with marginalized, vulnerable youth who have complex needs as well as a strong capacity to collaborate with community partners to affect positive outcomes with agency clients. Second, respondents voiced a strong commitment to what they recognized as organizational core values (i.e. accountability, engagement, empowerment, collaboration, diversity, and accessibility) and how these values provide a framework to the services they provide and the way they relate to their colleagues. Finally, respondents reported pride in their team’s efficacy. Namely, they described their teams as committed, experienced, engaged, motivated, responsible, and supportive. 4.7. What is necessary to enhance current agency supervision activities? To enhance supervision activities, five components were identified from the focus groups and key informant interviews. First, respondents encouraged the agency to capitalize within.

Directors, coordinators, and front line staff all emphasized the importance of maximizing competencies, strengths, expertise, knowledge, tools, and resources within the organization. Three particular ways were identified for this to occur. First, it was recommended that supervisory relationships be framed as a more collaborative endeavour. Second, when individual staff members receive training, there should be opportunities for these members to disseminate their new knowledge to colleagues within the agency. Third, senior agency staff members hold a wealth of expertise and various professional experiences, which could be of benefit other staff members. As such, the organization of group meetings where they can share their knowledge in the context of specific case presentations or they can provide feedback to colleagues on ongoing cases might be valuable. These three strategies all relate to mentorship and its impact on organizational development; coworkers learn to turn to appropriate resources within the organization, which then prevents isolation and increases the quality of the work performed by all staff members. Second, respondents stated the necessity of developing a common conceptualization of supervision. Consistent with other results presented thus far, most respondents commented on the

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

need for the agency to develop a common language concerning supervision and what it entails for each sector. Key findings revealed that the current issue is as much a lack of a common conceptualization, and also that it is not explicit throughout the agency. More attention needs to be devoted to creating and implementing an agency-wide definition of supervision, what it consists of, and how it is to be accomplished. Creating a document detailing this supervision framework that can be circulated throughout the organization could be a first step in this direction. Third, respondents stated that their agency should prioritize more structured and regular clinical supervision. In order to improve current supervision practices, they must become meaningful to all members of the agency by applying the following conditions: (1) supervision must happen regularly and follow-ups must be done as appropriate (i.e., if a meeting is cancelled, it is rescheduled); (2) during supervision, the clinical portion of the work is prioritized since clinical skills are what will make a difference with youth; and (3) a consistent framework must guide the organization’s supervision process. Fourth, respondents suggested separating clinical and administrative supervisions and fostering a more collaborative approach to clinical supervision. To do so, it was suggested that the agency create two positions per sector, one for a clinical director and another for an administrative director. The clinical director could be an expert who has strong clinical skills and a solid academic background to assist the agency members with clinical issues. On the other side, an administrative director could manage such responsibilities as budget, billing issues, and scheduling. Currently, supervision practices follow a hierarchical approach where feedback is provided from the top down. It was suggested that everyone receive feedback, including supervisors, and that a more collaborative approach to supervision be followed. Finally, respondents stressed the need for an organizational culture shift regarding supervision. Respondents discussed different ways in which this could be accomplished, such as sharing accountability within the team and between supervisors and supervisees, fostering a safe and respectful supervision environment, encouraging supervisors to complete self-assessments of their supervision practices and to engage in professional development activities, and increasing the focus on teamwork within supervision practices. This translated in their comments relating to increasing the sharing of resources and tools, focusing on strengthbuilding in supervisory feedback, developing staff members skills, increasing their understanding of what is needed to grow both individually and as an organization, and how to do so. 5. Recommendations for the development of enhanced clinical supervision practices The first goal of supervision is to ensure that clients receive quality services from their service provider. The second goal of supervision is to contribute to the professional development of supervisees so that they can learn to work competently and autonomously, while consulting their supervisor as needed to maintain efficacy in service provision. In this context, the supervisor acts as much as a gatekeeper (i.e., ensuring that services meet professional/ethical standards) and a mentor toward the supervisee (i.e., supporting the development and maintenance of professional competencies). The following recommendations were made by the research team to the agency, and were aimed at helping the agency develop a competency-based supervision culture that adopts a collaborative perspective toward supervision activities and practices. Furthermore, the recommendations formulated here are not agency-specific. In fact, these recommendations could serve as a framework for other community organizations to implement a successful competency-based

57

approach to clinical supervision. While recommendations have been presented separately, starting from those that are focused on the agency as a whole toward more specific areas/levels, and organizational levels (senior management – directors, middle management – coordinators, and front line staff), this should not be interpreted as a ‘top-down’ approach. In fact, the highest level of success will be achieved if these different recommendations are implemented concurrently. Additionally, the research team stressed the necessity of encouraging opportunities for groupbased supervision whenever it was appropriate/possible, as these have great potential to help develop and sustain a supervision culture that is collegial and collaborative, where peer mentorship can occur naturally. 5.1. Agency-wide recommendations 1. Senior and middle management: Create an agency-wide supervision policy through the development of a supervision manual that: a. Defines the scope of this professional activity, b. Establishes guidelines on how best to conduct this professional activity (alternatives based on setting specific contingencies and supervisee needs), c. Develops basic standards in terms of time and resources devoted to this professional activity (in line with current professional standards); and ensure that appropriate resources are in place within each service area. 2. Senior management, middle management, and front line staff: Provide the agency-wide training on competency-based supervision so both supervisors and supervisees understand the model. 3. Senior, middle management and front-line staff: Create a training committee that can: d. Identify future training needs within each service sector, e. Plan ongoing in-house and community-based training workshops based on feedback from the agency service providers and coordinators, f. Organize regular opportunities to disseminate knowledge within the agency about practices in other service areas (e.g., brown-bag or pizza lunches that feature best practices from an agency program to agency employees from other programs), g. Maintain an online repository of training resources available to all employees, regardless of their service area or work schedule, h. Manage the supervision review process (see next recommendation). 4. Senior and middle management: Create a confidential and anonymous supervision review process: i. That utilizes standard forms across service area, j. That happens on a regular basis (e.g., more than once a year), k. To gather data on the quality of supervision practices to inform future policy changes and program reviews; l. To provide aggregated feedback to supervisors about their supervision practices and competencies.

5.2. Service-area specific recommendations 1. Middle management, and front line staff: Create a committee to help develop updated competency benchmark within each sector so everyone has a clear sense of what is expected and what they should focus on: a. this committee should have representatives from both coordinators and front line staff, as well as the quality assurance representative,

58

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

b. one member of this committee should also serve on the training committee outlined above to ensure consistency between agreed-upon competency benchmarks within each sector area and training activities planned by the agency, c. review current supervision responsibilities accomplished by coordinators and create service-specific standards (duration, frequency, structure, and focus) for clinical and administrative supervision activities that can help to inform the development of the agency-wide supervision policy and associated supervision manual. 2. Middle management, and front line staff: Support peer supervision activities initiated by employees and encourage peer mentoring, by providing staff members with resources on structuring these activities: a. To accomplish this, create guidelines for these activities that are available to all the agency employees i. to make more explicit how someone who would like to mentor or who would like to be mentored can proceed, and ii. to clarify what the role of mentor is in comparison to the role of supervisor. 5.3. Senior and middle management/supervisor specific recommendations 1. Implement the agency-wide competency-based supervision framework to structure clinical supervision activities: a. Use service-specific competency benchmarks to collaboratively set professional development objectives with supervisees; monitor objectives for both supervisee and client outcomes; and use this information as part of performance evaluations. 2. Implement supervision contracts to help clarify roles and responsibilities within the supervisory relationship: a. Provide regular performance feedback to supervisees (i.e., ongoing evaluation) on both how supervisees are contributing to client outcomes and how they are meeting their professional development objectives; formal performance evaluations should not be the only time supervisees receive feedback on their work, b. Solicit regular feedback from supervisees about how supervision is meeting supervisee and client needs, and encourage supervisees to fill out regular evaluation forms on supervisor activities and practices. 3. Encourage supervisors to provide the agency training committee with regular feedback regarding training needs within their service area (through their representative on the training committee).

5.4. Front-line staff/supervisee-specific recommendations 1. Familiarize front line staff members to the competency-based supervision framework, 2. Encourage front line staff members to participate in the development of their supervision contract that outlines roles and responsibilities of supervisors and supervisees, 3. Encourage front line staff members to review servicespecific competency benchmarks and prepare a professional development plan to be shared with their supervisor regarding expectations and objectives of supervision activities, 4. Encourage front line staff members to participate in regular supervision review process implemented by the agency, 5. Encourage front line staff members to provide the agency training committee with regular feedback regarding training needs within their service area (through their representative on the training committee).

6. Discussion The purpose of the evaluation was to identify the needs and challenges of coordinators and front line staff, assess the efficiency of the supervision practices, and evaluate the supervisors’ and supervisees’ satisfaction with the current practices within a community-based agency serving youth. These findings formed the basis for the research team to formulate recommendations to the agency for the development of enhanced supervision practices across these three service areas. The evaluation completed by the evaluation team generated a large amount of data, which was analyzed using an inductive approach. Key findings indicated a number of important points that must be taken into consideration in defining enhancements for supervision practices at the agency. First, respondents reported a wide array of supervision activities across service areas, including individual supervision, staff meetings, performance appraisals, case management, training, crisis management, and peer supervision practices. Respondents also reported using a number of technological supports within supervision activities, including audio-recordings of sessions with clients, email, and phone support. This diversity in supervision practices appeared to reflect the complexity and variety of service settings, which would require flexible supervision practices and activities that are adapted to setting-specific contingencies. Second, respondents identified two core needs related to supervision practices across service settings at the agency: (1) Respondents from all levels reported a clear need for additional resources to improve the quality of supervision practices. (2) Additionally, respondents noted a need for the agency to develop a more competency-based supervision culture.

These core needs are not unique to the agency, and have been reported by other researchers who completed a meta-analysis of research articles regarding the impact of supervision on worker outcomes (Mor Barak et al., 2009). In their review, the authors found that effective supervision relied on three supervisory dimensions to ensure a positive work environment that contributes to worker effectiveness and quality service delivery: task assistance, social and emotional support, and interpersonal interaction. Of the three dimensions, task assistance had the greatest impact on positive staff outcomes, while supportive supervision and the quality of the supervisory alliance were associated with a reduction in staff-related issues (e.g. anxiety, stress, depression, somatic complaints, burnout, intention to leave and turnover). Given the reality that most agencies who provide mental health services to the population generally receive little on-the-job supervision, while having to contend with an increasingly complex workplace (multidisciplinary teams addressing increasingly complex client presentations), the need for regular quality supervision, competency-based supervision training and interdisciplinary competency training are paramount (Bogo et al., 2011; Hoge et al., 2011; Schoenwald et al., 2009). However, how this can be achieved remains a challenge. Most supervisors do not benefit from having received training in clinical supervision while they were completing their studies, and professional training in supervision remains scarce in most agencies (Milne & Reiser, 2012). Some have argued that a common factors view of supervision processes may represent the best model to address the complexities of supervising in the workplace, but what factors should be part of this approach and how to train supervisors remains a subject of some debate (Lampropoulos, 2002; Milne & Reiser, 2012).

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

Following this evaluation, the team put forth a series of recommendations to enhance current supervision practices at the agency, and in similar community organizations. We believe that the proposed framework could also be useful to other community organizations facing similar needs and challenges, because while providing a number of specific suggestions, it remains generally applicable to most community service providers. In particular, special attention was paid to including both supervisors and supervisees in the development of supervision training, to contribute to a more collaborative approach to supervision. In addition, the development of competency benchmarks that are specific to each service area was prioritized in order to create a supervision framework that is adapted to particular work contexts. It was also recommended that the agency develop a supervision review process, that is both anonymous and confidential, to create a safe space for both supervisors and supervisees to comment on supervision practices within the agency. The use of technologies, such as online surveys, could be useful and relatively affordable to implement in this process.

59

services accomplish their work mostly in a ‘drop-in’ context where teenagers come in crisis for immediate help and resources, while youth justice offers most of its services to clients that have been remanded in their care. Therefore, the challenge in completing this evaluation and developing recommendations for the agency resided in finding commonalities across service areas on which the agency can build a global supervision policy, while ensuring mechanisms can be put in place to create supervision models and processes that respect the idiosyncrasies of each service area. This evaluation process also underlined the importance of agency leadership for implementing an efficacious and coherent approach to data collection, and facilitating the research team’s work. Directors in each service area personally organized the team’s visits to each of their location, escorted the research team in a visit of their offices, and strongly encouraged employees to participate in the evaluation process, by giving them time to complete the online survey and participate in the focus groups. This ensured an overall good level of participation and helped the research team gain a richer understanding of the challenges and needs faced by the agency employees.

7. Lessons learned Acknowledgements Through the mixed-method approach, we gleaned a number of lessons. Obtaining a suitable response rate and thus a representative sample of stakeholders was challenging. The agency employs a large number of employees, most of whom work part-time. Additionally, the agency employs both Francophone and Anglophone individuals. As a result, we needed to ensure that all data collection material was available in both languages. We also needed a proactive and engaging strategy to encourage employees to participate in both the online survey and the focus groups. The response rate for the online staff survey was approximately 20%. Despite following recommended strategies (i.e. an introductory employee memo, a personalized invitation sent by email to each employee, email reminders), we still experienced a low rate of completed participations. We understand that these results are partly due to the large representation of part-time staff, many of whom work evenings, nights and weekends, and do not have regular contact with a supervisor. These same strategies proved more effective in recruiting participants for the focus groups. Many employees arrived at the focus group meetings with notes providing feedback from other employees who could not attend. This provided valuable information to include in our data analysis. Indeed, the focus group meetings and key informant interviews helped to contextualize survey information, as well as enrich the research team’s understanding of complex organizational dynamics relating to supervision. It also enabled to team to ensure a better representation of staff feedback from all programs and services within the agency, given the relatively low completion rate of the online staff survey. The final group of respondents for the online survey, interviews, and focus groups come from a wide range of professional backgrounds and they work for a variety of programs within the agency, making the results harder to interpret on an agency-wide level. Indeed, this limitation is illustrated in the lack of a common understanding of what clinical supervision constitutes and how it can be best exercised. This also likely reflects the complexity of exercising clinical supervision in real-world settings. The agency includes a number of disciplines working together in a variety of programs housed within three main services areas. Each of these areas (e.g. mental health, community, and youth justice) present with a particular work context that may bear little resemblance to the work context in the other two service areas. For example, mental health services mostly work by appointment with either teenagers or families who come in for counseling. Community

We would like to thank the agency and especially the directors, coordinators, and all the front line staff that we met during the evaluation process for the generosity and engagement in this project. The views expressed herein solely represent the authors. References Ballantyne, C. (2003). Measuring quality units: Considerations in choosing mandatory questions. Adelaide: Paper presented at the Evaluations and Assessment Conference: A Commitment to Quality, University of South Australia 24–25 November. Baker, S. B., Exum, H. A., & Tyler, R. E. (2002). The developmental process of clinical supervisors in training: An investigation of the supervisor complexity model. Counselor Education and Supervision, 42, 15–30. http://dx.doi.org/10.1002/j. 1556-6978.2002.tb01300.x Bernard, J. M. (2005). Tracing the development of clinical supervision. In D. L. Shulman & A. Safyer (Eds.), Supervision in counseling: Interdisciplinary issues and research (pp. 3–21) http://dx.doi.org/10.1300/J001v24n01_02 Bernard, J. M., & Goodyear, R. K. (2013). Fundamentals of clinical supervision (5th ed.). Pearson Education Inc. Bogo, M., Patterson, J., Tufford, L., & King, R. (2011). Interprofessional clinical supervision in mental health and addiction: Toward identifying common elements. The Clinical Supervisor, 30, 124–140. http://dx.doi.org/10.1080/07325223.2011.564961 Borders, L. D., Rainey, L. M., Crutchfield, L. B., & Martin, D. W. (1996). Impact of a counseling supervision course on doctoral students’ cognitions. Counselor Education and Supervision, 35, 204–217. http://dx.doi.org/10.1002/j. 1556-6978.1996. tb00224.x Canadian Interprofessional Health Collaborative (2010). A national interprofessional competency framework. Vancouver, Canada: College of Health Disciplines, University of British Columbia. D’Amour, D., Ferrada-Videla, M., Rodrigues, L. S. N., & Beaulieu, M.-D. (2005). The conceptual basis for interprofessional collaboration: Core concepts and theoretical frameworks. Journal of Interprofessional Care, 19(S1), 116–131. http://dx.doi.org/ 10.1080/13561820500082529 Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R. V., Thorpe, K. E., & Perrier, L. (2006). Accuracy of physician self-assessment competencies compared with observed measures of competence: A systematic review. Journal of the American Medical Association, 296(9), 1094–1102. http://dx.doi.org/10.1001/jama. 296.9. 1094 Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623. http://dx.doi.org/10.1080/02602930410001689171 Elmann, N. S., Illfelder-Kaye, J., & Robiner, W. N. (2005). Professional development: Training for professionalism as a foundation for competent practice in Psychology. Professional Psychology: Research & Practice, 36(4), 367–375. http://dx.doi.org/ 10.1037/0735-7028.36.4.367 Falander, C. A., & Shafranske, E. P. (2004). Clinical supervision: A competency-based approach. Washington, DC: American Psychological Association. Falander, C. A., & Shafranske, E. P. (2012). Getting the most out of clinical training and supervision: A guide for practicum students and interns. Washington, DC: American Psychological Association. Fouad, N. A., Grus, C. L., Hatcher, R. L., Kaslow, N. J., Hutchings, P. S., Madson, M. B., et al. (2009). Competency benchmarks: A model for understanding and measuring competence in professional psychology across training levels. Training and

60

J. Gosselin et al. / Evaluation and Program Planning 52 (2015) 50–60

Education in Professional Psychology, 3(4-suppl.), S5–S26. http://dx.doi.org/ 10.1037/a0015832 Giddings, M. M., Cleveland, P. H., & Smith, C. H. (2006). Responding to inadequate supervision: A model promoting integration for post-MSW practitioners. The Clinical Supervisor, 25(1–2), 105–126. http://dx.doi.org/10.1300/J001v25n01_08 Goodyear, R. K., & Bernard, J. M. (1998). Clinical supervision: Lessons from the literature. Counselor Education & Supervision, 38, 6–22. http://dx.doi.org/ 10.1002/j. 1556-6978.1998.tb00553.x Goodyear, R. K., Bunch, R., & Claiborn, C. D. (2005). Current supervision scholarship in psychology: A five year review. In L. Shulman & A. Safyer (Eds.), Supervision in counseling: Interdisciplinary issues and research (pp. 137–147). New York: Routledge http://dx.doi.org/10.1300/J001v24n01_07 Gosselin, J., Barker, K., Kogan, C., Pomerleau, M., & Pitre-D’Iorio, M.-P. (in press). Using a ‘best evidence synthesis’ approach to build an evidence-based model of psychotherapy supervisor development. Canadian Psychology. Haley, S. J. (2002). The influence of supervision training on supervisor self-efficacy among doctoral interns at university counseling centers. Haley, Sarah Jane: West Virginia U, US. ProQuest Dissertations and Theses database. (725930541). Hatcher, R. L., & Lassiter, K. D. (2007). Initial training in professional psychology: The practicum competencies outline. Training and Education in Professional Psychology, 1, 49–63. Heppner, P. P., & Handley, P. (1982). The relationship between supervisory behaviors and perceived supervisor expertness, attractiveness, or trustworthiness. Counselor Education and Supervision, 22(1), 37–46. http://dx.doi.org/10.1002/j. 15566978.1982.tb00928.x Hoge, M. A., Migdole, S., Farkas, M. S., Ponce, A. N., & Hunnicutt, C. (2011). Supervision in public sector behavioral health: A review. The Clinical Supervisor, 30, 183–203. http://dx.doi.org/10.1080/07325223.2011.604276 Kavanagh, D. J., Spence, S., Sturk, H., Strong, J., Wilson, J., & Worrall, L. (2008). Outcomes of training in supervision: Randomised controlled trial. Australian Psychologist, 43, 96–104. http://dx.doi.org/10.1080/00050060802056534 Kennard, D., Stewart, M. S., & Gluck, R. M. (1987). The supervision relationship: Variables contributing to positive versus negative experiences. Professional Psychology: Research and Practice, 18(2), 172–175. http://dx.doi.org/10.1037/07357028.18.2.172 Kilminster, S. M., & Jolly, B. C. (2000). Effective supervision in clinical practice settings: A literature review. Medical Education, 34, 827–840. http://dx.doi.org/10.1046/j. 1365-2923.2000.00758.x Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A metaanalysis. Clinical Psychology: Science and Practice, 10(3), 288–301. http://dx.doi.org/ 10.1093/clipsy.bpg025 Lampropoulos, G. (2002). A common factors view of counseling supervision process. The Clinical Supervisor, 21(1), 77–94. Lyon, R. C., Heppler, A., Leavitt, L., & Fisher, L. (2008). Supervisory training experiences and overall supervisory development in predoctoral interns. Clinical Supervisor, 27, 268–284. http://dx.doi.org/10.1080/07325220802490877 McMahon, M., & Simons, R. (2004). Supervision training for professional counselors: An exploratory study. Counselor Education and Supervision, 43, 301. http://dx.doi.org/ 10.1082/j. 1556-6978.2004.tb01854.x Milne, D., & Reiser, R. P. (2012). A rationale for evidence-based clinical supervision. Journal of Contemporary Psychotherapy, 42, 139–149. http://dx.doi.org/10.1007/ s10879-011-9199-8 Mor Barak, M. E., Travis, D. J., Pyun, H., & Xie, B. (2009). The impact of supervision on worker outcomes: A meta-analysis. Social Service Review, 83(1), 3–32. Nair, C. S., Wayland, C., & Soediro, S. (2005). Evaluating the student experience: A leap into the future. Sydney: Paper presented at the 2005 Australasian evaluations forum:

University learning and teaching: Evaluating and enhancing the experience, UNSW 28–29 November. Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33(3), 301–314. http:// dx.doi.org/10.1080/02602930701293231 Ogier, J. (2005). The response rates for online surveys—A hit and miss affair. Sydney: Paper presented at the 2005 Australasian evaluations forum: University learning and teaching: Evaluating and enhancing the experience, UNSW 28–29 November. Proctor, B. (1994). Supervision competence, confidence, accountability. British Journal of Guidance & Counselling, 22, 309–318. http://dx.doi.org/10.1080/03069889408 253676 QSR International Pty Ltd. (2012). NVivo qualitative data analysis software, Version 10. Schoenwald, S. K., Sheidow, A. J., & Chapman, J. E. (2009). Clinical supervision in treatment transport: Effects on adherence and outcomes. Journal of Counseling and Clinical Psychology, 77(3), 410–421. http://dx.doi.org/10.1037/a0013788 Stevens, D. T., Goodyear, R. K., & Robertson, P. (1998). Supervisor development: An exploratory study in changes in stance and emphasis. Clinical Supervisor, 16, 73–88. http://dx.doi.org/10.1300/J001v16n02_05 Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluations, 27, 237–246. http://dx.doi.org/10.1177/ 1098214005283748 Vidlak, N. W. (2002). Identifying important factors in supervisor development: An examination of supervisor experience, training, and attributes. Vidlak, Nicole White: U Nebraska - Lincoln, US. ProQuest Dissertations and Theses database. (727413951). Watkins, C. E. (1995). Psychotherapy supervision in the 1990: Some observations and reflections. American Journal of Psychotherapy, 49, 568–581. Webb, A., & Wheeler, S. (1998). How honest do counsellors dare to be in the supervisory relationship? An exploratory study. British Journal of Guidance & Couselling, 26(4), 509–524. http://dx.doi.org/10.1080/03069889808253860 Wheeler, S., & Richards, K. (2007). The impact of clinical supervision on counsellors and therapists, their practice and their clients: A systematic review of the literature. Counselling and Psychotherapy Research, 7(1), 54–65. http://dx.doi.org/10.1080/ 14733140601185274 Julie Gosselin is a clinical psychologist and an Associate professor at the University of Ottawa. She is also a principal researcher for the Centre for Research on Educational and Community Services (CRECS). Her areas of research include psychosocial adjustment in vulnerable families, and clinical supervision processes in mental health services. Sophie-Claire Valiquette-Tessier is a doctoral candidate in experimental psychology at the University of Ottawa. She is currently completing her doctoral thesis on stereotyped attitudes towards parenthood under the supervision of Dr. Gosselin. Marie-Pier Vandette is a doctoral candidate in clinical psychology at the University of Ottawa. She is currently completing her doctoral thesis on clinical supervision doctoral training under the supervision of Dr. Gosselin. Elisa Romano is a clinical psychologist and an Associate professor at the University of Ottawa. She is also a principal researcher for the Centre for Research on Educational and Community Services (CRECS). Her areas of research include children’s emotional and behavioural development and child maltreatment, specifically understanding its impact on children’s development and examining effective ways of intervening with children and families who have experienced maltreatment.