Parallel Processes: Using Motivational Interviewing as an Implementation Coaching Strategy Jennifer E. Hettema, PhD Denise Ernst, PhD Jessica Roberts Williams, PhD, MPH, RN Kristin J. Miller, MSW Abstract In addition to its clinical efficacy as a communication style for strengthening motivation and commitment to change, motivational interviewing (MI) has been hypothesized to be a potential tool for facilitating evidence-based practice adoption decisions. This paper reports on the rationale and content of MI-based implementation coaching Webinars that, as part of a larger active dissemination strategy, were found to be more effective than passive dissemination strategies at promoting adoption decisions among behavioral health and health providers and administrators. The Motivational Interviewing Treatment Integrity scale (MITI 3.1.1) was used to rate coaching Webinars from 17 community behavioral health organizations and 17 community health centers. The MITI coding system was found to be applicable to the coaching Webinars, and raters achieved high levels of agreement on global and behavior count measurements of fidelity to MI. Results revealed that implementation coaches maintained fidelity to the MI model, exceeding competency benchmarks for almost all measures. Findings suggest that it is feasible to implement MI as a coaching tool.
Address correspondence to Jennifer E. Hettema, PhD, Department of Family and Community Medicine, University of New Mexico, MSC 09 5040 ABQ, NM 87131, USA. Phone: +1-505-2722165; Fax: +1-505-2728045; Email: [email protected]. Denise Ernst, PhD, Denise Ernst Training and Consultation, 8695 SW 80th Avenue, Portland, OR 97223, USA. Phone: +1-503-9468320; Fax: +1-804-5274268 Jessica Roberts Williams, PhD, MPH, RN, University of Miami School of Nursing and Health Studies, P.O. Box 248153, Coral Gables, FL, USA. Phone: 571-633-9400; Fax: +1-571-633-9401; Email: [email protected] Jessica Roberts Williams, PhD, MPH, RN, MANILA Consulting Group, Inc., McLean, VA, USA. Phone: 571-633-9400; Fax: +1-571-633-9401; Email: [email protected] Kristin J. Miller, MSW, MANILA Consulting Group, Inc., McLean, VA, USA. Phone: +1-571-6339400; Fax: +1-571-633-9401
)
Journal of Behavioral Health Services & Research, 2013. 1–12. c 2013 Springer Science+Business Media, LLC (outside the USA). DOI 10.1007/s11414-013-9381-8
MI for Implementation Coaching
HETTEMA et al.
Introduction Motivational interviewing (MI) is a collaborative conversation style for strengthening a person's own motivation and commitment to change.1 This approach has been applied to the prevention and treatment of a variety of health conditions with promising findings.2 Like many evidence-based practices (EBPs), despite its strong evidence base, adoption and effective implementation of MI remains suboptimal.3,4 Interestingly, as a tool for helping individuals explore and resolve ambivalence about change, MI itself may be a promising strategy for disseminating successful interventions. This concept was supported by the parent study of the current special edition, which found that, as part of a larger active dissemination strategy, MI-based interactive implementation coaching Webinars are an effective tool for advancing adoption of EBPs.5 The purpose of the current report is to (1) describe the implementation coaching intervention and its consistency with the principles and practices of MI, (2) discuss the feasibility of applying an MI counseling fidelity instrument to implementation coaching sessions, and (3) report on the consistency of the implementation coaching sessions to the MI model.
Description of parent study As part of a study to evaluate the impact of different dissemination strategies on the adoption of EBPs, such as MI, community behavioral health organizations (CBHOs) and community health centers (CHCs) were randomly assigned to one of the following conditions using a nested experimental design that matched organizations on key characteristics: (1) passive dissemination through receipt of informational packets describing MI effectiveness and implementation resources or (2) active dissemination, which included the informational packet plus receipt of a large-group didactic Webinar and an MI-based interactive coaching Webinar. Historically, CHCs specialize in primary care and CBHOs in behavioral health services; however, recent efforts to integrate primary care and behavioral health have resulted in increased crossover and coordination of these services across these two types of organizations. Participants from the CBHOs and CHCs included administrators and front-line providers. For more details on the methodology, see the parent study in this special issue.5 Results of the study revealed that individuals from organizations that received active dissemination, including the MI-based interactive coaching Webinar, were more likely to report decisions to adopt MI, as measured by a stage of change-based question querying interest in and steps toward adopting MI.
Why might MI work well as an implementation coaching strategy? Conceptually, there are many reasons to believe that MI may be an effective dissemination tool. In fact, the Comparative Effectiveness Research Council's goal of developing implementation strategies that “empower patients, clinicians, and other stakeholders to make more informed decisions”6 aligns highly with the spirit and objectives of MI. MI co-developers Miller and Rollnick briefly discuss the application of MI to organizational change in the third edition of their book Motivational Interviewing: Helping People Change (pp. 345–346).1 The developers draw parallels between MI and appreciative inquiry, an organizational development method focused on developing and building on strengths.7 While Miller and Rollnick primarily focus on ways in which MI practices may be useful tools for managers and supervisors to motivate people within a workplace, the approach seems similarly applicable to consultative relationships involving implementation coaching. In MI, focus is shifted from the traditional model in which motivation must be instilled and solutions must come from an outside content expert, to a model in which
The Journal of Behavioral Health Services & Research
2013
empathy and evocation guide organizations to talk themselves into changing, and solutions are elicited from the system itself. MI and diffusion of innovations Several models are available to explain, predict, and promote dissemination of EBPs. While a variety of complex and interacting systems-level, economic, and external factors likely impact implementation, many models, including Rogers' Diffusion of Innovations,8 focus largely on the role of the individual in making decisions about adoption. Within Rogers' model, diffusion is believed to take place in five stages: knowledge, persuasion, decision, implementation, and confirmation. The model has been cited as roughly paralleling the transtheoretical stages of change.9,10 Conceptually, there is also an overlap between the stages of diffusion and the principles and practices of MI. Consistent with the aims of the parent research project that is the focus of this special issue, MI may be theorized to affect decision making during the knowledge and persuasion stages. In Rogers' first stage, knowledge, potential adopters become aware of an innovation. Rogers believed that a variety of factors impact the knowledge stage, including communication behavior. As a communication strategy itself, MI provides guidance on when and how much information to provide, as well as the style in which information is provided. In MI, there is an emphasis on the tendency to overestimate how much information or direction actually needs to be communicated to others. For example, in clinical practice, despite the fact that most patients are aware of the negative health impacts of smoking, providers are often tempted to take the expert role and lecture patients about the dangers of smoking and strategies to quit. Instead of encouraging patients to engage and explore their behavior, unneeded information provision and unsolicited advice often lead to withdrawal or defensiveness that runs contrary to the promotion of change.1 Similarly, through an MI lens, dissemination efforts may sometimes overemphasize the amount or extensiveness of information needed to promote knowledge and an ultimate decision to adopt. This is supported by research that suggests that years of education are not related to use of EBPs and that front-line providers make very limited use of scientific books and articles, instead learning through informal channels.10 MI may also inform the knowledge stage of diffusion by providing guidance on the style in which information is shared. In MI, communicating information collaboratively and respectfully is thought to maximize attention to and acquisition of information, as well as enhance the application of knowledge to decisions about change. For example, MI interventionists are encouraged to ask permission (and honor the response) before sharing information. Similarly, MI interventionists may use the elicit-provide-elicit technique, whereby they first ask individuals what they know about a topic, then share information in a tailored way, and follow up by asking how, if at all, the information has impacted the individuals' thinking about the topic. In an implementation coaching context, this may take the form of asking providers what they know about the potential benefits of implementing MI in their setting, underscoring these benefits or possibly sharing some new benefits, and then asking how, if at all, these benefits impact decisions about implementation. This technique may similarly be used to discuss knowledge about strategies for implementation. The MI model differs from typical expert-recipient models as it takes advantage of the expertise of providers and organizations and the unique and powerful perspectives they have on what will work in their specific settings. The second stage in Rogers' diffusion model is persuasion, whereby the individual develops positive or negative attitudes toward an innovation. Again, communication strategies and techniques used by disseminating agents can significantly impact persuasion. In MI, it is also acknowledged that the way in which an interventionist interacts with an individual can significantly impact his or her attitudes about change.
MI for Implementation Coaching
HETTEMA et al.
When forming attitudes about change or adoption, individuals often hold ambivalent feelings. There may be aspects of change that are positive and attractive, and, at the same time, there may be concerns about or drawbacks to change. Clinically, a smoker may be aware of the negative health effects of smoking and want to live a long life, and also really enjoy the feeling of smoking and appreciate its weight loss benefits. Similarly, when forming attitudes toward the adoption of MI, providers or organizations may have mixed feelings. While on the one hand a potential adopter may be aware of the strong evidence base for MI and align philosophically with its underlying principles, on the other hand, there may be concerns about the feasibility of implementation, as there is mounting evidence that to attain proficiency with MI, more is needed than initial training, such as follow-up feedback and coaching.11 MI is a type of conversation that is particularly useful in the exploration and resolution of ambivalence about change. While one is often tempted to persuade individuals into changing by trying to convince them of the benefits of change or the drawbacks of staying the same, such strategies typically make others feel angry, defensive, uncomfortable, or powerless. Instead, MI seeks to focus conversation in such a way that individuals talk themselves into changing. Evocation involves the selective elicitation of language that indicates movement toward change (change talk). In MI, specific interventionist behaviors have been found to evoke change talk, which in turn is predictive of positive outcome.12 Such language can include desire to change (e.g., “We want our patients to more effectively manage chronic disease.”), perceived ability to change (“We've successfully implemented other EBPs in the past.”), reasons to change (“MI is consistent with many of our goals as a Patient-Centered Medical Home.”), need to change (“We've got to do something to improve our rates of attendance.”), and commitment to change (“We're sending all of our front-line staff to an introductory MI training.”). By taking a nonjudgmental, empathic stance, MI interventionists encourage the sincere exploration of change. In addition, several strategic approaches are available to draw out, reinforce, and build upon change talk, including asking evocative questions, seeking elaboration, and differentially reflecting or affirming change talk. In Roger's model, the decision stage involves the choice of whether to reject or adopt an innovation. This represents the tipping point, at which time ambivalence about change is resolved in one direction or another. Promoting decisions to implement MI was the primary objective of the tested dissemination strategies in the parent study of this report. In the MI model, the act of decision making becomes the target change for the intervention. This, in turn, provides the interventionists with the goal toward which to guide the conversation and the key with which to evaluate the participant language. The MI communication strategies used in the knowledge and persuasion stages likely impact the probability that individuals will make a decision in favor of change.13 Similarly, engaging in collaborative, detailed planning following an implementation decision can help to consolidate and build commitment for the decision. Applications of MI to coaching While the conceptual links between MI and adoption decision making are clear, there has been little empirical investigation of the impact of MI on dissemination efforts. However, MI has been introduced as a strategy for coaching interventionists to improve their clinical skillfulness. For example, in a randomized trial of methods to help clinicians learn MI,11 providers who received post-training coaching in the spirit of MI were found to have larger gains in proficiency than those in the workshop only condition. Similarly, in the Motivational Interviewing Assessment: Supervisory Tools for Enhancing Proficiency,14 a collection of tools developed to aid organizations in supervising clinicians in the use of MI during clinical assessments that was developed as part of a National Institute on Drug Abuse/Substance Abuse and Mental Health Services Administration blending initiative, specific MI-consistent supervisory strategies are suggested. For example, supervisors are provided with guidelines that encourage them to “practice what you preach” by not
The Journal of Behavioral Health Services & Research
2013
taking the expert role, focusing on what went well, eliciting clinicians' ideas about improvement, and listening carefully to resistance as an opportunity to explore real implementation barriers. In addition to potentially increasing the effectiveness of performance coaching, adopting an MI style has a secondary educational benefit. By modeling MI, potential adopters are exposed to a firsthand application demonstrating how a collaborative communication technique might be used to promote change. While there is limited direct empirical evidence regarding the impact of MI practices in implementation coaching settings, initial evidence appears promising. In an early pilot study, Strang et al.15 provided telephone-based MI to general practitioners to increase their involvement in the care of opiate misusers and observed improvements in attitudinal and behavioral care factors. Similarly, in a small medical education pilot study,16 medical residents were assessed for their selfreported use of and attitudes toward substance abuse screening and brief intervention practices. They were then given normative feedback based on their responses as well as a motivational interview to explore ambivalence about engaging in substance abuse practice, promote engagement in a substance abuse curriculum, and build motivation for change. This study resulted in behavioral and attitudinal improvements in the expected direction. Based on the results of the parent trial, the conceptual applicability of MI to models of diffusion and implementation, and early research, the purpose of the current paper is to describe the content and structure of implementation coaching, determine the feasibility of measuring MI fidelity in implementation coaching, and describe the adherence of the coaching to the MI model.
Methods Interactive implementation coaching Webinars Coaching Webinars were collaboratively developed by the parent trial investigative team and three members of the Motivational Interviewing Network of Trainers (MINT). Two of the three MINT developers, including the first author, went on to conduct the coaching Webinars. Coaches received no explicit training as part of the current trial, but, as members of MINT, had previously received advanced training in MI as a clinical strategy, and had extensive experience training others in MI, supervising others in MI, and providing MI implementation consultation to agencies. Each coach conducted one pilot coaching Webinar prior to the start of the trial for which they received performance feedback from the parent trial investigative team; however, this feedback largely focused on adherence to the developed webinar protocol versus MI skillfulness. Coaching was designed to be consistent with the principles and practices of MI. The Webinars followed a semistructured format that was facilitated by a slideshow presentation and interactive polling software. Webinars began with a brief coach introduction and overview of the Webinar objectives to focus the interaction. Participants were then invited to introduce themselves, tell a little bit about their professional role and their organization, and describe any past experience with MI. Coaches actively listened, attending differentially to participant change talk, engaging participants, and developing rapport. Importance, confidence, and readiness rulers were then used by coaches to strategically evoke and respond to change talk regarding the implementation of MI. First, an importance ruler was presented, which asked participants to rate the importance of implementing MI on a scale from 1 to 7 using interactive polling software. Participants were then asked the evocative question: “Why did you select an X, instead of something lower like a Y?” To further build motivation for implementation, coaches used the elicit-provide-elicit technique to draw out change talk. For example, participants were asked: “What are some of the potential benefits of implementing MI in your setting?” Coaches actively listened to participants' responses, differentially reflecting and attempting to evoke change talk. Participants were then shown a pregenerated list of potential
MI for Implementation Coaching
HETTEMA et al.
benefits that were tied by the coach to what the participant had already stated. Lastly, participants were asked what benefits stood out to them most or would be most likely to lead them to implement MI. Coaches had the option to conduct similar ruler activities with confidence and readiness. Coaches had latitude in choosing whether to conduct one, two, or all three ruler activities with participants as well as determine the depth at which to explore each topic based on the participants' readiness level. For participants with high readiness, less time was spent conducting these motivation-building activities. If readiness to implement became sufficiently high during the Webinar to merit action planning, the coach transitioned to this stage. This involved asking permission to transition to action planning and then showing participants a table of concrete MI implementation goals of increasing intensity (e.g., learning more about the approach to determine its fit with the organization, training an internal champion as a trainer). Participants were asked to select one to three of these goals or generate their own goals to discuss. A change plan worksheet adapted from Ingersoll et al.17 was then completed for each goal using a collaborative and evocative approach. Coaches were given latitude to share resources, such as MI trainer listings, reading materials, or empirical information on effective training strategies, but were encouraged to use a collaborative information sharing style. At the end of change planning, coaches summarized the session and specific change plans, paying particular attention to the reasons for and specific steps to change. Instruments All available interactive implementation coaching Webinars were rated using the Motivational Interviewing Treatment Integrity Code 3.1.1 (MITI),18 a coding instrument designed to measure competence in MI. The MITI has two components, global scores and behavior counts, which map on to specific beginning proficiency and competency standards. This instrument was selected because it is the most widely used MI fidelity instrument, has demonstrated reliability and sensitivity,19 and has been found to predict treatment outcome across a range of behavioral domains.12 To date, the MITI has not been used to describe interpersonal interactions targeting implementation. Based on the format of the implementation coaching sessions, several modifications to the MITI were made to capture potentially unique and important aspects of the intervention. First, a structuring category was added. Structuring involves giving information about what will happen throughout the course of a session, in another session, or in transition from one part of a session to another. Though structuring is typically uncoded in the MITI, it was a construct in the MITI parent instrument, the Motivational Interviewing Skills Code.20 In addition, in the MITI, MI nonadherent behaviors include direct, confront, and advise without permission. Since advising without permission is qualitatively different from the other two categories and was anticipated to occur frequently in the context of the implementation coaching sessions, it was coded separately from the other two nonadherent behaviors. As is prescribed in the MITI, asking permission to give advice was coded as an MI adherent behavior. Lastly, though the MITI was developed to measure fidelity to MI in one-on-one interactions, in the current study, it was applied to group interactions in cases in which there was more than one organization representative on the call. Rating procedures The MITI was administered in a single pass to a randomly selected 20-min segment of each coaching interaction. The MITI requires one pass during which a trained rater actively codes behavior counts while listening to an audio-taped interaction, followed by the retrospective assessment of global characteristics. Raters in the current study had access to the transcriptions of coaching Webinars that could be used as a supplement to the recording if desired.
The Journal of Behavioral Health Services & Research
2013
Rater training Raters were three research assistants with master's degrees in social work or another counseling field. Raters received a 1-day introductory training on the use of the MITI from the first author, an experienced coder, coder trainer, and member of the Motivational Interviewing Network of Trainers. Training included didactic instruction and graded learning tasks, involving practice with feedback on the coding of clinical applications of MI, as well as several prerecorded mock implementation coaching sessions. Rating of actual coaching sessions did not begin until the rating team reached high levels of reliability with one another and agreement with the first author who shadow coded practice tapes. During rating of the actual sample, approximately one third of sessions were co-rated by all three raters, and weekly rater meetings were held to discuss inter-rater cases to prevent drift. To aid in consensus building, the first author also shadow coded five inter-rater cases from the sample.
Rater agreement As described above, all three raters co-rated 11 coaching sessions (32% of the sample) to allow for the assessment of inter-rater reliability (IRR). There are no detailed guidelines to inform the proportion of a sample that should be selected for IRR analysis. As the number of inter-rater cases increases, so does the confidence in the representativeness of the sample and the power of significance testing, though IRR is typically interpreted based on the absolute relationship between ratings versus the statistical significance of such relationships. While many studies fail to report the number of samples included in IRR estimates,21 those that report such data range from 13% to 18%.19,22 The subsample of 32% exceeds these standards. For continuous outcome variables, such as MITI behavior counts, intraclass correlation coefficients (ICCs) were calculated using a two-way mixed model for absolute agreement.23 ICCs are a more conservative estimate of reliability than simple Pearson correlations because they control for chance agreement between ratings.24 Cicchetti25 has recommended the following guidelines for interpretation of ICCs as follows: below 0.40=poor; 0.40 to 0.59=fair; 0.60 to 0.74= good; and 0.75 to 1.0=excellent. In rare cases, ICCs below 0 can be observed and should be interpreted as an indicator of poor reliability.26 For categorical variables, Randolph's free-marginal multi-rater kappa was used.27,28 This statistic also adjusts for chance agreement. Landis and Koch29 provide the following guidelines for interpretation of kappa: 0 to 0.20 as slight; 0.21 to 0.40 as fair; 0.41 to 0.60 as moderate; 0.61 to 0.80 as substantial; and 0.81 to 1 as almost perfect agreement. However, interpretation of the magnitude of kappa statistics is debatable, as this statistic is often suppressed when there is a heavily skewed frequency in the occurrence of categories.30 Previous studies of MI processes using the MITI instrument have shown us that, in highly controlled interventions with well-trained providers, there is a highly restricted range in global scores, skewing frequencies and making interpretation of kappa challenging. Anticipating this, the team also calculated the percentage of inter-rater cases with perfect agreement (range in scores across the three raters=0), the percentage whose scores differed by no more than 1 point, and the percentage whose scores differed by no more than 2 points. This latter measure likely gives a more accurate sense of reliability for this type of data.
Interpretation of results Results were analyzed to provide a descriptive account of the interpersonal processes and MI characteristics of the coaching Webinars.
MI for Implementation Coaching
HETTEMA et al.
Results Rated samples Of the 40 implementation coaching sessions conducted, 34 were available for rating. Half (N=17) of sessions were from CBHO settings and the other half were from CHC settings. The remaining sessions were uncodable because technical problems left recordings inaudible. On average, coaching Webinars lasted 70 min. Number of participants ranged from one to nine, with a mean of 2.5 (SD=2). Forty-eight percent of Webinars had only one participant. Feasibility of MITI application to implementation coaching webinars Overall, application of the MITI instrument was highly feasible, despite differences in the behavioral targets of the interactions (MI adoption versus clinical concerns) and the presence of more than one participant in more than half of the interactions. In developing the training and training practice samples, it was not challenging to create examples of coach behaviors that were consistent and inconsistent with the principles of MI. Rater training using a combination of clinical examples and examples tailored to implementation coaching interactions was well received by raters, who appeared to develop comfort and proficiency with the instrument at rates similar to raters involved on clinical coding teams. The presence of multiple participants in some interactions did not interfere with the application of the MITI. This may be due in part to the fact that the MITI only focuses on interventionist behaviors (versus participant behaviors). MITI feasibility in a group interaction may also have been promoted by the fact that, in general, organization representatives participating in the same call spoke to the coach rather than each other, and they acted largely as a unit elaborating on, continuing, or deepening each other's comments, versus sharing significantly different perspectives or opinions. Inter-rater agreement IRR data was available from all 3 raters for 11 interactions. As Table 1 describes, for MITI behavior counts, ICCs indicated excellent agreement between raters, ranging from ICC=0.803 to 0.909. The one exception involves MI-inconsistent behaviors, whose ICC was below 0 and should be interpreted as low. MI-inconsistent behaviors were observed in only four cases, and a total of
Table 1 Inter-rater agreement for MITI behavior counts MITI behavior count Giving information Structuring Mi adherent MI nonadherent (confront, direct) Advice without permission Closed questions Open questions Simple reflection Complex reflection
ICC (95% CI) ICC=0.909 ICC=0.904 ICC=0.785 ICCG0 ICC=0.937 ICC=0.826 ICC=0.803 ICC=0.821 ICC=0.896
(0.752, 0.973)* (0.730, 0.972)* (0.382, 0.938)* (0.831, (0.513, (0.462, (0.502, (0.719,
0.982)* 0.949)* 0.942)* 0.947)* 0.969)*
The Journal of Behavioral Health Services & Research
2013
*pG0.05
only 16 occurrences were recorded across the 33 observations. In three out of four cases with an MI-inconsistent observation, only one rater observed the MI-inconsistent behavior. IRR data for MITI Global Scores can be seen in Table 2. Kappa scores ranged from fair (kappa= 0.32) to substantial (kappa=0.77). However, in the current sample, there was limited range in MITI global scores, likely suppressing kappa scores and making interpretation of this statistic difficult. When looking at rates of agreement instead, it is clear that raters had high levels of agreement across all global scores. In the majority of ratings (94.5%), all three raters had perfect agreement or disagreed only by a score of 1. This means that in most cases, no rater differed more than 1 point from any other rater (e.g., 4, 5, 4). Consistency of implementation coaching webinars to MI As can be seen in Table 3, the interactive implementation coaching Webinars had high fidelity to the MI model. All global scores exceeded competency standards recommended in the MITI, with a mean MI spirit (average of evocation, collaboration, autonomy support) of 4.4 (SD=0.49). In addition, all behavior count thresholds fell between beginning proficiency and competency levels. Coaches used more reflections than questions and more complex reflections than simple reflections. When questions were used, they were more likely to be open-ended than closed-ended. MI nonadherent behaviors were rare, as was unsolicited advice.
Discussion Implementation efforts are often thwarted by individual, organizational, and system-wide factors including ambivalence about change and low self-efficacy to overcome seemingly insurmountable
Table 2 Inter-rater agreement for MITI global scores MITI global score
Agreement statistic
Agreement rates*
Evocation
Kappa=0.43
Collaboration
Kappa=0.32
Autonomy
Kappa=0.35
Direction
Kappa=0.77
Empathy
Kappa=0.54
Range=0:4 Range=1:6 Range=2:1 Range=0:3 Range=1:7 Range=2:1 Range=0:3 Range=1:7 Range=2:1 Range=0:8 Range=1:3 Range=2:0 Range=0:5 Range=1:6 Range=2:0
*Range=0 indicates perfect agreement across all three raters; Range=1 indicates that no rater differed more than 1 point from any other rater (e.g., 4, 5, 4); Range=2 indicates that no rater differed more than 2 points from any other rater (e.g., 3, 4, 5)
MI for Implementation Coaching
HETTEMA et al.
Table 3 Description of coaching calls MITI rating category
Mean (SD) range
Beginning proficiency
Competency
Evocation
4.4 (0.56) Range: 3–5 4.3 (0.68) Range: 3–5 4.4 (.66) Range: 3–5 4.8 (0.39) Range: 3–5 4.4 (0.49) Range: 3–5 4.4 (.49) Range: 3.33–5 1.2 (0.6) Range: 0.5–2.6 0.52 (0.18) Range: 0.14–1.00 0.42 (0.15) Range: 0.15–0.75 0.95 (0.11) Range: 0.56–1.00 5.5 (3.1) Range:1–3 3.1 (2.2) Range: 0–9 5.7 (3.6) Range: 1–17 0.7 (1.3) Range: 0–5 0.4 (0.8) Range: 0–3 5.8 (3.1) Range: 0–12 6.3 (2.8) Range: 1–14 7.9 (4.1) Range: 2–20 5.5 (3.0) Range: 1–14
3.5
4
3.5
4
3.5
4
3.5
4
3.5
4
Average of 3.5
Average of 4
1
2
50%
70%
40%
50%
90%
100%
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Collaboration Autonomy support Direction Empathy Global Spirit Reflection to question ratio Percent open questions Percent complex reflections Percent MI adherence Giving information Structuring MI adherent MI nonadherent (confront, direct) Advice without permission Closed questions Open questions Simple reflection Complex reflection
barriers.10 As a strategy for helping individuals explore and resolve ambivalence, MI has the potential to be applicable not only to the clinical issues patients face, but to the decisions of providers or administrators of patient care. The parent study to the current report found that, as part of a larger active
The Journal of Behavioral Health Services & Research
2013
dissemination strategy, MI-based interactive implementation coaching Webinars were effective in promoting progression in the adoption of MI as measured by stages of change-based instrument. These promising findings encouraged the authors to more comprehensively describe the content and style of the coaching Webinars and their consistency with the principles and practices of MI. To objectively assess the fidelity to the MI model, the MITI was applied to all available coaching sessions. Application of the MITI to this type of interaction was feasible, and rates of IRR on measured variables were good. Lastly, as predicted, fidelity to MI in the coaching Webinars was high across all measured constructs. The findings suggest that MI is likely an applicable and effective strategy for conducting implementation coaching. When interpreting the findings, several limitations should be considered. First, the current results are yielded from one sample of organizations and may not be broadly generalizable. For one, participants in the current study may already have had higher than average interest in patientcentered techniques such as MI or in EBPs in general, by virtue of their membership in the National Council for Community Behavioral Healthcare and the National Association for Community Health Centers (recruitment pools) or their decision to participate in a study on this topic. It is difficult to say whether the observed effects would be similar with different types of organizations. In addition, the coaches who conducted coaching Webinars in the current trial had extensive previous training and consultation experience. It is possible that coaches with less experience may not be as effective at implementing implementation coaching or adhere as consistently to the MI model. Additionally, the active dissemination strategy found to be effective in the parent trial had several components in addition to the MI-based interactive implementation coaching Webinar, including informational packets and a didactic Webinar. Although the results provide promising evidence of the potential effectiveness of MI as an implementation coaching strategy, it is possible that this was not the active component of the intervention. Additional research is needed to confirm the effectiveness of an MI approach and learn more about the generalizability of MI applications to implementation coaching. Future studies could examine outcome variables in addition to decision making that include further steps in implementation, or actual clinician performance or patient outcomes. Based on the research of MI in clinical settings, further work could also be done looking at the potential mechanisms of action of MI in implementation settings. Mechanisms of action are those active components of an intervention that are thought to causally impact results. This could include research on the relationship between MI fidelity in coaching interactions and outcomes to determine whether high or low adherence to the model, or specific components of the model, allow us to predict differences in outcome. In addition, future research could examine the language of the participants receiving coaching to determine whether mechanisms of action that appear to impact clinical outcomes, such as the expression of change talk, also drive the effectiveness of MI in implementation settings. Lastly, it may be helpful to address whether this model of implementation coaching is also useful for implementation efforts involving other EBPs or organizational changes. Making a decision to adopt an EBP can be influenced by communication strategies, including those used by implementation coaches or consultants. These communication efforts may first exert their effects in the knowledge stage of diffusion, during which potential adopters are exposed to an EBP. In some cases, this is an inherent alignment with the underlying spirit of MI. In other cases, due to the receipt of previous dissemination efforts, individuals may already be very or somewhat familiar with many of the practices of MI. Lastly, organization representatives have many ideas about what strategies will or will not be effective in implementing MI in their setting. When promoting knowledge, avoid unnecessary information provision and communicating information in a respectful and collaborative way; this may enhance the impact of this stage on later adoption decisions. Similarly, during the persuasion stage, communication strategies can influence whether individuals form positive or negative attitudes toward EBPs. In the MI model, it is acknowledged that when individuals are ambivalent about making a change, attempting to push them toward a
MI for Implementation Coaching
HETTEMA et al.
specific change typically results in withdrawal or resistance. Instead, evocation strategies can be used to draw out the person's own motivations for change and ideas about how change could occur.
Implications for Behavioral Health There appear to be some universal characteristics of the change process that occur regardless of the type of change being targeted. Just as MI practices may be effective in promoting change among patients, they may also be effective skills to use with those who provide care. The successful implementation of EBPs is a critical public health issue. The findings of the current report suggest that MI might be useful, not only as an EBP, but as a component of an effective strategy for facilitating implementation of other EBPs.
Acknowledgments Special thanks to Ali Hall and Robert Rhode for their work in developing and conducting the implementation Webinars and our SAMHSA Project Officer, Kevin Hennessy, Ph.D., for his thoughtful comments and feedback during the preparation of this manuscript. This study was supported by SAMHSA contract no. HHSS283200700040I/HHSS28342003T. Conflicts of Interest There are no conflicts of interest. Disclaimers This report was prepared under contract by MANILA Consulting Group, Inc., for SAMHSA. The content of this publication does not necessarily reflect the views or policies of SAMHSA or the US Department of Health and Human Services.
References 1. Miller WR, Rollnick S. Motivational interviewing: helping people change, 3rd ed. New York, NY: Guilford Press; 2013. 2. Hettema JE, Steele J, Miller WR. Motivational interviewing. Annual Review of Clinical Psychology. 2005;1:91–111. 3. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009. 4. Miller WR, Wilbourne PL, Hettema JE. In: Hester RK, Miller WR, eds. What works? A summary of alcohol treatment outcome research, 3rd ed. Boston, MA: Allyn & Bacon; 2003:13–63. 5. Williams JR, Tregear SJ, Dusablon T, et al. Increasing adoption of comparative effectiveness research in community behavioral health: methodology. Journal of Behavioral Health Services and Research. In press. 6. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: Department of Health and Human Services; 2009. 7. Cooperrider DL, Whitney D. Appreciative inquiry: a positive revolution in change. San Francisco: Berrett-Koehler Publishers; 2005. 8. Rogers EM. Diffusion of Innovations, 5th ed. New York, NY: Free Press; 2003. 9. Prochaska JO, DiClemente CC. The Transtheoretical Approach: Crossing Traditional Boundaries of Therapy. Homewood, IL: Dow Jones-Irwin; 1984. 10. Miller WR, Sorensen JL, Selzer JA, et al. Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. Journal of Substance Abuse Treatment. 2006;31:25–39. 11. Miller WR, Yahne CE, Moyers TB, et al. A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology. 2004;72:1050–1062. 12. Moyers TB, Martin T, Houck JM, et al. From in-session behaviors to drinking outcomes: a causal chain for motivational interviewing. Journal of Consulting and Clinical Psychology, 2009;77:113–1124 13. Miller WR, Benefield RG, Tonigan JS. Enhancing motivation for change in problem drinking: a controlled comparison of two therapist styles. Journal of Consulting and Clinical Psychology. 1993;61:455–461. 14. Martino S, Ball SA, Gallon SL, et al. Motivational interviewing assessment: supervisory tools for enhancing proficiency. Salem, OR: Northwest Frontier Addiction Technology Transfer Center, Oregon Health and Science University; 2006. 15. Strang J, McCambridge J, Platts S, et al. Engaging the reluctant GP in care of the opiate misuser: pilot study of change-orientated reflective listening (CORL). Family Practice. 2004;21:150–154.
The Journal of Behavioral Health Services & Research
2013
16. Hettema JE, Sorensen JL, Uy M, et al. Motivational enhancement therapy to increase resident physician engagement in substance abuse education. Substance Abuse. 2009;30:244–247. 17. Ingersoll KS, Wagner CC, Gharib S. Motivational groups for community substance abuse programs. 2002. Available at: http://people.uncw.edu/ogler/MI%20Groups%20for%20Com%20SA%20Prog.pdf. Accessed October 31, 2012. 18. Moyers TB, Martin T, Manuel JK, et al. Revised global scales: motivational interviewing treatment integrity 3.1.1 (MITI 3.1.1). 2010. Available at: http://casaa.unm.edu/download/MITI3_1.pdf Accessed October 31, 2012. 19. Moyers TB, Martin T, Manuel JK, et al. Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment. 2005;28:19–26. 20. Miller WR, Moyers TB, Ernst D, et al. Manual for the Motivational Interviewing Skills Code (MISC), Version 2.1. 2008. Available at: http://casaa.unm.edu/download/misc.pdf Accessed October 31, 2012. 21. Tollison SJ, Lee CM, Neighbors C, et al. Questions and reflections: The use of the motivational interviewing microskills in a peer-led brief alcohol intervention for college students. Behavior Therapy. 2007;39:183–194. 22. Moyers TB, Martin T. Therapist influence on client language during motivational interviewing sessions. Journal of Substance Abuse Treatment. 2006;30:245–251. 23. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86:420–428. 24. Cicchetti DV, Sparrow SA. Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency. 1981;2:127–137. 25. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Special section: normative assessment. Psychological Assessment. 1994;6:284–290. 26. Tammemagi M, Frank J, Streiner D. Negative values of the intraclass correlation coefficient are not theoretically possible: response. Journal of Clinical Epidemiology. 1996;49:1206. 27. Randolph JJ. Free-marginal multirater kappa: an alternative to Fleiss' fixed-marginal multirater kappa. Paper presented at: The Joensuu University Learning and Instruction Symposium; October 14–15, 2005; Joensuu, Finland. (ERIC Document Reproduction Service No. ED490661.) 28. Randolph JJ. Online kappa calculator. 2008. Available at: http://justus.randolph.name/kappa. Accessed October 31, 2012. 29. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. 30. Bakeman R, Quera V, McArthur D, et al. Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods. 1997;2:357–370.
MI for Implementation Coaching
HETTEMA et al.
Address correspondence to Jennifer E. Hettema, PhD, Department of Family and Community Medicine, University of New Mexico, MSC 09 5040 ABQ, NM 87131, USA. Phone: +1-505-2722165; Fax: +1-505-2728045; Email: [email protected]. Denise Ernst, PhD, Denise Ernst Training and Consultation, 8695 SW 80th Avenue, Portland, OR 97223, USA. Phone: +1-503-9468320; Fax: +1-804-5274268 Jessica Roberts Williams, PhD, MPH, RN, University of Miami School of Nursing and Health Studies, P.O. Box 248153, Coral Gables, FL, USA. Phone: 571-633-9400; Fax: +1-571-633-9401; Email: [email protected] Jessica Roberts Williams, PhD, MPH, RN, MANILA Consulting Group, Inc., McLean, VA, USA. Phone: 571-633-9400; Fax: +1-571-633-9401; Email: [email protected] Kristin J. Miller, MSW, MANILA Consulting Group, Inc., McLean, VA, USA. Phone: +1-571-6339400; Fax: +1-571-633-9401
)
Journal of Behavioral Health Services & Research, 2013. 1–12. c 2013 Springer Science+Business Media, LLC (outside the USA). DOI 10.1007/s11414-013-9381-8
MI for Implementation Coaching
HETTEMA et al.
Introduction Motivational interviewing (MI) is a collaborative conversation style for strengthening a person's own motivation and commitment to change.1 This approach has been applied to the prevention and treatment of a variety of health conditions with promising findings.2 Like many evidence-based practices (EBPs), despite its strong evidence base, adoption and effective implementation of MI remains suboptimal.3,4 Interestingly, as a tool for helping individuals explore and resolve ambivalence about change, MI itself may be a promising strategy for disseminating successful interventions. This concept was supported by the parent study of the current special edition, which found that, as part of a larger active dissemination strategy, MI-based interactive implementation coaching Webinars are an effective tool for advancing adoption of EBPs.5 The purpose of the current report is to (1) describe the implementation coaching intervention and its consistency with the principles and practices of MI, (2) discuss the feasibility of applying an MI counseling fidelity instrument to implementation coaching sessions, and (3) report on the consistency of the implementation coaching sessions to the MI model.
Description of parent study As part of a study to evaluate the impact of different dissemination strategies on the adoption of EBPs, such as MI, community behavioral health organizations (CBHOs) and community health centers (CHCs) were randomly assigned to one of the following conditions using a nested experimental design that matched organizations on key characteristics: (1) passive dissemination through receipt of informational packets describing MI effectiveness and implementation resources or (2) active dissemination, which included the informational packet plus receipt of a large-group didactic Webinar and an MI-based interactive coaching Webinar. Historically, CHCs specialize in primary care and CBHOs in behavioral health services; however, recent efforts to integrate primary care and behavioral health have resulted in increased crossover and coordination of these services across these two types of organizations. Participants from the CBHOs and CHCs included administrators and front-line providers. For more details on the methodology, see the parent study in this special issue.5 Results of the study revealed that individuals from organizations that received active dissemination, including the MI-based interactive coaching Webinar, were more likely to report decisions to adopt MI, as measured by a stage of change-based question querying interest in and steps toward adopting MI.
Why might MI work well as an implementation coaching strategy? Conceptually, there are many reasons to believe that MI may be an effective dissemination tool. In fact, the Comparative Effectiveness Research Council's goal of developing implementation strategies that “empower patients, clinicians, and other stakeholders to make more informed decisions”6 aligns highly with the spirit and objectives of MI. MI co-developers Miller and Rollnick briefly discuss the application of MI to organizational change in the third edition of their book Motivational Interviewing: Helping People Change (pp. 345–346).1 The developers draw parallels between MI and appreciative inquiry, an organizational development method focused on developing and building on strengths.7 While Miller and Rollnick primarily focus on ways in which MI practices may be useful tools for managers and supervisors to motivate people within a workplace, the approach seems similarly applicable to consultative relationships involving implementation coaching. In MI, focus is shifted from the traditional model in which motivation must be instilled and solutions must come from an outside content expert, to a model in which
The Journal of Behavioral Health Services & Research
2013
empathy and evocation guide organizations to talk themselves into changing, and solutions are elicited from the system itself. MI and diffusion of innovations Several models are available to explain, predict, and promote dissemination of EBPs. While a variety of complex and interacting systems-level, economic, and external factors likely impact implementation, many models, including Rogers' Diffusion of Innovations,8 focus largely on the role of the individual in making decisions about adoption. Within Rogers' model, diffusion is believed to take place in five stages: knowledge, persuasion, decision, implementation, and confirmation. The model has been cited as roughly paralleling the transtheoretical stages of change.9,10 Conceptually, there is also an overlap between the stages of diffusion and the principles and practices of MI. Consistent with the aims of the parent research project that is the focus of this special issue, MI may be theorized to affect decision making during the knowledge and persuasion stages. In Rogers' first stage, knowledge, potential adopters become aware of an innovation. Rogers believed that a variety of factors impact the knowledge stage, including communication behavior. As a communication strategy itself, MI provides guidance on when and how much information to provide, as well as the style in which information is provided. In MI, there is an emphasis on the tendency to overestimate how much information or direction actually needs to be communicated to others. For example, in clinical practice, despite the fact that most patients are aware of the negative health impacts of smoking, providers are often tempted to take the expert role and lecture patients about the dangers of smoking and strategies to quit. Instead of encouraging patients to engage and explore their behavior, unneeded information provision and unsolicited advice often lead to withdrawal or defensiveness that runs contrary to the promotion of change.1 Similarly, through an MI lens, dissemination efforts may sometimes overemphasize the amount or extensiveness of information needed to promote knowledge and an ultimate decision to adopt. This is supported by research that suggests that years of education are not related to use of EBPs and that front-line providers make very limited use of scientific books and articles, instead learning through informal channels.10 MI may also inform the knowledge stage of diffusion by providing guidance on the style in which information is shared. In MI, communicating information collaboratively and respectfully is thought to maximize attention to and acquisition of information, as well as enhance the application of knowledge to decisions about change. For example, MI interventionists are encouraged to ask permission (and honor the response) before sharing information. Similarly, MI interventionists may use the elicit-provide-elicit technique, whereby they first ask individuals what they know about a topic, then share information in a tailored way, and follow up by asking how, if at all, the information has impacted the individuals' thinking about the topic. In an implementation coaching context, this may take the form of asking providers what they know about the potential benefits of implementing MI in their setting, underscoring these benefits or possibly sharing some new benefits, and then asking how, if at all, these benefits impact decisions about implementation. This technique may similarly be used to discuss knowledge about strategies for implementation. The MI model differs from typical expert-recipient models as it takes advantage of the expertise of providers and organizations and the unique and powerful perspectives they have on what will work in their specific settings. The second stage in Rogers' diffusion model is persuasion, whereby the individual develops positive or negative attitudes toward an innovation. Again, communication strategies and techniques used by disseminating agents can significantly impact persuasion. In MI, it is also acknowledged that the way in which an interventionist interacts with an individual can significantly impact his or her attitudes about change.
MI for Implementation Coaching
HETTEMA et al.
When forming attitudes about change or adoption, individuals often hold ambivalent feelings. There may be aspects of change that are positive and attractive, and, at the same time, there may be concerns about or drawbacks to change. Clinically, a smoker may be aware of the negative health effects of smoking and want to live a long life, and also really enjoy the feeling of smoking and appreciate its weight loss benefits. Similarly, when forming attitudes toward the adoption of MI, providers or organizations may have mixed feelings. While on the one hand a potential adopter may be aware of the strong evidence base for MI and align philosophically with its underlying principles, on the other hand, there may be concerns about the feasibility of implementation, as there is mounting evidence that to attain proficiency with MI, more is needed than initial training, such as follow-up feedback and coaching.11 MI is a type of conversation that is particularly useful in the exploration and resolution of ambivalence about change. While one is often tempted to persuade individuals into changing by trying to convince them of the benefits of change or the drawbacks of staying the same, such strategies typically make others feel angry, defensive, uncomfortable, or powerless. Instead, MI seeks to focus conversation in such a way that individuals talk themselves into changing. Evocation involves the selective elicitation of language that indicates movement toward change (change talk). In MI, specific interventionist behaviors have been found to evoke change talk, which in turn is predictive of positive outcome.12 Such language can include desire to change (e.g., “We want our patients to more effectively manage chronic disease.”), perceived ability to change (“We've successfully implemented other EBPs in the past.”), reasons to change (“MI is consistent with many of our goals as a Patient-Centered Medical Home.”), need to change (“We've got to do something to improve our rates of attendance.”), and commitment to change (“We're sending all of our front-line staff to an introductory MI training.”). By taking a nonjudgmental, empathic stance, MI interventionists encourage the sincere exploration of change. In addition, several strategic approaches are available to draw out, reinforce, and build upon change talk, including asking evocative questions, seeking elaboration, and differentially reflecting or affirming change talk. In Roger's model, the decision stage involves the choice of whether to reject or adopt an innovation. This represents the tipping point, at which time ambivalence about change is resolved in one direction or another. Promoting decisions to implement MI was the primary objective of the tested dissemination strategies in the parent study of this report. In the MI model, the act of decision making becomes the target change for the intervention. This, in turn, provides the interventionists with the goal toward which to guide the conversation and the key with which to evaluate the participant language. The MI communication strategies used in the knowledge and persuasion stages likely impact the probability that individuals will make a decision in favor of change.13 Similarly, engaging in collaborative, detailed planning following an implementation decision can help to consolidate and build commitment for the decision. Applications of MI to coaching While the conceptual links between MI and adoption decision making are clear, there has been little empirical investigation of the impact of MI on dissemination efforts. However, MI has been introduced as a strategy for coaching interventionists to improve their clinical skillfulness. For example, in a randomized trial of methods to help clinicians learn MI,11 providers who received post-training coaching in the spirit of MI were found to have larger gains in proficiency than those in the workshop only condition. Similarly, in the Motivational Interviewing Assessment: Supervisory Tools for Enhancing Proficiency,14 a collection of tools developed to aid organizations in supervising clinicians in the use of MI during clinical assessments that was developed as part of a National Institute on Drug Abuse/Substance Abuse and Mental Health Services Administration blending initiative, specific MI-consistent supervisory strategies are suggested. For example, supervisors are provided with guidelines that encourage them to “practice what you preach” by not
The Journal of Behavioral Health Services & Research
2013
taking the expert role, focusing on what went well, eliciting clinicians' ideas about improvement, and listening carefully to resistance as an opportunity to explore real implementation barriers. In addition to potentially increasing the effectiveness of performance coaching, adopting an MI style has a secondary educational benefit. By modeling MI, potential adopters are exposed to a firsthand application demonstrating how a collaborative communication technique might be used to promote change. While there is limited direct empirical evidence regarding the impact of MI practices in implementation coaching settings, initial evidence appears promising. In an early pilot study, Strang et al.15 provided telephone-based MI to general practitioners to increase their involvement in the care of opiate misusers and observed improvements in attitudinal and behavioral care factors. Similarly, in a small medical education pilot study,16 medical residents were assessed for their selfreported use of and attitudes toward substance abuse screening and brief intervention practices. They were then given normative feedback based on their responses as well as a motivational interview to explore ambivalence about engaging in substance abuse practice, promote engagement in a substance abuse curriculum, and build motivation for change. This study resulted in behavioral and attitudinal improvements in the expected direction. Based on the results of the parent trial, the conceptual applicability of MI to models of diffusion and implementation, and early research, the purpose of the current paper is to describe the content and structure of implementation coaching, determine the feasibility of measuring MI fidelity in implementation coaching, and describe the adherence of the coaching to the MI model.
Methods Interactive implementation coaching Webinars Coaching Webinars were collaboratively developed by the parent trial investigative team and three members of the Motivational Interviewing Network of Trainers (MINT). Two of the three MINT developers, including the first author, went on to conduct the coaching Webinars. Coaches received no explicit training as part of the current trial, but, as members of MINT, had previously received advanced training in MI as a clinical strategy, and had extensive experience training others in MI, supervising others in MI, and providing MI implementation consultation to agencies. Each coach conducted one pilot coaching Webinar prior to the start of the trial for which they received performance feedback from the parent trial investigative team; however, this feedback largely focused on adherence to the developed webinar protocol versus MI skillfulness. Coaching was designed to be consistent with the principles and practices of MI. The Webinars followed a semistructured format that was facilitated by a slideshow presentation and interactive polling software. Webinars began with a brief coach introduction and overview of the Webinar objectives to focus the interaction. Participants were then invited to introduce themselves, tell a little bit about their professional role and their organization, and describe any past experience with MI. Coaches actively listened, attending differentially to participant change talk, engaging participants, and developing rapport. Importance, confidence, and readiness rulers were then used by coaches to strategically evoke and respond to change talk regarding the implementation of MI. First, an importance ruler was presented, which asked participants to rate the importance of implementing MI on a scale from 1 to 7 using interactive polling software. Participants were then asked the evocative question: “Why did you select an X, instead of something lower like a Y?” To further build motivation for implementation, coaches used the elicit-provide-elicit technique to draw out change talk. For example, participants were asked: “What are some of the potential benefits of implementing MI in your setting?” Coaches actively listened to participants' responses, differentially reflecting and attempting to evoke change talk. Participants were then shown a pregenerated list of potential
MI for Implementation Coaching
HETTEMA et al.
benefits that were tied by the coach to what the participant had already stated. Lastly, participants were asked what benefits stood out to them most or would be most likely to lead them to implement MI. Coaches had the option to conduct similar ruler activities with confidence and readiness. Coaches had latitude in choosing whether to conduct one, two, or all three ruler activities with participants as well as determine the depth at which to explore each topic based on the participants' readiness level. For participants with high readiness, less time was spent conducting these motivation-building activities. If readiness to implement became sufficiently high during the Webinar to merit action planning, the coach transitioned to this stage. This involved asking permission to transition to action planning and then showing participants a table of concrete MI implementation goals of increasing intensity (e.g., learning more about the approach to determine its fit with the organization, training an internal champion as a trainer). Participants were asked to select one to three of these goals or generate their own goals to discuss. A change plan worksheet adapted from Ingersoll et al.17 was then completed for each goal using a collaborative and evocative approach. Coaches were given latitude to share resources, such as MI trainer listings, reading materials, or empirical information on effective training strategies, but were encouraged to use a collaborative information sharing style. At the end of change planning, coaches summarized the session and specific change plans, paying particular attention to the reasons for and specific steps to change. Instruments All available interactive implementation coaching Webinars were rated using the Motivational Interviewing Treatment Integrity Code 3.1.1 (MITI),18 a coding instrument designed to measure competence in MI. The MITI has two components, global scores and behavior counts, which map on to specific beginning proficiency and competency standards. This instrument was selected because it is the most widely used MI fidelity instrument, has demonstrated reliability and sensitivity,19 and has been found to predict treatment outcome across a range of behavioral domains.12 To date, the MITI has not been used to describe interpersonal interactions targeting implementation. Based on the format of the implementation coaching sessions, several modifications to the MITI were made to capture potentially unique and important aspects of the intervention. First, a structuring category was added. Structuring involves giving information about what will happen throughout the course of a session, in another session, or in transition from one part of a session to another. Though structuring is typically uncoded in the MITI, it was a construct in the MITI parent instrument, the Motivational Interviewing Skills Code.20 In addition, in the MITI, MI nonadherent behaviors include direct, confront, and advise without permission. Since advising without permission is qualitatively different from the other two categories and was anticipated to occur frequently in the context of the implementation coaching sessions, it was coded separately from the other two nonadherent behaviors. As is prescribed in the MITI, asking permission to give advice was coded as an MI adherent behavior. Lastly, though the MITI was developed to measure fidelity to MI in one-on-one interactions, in the current study, it was applied to group interactions in cases in which there was more than one organization representative on the call. Rating procedures The MITI was administered in a single pass to a randomly selected 20-min segment of each coaching interaction. The MITI requires one pass during which a trained rater actively codes behavior counts while listening to an audio-taped interaction, followed by the retrospective assessment of global characteristics. Raters in the current study had access to the transcriptions of coaching Webinars that could be used as a supplement to the recording if desired.
The Journal of Behavioral Health Services & Research
2013
Rater training Raters were three research assistants with master's degrees in social work or another counseling field. Raters received a 1-day introductory training on the use of the MITI from the first author, an experienced coder, coder trainer, and member of the Motivational Interviewing Network of Trainers. Training included didactic instruction and graded learning tasks, involving practice with feedback on the coding of clinical applications of MI, as well as several prerecorded mock implementation coaching sessions. Rating of actual coaching sessions did not begin until the rating team reached high levels of reliability with one another and agreement with the first author who shadow coded practice tapes. During rating of the actual sample, approximately one third of sessions were co-rated by all three raters, and weekly rater meetings were held to discuss inter-rater cases to prevent drift. To aid in consensus building, the first author also shadow coded five inter-rater cases from the sample.
Rater agreement As described above, all three raters co-rated 11 coaching sessions (32% of the sample) to allow for the assessment of inter-rater reliability (IRR). There are no detailed guidelines to inform the proportion of a sample that should be selected for IRR analysis. As the number of inter-rater cases increases, so does the confidence in the representativeness of the sample and the power of significance testing, though IRR is typically interpreted based on the absolute relationship between ratings versus the statistical significance of such relationships. While many studies fail to report the number of samples included in IRR estimates,21 those that report such data range from 13% to 18%.19,22 The subsample of 32% exceeds these standards. For continuous outcome variables, such as MITI behavior counts, intraclass correlation coefficients (ICCs) were calculated using a two-way mixed model for absolute agreement.23 ICCs are a more conservative estimate of reliability than simple Pearson correlations because they control for chance agreement between ratings.24 Cicchetti25 has recommended the following guidelines for interpretation of ICCs as follows: below 0.40=poor; 0.40 to 0.59=fair; 0.60 to 0.74= good; and 0.75 to 1.0=excellent. In rare cases, ICCs below 0 can be observed and should be interpreted as an indicator of poor reliability.26 For categorical variables, Randolph's free-marginal multi-rater kappa was used.27,28 This statistic also adjusts for chance agreement. Landis and Koch29 provide the following guidelines for interpretation of kappa: 0 to 0.20 as slight; 0.21 to 0.40 as fair; 0.41 to 0.60 as moderate; 0.61 to 0.80 as substantial; and 0.81 to 1 as almost perfect agreement. However, interpretation of the magnitude of kappa statistics is debatable, as this statistic is often suppressed when there is a heavily skewed frequency in the occurrence of categories.30 Previous studies of MI processes using the MITI instrument have shown us that, in highly controlled interventions with well-trained providers, there is a highly restricted range in global scores, skewing frequencies and making interpretation of kappa challenging. Anticipating this, the team also calculated the percentage of inter-rater cases with perfect agreement (range in scores across the three raters=0), the percentage whose scores differed by no more than 1 point, and the percentage whose scores differed by no more than 2 points. This latter measure likely gives a more accurate sense of reliability for this type of data.
Interpretation of results Results were analyzed to provide a descriptive account of the interpersonal processes and MI characteristics of the coaching Webinars.
MI for Implementation Coaching
HETTEMA et al.
Results Rated samples Of the 40 implementation coaching sessions conducted, 34 were available for rating. Half (N=17) of sessions were from CBHO settings and the other half were from CHC settings. The remaining sessions were uncodable because technical problems left recordings inaudible. On average, coaching Webinars lasted 70 min. Number of participants ranged from one to nine, with a mean of 2.5 (SD=2). Forty-eight percent of Webinars had only one participant. Feasibility of MITI application to implementation coaching webinars Overall, application of the MITI instrument was highly feasible, despite differences in the behavioral targets of the interactions (MI adoption versus clinical concerns) and the presence of more than one participant in more than half of the interactions. In developing the training and training practice samples, it was not challenging to create examples of coach behaviors that were consistent and inconsistent with the principles of MI. Rater training using a combination of clinical examples and examples tailored to implementation coaching interactions was well received by raters, who appeared to develop comfort and proficiency with the instrument at rates similar to raters involved on clinical coding teams. The presence of multiple participants in some interactions did not interfere with the application of the MITI. This may be due in part to the fact that the MITI only focuses on interventionist behaviors (versus participant behaviors). MITI feasibility in a group interaction may also have been promoted by the fact that, in general, organization representatives participating in the same call spoke to the coach rather than each other, and they acted largely as a unit elaborating on, continuing, or deepening each other's comments, versus sharing significantly different perspectives or opinions. Inter-rater agreement IRR data was available from all 3 raters for 11 interactions. As Table 1 describes, for MITI behavior counts, ICCs indicated excellent agreement between raters, ranging from ICC=0.803 to 0.909. The one exception involves MI-inconsistent behaviors, whose ICC was below 0 and should be interpreted as low. MI-inconsistent behaviors were observed in only four cases, and a total of
Table 1 Inter-rater agreement for MITI behavior counts MITI behavior count Giving information Structuring Mi adherent MI nonadherent (confront, direct) Advice without permission Closed questions Open questions Simple reflection Complex reflection
ICC (95% CI) ICC=0.909 ICC=0.904 ICC=0.785 ICCG0 ICC=0.937 ICC=0.826 ICC=0.803 ICC=0.821 ICC=0.896
(0.752, 0.973)* (0.730, 0.972)* (0.382, 0.938)* (0.831, (0.513, (0.462, (0.502, (0.719,
0.982)* 0.949)* 0.942)* 0.947)* 0.969)*
The Journal of Behavioral Health Services & Research
2013
*pG0.05
only 16 occurrences were recorded across the 33 observations. In three out of four cases with an MI-inconsistent observation, only one rater observed the MI-inconsistent behavior. IRR data for MITI Global Scores can be seen in Table 2. Kappa scores ranged from fair (kappa= 0.32) to substantial (kappa=0.77). However, in the current sample, there was limited range in MITI global scores, likely suppressing kappa scores and making interpretation of this statistic difficult. When looking at rates of agreement instead, it is clear that raters had high levels of agreement across all global scores. In the majority of ratings (94.5%), all three raters had perfect agreement or disagreed only by a score of 1. This means that in most cases, no rater differed more than 1 point from any other rater (e.g., 4, 5, 4). Consistency of implementation coaching webinars to MI As can be seen in Table 3, the interactive implementation coaching Webinars had high fidelity to the MI model. All global scores exceeded competency standards recommended in the MITI, with a mean MI spirit (average of evocation, collaboration, autonomy support) of 4.4 (SD=0.49). In addition, all behavior count thresholds fell between beginning proficiency and competency levels. Coaches used more reflections than questions and more complex reflections than simple reflections. When questions were used, they were more likely to be open-ended than closed-ended. MI nonadherent behaviors were rare, as was unsolicited advice.
Discussion Implementation efforts are often thwarted by individual, organizational, and system-wide factors including ambivalence about change and low self-efficacy to overcome seemingly insurmountable
Table 2 Inter-rater agreement for MITI global scores MITI global score
Agreement statistic
Agreement rates*
Evocation
Kappa=0.43
Collaboration
Kappa=0.32
Autonomy
Kappa=0.35
Direction
Kappa=0.77
Empathy
Kappa=0.54
Range=0:4 Range=1:6 Range=2:1 Range=0:3 Range=1:7 Range=2:1 Range=0:3 Range=1:7 Range=2:1 Range=0:8 Range=1:3 Range=2:0 Range=0:5 Range=1:6 Range=2:0
*Range=0 indicates perfect agreement across all three raters; Range=1 indicates that no rater differed more than 1 point from any other rater (e.g., 4, 5, 4); Range=2 indicates that no rater differed more than 2 points from any other rater (e.g., 3, 4, 5)
MI for Implementation Coaching
HETTEMA et al.
Table 3 Description of coaching calls MITI rating category
Mean (SD) range
Beginning proficiency
Competency
Evocation
4.4 (0.56) Range: 3–5 4.3 (0.68) Range: 3–5 4.4 (.66) Range: 3–5 4.8 (0.39) Range: 3–5 4.4 (0.49) Range: 3–5 4.4 (.49) Range: 3.33–5 1.2 (0.6) Range: 0.5–2.6 0.52 (0.18) Range: 0.14–1.00 0.42 (0.15) Range: 0.15–0.75 0.95 (0.11) Range: 0.56–1.00 5.5 (3.1) Range:1–3 3.1 (2.2) Range: 0–9 5.7 (3.6) Range: 1–17 0.7 (1.3) Range: 0–5 0.4 (0.8) Range: 0–3 5.8 (3.1) Range: 0–12 6.3 (2.8) Range: 1–14 7.9 (4.1) Range: 2–20 5.5 (3.0) Range: 1–14
3.5
4
3.5
4
3.5
4
3.5
4
3.5
4
Average of 3.5
Average of 4
1
2
50%
70%
40%
50%
90%
100%
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Collaboration Autonomy support Direction Empathy Global Spirit Reflection to question ratio Percent open questions Percent complex reflections Percent MI adherence Giving information Structuring MI adherent MI nonadherent (confront, direct) Advice without permission Closed questions Open questions Simple reflection Complex reflection
barriers.10 As a strategy for helping individuals explore and resolve ambivalence, MI has the potential to be applicable not only to the clinical issues patients face, but to the decisions of providers or administrators of patient care. The parent study to the current report found that, as part of a larger active
The Journal of Behavioral Health Services & Research
2013
dissemination strategy, MI-based interactive implementation coaching Webinars were effective in promoting progression in the adoption of MI as measured by stages of change-based instrument. These promising findings encouraged the authors to more comprehensively describe the content and style of the coaching Webinars and their consistency with the principles and practices of MI. To objectively assess the fidelity to the MI model, the MITI was applied to all available coaching sessions. Application of the MITI to this type of interaction was feasible, and rates of IRR on measured variables were good. Lastly, as predicted, fidelity to MI in the coaching Webinars was high across all measured constructs. The findings suggest that MI is likely an applicable and effective strategy for conducting implementation coaching. When interpreting the findings, several limitations should be considered. First, the current results are yielded from one sample of organizations and may not be broadly generalizable. For one, participants in the current study may already have had higher than average interest in patientcentered techniques such as MI or in EBPs in general, by virtue of their membership in the National Council for Community Behavioral Healthcare and the National Association for Community Health Centers (recruitment pools) or their decision to participate in a study on this topic. It is difficult to say whether the observed effects would be similar with different types of organizations. In addition, the coaches who conducted coaching Webinars in the current trial had extensive previous training and consultation experience. It is possible that coaches with less experience may not be as effective at implementing implementation coaching or adhere as consistently to the MI model. Additionally, the active dissemination strategy found to be effective in the parent trial had several components in addition to the MI-based interactive implementation coaching Webinar, including informational packets and a didactic Webinar. Although the results provide promising evidence of the potential effectiveness of MI as an implementation coaching strategy, it is possible that this was not the active component of the intervention. Additional research is needed to confirm the effectiveness of an MI approach and learn more about the generalizability of MI applications to implementation coaching. Future studies could examine outcome variables in addition to decision making that include further steps in implementation, or actual clinician performance or patient outcomes. Based on the research of MI in clinical settings, further work could also be done looking at the potential mechanisms of action of MI in implementation settings. Mechanisms of action are those active components of an intervention that are thought to causally impact results. This could include research on the relationship between MI fidelity in coaching interactions and outcomes to determine whether high or low adherence to the model, or specific components of the model, allow us to predict differences in outcome. In addition, future research could examine the language of the participants receiving coaching to determine whether mechanisms of action that appear to impact clinical outcomes, such as the expression of change talk, also drive the effectiveness of MI in implementation settings. Lastly, it may be helpful to address whether this model of implementation coaching is also useful for implementation efforts involving other EBPs or organizational changes. Making a decision to adopt an EBP can be influenced by communication strategies, including those used by implementation coaches or consultants. These communication efforts may first exert their effects in the knowledge stage of diffusion, during which potential adopters are exposed to an EBP. In some cases, this is an inherent alignment with the underlying spirit of MI. In other cases, due to the receipt of previous dissemination efforts, individuals may already be very or somewhat familiar with many of the practices of MI. Lastly, organization representatives have many ideas about what strategies will or will not be effective in implementing MI in their setting. When promoting knowledge, avoid unnecessary information provision and communicating information in a respectful and collaborative way; this may enhance the impact of this stage on later adoption decisions. Similarly, during the persuasion stage, communication strategies can influence whether individuals form positive or negative attitudes toward EBPs. In the MI model, it is acknowledged that when individuals are ambivalent about making a change, attempting to push them toward a
MI for Implementation Coaching
HETTEMA et al.
specific change typically results in withdrawal or resistance. Instead, evocation strategies can be used to draw out the person's own motivations for change and ideas about how change could occur.
Implications for Behavioral Health There appear to be some universal characteristics of the change process that occur regardless of the type of change being targeted. Just as MI practices may be effective in promoting change among patients, they may also be effective skills to use with those who provide care. The successful implementation of EBPs is a critical public health issue. The findings of the current report suggest that MI might be useful, not only as an EBP, but as a component of an effective strategy for facilitating implementation of other EBPs.
Acknowledgments Special thanks to Ali Hall and Robert Rhode for their work in developing and conducting the implementation Webinars and our SAMHSA Project Officer, Kevin Hennessy, Ph.D., for his thoughtful comments and feedback during the preparation of this manuscript. This study was supported by SAMHSA contract no. HHSS283200700040I/HHSS28342003T. Conflicts of Interest There are no conflicts of interest. Disclaimers This report was prepared under contract by MANILA Consulting Group, Inc., for SAMHSA. The content of this publication does not necessarily reflect the views or policies of SAMHSA or the US Department of Health and Human Services.
References 1. Miller WR, Rollnick S. Motivational interviewing: helping people change, 3rd ed. New York, NY: Guilford Press; 2013. 2. Hettema JE, Steele J, Miller WR. Motivational interviewing. Annual Review of Clinical Psychology. 2005;1:91–111. 3. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009. 4. Miller WR, Wilbourne PL, Hettema JE. In: Hester RK, Miller WR, eds. What works? A summary of alcohol treatment outcome research, 3rd ed. Boston, MA: Allyn & Bacon; 2003:13–63. 5. Williams JR, Tregear SJ, Dusablon T, et al. Increasing adoption of comparative effectiveness research in community behavioral health: methodology. Journal of Behavioral Health Services and Research. In press. 6. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: Department of Health and Human Services; 2009. 7. Cooperrider DL, Whitney D. Appreciative inquiry: a positive revolution in change. San Francisco: Berrett-Koehler Publishers; 2005. 8. Rogers EM. Diffusion of Innovations, 5th ed. New York, NY: Free Press; 2003. 9. Prochaska JO, DiClemente CC. The Transtheoretical Approach: Crossing Traditional Boundaries of Therapy. Homewood, IL: Dow Jones-Irwin; 1984. 10. Miller WR, Sorensen JL, Selzer JA, et al. Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. Journal of Substance Abuse Treatment. 2006;31:25–39. 11. Miller WR, Yahne CE, Moyers TB, et al. A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology. 2004;72:1050–1062. 12. Moyers TB, Martin T, Houck JM, et al. From in-session behaviors to drinking outcomes: a causal chain for motivational interviewing. Journal of Consulting and Clinical Psychology, 2009;77:113–1124 13. Miller WR, Benefield RG, Tonigan JS. Enhancing motivation for change in problem drinking: a controlled comparison of two therapist styles. Journal of Consulting and Clinical Psychology. 1993;61:455–461. 14. Martino S, Ball SA, Gallon SL, et al. Motivational interviewing assessment: supervisory tools for enhancing proficiency. Salem, OR: Northwest Frontier Addiction Technology Transfer Center, Oregon Health and Science University; 2006. 15. Strang J, McCambridge J, Platts S, et al. Engaging the reluctant GP in care of the opiate misuser: pilot study of change-orientated reflective listening (CORL). Family Practice. 2004;21:150–154.
The Journal of Behavioral Health Services & Research
2013
16. Hettema JE, Sorensen JL, Uy M, et al. Motivational enhancement therapy to increase resident physician engagement in substance abuse education. Substance Abuse. 2009;30:244–247. 17. Ingersoll KS, Wagner CC, Gharib S. Motivational groups for community substance abuse programs. 2002. Available at: http://people.uncw.edu/ogler/MI%20Groups%20for%20Com%20SA%20Prog.pdf. Accessed October 31, 2012. 18. Moyers TB, Martin T, Manuel JK, et al. Revised global scales: motivational interviewing treatment integrity 3.1.1 (MITI 3.1.1). 2010. Available at: http://casaa.unm.edu/download/MITI3_1.pdf Accessed October 31, 2012. 19. Moyers TB, Martin T, Manuel JK, et al. Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment. 2005;28:19–26. 20. Miller WR, Moyers TB, Ernst D, et al. Manual for the Motivational Interviewing Skills Code (MISC), Version 2.1. 2008. Available at: http://casaa.unm.edu/download/misc.pdf Accessed October 31, 2012. 21. Tollison SJ, Lee CM, Neighbors C, et al. Questions and reflections: The use of the motivational interviewing microskills in a peer-led brief alcohol intervention for college students. Behavior Therapy. 2007;39:183–194. 22. Moyers TB, Martin T. Therapist influence on client language during motivational interviewing sessions. Journal of Substance Abuse Treatment. 2006;30:245–251. 23. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86:420–428. 24. Cicchetti DV, Sparrow SA. Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency. 1981;2:127–137. 25. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Special section: normative assessment. Psychological Assessment. 1994;6:284–290. 26. Tammemagi M, Frank J, Streiner D. Negative values of the intraclass correlation coefficient are not theoretically possible: response. Journal of Clinical Epidemiology. 1996;49:1206. 27. Randolph JJ. Free-marginal multirater kappa: an alternative to Fleiss' fixed-marginal multirater kappa. Paper presented at: The Joensuu University Learning and Instruction Symposium; October 14–15, 2005; Joensuu, Finland. (ERIC Document Reproduction Service No. ED490661.) 28. Randolph JJ. Online kappa calculator. 2008. Available at: http://justus.randolph.name/kappa. Accessed October 31, 2012. 29. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. 30. Bakeman R, Quera V, McArthur D, et al. Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods. 1997;2:357–370.
MI for Implementation Coaching
HETTEMA et al.
