Clinical Rehabilitation http://cre.sagepub.com/
An evaluation of treatment integrity in a randomised controlled trial of memory rehabilitation for people with multiple sclerosis Kathryn J Smale, Sara E Carr, Annette F Schwartz, Roshan das Nair and Nadina B Lincoln Clin Rehabil published online 19 September 2014 DOI: 10.1177/0269215514548733 The online version of this article can be found at: http://cre.sagepub.com/content/early/2014/09/18/0269215514548733
Published by: http://www.sagepublications.com
Additional services and information for Clinical Rehabilitation can be found at: Email Alerts: http://cre.sagepub.com/cgi/alerts Subscriptions: http://cre.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav
>> OnlineFirst Version of Record - Sep 19, 2014 What is This?
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
548733
research-article2014
CRE0010.1177/0269215514548733Clinical RehabilitationSmale et al.
CLINICAL REHABILITATION
Article
An evaluation of treatment integrity in a randomised controlled trial of memory rehabilitation for people with multiple sclerosis
Clinical Rehabilitation 1–7 © The Author(s) 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/0269215514548733 cre.sagepub.com
Kathryn J Smale1, Sara E Carr2, Annette F Schwartz2, Roshan das Nair1 and Nadina B Lincoln1
Abstract Objective: To assess the treatment integrity of a memory rehabilitation programme for people with multiple sclerosis. Subjects: Data were drawn from the intervention group of a randomised controlled trial of memory rehabilitation. This comprised 24 participants with multiple sclerosis. Measures: Four core session components were identified from the treatment manual: recap, activities, take-home activity and other. One video-recording of each of ten intervention sessions was transcribed and amount of time spent on components recorded. Results: There were no significant differences between early and late stages of the programme in time spent on the core components (recap Z=–0.87, P=0.49; activities Z = –0.29, P=0.89; take-home activity Z = –0.59, P=0.69; other Z = –0.58, P=0.69). Thus, adherence to the manual was good with no evidence of programme drift. Conclusions: Good adherence indicates the intervention was delivered as described in the manual and strengthens confidence in the findings of the randomised controlled trial. Keywords Memory rehabilitation, treatment integrity, multiple sclerosis, process evaluation Received: 17 December 2013; accepted: 26 July 2014
Introduction Memory rehabilitation in multiple sclerosis (MS) is in its relative infancy. As such, documenting treatment integrity is a key issue in research and clinical practice. Treatment integrity refers to the degree to which an intervention is delivered as intended. Monitoring treatment integrity increases confidence
1Division
of Rehabilitation and Ageing, University of Nottingham, UK 2Central Surrey Health, Surrey, UK Corresponding author: Kathryn Smale, Division of Rehabilitation and Ageing, University of Nottingham, Nottingham, NG7 2UH, UK. Email: [email protected]
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
2
Clinical Rehabilitation
that significant results are due to effective treatment, or that non-significant results are due to an ineffective intervention, rather than poor implementation. Treatment integrity is typically achieved through comprehensive training and supervision, and demonstrated by measuring therapist adherence to the manual.1 There are a variety of possible methods for evaluating adherence. The ideal method is to observe sessions and code them on a recording form according to predetermined criteria.2,3 Time-sampling by independent observers provides the most objective and systematic documentation of content.4 Assessment of treatment integrity in memory rehabilitation studies is important, since these studies typically lack detailed information on the amount of time spent on session components and adherence to treatment manuals. Memory rehabilitation programmes aim to improve everyday memory functioning, a common problem for people with multiple sclerosis.5 The current research base from which to draw inferences for the use of memory rehabilitation in clinical practice is small and further studies with better methodological quality are required.6–8 Accordingly, this study evaluated the treatment integrity of a randomised controlled trial of memory rehabilitation for people with MS.9 In this study participants with memory problems and multiple sclerosis were randomly allocated to receive a group memory rehabilitation programme or to a waiting list control group. The intervention was run by an assistant psychologist who received weekly supervision from a consultant clinical neuropsychologist. Comparison of outcomes showed a significant difference between the intervention group and control group on a measure of mood, the General Health Questionnaire -2810, but no significant differences in memory problems in daily life as assessed on the Everyday Memory Questionnaire11 or on the MS Impact Scale12. Feedback from participants during the final treatment session was positive, indicating that the intervention made a difference to how they coped with memory difficulties, including being able to use specific strategies, improved confidence and reduced feelings of being the only one with memory problems.
Accordingly, the aims of this study were to: 1) Assess the extent to which the memory rehabilitation treatment manual was adhered to in the randomised controlled trial. 2) Establish benchmark timings for treatment components. 3) Assess the opinion of the therapist delivering the treatment regarding the content and format of the intervention.
Methods Ethical approval for the study was obtained from the South West London Research Ethics Committee. Participants were recruited from those who attended a community neurorehabilitation service and informed consent was obtained on recruitment to the trial. Participants also gave consent for video recording of treatment sessions. Details of the trial are provided elsewhere,9 but in summary, participants with multiple sclerosis were recruited who reported memory problems in daily life, were more than 12 months since diagnosis, able to give informed consent, able to speak and understand conversational English, and able to attend the outpatient unit where the treatment sessions were delivered. Patients with very severe memory problems who were considered by the consultant clinical psychologist or multiple sclerosis specialist nurse not to be able to cope with group sessions were excluded. Participants were randomly allocated to memory rehabilitation groups, each with eight group members, or to a waiting list control group. The programme consisted of 10 sessions, each lasting approximately 90 minutes. The 10 sessions included an introductory session, a session on memory and memory problems, two sessions on attention, three on internal memory aids and two on external memory aids. The tenth session was a review of the programme. Participants completed a questionnaire about the intervention at the end of the last session. Outcomes were assessed at four or eight months after randomisation. There were no significant differences between the two groups on the Everyday Memory Questionnaire or MS Impact Scale (P >
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
3
Smale et al. 0.05) at either outcome time-point. However, the intervention group reported significantly better mood than controls on the General Health Questionnaire -28 at eight months (P = 0.04). The feedback from participants who received the intervention was generally positive and some suggestions for changes to the programme were provided. Adherence to the treatment manual was examined by time-sampling of video recordings of intervention sessions. One of each of the 10 sessions was chosen to be analysed using a computerised random number generator. This ensured that adherence to the whole manual was assessed. A recording form was developed by studying the manual for key components prior to observation. Four key session components were identified: recap, activities, takehome activity, and other. These were further broken down into different subcomponents for each session. Recap incorporated a summary of the previous session and a review of the take-home activity. Activities formed the bulk of the manual content and incorporated strategies to address memory problems, case examples, and group and individual tasks. A take-home activity was set for the group each week to provide an opportunity to put what was learnt into practice in their everyday lives. As the introduction to the programme, session one did not include a recap section. Similarly, the final session did not include a take-home activity. Videos of the sessions were observed and transcribed verbatim. The content of each one-minute unit was documented on a record form. More than one component could be present in each one-minute interval, but in order to be recorded as content it needed to be discussed in at least one complete sentence during the time period. Since this was a pilot study, there was no ‘gold standard’ against which to test the amount of time spent on each component of treatment. Programme drift, whereby therapists deviate from the treatment manual, was, therefore, the main measure used to assess adherence to the manual. Systematic documentation of therapy content provided a benchmark against which to assess treatment integrity in future trials. Data were tested for normality using the Shapiro-Wilk test, since the study had less than 50
participants. Three of the categories were normally distributed, but the ‘other’ category was not, therefore Mann-Whitney comparisons were conducted. Given there are not yet any benchmark timings against which to test for programme drift, this was assessed by comparing early (sessions 2-5) and late (sessions 6-9) stages of the programme. Sessions one and 10 were excluded from analysis of programme drift as their core components varied slightly from the rest of the programme. All data were analysed using SPSS version 19. In addition, an interview was conducted with the therapist. This included questions about the format of the manual, the participants, changes that might be made to the manual, the selection of participants, the venue, and the organisation of sessions. Notes were made during the interview to summarise the key points.
Results Twenty-four participants took part in the memory rehabilitation programme. They were aged between 35 and 72 years (mean 55.8; SD 10.2), with 18 (75%) women. Length of time spent on treatment components in early and late stages of the programme are shown in Table 1. Mann-Whitney comparisons indicated there were no significant differences between the early and late stages in the time spent on any of the four core components. Thus, there was no evidence of programme drift. All the elements in the manual were included in the sessions. The only slight alteration was in session one where group members discussed the efficacy of strategies rather than numerically rating them as recommended in the manual. The amount of time spent on the core components of each session is shown in Table 2. Time as a percentage of the total content is also given, since more than one component could occur within the one-minute intervals. Most time was spent on ‘activities’ and ‘other’ components. General conversation accounted for the majority of the ‘other’ category. This tended to
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
4
Clinical Rehabilitation
Table 1. Median length of time (in minutes) spent on treatment components across early and late stages of the memory rehabilitation programme. Component of therapy
Recap Activities Take-home activity Other
Early (sessions 2-5)
Late (sessions 6-9)
Comparison using Mann Whitney U
Median (IQR)
Median (IQR)
Z
P
19.5 (8) 58 (17) 4.5 (2) 29.5 (28.25)
16 (6.75) 61.5 (11.5) 2.5 (2.25) 19 (8.5)
−0.87 −0.29 −0.59 −0.58
0.49 0.89 0.69 0.69
Table 2. Time (in minutes) spent on each session component. Component of intervention
Session
Minutes
1
2
3
4
5
6
7
8
9
10
Mean
SD
Mean
SD
Recap Activities Take-home activity Other Total content Length of session
– 72 3 36 111 87
12 34 7 44 97 84
23 53 4 43 123 93
23 63 5 12 103 90
16 72 2 16 106 95
22 50 2 23 97 80
18 69 2 13 102 90
14 57 3 23 97 83
7 66 8 15 96 91
6 55 – 31 92 88
15.67 59.10 4.00 25.60
6.50 11.86 2.24 12.20
15.23 57.86 3.93 24.90
5.83 11.35 2.38 11.36
centre on memory and topics discussed in previous sessions. Group members also compared other symptoms of MS, medication and experiences of living with the condition. Extra activities were included in sessions two and three. For instance, extra case examples and a video clip were used in session 2, and session 3, which focussed on attention, included a face recognition activity not in the manual. In session 7, the group ran out of time so one of the activities was done in session 8 instead. In both cases, these additional activities were recorded in the ‘other’ category. According to the therapist, the manual was well structured and provided useful background to memory problems. The overall programme was considered to be worthwhile. Although PowerPoint presentations were provided for each session, these were only found to be useful in the first session and it was easier to present the materials on a flip chart. This was partly due to lack of space at the venue. The sequence of sessions was considered to be satisfactory, although people wanted to talk about
% of total content
external memory aids early on because they were easy to understand. The reading level required was generally acceptable. Some language was changed from the draft manual and illustrations were added to reduce the text. Further strategies were suggested for inclusion in the manual. These were to include a categorisation task when discussing internal memory aids. In addition, time management was discussed, i.e. plan, prioritise, and pace, which was requested by the groups as many experienced fatigue. Case examples were useful and discussed in group sessions in order to help participants apply new strategies to their daily lives. Although sessions were kept to 1.5 hours, which was a time frame set at the beginning of the study, the therapist considered it would be better if there was scope for longer sessions i.e. up to 2 hours. In particular, more time was needed for the discussion of homework, and to allow extra discussion around topics that were pertinent to each group (such as how fatigue interacts with memory).
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
5
Smale et al. Most participants reported living at home alone and the groups gave them something to do. Many wanted information about memory problems. There were very different levels of memory ability within the group, and it was considered by the therapist that it might have been beneficial to have separate high level and low level memory ability groups. Specifically, those with more severe memory problems may have benefitted from being in a smaller group which was more geared to their level of functioning as they tended to work slower and needed more individual help. For instance, severe memory difficulties were sometimes coupled with more severe physical difficulties and the therapist was required to help write in participants’ booklets, or repeat flip chart slides and discussion points. However, the therapist reported that those with milder memory problems were sometimes good for the smooth running of the group as they were able to keep the discussions going and would often help the less able participants. However, the participants with mild memory difficulties expressed concern at being placed in the mixed memory ability groups due to fear of how their MS would develop over time (avoidance of thinking about the development of MS was a reported coping strategy). The therapist and participants suggested several changes to the programme. One was to call tasks set ‘take-home activities’ rather than homework, due to associations (negative or otherwise) with school. The letter cancellation, attention training task, was seen as tedious to do and participants did not like it. The suggestion was to use Sudoku or crosswords as an alternative but these might have lesser attentional demands. There were practical problems because the room used on some occasions was too small. This also meant that it was difficult to have a relaxing break mid-session and so participants left the room whenever they needed a break. The venue also changed and it would have been better to have a consistent venue. The group size affected the success of the programme. The intended group size was six to eight people. However, when all eight were present this was too many and the ideal group size was six. Similarly, the smallest attendance at a group session was three people and this was not enough as
there was very little discussion and material was covered too quickly. Attendance was mainly good with attendance rates of 80-90%. However, one group only had 66% attendance which made delivering the programme difficult. In this group two dropped out before group started, one was ill, and one was too busy to attend. Delivery of the programme was affected by relapses. For research purposes relapses needed to be recorded both during intervention period and at assessment. Although face-to-face sessions were 1.5 hours, it was estimated that it took two to six hours preparing for each session. Facilitator notes were written for each session. In addition, preparation was needed in order to tailor contents to individual participant abilities and circumstances. If participants missed sessions they could make up some of the missed materials by attending early for the following session. This also took up additional therapist time.
Discussion This study revealed that, overall, adherence to the treatment manual was good. Systematic documentation of the intervention content indicated that the manual was largely acceptable. This study employed a thorough and replicable way of assessing treatment integrity. All the core components identified in advance were included and there was no evidence of programme drift between the early and late stages of the programme. This improves confidence in the therapist’s adherence to the manual as well as indicating that training and supervision were adequate.13 It is also consistent with previous observations of memory rehabilitation that suggest the interventions can be delivered in the manner intended from a manual14 and that this is not something specific to interventions with people with multiple sclerosis. Manualisation is important in reducing variance in treatment delivery and enabling replication of research. Nevertheless, flexibility within fidelity is important in therapeutic interventions;15,16 clinical skill, creativity and sensitivity to the needs of group members play a vital role in the proper implementation.15,17 A strength of this manual is that it gave the therapist a degree of flexibility. Importantly it allowed plenty of time for discussion which
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
6
Clinical Rehabilitation
provided the opportunity to learn from each other and normalise the consequences of illness, a wellestablished benefit of delivering interventions in a group setting.17 However, several additional activities were included and whether these should be incorporated into the manual or whether it is more appropriate to allow the therapist flexibility in adding or tailoring activities needs consideration. There are a number of limitations with this study. The small sample size reduces confidence in the findings and compromises statistical power. In addition, having more than one rater would establish inter-rater reliability, help improve accuracy and confirm descriptions of content.18,19 Similarly, although the structure of the manual was clear, ideally a second person should have agreed on the session components. There was only one therapist and one site, and this may have reduced the variance seen in the adherence to the manual. We have to some extent mitigated this by randomly selecting the sessions from the various groups in our analysis. Using programme drift as the sole quantitative measure of adherence offers only an indication of treatment integrity. Nevertheless, this study provides a benchmark against which future studies can evaluate treatment integrity. The interview with the therapist was not audio recorded and so interpretation relied on notes made by two interviewers at the time of the interview. Despite these limitations, this study provides a framework for documenting integrity of manualised treatments of complex interventions. Clinical messages •• Adherence to the treatment manual was good. •• Consideration of flexibility within fidelity and therapist tailoring of activities needs consideration. •• Feedback from the therapist and participants highlighted the need for changes to the manual and practical difficulties in delivering the programme.
Acknowledgements We would like to express our thanks to Liz Wilkinson, MS Specialist Nurse at Central Surrey Health,
for her involvement in the study and identifying individuals to participate. We also thank all the participants for their involvement in the study. Conflict of interest The authors declare that there is no conflict of interest.
Funding The research was supported by a research grant from Biogen Idec Limited, Maidenhead, Berkshire.
References 1. Hildebrand MW, Host HH, Binder EF, et al. Measuring treatment fidelity in a rehabilitation intervention study. Am J Phys Med Rehabil 2012; 91: 715–724. 2. Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychol 2004; 23: 443–451. 3. Thomas SA, Russell C, Seed R, et al. An evaluation of treatment integrity in a randomised trial of behavioural therapy for low mood in stroke patients with aphasia. Clin Rehabil 2013; 27: 1097–1106. 4. O’Brien MC, das Nair R and Lincoln NB. A comparison of the content of memory rehabilitation groups for patients with neurological disabilities. Neuropsychol Rehabil 2013; 23: 321–332. 5. Prakash RS, Snook EM, Lewis JM, et al. Cognitive impairments in relapsing-remitting multiple sclerosis: a meta-analysis. Mult Scler 2008;14:1250–1261. 6. Das Nair R, Ferguson H, Stark DL, et al. Memory Rehabilitation for people with multiple sclerosis. Cochrane Database Syst Rev 2012; 3: CD008754. 7. Das Nair R and Lincoln NB. Evaluation of Rehabilitation of Memory in Neurological Disabilities (ReMiND): a randomized controlled trial. Clin Rehabil 2012; 26: 894–903. 8. O’Brien AR, Chiaravallod N, Goverover Y, et al. Evidenced-based cognitive rehabilitation for persons with multiple sclerosis: A review of the literature. Arch Physl Med Rehabil 2008; 89: 761–769. 9. Carr S, das Nair R, Schwartz A, et al. Group memory rehabilitation for people with multiple sclerosis: a pilot randomised controlled trial. Clin Rehabil 2014; 28: 552–561. 10. Goldberg D and Williams P. A Users Guide to the General Health Questionnaire. Berkshire: Nfer-Nelson Ltd., 1988. 11. Sunderland A, Harris J and Baddeley A. Assessing everyday memory after severe head injury. In: Harris J and Morris P (eds) Everyday memory, actions and absentmindedness. London: Academic Press, 1984, pp. 191–206. 12. Hobart J, Lamping D, Fitzpatrick R, Riazi A and Thompson A. The Multiple Sclerosis Impact Scale (MSIS-29): a new patient-based outcome measure. Brain 2001; 124: 962–973.
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
7
Smale et al. 13. Moncher FJ and Prinz RJ. Treatment fidelity in outcome studies. Clin Psychol Rev 1991;11: 247–266. 14. O’Brien MC, Das Nair R and Lincoln NB. A comparison of the content of memory rehabilitation groups for patients with neurological disabilities. Neuropsychological Rehabilitation 2013; 23: 321–332. 15. Kendall PC and Beidas RS. Smoothing the trail for dissemination of evidence-based practices for youth: flexibility within fidelity. Prof Psychol Res Pr 2007; 38:13–20. 16. Kendall PC, Gosch E, Furr JM, et al. Flexibility within fidelity. J Am Acad Child Adolesc Psychiatry 2008; 47: 987–993.
17. Yalom I and Leszcz M. The theory and practice of group psychotherapy. 5th ed. New York: Basic Books, 2005. 18. Lombard M, Snyder-Duch J and Bracken CC. Content analysis in mass communication: assessment and reporting of intercoder reliability. Hum Commun Res 2002; 28: 587–604. 19. De Wit L, Kamsteegt H, Yadav B, et al. Defining the content of individual physiotherapy and occupational therapy sessions for stroke patients in an inpatient rehabilitation setting. Development, validation and inter-rater reliability of a scoring list. Clin Rehabil 2007; 21: 450–459.
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
An evaluation of treatment integrity in a randomised controlled trial of memory rehabilitation for people with multiple sclerosis Kathryn J Smale, Sara E Carr, Annette F Schwartz, Roshan das Nair and Nadina B Lincoln Clin Rehabil published online 19 September 2014 DOI: 10.1177/0269215514548733 The online version of this article can be found at: http://cre.sagepub.com/content/early/2014/09/18/0269215514548733
Published by: http://www.sagepublications.com
Additional services and information for Clinical Rehabilitation can be found at: Email Alerts: http://cre.sagepub.com/cgi/alerts Subscriptions: http://cre.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav
>> OnlineFirst Version of Record - Sep 19, 2014 What is This?
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
548733
research-article2014
CRE0010.1177/0269215514548733Clinical RehabilitationSmale et al.
CLINICAL REHABILITATION
Article
An evaluation of treatment integrity in a randomised controlled trial of memory rehabilitation for people with multiple sclerosis
Clinical Rehabilitation 1–7 © The Author(s) 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/0269215514548733 cre.sagepub.com
Kathryn J Smale1, Sara E Carr2, Annette F Schwartz2, Roshan das Nair1 and Nadina B Lincoln1
Abstract Objective: To assess the treatment integrity of a memory rehabilitation programme for people with multiple sclerosis. Subjects: Data were drawn from the intervention group of a randomised controlled trial of memory rehabilitation. This comprised 24 participants with multiple sclerosis. Measures: Four core session components were identified from the treatment manual: recap, activities, take-home activity and other. One video-recording of each of ten intervention sessions was transcribed and amount of time spent on components recorded. Results: There were no significant differences between early and late stages of the programme in time spent on the core components (recap Z=–0.87, P=0.49; activities Z = –0.29, P=0.89; take-home activity Z = –0.59, P=0.69; other Z = –0.58, P=0.69). Thus, adherence to the manual was good with no evidence of programme drift. Conclusions: Good adherence indicates the intervention was delivered as described in the manual and strengthens confidence in the findings of the randomised controlled trial. Keywords Memory rehabilitation, treatment integrity, multiple sclerosis, process evaluation Received: 17 December 2013; accepted: 26 July 2014
Introduction Memory rehabilitation in multiple sclerosis (MS) is in its relative infancy. As such, documenting treatment integrity is a key issue in research and clinical practice. Treatment integrity refers to the degree to which an intervention is delivered as intended. Monitoring treatment integrity increases confidence
1Division
of Rehabilitation and Ageing, University of Nottingham, UK 2Central Surrey Health, Surrey, UK Corresponding author: Kathryn Smale, Division of Rehabilitation and Ageing, University of Nottingham, Nottingham, NG7 2UH, UK. Email: [email protected]
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
2
Clinical Rehabilitation
that significant results are due to effective treatment, or that non-significant results are due to an ineffective intervention, rather than poor implementation. Treatment integrity is typically achieved through comprehensive training and supervision, and demonstrated by measuring therapist adherence to the manual.1 There are a variety of possible methods for evaluating adherence. The ideal method is to observe sessions and code them on a recording form according to predetermined criteria.2,3 Time-sampling by independent observers provides the most objective and systematic documentation of content.4 Assessment of treatment integrity in memory rehabilitation studies is important, since these studies typically lack detailed information on the amount of time spent on session components and adherence to treatment manuals. Memory rehabilitation programmes aim to improve everyday memory functioning, a common problem for people with multiple sclerosis.5 The current research base from which to draw inferences for the use of memory rehabilitation in clinical practice is small and further studies with better methodological quality are required.6–8 Accordingly, this study evaluated the treatment integrity of a randomised controlled trial of memory rehabilitation for people with MS.9 In this study participants with memory problems and multiple sclerosis were randomly allocated to receive a group memory rehabilitation programme or to a waiting list control group. The intervention was run by an assistant psychologist who received weekly supervision from a consultant clinical neuropsychologist. Comparison of outcomes showed a significant difference between the intervention group and control group on a measure of mood, the General Health Questionnaire -2810, but no significant differences in memory problems in daily life as assessed on the Everyday Memory Questionnaire11 or on the MS Impact Scale12. Feedback from participants during the final treatment session was positive, indicating that the intervention made a difference to how they coped with memory difficulties, including being able to use specific strategies, improved confidence and reduced feelings of being the only one with memory problems.
Accordingly, the aims of this study were to: 1) Assess the extent to which the memory rehabilitation treatment manual was adhered to in the randomised controlled trial. 2) Establish benchmark timings for treatment components. 3) Assess the opinion of the therapist delivering the treatment regarding the content and format of the intervention.
Methods Ethical approval for the study was obtained from the South West London Research Ethics Committee. Participants were recruited from those who attended a community neurorehabilitation service and informed consent was obtained on recruitment to the trial. Participants also gave consent for video recording of treatment sessions. Details of the trial are provided elsewhere,9 but in summary, participants with multiple sclerosis were recruited who reported memory problems in daily life, were more than 12 months since diagnosis, able to give informed consent, able to speak and understand conversational English, and able to attend the outpatient unit where the treatment sessions were delivered. Patients with very severe memory problems who were considered by the consultant clinical psychologist or multiple sclerosis specialist nurse not to be able to cope with group sessions were excluded. Participants were randomly allocated to memory rehabilitation groups, each with eight group members, or to a waiting list control group. The programme consisted of 10 sessions, each lasting approximately 90 minutes. The 10 sessions included an introductory session, a session on memory and memory problems, two sessions on attention, three on internal memory aids and two on external memory aids. The tenth session was a review of the programme. Participants completed a questionnaire about the intervention at the end of the last session. Outcomes were assessed at four or eight months after randomisation. There were no significant differences between the two groups on the Everyday Memory Questionnaire or MS Impact Scale (P >
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
3
Smale et al. 0.05) at either outcome time-point. However, the intervention group reported significantly better mood than controls on the General Health Questionnaire -28 at eight months (P = 0.04). The feedback from participants who received the intervention was generally positive and some suggestions for changes to the programme were provided. Adherence to the treatment manual was examined by time-sampling of video recordings of intervention sessions. One of each of the 10 sessions was chosen to be analysed using a computerised random number generator. This ensured that adherence to the whole manual was assessed. A recording form was developed by studying the manual for key components prior to observation. Four key session components were identified: recap, activities, takehome activity, and other. These were further broken down into different subcomponents for each session. Recap incorporated a summary of the previous session and a review of the take-home activity. Activities formed the bulk of the manual content and incorporated strategies to address memory problems, case examples, and group and individual tasks. A take-home activity was set for the group each week to provide an opportunity to put what was learnt into practice in their everyday lives. As the introduction to the programme, session one did not include a recap section. Similarly, the final session did not include a take-home activity. Videos of the sessions were observed and transcribed verbatim. The content of each one-minute unit was documented on a record form. More than one component could be present in each one-minute interval, but in order to be recorded as content it needed to be discussed in at least one complete sentence during the time period. Since this was a pilot study, there was no ‘gold standard’ against which to test the amount of time spent on each component of treatment. Programme drift, whereby therapists deviate from the treatment manual, was, therefore, the main measure used to assess adherence to the manual. Systematic documentation of therapy content provided a benchmark against which to assess treatment integrity in future trials. Data were tested for normality using the Shapiro-Wilk test, since the study had less than 50
participants. Three of the categories were normally distributed, but the ‘other’ category was not, therefore Mann-Whitney comparisons were conducted. Given there are not yet any benchmark timings against which to test for programme drift, this was assessed by comparing early (sessions 2-5) and late (sessions 6-9) stages of the programme. Sessions one and 10 were excluded from analysis of programme drift as their core components varied slightly from the rest of the programme. All data were analysed using SPSS version 19. In addition, an interview was conducted with the therapist. This included questions about the format of the manual, the participants, changes that might be made to the manual, the selection of participants, the venue, and the organisation of sessions. Notes were made during the interview to summarise the key points.
Results Twenty-four participants took part in the memory rehabilitation programme. They were aged between 35 and 72 years (mean 55.8; SD 10.2), with 18 (75%) women. Length of time spent on treatment components in early and late stages of the programme are shown in Table 1. Mann-Whitney comparisons indicated there were no significant differences between the early and late stages in the time spent on any of the four core components. Thus, there was no evidence of programme drift. All the elements in the manual were included in the sessions. The only slight alteration was in session one where group members discussed the efficacy of strategies rather than numerically rating them as recommended in the manual. The amount of time spent on the core components of each session is shown in Table 2. Time as a percentage of the total content is also given, since more than one component could occur within the one-minute intervals. Most time was spent on ‘activities’ and ‘other’ components. General conversation accounted for the majority of the ‘other’ category. This tended to
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
4
Clinical Rehabilitation
Table 1. Median length of time (in minutes) spent on treatment components across early and late stages of the memory rehabilitation programme. Component of therapy
Recap Activities Take-home activity Other
Early (sessions 2-5)
Late (sessions 6-9)
Comparison using Mann Whitney U
Median (IQR)
Median (IQR)
Z
P
19.5 (8) 58 (17) 4.5 (2) 29.5 (28.25)
16 (6.75) 61.5 (11.5) 2.5 (2.25) 19 (8.5)
−0.87 −0.29 −0.59 −0.58
0.49 0.89 0.69 0.69
Table 2. Time (in minutes) spent on each session component. Component of intervention
Session
Minutes
1
2
3
4
5
6
7
8
9
10
Mean
SD
Mean
SD
Recap Activities Take-home activity Other Total content Length of session
– 72 3 36 111 87
12 34 7 44 97 84
23 53 4 43 123 93
23 63 5 12 103 90
16 72 2 16 106 95
22 50 2 23 97 80
18 69 2 13 102 90
14 57 3 23 97 83
7 66 8 15 96 91
6 55 – 31 92 88
15.67 59.10 4.00 25.60
6.50 11.86 2.24 12.20
15.23 57.86 3.93 24.90
5.83 11.35 2.38 11.36
centre on memory and topics discussed in previous sessions. Group members also compared other symptoms of MS, medication and experiences of living with the condition. Extra activities were included in sessions two and three. For instance, extra case examples and a video clip were used in session 2, and session 3, which focussed on attention, included a face recognition activity not in the manual. In session 7, the group ran out of time so one of the activities was done in session 8 instead. In both cases, these additional activities were recorded in the ‘other’ category. According to the therapist, the manual was well structured and provided useful background to memory problems. The overall programme was considered to be worthwhile. Although PowerPoint presentations were provided for each session, these were only found to be useful in the first session and it was easier to present the materials on a flip chart. This was partly due to lack of space at the venue. The sequence of sessions was considered to be satisfactory, although people wanted to talk about
% of total content
external memory aids early on because they were easy to understand. The reading level required was generally acceptable. Some language was changed from the draft manual and illustrations were added to reduce the text. Further strategies were suggested for inclusion in the manual. These were to include a categorisation task when discussing internal memory aids. In addition, time management was discussed, i.e. plan, prioritise, and pace, which was requested by the groups as many experienced fatigue. Case examples were useful and discussed in group sessions in order to help participants apply new strategies to their daily lives. Although sessions were kept to 1.5 hours, which was a time frame set at the beginning of the study, the therapist considered it would be better if there was scope for longer sessions i.e. up to 2 hours. In particular, more time was needed for the discussion of homework, and to allow extra discussion around topics that were pertinent to each group (such as how fatigue interacts with memory).
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
5
Smale et al. Most participants reported living at home alone and the groups gave them something to do. Many wanted information about memory problems. There were very different levels of memory ability within the group, and it was considered by the therapist that it might have been beneficial to have separate high level and low level memory ability groups. Specifically, those with more severe memory problems may have benefitted from being in a smaller group which was more geared to their level of functioning as they tended to work slower and needed more individual help. For instance, severe memory difficulties were sometimes coupled with more severe physical difficulties and the therapist was required to help write in participants’ booklets, or repeat flip chart slides and discussion points. However, the therapist reported that those with milder memory problems were sometimes good for the smooth running of the group as they were able to keep the discussions going and would often help the less able participants. However, the participants with mild memory difficulties expressed concern at being placed in the mixed memory ability groups due to fear of how their MS would develop over time (avoidance of thinking about the development of MS was a reported coping strategy). The therapist and participants suggested several changes to the programme. One was to call tasks set ‘take-home activities’ rather than homework, due to associations (negative or otherwise) with school. The letter cancellation, attention training task, was seen as tedious to do and participants did not like it. The suggestion was to use Sudoku or crosswords as an alternative but these might have lesser attentional demands. There were practical problems because the room used on some occasions was too small. This also meant that it was difficult to have a relaxing break mid-session and so participants left the room whenever they needed a break. The venue also changed and it would have been better to have a consistent venue. The group size affected the success of the programme. The intended group size was six to eight people. However, when all eight were present this was too many and the ideal group size was six. Similarly, the smallest attendance at a group session was three people and this was not enough as
there was very little discussion and material was covered too quickly. Attendance was mainly good with attendance rates of 80-90%. However, one group only had 66% attendance which made delivering the programme difficult. In this group two dropped out before group started, one was ill, and one was too busy to attend. Delivery of the programme was affected by relapses. For research purposes relapses needed to be recorded both during intervention period and at assessment. Although face-to-face sessions were 1.5 hours, it was estimated that it took two to six hours preparing for each session. Facilitator notes were written for each session. In addition, preparation was needed in order to tailor contents to individual participant abilities and circumstances. If participants missed sessions they could make up some of the missed materials by attending early for the following session. This also took up additional therapist time.
Discussion This study revealed that, overall, adherence to the treatment manual was good. Systematic documentation of the intervention content indicated that the manual was largely acceptable. This study employed a thorough and replicable way of assessing treatment integrity. All the core components identified in advance were included and there was no evidence of programme drift between the early and late stages of the programme. This improves confidence in the therapist’s adherence to the manual as well as indicating that training and supervision were adequate.13 It is also consistent with previous observations of memory rehabilitation that suggest the interventions can be delivered in the manner intended from a manual14 and that this is not something specific to interventions with people with multiple sclerosis. Manualisation is important in reducing variance in treatment delivery and enabling replication of research. Nevertheless, flexibility within fidelity is important in therapeutic interventions;15,16 clinical skill, creativity and sensitivity to the needs of group members play a vital role in the proper implementation.15,17 A strength of this manual is that it gave the therapist a degree of flexibility. Importantly it allowed plenty of time for discussion which
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
6
Clinical Rehabilitation
provided the opportunity to learn from each other and normalise the consequences of illness, a wellestablished benefit of delivering interventions in a group setting.17 However, several additional activities were included and whether these should be incorporated into the manual or whether it is more appropriate to allow the therapist flexibility in adding or tailoring activities needs consideration. There are a number of limitations with this study. The small sample size reduces confidence in the findings and compromises statistical power. In addition, having more than one rater would establish inter-rater reliability, help improve accuracy and confirm descriptions of content.18,19 Similarly, although the structure of the manual was clear, ideally a second person should have agreed on the session components. There was only one therapist and one site, and this may have reduced the variance seen in the adherence to the manual. We have to some extent mitigated this by randomly selecting the sessions from the various groups in our analysis. Using programme drift as the sole quantitative measure of adherence offers only an indication of treatment integrity. Nevertheless, this study provides a benchmark against which future studies can evaluate treatment integrity. The interview with the therapist was not audio recorded and so interpretation relied on notes made by two interviewers at the time of the interview. Despite these limitations, this study provides a framework for documenting integrity of manualised treatments of complex interventions. Clinical messages •• Adherence to the treatment manual was good. •• Consideration of flexibility within fidelity and therapist tailoring of activities needs consideration. •• Feedback from the therapist and participants highlighted the need for changes to the manual and practical difficulties in delivering the programme.
Acknowledgements We would like to express our thanks to Liz Wilkinson, MS Specialist Nurse at Central Surrey Health,
for her involvement in the study and identifying individuals to participate. We also thank all the participants for their involvement in the study. Conflict of interest The authors declare that there is no conflict of interest.
Funding The research was supported by a research grant from Biogen Idec Limited, Maidenhead, Berkshire.
References 1. Hildebrand MW, Host HH, Binder EF, et al. Measuring treatment fidelity in a rehabilitation intervention study. Am J Phys Med Rehabil 2012; 91: 715–724. 2. Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychol 2004; 23: 443–451. 3. Thomas SA, Russell C, Seed R, et al. An evaluation of treatment integrity in a randomised trial of behavioural therapy for low mood in stroke patients with aphasia. Clin Rehabil 2013; 27: 1097–1106. 4. O’Brien MC, das Nair R and Lincoln NB. A comparison of the content of memory rehabilitation groups for patients with neurological disabilities. Neuropsychol Rehabil 2013; 23: 321–332. 5. Prakash RS, Snook EM, Lewis JM, et al. Cognitive impairments in relapsing-remitting multiple sclerosis: a meta-analysis. Mult Scler 2008;14:1250–1261. 6. Das Nair R, Ferguson H, Stark DL, et al. Memory Rehabilitation for people with multiple sclerosis. Cochrane Database Syst Rev 2012; 3: CD008754. 7. Das Nair R and Lincoln NB. Evaluation of Rehabilitation of Memory in Neurological Disabilities (ReMiND): a randomized controlled trial. Clin Rehabil 2012; 26: 894–903. 8. O’Brien AR, Chiaravallod N, Goverover Y, et al. Evidenced-based cognitive rehabilitation for persons with multiple sclerosis: A review of the literature. Arch Physl Med Rehabil 2008; 89: 761–769. 9. Carr S, das Nair R, Schwartz A, et al. Group memory rehabilitation for people with multiple sclerosis: a pilot randomised controlled trial. Clin Rehabil 2014; 28: 552–561. 10. Goldberg D and Williams P. A Users Guide to the General Health Questionnaire. Berkshire: Nfer-Nelson Ltd., 1988. 11. Sunderland A, Harris J and Baddeley A. Assessing everyday memory after severe head injury. In: Harris J and Morris P (eds) Everyday memory, actions and absentmindedness. London: Academic Press, 1984, pp. 191–206. 12. Hobart J, Lamping D, Fitzpatrick R, Riazi A and Thompson A. The Multiple Sclerosis Impact Scale (MSIS-29): a new patient-based outcome measure. Brain 2001; 124: 962–973.
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
7
Smale et al. 13. Moncher FJ and Prinz RJ. Treatment fidelity in outcome studies. Clin Psychol Rev 1991;11: 247–266. 14. O’Brien MC, Das Nair R and Lincoln NB. A comparison of the content of memory rehabilitation groups for patients with neurological disabilities. Neuropsychological Rehabilitation 2013; 23: 321–332. 15. Kendall PC and Beidas RS. Smoothing the trail for dissemination of evidence-based practices for youth: flexibility within fidelity. Prof Psychol Res Pr 2007; 38:13–20. 16. Kendall PC, Gosch E, Furr JM, et al. Flexibility within fidelity. J Am Acad Child Adolesc Psychiatry 2008; 47: 987–993.
17. Yalom I and Leszcz M. The theory and practice of group psychotherapy. 5th ed. New York: Basic Books, 2005. 18. Lombard M, Snyder-Duch J and Bracken CC. Content analysis in mass communication: assessment and reporting of intercoder reliability. Hum Commun Res 2002; 28: 587–604. 19. De Wit L, Kamsteegt H, Yadav B, et al. Defining the content of individual physiotherapy and occupational therapy sessions for stroke patients in an inpatient rehabilitation setting. Development, validation and inter-rater reliability of a scoring list. Clin Rehabil 2007; 21: 450–459.
Downloaded from cre.sagepub.com at TEXAS SOUTHERN UNIVERSITY on October 27, 2014
