709975

research-article2017

TPP0010.1177/2045125317709975Therapeutic Advances in PsychopharmacologyM Sawicka and DK Tracy

Therapeutic Advances in Psychopharmacology

Review

Naltrexone efficacy in treating alcohol-use disorder in individuals with comorbid psychosis: a systematic review

Ther Adv Psychopharmacol 2017, Vol. 7(8-9) 211­–224 https://doi.org/10.1177/2045125317709975 DOI: 10.1177/ https://doi.org/10.1177/2045125317709975 2045125317709975

© The Author(s), 2017. Reprints and permissions: http://www.sagepub.co.uk/ journalsPermissions.nav

Martyna Sawicka and Derek K Tracy

Abstract Background: Psychotic illnesses, such as schizophrenia, are typically enduring and disabling conditions, impacting individual, family, and societal outcomes. Individuals with these face greater vulnerabilities in developing alcohol-use disorder (AUD). Furthermore, the nature of psychoses, often manifesting with paranoia, cognitive impairment, a lack of insight, suboptimal treatment adherence, and stigma from others, means that they can pose unique treatment challenges when these two conditions comorbidly occur. These challenges mean that the standard literature on the effectiveness of the opioid antagonist naltrexone in AUD does not necessarily translate to this vulnerable population. Methods: Following PRISMA guidelines, we herein systematically reviewed the evidence for naltrexone in individuals with both psychosis and AUD. Overall, there is a paucity of research in this important area, with only nine reports meeting search criteria, only four of which were randomized control trials. Studies compared naltrexone with: placebo, another pharmaceutical agent, or upon changes to baseline drinking behaviour. One study evaluated the long-acting injectable formulation of this drug. Results: Most studies, including the methodologically more robust ones, supported naltrexone’s effectiveness over placebo in terms of reduction in drinking days and numbers of drinks consumed on such days in this cohort. Work comparing naltrexone to other pharmaceutical interventions showed approximate equivalence with disulfiram, and modest superiority over acamprosate. Conclusions: On this limited evidence base, this review endorses the use of naltrexone as both safe and effective in those with both psychotic illnesses and AUD. Several key issues remain to be elucidated. Critically, study designs meant that they were limited to individuals with good engagement with services, and levels of adherence were attained that are unlikely to be replicated in this cohort in real-world settings. Finally, effects of specific psychosis symptomatology, not least paranoia and insight, upon naltrexone use, and the reverse directional potential of ‘double dysphoria’ from an opioid antagonist remain largely unexplored.

Received: 31 August 2016; revised manuscript accepted: 23 February 2017

Correspondence to: Dr Derek Tracy Consultant Psychiatrist, The Memorial Hospital, Woolwich, London SE18 3RG, UK [email protected]

Introduction Psychotic disorders have a lifetime prevalence of about 3%, with schizophrenia being the most common.1 The overall annual financial burden of

Martyna Sawicka Cognition, Schizophrenia and Imaging Laboratory, Department of Psychosis Studies, The Institute of Psychiatry, Psychology and Neuroscience, King’s College London, UK Department of Psychology, Arizona State University

Keywords:  alcohol, alcohol-use disorder, comorbid psychosis, dependency, naltrexone, schizophrenia

schizophrenia has been estimated at 63.7 billion dollars in the US2 and 6.7 billion pounds in England,3 the majority of which are indirect costs due to societal losses of productivity.

journals.sagepub.com/home/tpp 211

Therapeutic Advances in Psychopharmacology 7(8-9) These complex conditions are further complicated for many with comorbid substance misuse, of which alcohol is the most frequent.4 A recent meta-analysis reported an alcohol-use disorder (AUD) prevalence of 20% in individuals with schizophrenia.5 Individuals with psychoses may face additional risks for alcohol misuse due to increased social disadvantage, vulnerability,6 and self-medication to ease symptom distress,7 though research demonstrates that many experience an exacerbation of symptoms through alcohol use.7–9 Alcohol use in this population is associated with more physical illness, medication nonadherence, earlier hospital readmission, greater rates of suicide, interpersonal and family problems, and homelessness,10–16 though these works have varied in their use of alcohol dependency, harmful use, and AUD as relevant criteria. Furthermore, in terms of treatment, interventions with those with schizophrenia may face additional challenges such as cognitive impairment, paranoia and lack of insight, which could hinder an ability to actively engage in an intervention.17,18 In the UK, the National Institute for Health and Care Excellence (NICE) guidelines recommend a stepped approach to treatment of AUD, starting with motivational interviewing, in order to build insight into the problems related to drinking, followed by a psychological intervention such as cognitive behavioural therapy; the addition of acamprosate or oral naltrexone is recommended for harmful drinkers who do not respond to psychological intervention alone.19 This integrated treatment approach is mirrored in the NICE guidelines for comorbid psychosis and substance use; the delineating factor between the two guidelines is the additional use of antipsychotic medications where appropriate. No drug has been specifically approved for use in populations with dual diagnoses, though having a concomitant psychotic illness is not a contraindication to use, except in the case of disulfiram.20 Naltrexone, an opioid antagonist, initially used primarily for opioid addiction, has been shown to have promising effects in the treatment of AUD, and it is licensed for such use in the UK and in the US.21–25 Further, the Combined Pharmacotherapies and Behavioural Interventions for Alcohol Dependence (COMBINE) study demonstrated that not only is naltrexone effective in reducing alcohol consumption, it is also superior to an alternative pharmacological treatment, acamprosate, and behavioural interventions.26

While naltrexone’s mechanisms of action in AUD are not fully understood, it is postulated that it works by reducing craving, decreasing subjective pleasure from alcohol consumption, and stemming endorphin-mediated reinforcement effects.27–30 Meta-analyses on the efficacy of naltrexone in alcohol dependent/abusing populations generally show a significant reduction in relapse and heavy drinking rates, though the evidence varies between trials, with some large robust studies failing to find a significant treatment effect.31–34 Interestingly, though limited, there are data to show that clinicians may be more likely to utilize it in those with comorbid psychiatric conditions35,36; the rationale for this may vary, though it might represent greater clinical contact and use of medication, and, speculatively, a more interventionist approach in a vulnerable cohort. However, these findings are not always consistent, and further research is necessary. Given these unique population challenges, the aim of this systematic review is to assess and summarize the current available evidence regarding the treatment effect of naltrexone in individuals with comorbid psychosis spectrum and AUD. Methods Data acquisition The aim was to identify and critically appraise all randomized controlled trials (RCTs) and non-RCTs available to review by August 2016, in which the effect of naltrexone on alcohol consumption was investigated in a population with comorbid psychosis spectrum disorder and an AUD. The PRISMA guidelines were utilized to gather studies, extract data, and synthesize information.37 Search strategy Electronic database and manual searches were performed to locate references. The following databases were searched: Global Health (1973 to week of 5 August 2016), International Pharmaceutical Abstracts (1970 to week of 5 August 2016), Journals @ Ovid Full Text (15 January 2016), Ovid MEDLINE (1946 to week of 5 August 2016), PsycARTICLES Full Text, PsycINFO (1806 to week of 5 August 2016), Embase Classic and Embase (1947 to week of 5 August 2016). The search criteria were ‘(naltrexone OR Revia OR Depade OR Vivitrol) AND (Schizophrenia OR

212 journals.sagepub.com/home/tpp

M Sawicka and DK Tracy

Figure 1.  Flow diagram demonstrating process of inclusion of studies for review.

Psychosis OR Psychotic) AND Alcohol’. Results were then de-duplicated.

naltrexone, study design, and drinking behaviour outcome measures.

Inclusion and exclusion criteria Both RCTs and non-RCTs were included. Only studies with at least one participant group composed of dually diagnosed (a psychosis spectrum disorder and an AUD) individuals were included; animal studies were excluded. Studies were required to have at least one primary outcome of drinking behaviour post-naltrexone administration. Studies were restricted to the English language. Due to the scarcity of research with this specific population, poster presentations were included if sufficient information was available. Books, letters, comments, and editorials were excluded.

Results The database search yielded 3879 reports. A further search through reference lists in these search results added three additional reports. After dismissal by title, 108 abstracts were reviewed. In total, nine reports were identified (five journal articles and four detailed poster presentation abstracts) that met inclusion and exclusion criteria, see Figure 1, with a total of 798 participants, all diagnosed with an AUD. Study characteristics are in Table 1. Of these participants, 273 were diagnosed with a psychotic disorder. When participants in all the studies are considered together, 39.97% were allocated to a treatment condition wherein they were prescribed solely naltrexone to treat their AUD; the remaining participants were allocated to various conditions such as placebo, other medication(s), and psychotherapy. The average age of participants, where provided, was 44.24. Overall, the studies demonstrate a gender

Data extraction The following information was extracted from each study (where available): number of participants, participant age, sex, diagnosis, dosage of

journals.sagepub.com/home/tpp 213

100 mg 2 × a week, 150 mg 1 × weekly

100 mg 2 × a week, 150 mg 1 × weekly 380 mg monthly injection 50 mg

Not reported

50 mg

50 mg

50 mg

50 mg

Prospective open label

Randomized controlled trial

Prospective open label

Prospective open label

Retrospective chart review

Randomized controlled trial

Randomized controlled trial (w/ only disulfiram open label)

Randomized controlled trial

Prospective open label

Batki et al.45

Batki et al.16*

Batki et al.44*

Bratu and Sopterean42*

Maxwell and Shinderman46

Petrakis et al.38

Petrakis et al.41

Petrakis et al.39

Vasile et al.43*

Disulfiram

(1) Disulfiram and placebo (2) Naltrexone and disulfiram (3) Placebo

(1) Placebo (2) Open-label disulfiram and blinded naltrexone (3) Open-label disulfiram and blinded placebo

15 placebo

n/a

(1) Acamprosate Addiction counselling

None

Placebo

n/a

Comparison group

*Denotes a poster presentation abstract. CBT, cognitive behavioural therapy; n/a, not applicable.

Naltrexone dose

Design

Study

Table 1.  Study characteristics.

Antipsychotic medication

Antipsychotic medication

Participants in substanceabuse treatment with intensive rehabilitation, antipsychotic medication

CBT drug relapse prevention strategies, antipsychotic medication

Antipsychotic medication 32% of patients in outpatient treatment

Antipsychotic medication

Motivational counselling, antipsychotic medication

Motivational counselling

Antipsychotic medication

Concurrent intervention

24

12

12

12

 8

24

12

12

 8

Study period (weeks)

Overall alcohol consumption

Days of abstinence Drinking days per week Heavy drinking days

Days of abstinence Drinking days per week Heavy drinking days

Drinking days Heavy drinking days Number of standard drinks

Overall alcohol consumption

Days of abstinence Heavy drinking days

Drinking days Heavy drinking days Drinks per week

Drinking days Heavy drinking days Drinks per week

Drinking days Drinking to intoxication (heavy drinking) Number of standard drinks Number of drinks per drinking day

Primary alcohol outcomes

Therapeutic Advances in Psychopharmacology 7(8-9)

214 journals.sagepub.com/home/tpp

M Sawicka and DK Tracy Table 2.  Demographics and clinical characteristics of participants. Study

Age (years)

Sex males

Batki et al.45

43

15

Batki et al.16

42.5

Batki et al.44

Sex females

Total participants

Number of participants receiving naltrexone only

Participants with psychosis diagnosis

Other diagnoses included

4

19

19

11 schizophrenia 7 schizoaffective 1 psychotic disorder NOS



64

26

90

45

45 schizophrenia 45 schizoaffective



44

13

12

25

25

6 schizophrenia 11 schizoaffective

8 bipolar

Bratu and Sopterean42

18–65

24

12

36

12

36 (schizophrenia)



Maxwell and Shinderman46

Not reported

51

21

72

72

17 Schizophrenia 7 schizoaffective

37 MDD, 4 genderidentity disorder, 11 bipolar

Petrakis et al.38

46

31

0

31

16

18 schizophrenia 13 schizoaffective



Petrakis et al.41

47

247

7

254

59

18 schizophrenia or schizoaffective

178 MDD, 109 PTSD, 50 cocaine abuse, 57 GAD, 49 bipolar

Petrakis et al.39

46.98

244

7

251

63

11 schizophrenia 7 schizoaffective

150 MDD, 86 PTSD, 26 panic disorder, 20 GAD, 48 bipolar (exceeded count due to comorbidities)

Vasile et al.43

40.2

9

11

20

8

20 (schizophrenia)



Abbreviations: NOS, not otherwise specified; MDD, major depressive disorder; PTSD, post-traumatic stress disorder; GAD, generalized anxiety disorder.

bias, with approximately 87% of participants being male. Further participant details are listed in Table 2. Drinking outcomes Due to the varied designs adopted, three outcome groupings have been utilized. Naltrexone’s effects on drinking behaviour were compared: with placebo in four studies38–41; with another pharmaceutical agent in four studies39,41–43; and against baseline drinking behaviour in three studies, one of which utilized the long-acting injectable form of the medication.44–46 Naltrexone versus placebo All four studies16,38,39,41 showed significant superiority of naltrexone over placebo, though the placebo groups did show improvement in comparison

to baseline drinking behaviour. Further details may be found in Table 3. In the first placebocontrolled study on the topic38 (n = 31; male = 31), by the 12-week intervention endpoint, individuals who received naltrexone (n = 16) drank, on average, 7.3 fewer days (p < 0.0001), 26.4 fewer drinks (p < 0.05), had 0.44 fewer heavy drinking days (p = 0.003), and reported more abstinent days (p = 0.04) than those receiving placebo (n = 15).38 The findings of this pilot study led to a much larger, but equal-duration follow-up study by the same research group41, consisting of 254 (male = 247) military veterans with alcohol dependency and mental illness (not limited to psychosis). It adopted a more sophisticated design, with four study arms: placebo, naltrexone, disulfiram, and naltrexone plus disulfiram. The analysis thus looked not only at the efficacy of naltrexone versus placebo, but also of naltrexone versus disulfiram and monopharmacotherapy versus

journals.sagepub.com/home/tpp 215

Therapeutic Advances in Psychopharmacology 7(8-9) Table 3. Naltrexone versus placebo. Study

Self-reported days of abstinence

Self-reported heavy drinking days

Drinks

Drinking days

Drinks per drinking day

Batki et al.16

Not reported

Decreased, p < 0.04

Not reported

Decreased, p < 0.08

Not reported

Petrakis et al.38

Not reported

Decreased, p = 0.003

Decrease of 26.4, p value not stated

Decreased, p < 0.0001

Not reported

Petrakis et al.41

Decreased, p = 0.04

No significant change

Not reported

Decreased, p = 0.02

Not reported due to high rates of abstinence

Petrakis et al.39

Decreased, p = 0.04

Decreased, p = 0.02

Not reported

Decreased, p = 0.02

Not reported

Table 4. Naltrexone versus other pharmacological treatment. Study

Intervention

Self-reported days of abstinence

Self-reported heavy drinking days

Global alcohol dependence

Drinks per week

Drinking days per week

Bratu and Sopterean42

Naltrexone versus acamprosate

+23.6 days, p < 0.01*

–5.7 days, p = 0.043

Decreased, p = 0.122

Not measured

Not measured

Petrakis et al.39

Disulfiram versus naltrexone

Decreased, p = 0.37

Decreased, p = 0.42

Not measured

Not reported

Not reported



Disulfiram and naltrexone combined versus disulfiram or naltrexone

Decreased, p = 0.29

Decreased, p = 0.16

Not measured

Not reported

Not reported

Petrakis et al.41

Disulfiram versus naltrexone

Decreased, p = 0.68

Decreased, p = 0.65

Not measured

Not reported

Decreased, p not reported



Disulfiram and naltrexone combined versus disulfiram or naltrexone

Decreased, p = 0.94

Decreased, p = 0.76

Not measured

Not measured

Decreased, p not reported

Vasile et al.43

Disulfiram versus naltrexone

Not measured

Not measured

Decreased, p < 0.01 in both groups; group differences, p = 0.144

Not measured

Not measured

*Denotes superiority of comparison drug.

combination therapy. The findings of the pharmaceutical comparisons are presented in the next section. When comparing naltrexone efficacy against placebo, this larger trial found similar reductions in drinking days (p = 0.02).39,41 In a post hoc analysis, Petrakis et  al.39 reviewed their 2005 study by investigating medication effects through grouping participants on the basis of psychosis spectrum disorders versus nonpsychotic disorders.39 In this analysis, they found that when

compared with individuals with nonpsychotic disorders, individuals with psychosis spectrum disorders relapsed more (p < 0.05), but experienced significantly more days of abstinence (p = 0.01) and fewer heavy drinking days (p = 0.02).39 In a double-blind randomized controlled trial with 90 participants (male = 64), Batki et  al.40 directly administered oral naltrexone three times weekly with a dosing schedule of 100 mg on Mondays, 100 mg on Wednesdays, and 150 mg on Fridays.

216 journals.sagepub.com/home/tpp

M Sawicka and DK Tracy Table 5. Naltrexone versus baseline. Study

Self-reported days of abstinence

Self-reported heavy drinking days

Drinks per week

Drinking days per week

Drinks per drinking day

Global alcohol intake

Batki et al.44

Not measured

Decreased, p < 0.02

Decreased, p < 0.02

Decreased, p < 0.02

Decreased, p < 0.02

Not measured

Batki et al.45

Not measured

Decreased, p = 0.043

Decreased, p = 0.016

Decreased, p = 0.034

Decreased, p = 0.048

Not measured

Maxwell and Shinderman46

Not measured

Not measured

Not measured

Not measured

Not measured

70.8% patients reduced by 90%, 11.2% reduced by 75–90%

This study also found that heavy drinking was reduced at the 12-week endpoint of naltrexone treatment (p < 0.04). While not statistically significant (p < 0.08), they found a trend for a decrease in fewer standard drinks per week. However, they failed to replicate the finding of reduction in number of drinking days. In the three studies that measured this, there were no differences between naltrexone and placebo in terms of study retention and adherence to drug administration.38,41 In the post hoc analysis that contained individuals with nonpsychotic disorders39, there were also no differences in study retention or adherence between the clinical groups.39 Naltrexone versus other pharmacological treatment Table 4 outlines the results of various pharamacological comparions. One study compared naltrexone with acamprosate. Bratu and Sopterean42 randomized 36 individuals with psychosis (male = 24) into three groups: naltrexone 50mg daily, acamprosate 999 mg daily, and counselling focused on addictive behaviour. Naltrexone yielded superior effect on the reduction of both heavy drinking days and maintenance of sobriety when compared with counselling targeting addictive behaviour (p < 0.01).42 Over the 24-week intervention, they found that while naltrexone had a significant effect on the maintenance of abstinence, acamprosate treatment resulted in a larger effect size, yielding an additional 23.6 sober days (p < 0.01).42 However, naltrexone was more effective in reducing the number of heavy drinking days (–5.7 days; p = 0.043). Furthermore, while insignificant (p = 0.122), the severity of alcohol dependence (as measured by the Inventory of Drug Taking Situations, alcohol-focused version) decreased most in the naltrexone group. In sum, naltrexone was effective in maintaining

abstinence, though acamprosate yielded longer abstinence periods, and was superior to acamprosate in reducing the number of heavy drinking days. Three studies compared naltrexone with disulfiram, with results showing approximate equivalence between them. Vasile et  al.43 utilized an open-label design to compare the efficacy of naltrexone with that of disulfiram in 20 participants with a 24-week intervention. Naltrexone demonstrated equal reduction of global alcohol symptomology and severity (p < 0.01). Furthermore, disulfiram had significantly more individuals discontinue treatment due to adverse events (p < 0.05), suggesting that naltrexone may be better tolerated than disulfiram. In the aforementioned four-arm comparison study, Petrakis et al.41 evaluated the effects of placebo, open-label disulfiram and blinded placebo, blinded naltrexone and placebo, and open-label disulfiram and blinded naltrexone. Due to the potentially dangerous interaction of disulfiram and alcohol, as well as the temptation to sample alcohol to test a blind condition of disulfiram, the authors chose to dispense the disulfiram by open label. This study found that naltrexone significantly decreased drinking days per week (p = 0.02) and increased consecutive days of abstinence (p = 0.04) compared with placebo, though these decreases were not significantly different from those found with the other active arm, disulfiram. Post hoc analysis of this same study grouped participants according to presence of psychosis spectrum diagnoses39 but again showed no differences between medications. Specifically, individuals with psychotic spectrum disorders fared worse; they experienced more relapse, more heavy drinking days, fewer days of consecutive abstinence, and fewer days of total abstinence (p < 0.05).

journals.sagepub.com/home/tpp 217

Therapeutic Advances in Psychopharmacology 7(8-9) While medications proved to be more efficacious than placebo, the combination of the two drugs did not yield a significant advantage over either medication alone in any clinical group.39,41 Drinking behaviours at baseline versus postnaltrexone (no comparison condition) Only three studies observed naltrexone outcomes by solely comparing drinking behaviour pre and post naltrexone (see Table 5). In the first study on this topic, a retrospective chart review, Maxwell and Shinderman46 found that with the caveats of this method of analysis, naltrexone yielded an overall reduction in drinking, with 70.8% of participants (n = 31) reducing their drinking by more than 90%, and 11.2% reduced drinking by 75–90% over an 8-week period in an outpatient community mental health centre. An 8-week open-label pilot study (n = 19)45 also compared drinking behaviours of individuals with schizophrenia and AUD at baseline with postnaltrexone administration. In this study, directly observed oral doses of 100 mg of naltrexone were given twice a week and 150 mg once a week, with no additional supports. This alternative dosing schedule did not produce differing results, with reductions in heavy drinking days (p = 0.043), drinks per week (p = 0.16), drinking days per week (p = 0.034), and drinks per drinking day (p = 0.048) observed. Long-acting injection naltrexone.  One study evaluated the feasibility and effects of the long-acting injectable form of naltrexone on baseline drinking behaviour over the course of 12 weeks.44 When participants (n = 25; males = 13) received all three monthly depot injections of naltrexone (380 mg/injection), significant reductions were found in drinks per week (p < 0.02), drinking days per week (p < 0.02), heavy drinking days (p < 0.02), and drinks per drinking day (p < 0.03).44 Outcomes on psychosis pathology and cravings Across the three groupings, five of the studies reported on changes in psychopathology. Whilst an opioid antagonist would not be expected to have direct antipsychotic actions, any reduction in harmful alcohol consumption is postulated to lead to secondary changes in mental state. From the limited evidence available, any such effects on general psychopathology remain inconclusive. While some studies found naltrexone to not have a

significant effect on general psychopathology,38,39,42 two found that naltrexone significantly deceased psychopathology.41,45 Similarly, studies found contradictory evidence for naltrexone’s effect on alcohol craving, with three studies finding significant decreases,38,41,45 and two not finding this association.39,40 Interestingly, Petrakis et  al. found the decrease in craving when grouping together individuals with a mental health diagnosis,41 yet this significance dissipated when the analyses were conducted by grouping participants based on the presence of a psychotic disorder.39 This may suggest that naltrexone has a differential effect on craving, based on the presence of psychotic symptoms. However, further research would be necessary to fully illuminate this relationship. Discussion In total, nine studies were located that investigated the effect of naltrexone on drinking behaviour in a comorbid schizophrenia and AUD population. The most favourable evidence for naltrexone’s efficacy was in the reduction of heavy drinking days,38–40,42,44,45 followed by maintenance of abstinence39,41 and reduction in the number of drinking days.38,44 Further evidence suggests that naltrexone may be effective in reduction of drinks per week44 and the number of drinks per drinking day.44 While studies indicate that the effect of disulfiram is approximately equivalent to that of naltrexone, fitting with the wider literature on the topic, naltrexone is shown to be better tolerated in patients suffering from comorbid alcohol use and psychotic disorders,38,42,43 though mild side effects have been reported.45 In the one study that tested for such an effect, combining these treatments did not offer any advantage, though it did produce more side effects than found in the administration of either drug alone.39,41 Furthermore, it has been argued that because naltrexone does not produce an aversive reaction as disulfiram does, individuals may be more willing to try it due to less fear of punitive relapse consequences.47 Interestingly, studies that offered a psychosocial intervention alongside naltrexone (concurrent interventions are listed in Table 1) did not yield further advantages over studies that prescribed naltrexone alone.38,40,41,44,46 While magnitude and longevity of gains remain to be discovered, even small changes in drinking behaviour may have a large subjective impact on the lives of those with comorbid psychosis and AUD, since

218 journals.sagepub.com/home/tpp

M Sawicka and DK Tracy these individuals suffer the same consequences of alcohol as their mentally healthy counterparts, but with less intake.48 Two studies had contrary findings. While Batki and colleagues40 did not find a significant decrease in drinking days, authors of the other three studies in the naltrexone versus placebo category found a significant effect in this measure.38,39,41 However, Batki et  al. did find a nonsignificant trend for fewer drinks per week in the naltrexone group. In contrast to a dosage of 50 mg used by the other three studies in this category, Batki and colleagues used a dosing schedule of 100 mg twice a week and 150 mg once a week. Since the trend found in other studies is present in the authors’ findings, it is conceivable that the dosing influenced the outcome. Indeed, the recommended dosage of naltrexone per general prescribing guidelines is 50 mg.20,49 However, the authors note that the most salient part of their study was that dosing was directly observed, ensuring adherence on days that subjects attended, which was thus considerably better than many trials that had more inexact methods, such as pill count or self-report. The second inconsistency is the finding by Petrakis et al.41 that naltrexone was ineffective for reduction in heavy drinking days. This is not only inconsistent with the studies comparing naltrexone with placebo, but to all studies in this review that measured heavy drinking, suggesting that this is an isolated finding to the participants enrolled in the study by Petrakis et  al.41. This sample was composed of highly motivated individuals who were already enrolled in a treatment program and demonstrated high rates of abstinence. Evidence indicates that naltrexone may be most effective in conjunction with alcohol consumption, rather than sobriety.50 High abstinence may explain the lack of significant effect on heavy drinking days in the sample studied by Petrakis et al.41.

be transferable to clinical practice. Interestingly, Batki and colleagues employed a model of adherence in an interim analysis of their 2009 study where they found alcohol reductions even in individuals who were nonadherent (taking less than 80% of the suggested dose of naltrexone).51 Longacting ‘depot’ injection side-steps this issue, but the evidence base is currently limited.52 Concurrent psychosocial ordinary care was not possible to control for in several works,38,40,41,44 though studies with no such interventions still observed significant drinking reductions.39,43,45,46 A total of 87% of the participants were male; women have been traditionally under-represented in this field53 but appreciation for sex-specific differences has increased,54,55 albeit that the general literature suggests men and women do not differ in their response to naltrexone.56 Three studies utilized samples from the Veterans Administration in the USA,39,41 with military personnel shown to often have differing personality traits from the general population,57–60 and results were not stratified by ethnicity. Eastern-Asian and African-American individuals have been shown to respond differentially to naltrexone.61,62 Therefore, the results may only be pertinent to the specific populations sampled, part of the wider challenge of stratified medicine; which interventions work in which subgroups.63

Methodology, design, and limitations The studies included in this review have several limitations. The majority of studies had participants stabilized on antipsychotic medications,38,39,41–46 putatively impacting dopaminergic reward systems, though it enhances the transferability of the findings.

Overall, the literature is notable for its small size, with several of the studies coming from a single research team. One study was a retrospective chart review,46 four studies42–45 were open-label trials, and three were randomized controlled trials.38,40,41 The inclusion of a retrospective chart review yields weak evidence, as the intervention was not controlled, nor were specific drinking measures allowed to be taken. Open-label trials leave the possibilities of placebo effect and researcher biases open through unblinding. Several of these studies did not include a control group. In addition, the use of open-label designs may attract those who are more motivated to gain sobriety as patients are aware of receiving active medication. RCTs, while the ‘gold standard’, also suffer this same limitation by recruiting participants willing to receive a drug. As a consequence, external validity is compromised.64

Two studies administered naltrexone,40,45 yielding high adherence and low attrition rates that may not

The variety of designs necessitated different statistical analyses to be employed to interpret the raw

journals.sagepub.com/home/tpp 219

Therapeutic Advances in Psychopharmacology 7(8-9) data. Analysis of variance,39,41,45 analysis of covariance,45 zero-inflated negative-binomial regression,40,44 random regression analysis,38,41 an unspecified effect size,42 and a qualitative scale46 were all implemented in data analysis. Due to the lack of consistency across studies, a meta-analysis, which may provide ‘the best estimates of treatment effects’,65 is neither currently feasible nor practical. The synthesis of findings found in this review, while qualitatively important, does not produce results about the magnitude of the pooled effects. Lastly, inclusion of poster presentations does not allow for the critical analysis of methodology, as this is not discussed to similar depth as of that in journal articles. Not only is this lack of information a limitation intrinsically, but it also opens the possibility of further unknown disadvantages. Future directions While this review demonstrates safety and efficacy of naltrexone in a comorbid psychosis and AUD population, the small number of studies identified reflects a shortage of examination in this specific area; further research is necessary to overcome the present limitations and strengthen findings. In order to adequately evaluate the effect of naltrexone, future research should aim to adopt designs, outcomes, and statistical analyses that allow for direct comparison across studies. None of these studies have examined the durability of drinking behaviour reductions, with follow-up periods of a maximum of 24 weeks. Future research should investigate whether the effects produced by naltrexone are sustainable and lead to long-lasting change. This necessity is highlighted by findings that AUD-diagnosed individuals treated with naltrexone did not differ in drinking habits to the placebo-treated group at follow ups ranging from 6 to 12 months.66,67 However, individuals with psychosis differ in significant ways to the general population, even further warranting additional study. So-called ‘double dysphoria’ is another concern with this population. Individuals with psychosis are more prone to states of dysphoria, both due to the illness, but also due to social factors such as social isolation, financial difficulties and decreases in functioning. These factors lead individuals to attempt to reduce dysphoric states, commonly with illicit substances,68 notably cannabis; a particular contemporary concern is the rise in use of synthetic cannabinoid receptor agonists, a class of

novel psychoactive substances that appear particularly potent.69–71 Due to naltrexone’s blockade of opioid receptors, there is concern about an exacerbation of dysphoria. While a recent metaanalysis indicates that dysphoria is not a serious side effect produced by naltrexone,72 this review did not include individuals with psychosis who have may have a higher dysphoric baseline prior to medication prescription. However, this review did include individuals with depression and demonstrated that the addition of an antidepressant yielded positive results. Further research should consider extending this line of research to populations with psychosis. An additional factor to consider in this population is the concomitant necessity of antipsychotic medications. While the pharmacokinetic properties of naltrexone make an adverse interaction with antipsychotics unlikely, few studies have been conducted on this subject.73 However, there have been reports of an adverse interaction between naltrexone and thiorizadine causing ‘intense lethargy’74 warranting additional investigation, though it should be noted that these are primarily from case reports, involve an antipsychotic seldom prescribed in contemporary practice, and are now relatively old. Conversely, the majority of studies within this systematic review included patients that were stabilized on antipsychotic medications suggesting that interactions are unlikely. In fact, pilot research suggests that naltrexone may actually be a beneficial addition to antipsychotics in combatting the unwanted side effect of weight gain.75 Despite a number of additional concerns associated with remediation of alcohol use in this population, effective interventions are vital to the safety of the individuals themselves and the public. A recent systematic review of schizophrenia and risk factors found that dually diagnosed patients are at an increased risk for suicide and suicidal behaviour.12 Furthermore, while levels of crime committed by individuals with schizophrenia are approximately equal to the general population, dually diagnosed individuals are over four times more likely to be convicted of a violent crime.76,77 Rural areas have a disproportionately higher rate of alcohol misuse and co-occurring disorders but fewer treatment resources, despite sizable populations.78 When considered together, these findings highlight the need for empirically supported pharmacological interventions, as medications are easily accessible and relatively fast acting.

220 journals.sagepub.com/home/tpp

M Sawicka and DK Tracy Robust results in this population may lead to the licensing of naltrexone explicitly for a comorbid psychosis and AUD population, leading to more access to this treatment and consequently, a better quality of life. In order for a regulatory body to grant approval for a target population, thousands of participants must typically undergo clinical trials to indicate an effect at the population level.49,79 Future studies should therefore include much larger sample sizes. Not only should studies aim for considerable size, sex equality needs to be incorporated as present studies have almost exclusively explored the effect of naltrexone in men. Recent work has shown that ‘sweet liking’ preference predicts naltrexone effectiveness in individuals with AUD without psychosis,80 the scientific rationale being that it is a marker for the functioning of the opioid system; this should be tested in future work on psychosis populations. In conclusion, our findings support the use of naltrexone as safe and effective in those with AUD and psychosis, though the underlying evidence base is modest. Guidelines for AUD in the general population typically recommend psychosocial interventions before pharmacological ones, but this has added challenges in individuals with psychoses. On the one hand, medication might offer a ‘simpler’ choice than more complex interventions; on the other, illness insight, attitudes towards, and adherence with, medication can hinder pharmacological treatment. Clinicians should try offering informed choices mindful of these factors, the additional vulnerabilities those with psychoses have for developing AUD, and the extra challenges they may face in its treatment. Whilst currently generally unexplored in the scientific literature, the impact of psychopathology, including paranoid or delusional ideas, on a treatment regime should be considered, as should likely issues around adherence, whether unintentional through cognitive difficulties or as part of a wider pattern of noncompliance. Further, those with psychotic illnesses can suffer from antipsychotic-induced dysphoria, and an opioid antagonist risks exacerbating this. In summary, those with these comorbidities need intervention at least as much as anyone else, and the use of naltrexone can be supported, but clinicians should be mindful of unique potential pitfalls. Funding This research received no specific grant from any funding agency in the public, commercial, or notfor-profit sectors.

Conflict of interest statement The authors declare that there is no conflict of interest.

References   1. Perala J, Suvisaari J and Saarni SI. Lifetime prevalence of psychotic and bipolar disorders in a general population. Arch Gen Psychiatry 2007; 64: 19–28.   2. Wu EQ, Birnbaum HG, Shi L, et al. The economic burden of schizophrenia in the United States in 2002. J Clin Psychiatry 2005; 66: 1122–1129.   3. Mangalore R and Knapp M. Cost of schizophrenia in England. J Ment Health Policy Econ 2007; 10: 23–41.   4. Drake R and Mueser K. Co-occurring alcohol use disorder and schizophrenia. Alcohol Res Health 2002; 26: 99–102.   5. Koskinen J, Löhönen J, Koponen H, et al. Prevalence of alcohol use disorders in schizophrenia: a systematic review and metaanalysis. Acta Psychiatr Scand 2009; 120: 85–96.   6. Tracy DK and Shergill SS. Treatment-refractory schizophrenia: definition and assessment. In: Buckley PF and Gaughran F (eds) Treatmentrefractory schizophrenia: a clinical conundrum. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014, pp.1–19.   7. Pristach CA and Smith CM. Self-reported effects of alcohol use on symptoms of schizophrenia. Psychiatr Serv 1996; 47: 421–423.   8. National Institute for Health and Clinical Excellence. Psychosis with coexisting substance misuse: assessment and management in adults and young people. NICE clinical guideline, no. 120. Leicester, UK: British Psychological Society, www.guidance.nice.org.uk/CG120, 2011.   9. Bellido I, Lopez C, Bellido MV, et al. Risk and protective factors for relapse in outpatients with schizophrenia. Eur Psychiatry 2015; 30(Suppl. 1): 884. 10. Dixon L. Dual diagnosis of substance abuse in schizophrenia: prevalence and impact on outcomes. Schizophr Res 1999; 35(Suppl. 1): S93–S100. 11. Salyers MP and Mueser KT. Social functioning, psychopathology, and medication side effects in relation to substance use and abuse in schizophrenia. Schizophr Res 2001; 48: 109–123.

journals.sagepub.com/home/tpp 221

Therapeutic Advances in Psychopharmacology 7(8-9) 12. Hor K and Taylor M. Suicide and schizophrenia: a systematic review of rates and risk factors. J Psychopharmacol 2010; 24: 81–90. 13. Lorine K, Goenjian H, Kim S, et al. Risk factors associated with psychiatric readmission. J Nerv Ment Dis 2015; 203: 425–430. 14. Margolese HC, Malchy L, Negrete JC, et al. Drug and alcohol use among patients with schizophrenia and related psychoses: levels and consequences. Schizophr Res 2004; 67: 157–166. 15. Ochoa S, Suarez D, Novick D, et al. Factors predicting hostility in outpatients with schizophrenia: 36-month results from the SOHO study. J Nerv Ment Dis 2013; 201: 464–470. 16. Batki SL, Meszaros ZS, Strutynski K, et al. Medical comorbidity in patients with schizophrenia and alcohol dependence. Schizophr Res 2009; 107: 139–146. 17. Ziedonis DM, Smelson D, Rosenthal R, et al. Improving the care of individuals with schizophrenia and substance use disorders: consensus recommendations. J Psychiatr Pract 2005; 11: 315–339. 18. Bellack AS and Gearon JS. Substance abuse treatment for people with schizophrenia. Addict Behav 1998; 23: 749–766. 19. National Institute for Health and Clinical Excellence. Alcohol-use disorders: diagnosis, assessment and management of harmful drinking and alcohol dependence. NICE clinical guidelines, no. 115. Leicester, UK: British Psychological Society, www.nice.org.uk/guidance/CG115, 2011. 20. Joint Formulary Committee. British National Formulary. London, UK: Pharmaceutical Press, 2013. 21. O’Malley SS, Jaffe AJ, Chang G, et al. Naltrexone and coping skills therapy for alcohol dependence: a controlled study. Arch Gen Psychiatry 1992; 49: 881–887. 22. Volpicelli JR, Watson NT, King AC, et al. Effect of naltrexone on alcohol ‘high’ in alcoholics. Am J Psychiatry 1995; 152: 613–615. 23. Chick J, Anton R, Checinski K, et al. A multicentre, randomized, double-blind, placebocontrolled trial of naltrexone in the treatment of alcohol dependence or abuse. Alcohol 2000; 35: 587–593. 24. Morley KC, Teesson M, Reid SC, et al. Naltrexone versus acamprosate in the treatment of alcohol dependence: a multi-centre, randomized, double-blind, placebo-controlled trial. Addiction 2006; 101: 1451–1462.

25. Streeton C and Whelan G. Naltrexone, a relapse prevention maintenance treatment of alcohol dependence: a meta-analysis of randomized controlled trials. Alcohol 2001; 36: 544–552. 26. Anton RF, O’Malley SS, Ciraulo DA, et al. Combined pharmacotherapies and behavioral interventions for alcohol dependence: the COMBINE study: a randomized controlled trial. JAMA 2006; 295: 2003–2017. 27. Anton RF, Drobes DJ, Voronin K, et al. Naltrexone effects on alcohol consumption in a clinical laboratory paradigm: temporal effects of drinking. Psychopharmacology (Berl) 2004; 173: 32–40. 28. Drobes DJ, Anton RF, Thomas SE, et al. Effects of naltrexone and nalmefene on subjective response to alcohol among non-treatment-seeking alcoholics and social drinkers. Alcohol Clin Exp Res 2004; 28: 1362–1370. 29. King AC, Volpicelli JR, Frazer A, et al. Effect of naltrexone on subjective alcohol response in subjects at high and low risk for future alcohol dependence. Psychopharmacology (Berl) 1997; 129: 15–22. 30. McCaul ME, Wand GS, Stauffer R, et al. Naltrexone dampens ethanol-induced cardiovascular and hypothalamic–pituitary– adrenal axis activation. Neuropsychopharmacology 2001; 25: 537–547. 31. Jonas DE, Amick HR, Feltner C, et al. Pharmacotherapy for adults with alcohol use disorders in outpatient settings: a systematic review and meta-analysis. JAMA 2014; 311: 1889–1900. 32. Hendershot CS, Wardell JD, Samokhvalov AV, et al. Effects of naltrexone on alcohol selfadministration and craving: meta-analysis of human laboratory studies. Addict Biol 2016. DOI: 10.1111/adb.12425. 33. Srisurapanont M and Jarusuraisin N. Opioid antagonists for alcohol dependence. Cochrane Database Syst Rev 2005; CD001867. 34. Krystal JH, Cramer JA, Krol WF, et al. Naltrexone in the treatment of alcohol dependence. N Engl J Med 2001; 345: 1734– 1739. 35. Iheanacho T, Issa M, Marienfeld C, et al. Use of naltrexone for alcohol use disorders in the Veterans’ Health Administration: a national study. Drug Alcohol Depend 2013; 132: 122–126. 36. Petrakis IL, Leslie DF and Rosenheck R. Use of naltrexone in the treatment of alcoholism nationally in the Department of Veterans Affairs. Alcohol Clin Exp Res 2003; 27: 1780–1784.

222 journals.sagepub.com/home/tpp

M Sawicka and DK Tracy 37. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statement. J Clin Epidemiol 2009; 62: 1006–1012. 38. Petrakis IL, O’Malley S, Rounsaville B, et al. Naltrexone augmentation of neuroleptic treatment in alcohol abusing patients with schizophrenia. Psychopharmacology (Berl) 2004; 172: 291–297. 39. Petrakis IL, Nich C and Ralevski E. Psychotic spectrum disorders and alcohol abuse: a review of pharmacotherapeutic strategies and a report on the effectiveness of naltrexone and disulfiram. Schizophr Bull 2006; 32: 644–654. 40. Batki SL, Dimmock JA, PloutzSnyder R, et al. Directly monitored naltrexone reduces heavy drinking in schizophrenia: preliminary analysis of a controlled trial. Alcohol Clin Exp Res 2009; 33(Suppl. 1): 113A. 41. Petrakis IL, Poling J, Levinson C, et al. Naltrexone and disulfiram in patients with alcohol dependence and comorbid psychiatric disorders. Biol Psychiatry 2005; 57: 1128–1137. 42. Bratu ER and Sopterean GA. Comparison of the efficacies of naltrexone and acamprosate in the treatment of patients with chronic schizophrenia who are alcohol dependent. Eur Neuropsychopharmacol 2014; 24: S71–S72. 43. Vasile D, Vasiliu O, Mangalagiu AG, et al. Pharmacological management of alcohol dependence in chronic schizophrenia. Eur Neuropsychopharmacol 2013; 23: S429. 44. Batki SL, Dimmock JA, Meszaros ZS, et al. Extended-release naltrexone for alcohol dependence in patients with serious mental illness. Alcohol Clin Exp Res 2010; 34: 176A. 45. Batki SL, Dimmock JA, Wade M, et al. Monitored naltrexone without counseling for alcohol abuse/dependence in schizophreniaspectrum disorders. Am J Addict 2007; 164: 253–259.

49. US Food and Drug Administration. Clinical research, https://www.fda.gov/forpatients/ approvals/drugs/ucm405622.htm, 2015. 50. Sinclair JD. Evidence about the use of naltrexone and for different ways of using it in the treatment of alcoholism. Alcohol Alcohol 2001; 36: 2–10. 51. Batki SL, Dimmock JA, Ploutz-Snyder R, et al. Effectiveness of monitored naltrexone treatment in adherent vs. non-adherent patients with alcohol dependence and schizophrenia: preliminary analysis. Alcohol Clin Exp Res 2008; 32: 257A. 52. Garbutt JC, Kranzler HR, O’Malley SS, et al. Efficacy and tolerability of long-acting injectable naltrexone for alcohol dependence: a randomized controlled trial. JAMA 2005; 293: 1617–1625. 53. Vannicelli M and Nash L. Effect of sex bias on women’s studies on alcoholism. Alcohol Clin Exp Res 1984; 8: 334–336. 54. Walitzer KS and Dearing RL. Gender differences in alcohol and substance use relapse. Clin Psychol Rev 2006; 26: 128–148. 55. Probst C, Roerecke M, Behrendt S, et al. Gender differences in socioeconomic inequality of alcohol-attributable mortality: a systematic review and meta-analysis. Drug Alcohol Rev 2015; 34: 267–277. 56. Greenfield SF, Pettinati HM, O’Malley S, et al. Gender differences in alcohol treatment: an analysis of outcome from the COMBINE study. Alcohol Clin Exp Res 2010; 34: 1803–1812. 57. Eighmey J. Why do youth enlist? Identification of underlying themes. Armed Forces Soc 2006; 32: 307–328. 58. Kleykamp MA. College, jobs, or the military? Enlistment during a time of war. Soc Sci Q 2006; 87: 272–290. 59. Segal MW, Segal DR, Bachman JG, et al. Gender and the propensity to enlist in the U.S. military. Gend Issues 1998; 16: 65–87.

46. Maxwell S and Shinderman MS. Use of naltrexone in the treatment of alcohol use disorders in patients with concomitant major mental illness. J Addict Dis 2000; 19: 61–90.

60. Woodruff T, Kelty R and Segal DR. Propensity to serve and motivation to enlist among American combat soldiers. Armed Forces Soc 2006; 32: 353–366.

47. Petrakis IL, Gonzalez MD, Rosenheck MD, et al. Comorbidity of alcoholism and psychiatric disorders. Alcohol Res Health 2002; 26: 81–89.

61. Plebani JG, Oslin DW and Lynch KG. Examining naltrexone and alcohol effects in a minority population: results from an initial human laboratory study. Am J Addict 2011; 20: 330–336.

48. Kavanagh DJ, McGrath J, Saunders JB, et al. Substance misuse in patients with schizophrenia: epidemiology and management. Drugs 2002; 62: 743–755.

62. Ray LA, Bujarski S, Chin PF, et al. Pharmacogenetics of naltrexone in Asian Americans: a randomized placebo-controlled

journals.sagepub.com/home/tpp 223

Therapeutic Advances in Psychopharmacology 7(8-9) laboratory study. Neuropsychopharmacology 2012; 37: 445–455. 63. Joyce DW, Kehagia AA, Tracy DK, et al. Realising stratified psychiatry using multidimensional signatures and trajectories. J Transl Med 2017; 15: 15. 64. Rothwell PM. External validity of randomised controlled trials: ‘To whom do the results of this trial apply?’ Lancet 2005; 365: 82–93. 65. Flather MD, Farkouh ME, Pogue JM, et al. Strengths and limitations of meta-analysis: larger studies may be more reliable. Control Clin Trials 1997; 12: 568–579. 66. Krystal JH, Cramer JA, Krol WF, et al.; Veterans Affairs Naltrexone Cooperative Study 425 Group. Naltrexone in the treatment of alcohol dependence. N Engl J Med 2001; 345: 1734– 1739. 67. West SL, Garbutt JC, Carey TS, et al. Pharmacotherapy for alcohol dependence. Evid Rep Technol Assess (Summ) 1999; 1–5. 68. Phillips P and Johnson S. How does drug and alcohol misuse develop among people with psychotic illness? A literature review. Soc Psychiatry Psychiatr Epidemiol 2001; 36: 269–276. 69. Baumeister D, Tojo LM and Tracy DK. Legal highs: staying on top of the flood of novel psychoactive substances. Ther Adv Psychopharmacol 2015; 5: 97–132. 70. Tracy DK, Wood DM and Baumeister D. Novel psychoactive substances: identifying and managing acute and chronic harmful use. BMJ 2017a; 356: i6814.

Visit SAGE journals online journals.sagepub.com/ home/tpp

SAGE journals

71. Tracy DK, Wood DM and Baumeister D. Novel psychoactive substances: types, mechanisms of action, and effects. BMJ 2017b; 356: i6848. 72. Miotto K, McCann M, Basch J, et al. Naltrexone and dysphoria: fact or myth? Am J Addict 2002; 11: 151–160.

73. Saber-Tehrani AS, Bruce RD and Altice FL. Pharmacokinetic drug interactions and adverse consequences between psychotropic medications and pharmacotherapy for the treatment of opioid dependence. Am J Drug Alcohol Abuse 2011; 37: 1–11. 74. Maany I, O’Brien CP and Woody G. Interaction between thioridazine and naltrexone. Am J Psychiatry 1987; 144: 966. 75. Tek C, Guloksuz S, Srihari VH, et al. Investigating the safety and efficacy of naltrexone for anti-psychotic-induced weight gain in severe mental illness: study protocol of a double-blind, randomized, placebo-controlled trial. BMC Psychiatry 2013; 13: 1–10. 76. Fazel S, Langstom N, Hjern A, et al. Schizophrenia, substance abuse, and violent crime. JAMA 2009; 301: 2016–2023. 77. Tracy DK and Gaughran F. Treatment with medication: side-effects, adherence and risk. In: Buchanan A and Wooton L (eds) Care of the mentally disordered offender in the community. 2nd ed. Oxford, UK: Oxford University Press, 2017. 78. The Substance Abuse and Mental Health Services Administration. Report to congress on the prevention and treatment of co-occurring substance abuse disorders and mental disorders. November 2002. New York, NY: New York State Health Foundation. 79. Medicines and Healthcare Products Regulatory Agency. Clinical trials for medicines: apply for authorisation in the UK, https://www.gov.uk/ guidance/clinical-trials-for-medicines-apply-forauthorisation-in-the-uk, 2016. 80. Garbutt JC, Kampov-Polevoy AB, Kalka-Juhl LS, et al. Association of the sweet-liking phenotype and craving for alcohol with the response to naltrexone treatment in alcohol dependence: a randomized clinical trial. JAMA Psychiatry 2016; 73: 1056–1063.

224 journals.sagepub.com/home/tpp

Naltrexone efficacy in treating alcohol-use disorder in individuals with comorbid psychosis: a systematic review.

Psychotic illnesses, such as schizophrenia, are typically enduring and disabling conditions, impacting individual, family, and societal outcomes. Indi...
520KB Sizes 1 Downloads 9 Views