Evaluation of Drug Dependence Treatment In VA Hospitals THEODORE W. LOREI, MSW, GLORIA N. FRANCKE, PHARMD, AND PATRICIA S. HARGER, BA
Abstract: A nationwide evaluation of 49 Veterans Administration drug dependence treatment programs was conducted using a sample of 1,655 patients. Their status with reference to seven treatment goals was assessed at admission and again 11 months later to determine change. The following changes were observed: a large decrease in heroin use; a small increase in the use of alcohol; moderate decline of involvement in the drug culture; moderate increase in psychological well-
being; and small increase in economic independence. No statistically significant changes were observed in frequency of arrests, in establishment of stable living arrangements, in drug related medical problems, or in the ability to relate to other people. Implications of the findings for further research and program modification are discussed. (Am. J. Public Health 68:39-43, 1978).
In 1973, the Veterans Administration (VA) began a study to measure patient improvement in its 49 drug-dependence treatment programs located throughout the country. The effectiveness of the nationwide program and of individual programs was to be inferred from the type and degree of improvement observed. This study differed in several ways from previously reported "follow-up" studies.1 It involved multiple rather than single programs; its focus was on measuring the effectiveness of total programs rather than on single modalities (such as methadone maintenance); and, although patient changes were studied, such changes were primarily of interest as indicators of treatment effectiveness rather than as measures of the course of addiction. Methodologically, this study was distinguished by its: a) emphasis on explicit treatment goal definition as a prelude to evaluation, b) the use of data collection instruments at both admission and follow-up to measure treatment goal attainment, and c) computation of follow-up time-point from admission rather than from treatment termination.
tween July and December 1973. This report focuses on a subsample of 1,655 patients who were randomly selected for follow-up.* Patients who were incarcerated at the time of follow-up were not interviewed because it seemed doubtful that private interviews could regularly be arranged for in prisons. The average age of the sample of 1,655 was 28.2 years; 57 per cent were white, 19 per cent had some education beyond high school, 32 per cent were on probation or parole, and 6: per cent had been admitted to drug dependence treatment at least once before. Heroin was rated as the primary drug of abuse for 58 per cent of the sample. A comparison of the admission characteristics of the follow-up sample with the total admission group showed no statistically significant differences Q2 test, .05 level of probability) on age, race, education, number of previous admissions for drug abuse, frequency of arrests, and self-support. The follow-up group did have more stable living arrangements (before admission), were less often referred directly from military service, and were rated as having had higher motivation to stop using drugs.
Method
Instrumentation Two fixed alternative interview schedules, the Intake Fonn and the Eleven Month Follow-up Form, were written to assess patient status relative to the following treatment goals: 1) elimination of nonprescribed drug use, 2) development of attitudes and skills conducive to being economi-
Subjects Admission data were collected on 4,946 patients admitted to all 49 VA drug dependence treatment programs beFrom the Veterans Administration, Washington, DC. Address reprint requests to Theodore W. Lorei, MSW, Veterans Administration, 810 Vermont Avenue, NW, Washington, DC 20420. This paper, submitted to the Journal March 17, 1976, was revised and accepted for publication April 5, 1977.
AJPH January, 1978, Vol. 68, No. 1
*Sixty four per cent of the 2,607 patients originally selected (15 per cent not located, 8 per cent incarcerated with private interviews not possible, 2 per cent dead, 7 per cent miscellaneous losses, 4 per cent refusals). 39
LOREI, ET AL.
cally self-supporting, 3) elimination of antisocial behavior, 4) establishment of stable living arrangements, 5) improvement of health, 6) improvement of social interactional skills, and 7) promotion of a sense of psychological well-being independent of the drug culture. Goal relevant items in both forms were identical, thereby permitting comparison of patient status at both time-points. The Intake Form also contained items for identifying data and characteristics hypothesized to be related to treatment outcome. The Eleven Month Followup Form contained items for describing length of treatment received, as well as some additional items for assessing goal status (for which there were no parallel items in the Intake Form).t
Procedure The Intake Form was completed by clinical staff; the Eleven Month Follow-up Form was completed by survey research interviewers employed by Macro Systems, Inc., and Opinion Research Corporation. Urine samples were requested at the conclusion of the follow-up interviews and $10 incentives were paid. Urinalyses were performed by The Laboratory of Chromatography, Flushing, NY.
TABLE 1-Changes in Total Drug Use Category
Used
Admission % using
Follow-up % using
tb
dc
88.1 a
69.1
-18.68*
-.46
86.0
61.7
-21.88*
-.54
81.0
32.9
-25.63*
-.63
one or more
drugs Used one or more drugs other
than alcohol Used one or more drugs other than alcohol or
marijuana
a One would expect that this entry would be 100 per cent for patients admitted to drug dependence treatment. The fact that it is not apparently reflects the fact that some patients were not using a drug "on the average of twice a week during the first four weeks" as was required to be checked "yes" on the drug items. b t's were computed using the three types of total drug use scores as the basic observations. c d is a standardized effect size measure proposed by Cohen.2 Conceptually, it is the mean of the difference scores between admission and follow-up mean divided by the standard deviation of these scores. The computational formula suggested by Cohen d = t/\/i (2.5.9) was used. In the absence of any more appropriate standards, he suggests that d values of .20, .50, and .80 indicate small, medium, and large differences, respectively.
'p < .01.
Results Functioning at Follow-up (Total Group) Drug use. The description of drug use during the four weeks prior to the follow-up interview is based on patient report. Some evidence for the validity of these data is provided by the 95 per cent agreement between patient reported drug use in the past 48 hours (a period "within reach" of laboratory tests) and urinalyses for five drugs.** According to patient report, 69 per cent were using at least one of the drugs listedtt on the average of twice a week during the four weeks prior to the follow-up interview, 62 per cent were using one or more drugs other than alcohol (to the point of intoxication), and 33 per cent were using one or more drugs other than alcohol or marijuana (Table 1). Considering each drug individually, the results appear more favorable (Table 2). The degree of improvement on each of the "total" measures in Table 1 was statistically significant and "medium" in size according to the effect size measure (d) suggested by Cohen.2 The degree of improvement for heroin use (Table 2) was close to large. The reader may, of course, determine his or her own definitions of what constitutes a small, medium, or large change. The d statistic is intended only as a guide when more appropriate standards are not
available. The use of marijuana did not change to a statistically significant extent, and the use of alcohol increased slightly (Table 2). Eighty-nine per cent of the patients said they used drugs
*Copies of the Intake and Eleven Month Follow-up Forms are available on request from the senior author. **Eighty-four per cent provided urine samples. ltSee Table 2 for list. 40
TABLE 2-Changes in Individual Drug Use Drug
Methadone (nonprescribed) Heroin Other opiates, opium preparations, and synthetics Cocaine Barbiturates Other sedatives, hypnotics and
tranquilizers Amphetamines Cannabis sativa (marijuana or hashish) Hallucinogens Alcohol (to the point of intoxication) Others Note. N
Admission % using
Follow-up % using
ta
db
11.0 55.3
3.2 16.1
-8.96* -28.42*
-.22 -.70
15.6 17.6 24.2
4.0 6.2 7.7
-1 1.73* -10.47* -14.16*
-.29 -.26 -.35
19.8 21.3
9.0 8.3
-9.20* -11.67*
-.23 -.29
49.7 10.5
52.3 2.0
1.72 -10.61*
.04 -.26
26.1
32.1 3.5
4.05* -3.15*
.10 -.08
5.7 =
1,655
a Although the percentages of 2's (using drugs, as opposed to
l's, not
using drugs) are presented for convenience, t's were computed using the di-
chotomous scores. b d values of .20, .50, and .80 indicate small, medium and large differences respectively, between admission and follow-up.
*p
< .01.
less often at follow-up than at admission, 7 per cent said they used them as often, and 4 per cent said they used them more often. Self-support. At follow-up, 60 per cent reported that they were self-supporting (the mean of 1.60 in the follow-up column of Table 3 indicates that 60 per cent were rated "2", AJPH January, 1978, Vol. 68, No. 1
EVALUATION OF DRUG DEPENDENCE TREATMENT TABLE 4-Changes in Patient Behavior Relevant to other Treatment Goals
TABLE 3-Changes in Self-Support Behavior Follow-up
Admission Item
Supported self from employment (1 = No;2 = Yes) Supported dependents, if applicable (1 = No; 2 = Yes) Hours of employment per week Weekly incomea Attended school or job training (1 = No;2 = Yes)
t
X
d
1.45
1.60
9.94*
.24
1.39
1.54
6.74*
.24
17.84 $66.37
20.93 $76.32
4.88* 3.50*
.12 .09
1.07
1.17
9.21*
.23
N= 1,655 The means for this variable were computed including patients with no income. b d values of .20, .50, and .80 indicate small, medium, and large differences, respectively. *p < .01. a
i.e., self-supporting). The average number of hours of employment per week was 21, and the average weekly income was $76. When these last two statistics were computed, for only those individuals who were employed (56 per cent), the average number of hours of employment per week was 37 and the average income was $137.* As noted above, there were goal relevant items on the Eleven Month Follow-up Form which had no parallel on the Intake Form. Responses, as they relate to each goal, are discussed in the text. Relative to self-support, 13 per cent reported receiving money from public welfare, and 10 per cent from illegal activities. Eight per cent reported volunteer activities. Antisocial behavior. At follow-up, the average number of arrests reported by patients for the previous six months was .65 (Table 4). Thirty-six per cent had been arrested at least once for something other than traffic violations. There was no statistically significant change from admission (when arrests during the previous six months were recorded) in the mean number of arrests nor in the proportion that was arrested. The follow-up percentage may be an underestimate of the true figure because patients in jail were not interviewed. (Of course, not all who were in jail were necessarily arrested in the six month interval prior to the intended follow-up date). Stable living arrangements. At follow-up, 83 per cent were rated as having "some form of stable, noninstitutional living arrangements" (Table 4).t This percentage was not different from the admission percentage to a statistically significant degree. Follow-up items (not included in the admission form) indicated that 20 per cent had lived in more than one place during the past month, 12 per cent had had nights when they *Some statistics presented in the text do not appear in tables. tAs noted above, the means of dichotomous variables are readily converted to percentages. AJPH January, 1978, Vol. 68, No. 1
Admission Follow-up X X
Item
Mean number of arrests in last six months Stable living arrangements(1 = No;2 = Yes) Physical distress and disability (1 = Not at all; ... 4 = Severely) Difficulty in getting along with people (1 = Not at all; ... 4= Very definitely) Able to feel good without drugs (1 = Not at all; ... 4 = Very definitely) Involvement in the drug culture (1 = Not involved; ... 4= Heavily involved)
t
da
.59
.65
1.24
.03
1.81
1.83
1.45
.04
1.71
1.77
2.07
.05
1.98
1.92
-1.75
-.04
2.03
2.77
22.77*
2.81
2.05
.56
-22.87* -.56
N = 1,655
ad values of .20, .50, ferences, respectively.
and .80 indicate small, medium, and large dif-
*p < .01.
did not know whether they would have a place to sleep, 22 per cent had spent at least one night in an institution, 7 per cent felt they had not had adequate shelter, 10 per cent had serious problems in getting enough to eat, and 10 per cent had problems in getting adequate clothing. These percentages were not mutually exclusive. Medical problems. At follow-up, the average rating of the extent to which the patient had been "troubled by physical distress or disability from drug-related medical problems" was 1.77 on a four-point scale on which 2.00 was defined as "somewhat."** As can be seen from Table 4, this mean did not differ from the admission mean to a statistically significant degree. At follow-up, the most frequently reported "medical difficulties" were: insomnia (45 per cent), constipation (24 per cent), and withdrawal symptoms (22 per cent). Fortyseven per cent said they had no medical problems. Interpersonal adjustment. At follow-up, the average rating of the veteran's "difficulty in getting along with people in his immediate environment, such as wife, girl friend, parents, employer, co-worker" was 1.92 on a four-point scale on which 2.00 was defined as "somewhat".#4 This mean was not significantly different from the mean of 1.98 for the admission ratings (Table 4). Twenty per cent reported that others caused them "quite a bit" or "a great deal" of difficulty. Fourteen per cent reported that they caused others "quite a bit" or "'a
**The four scale points were defined as follows: 1=not at all, 2=somewhat, 3=moderately, 4=very definitely. tlThe four scale points were defined as follows: l=not at all, 2= somewhat, 3 =moderately, 4= very definitely.
LOREI, ET AL.
great deal" of trouble. Fifteen per cent reported getting irritated with others "very easily", and 13 per cent reported having gotten into a physical fight with someone. Psychological well-being. This concept was measured by two questions at both admission and follow-up (Table 4). The follow-up mean on "ability to achieve a sense of psychological well-being apart from taking drugs and being a part of the drug culture" was 2.77 on a four-point scale on which 3.00 was defined as "moderately".t# The mean "involvement with the drug culture" was 2.05 on a four-point scale on which 2.00 was defined as "somewhat".#4 Both of these means indicated significant improvement over admission status and the effect sizes were medium.
Functioning at Follow-up (Patients Grouped by Program) The basic question was whether some programs produced better results than others. The lack of random assignment of patients to programs precluded answering this question definitively, but the analyses were intended to develop evidence regarding possible differential program effectiveness. Seven analyses of multiple covariance (BMD 04V) were performed.3 The dependent variables were follow-up status on total drug use, self-support, number of arrests, stable living arrangements, physical distress, interpersonal difficulty, and psychological well-being. This analyses of covariance yielded significant F values (p < .01) for all dependent variables except self-support. In other words, the outcomes for patient cohorts from each program differed significantly, even when adjustments had been made for initial characteristics that might have influenced those outcomes. The interprogram differences were all small with the exception of patients' reported sense of psychological well-being, which were of medium size (using the effect size standards suggested by Cohen2).
Discussion The contributions of the present study toward answering questions about program effectiveness are discussed below, with the recognition that additional studies are necessary to answer these questions definitively. This study should be regarded as a treatment program evaluation rather than a treatment type evaluation because it was concerned with the total set of treatment activities rather than with specific modalities such as methadone maintenance. The following description of treatment program evaluation is provided by Sells.5 Stated most generally, in treatment program evaluation, the "treatment" is defined, in the logician's terms, by extension; it is the treatment offered at a specific organization (agency, hospital, clinic, or treatment center) and the research is concerned with the results obtained by that organization, measured against its specific goals, in administering its particular treatment program to a given population of patients.
When the Veterans Administration's treatment program is viewed as a whole (rather than by individual treatment cen42
ters, or by modalities, such as methadone maintenance), the study results suggest its treatment was helpful in reducing drug abuse or misuse, in reducing involvement in the drug culture, in increasing a sense of psychological well-being, and in increasing self-support. These results also suggest that the treatment program was not effective in reducing the use of alcohol, distress from drug-related medical problems, frequency of arrests, interpersonal difficulty, and instability of living arrangements.
Although statistically significant improvements
were
observed in each of the areas just mentioned at some local programs, and although such improvements may have occurred for certain unidentified patient subgroups, the implications of the negative findings for the overall program should not be ignored. Treatment strategies need to be reexamined to determine why they are not more effective in the areas identified above. Also, it should be noted that, although there was statistically significant increase in the percentage of patients who were self-supporting, the large percentage who were not self-supporting at follow-up suggests that treatment efforts need to be reconsidered in this area. Utilization of the findings of this study makes it possible to define program goals more precisely and thus to state the question about effectiveness more specifically. The goals formulated to guide the present study, although more explicitly stated than those used in most studies, were quite general. For example, one of these goals was "to eliminate the nonprescribed use of drugs." But, is this realistic? Probably most program directors would feel that, even under the best conditions, some patients would continue to use drugs. The question really is: what degree of elimination of drug use is achievable in the present state of treatment technology? The results make it possible to set provisional, specific goals in the areas considered or, alternatively, to provide baselines against which to judge future treatment efforts. Because of major differences in variable definition and study design, it is almost impossible to make meaningful comparisons between these results and those of other investigators. Nevertheless, it is interesting to note, that in studying change from before to during treatment, Sells5 found a large reduction in opiate use, a finding that is similar to the large reduction in heroin use reported here. Contrary to the present results, Sells5 found a small decrease in the use of alcohol and a small increase in unemployment. Comparison of the number of arrests was not possible between the studies. In a study for the National Institute on Drug Abuse, Macro Systems, Inc.,6 found large decreases in drug use, criminal activities, and unemployment. As is apparent, there is agreement between the findings of the present study and the two studies just mentioned in that participation in treatment was associated with a large decrease in drug use. The results on the other outcome measures were not consistent across studies. From the vantage point of research, the present study had many weaknesses. The design problems have already been discussed. The instruments were crude, partly because they had to be kept brief and suitable for use by clinical staff with diverse professional backgrounds. The time period for follow-up was short, possibly accounting for the fact that AJPH January, 1978, Vol. 68, No. 1
EVALUATION OF DRUG DEPENDENCE TREATMENT
variables such as number of arrests did not decrease. From the second vantage point, that of program evaluation as practiced by some government agencies, the study appears stronger. We feel that the Veterans Administration engaged in a pioneering attempt to assess the consequences of one of its newer programs and to provide feedback to staff at all levels of responsibility. When one considers that many evaluation studies completely neglect outcome measures in favor of workload or other process measures,7 the VA effort looks impressive despite its limitations. Information gleaned for this study obviously is a prerequisite for the decision to search for and implement alternative treatment approaches.
REFERENCES 1. Mensh, I. N. Drug addiction. In C. G. Costello, Ed. Symptoms of Psychopathology. New York: Wiley, 1970. 2. Cohen, J. Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press, 1969.
3. Dixon, W. J., Ed. Biomedical Computer Programs. Los Angeles: University of California Press, 1967. 4. Campbell, D. T., & Stanley, J. C. Experimental and Quasiexperimental Designs for Research. Chicago: Rand McNally, 1963. 5. Sells, S. B., Ed. Evaluation of Treatment (Vol. 1). Cambridge, MA: Ballinger, 1974. 6. Macro Systems, Inc. Three-year Follow-up Study of Clients Enrolled in Treatment Programs in New York City (Phase III Final Report). Silver Spring, MD: Macro, 1975. 7. Hatry, H. P. Measuring the effectiveness of nondefense public programs. Operations Research, 1970, 18:772-784.
ACKNOWLEDGMENTS The authors express their indebtedness to the following: Mansell Piper, Harry McKnight, Stewart Baker, Louise King, Brian Ochs, Jack Collier, Robert Normand, Wanda McKinney (VA Central Office staff), the VA field staff who participated in the study design and data collection, and the staffs of Macro Systems, Inc., and Opinion Research Corporation.
National Qualifying Exam for Pediatric Nurse Practitioners and Associates to be Held in April The second National Qualifying Examination for Pediatric Nurse Practitioners and Associates will be administered on Friday, April 14, 1978. Jointly developed by the professions of nursing and medicine, this voluntary certification examination is sponsored by the National Board of Pediatric Nurse Practitioners and Associates. Members of the Board are the National Association of Pediatric Nurse Associates and Practitioners and the American Academy of Pediatrics. The National Qualifying Examination is a one-day written examination that evaluates entry-level competence in the pediatric nurse practitioner/associate role. Certification by the National Board of Pediatric Nurse Practitioners and Associates provides formal recognition of the nurse's qualifications to practice as a pediatric nurse practitioner/associate. Assistance in examination development, scoring and analysis, and evaluation is provided by the National Board of Medical Examiners. Applications and detailed information regarding the 1978 National Qualifying Examination should be directed to: Mary Kaye Willian, RN, PNA, Executive Director, National Board of Pediatric Nurse Practitioners and Associates, P.O. Box 1034, Evanston, IL 60204
AJPH January, 1978, Vol. 68, No. 1
43
Abstract: A nationwide evaluation of 49 Veterans Administration drug dependence treatment programs was conducted using a sample of 1,655 patients. Their status with reference to seven treatment goals was assessed at admission and again 11 months later to determine change. The following changes were observed: a large decrease in heroin use; a small increase in the use of alcohol; moderate decline of involvement in the drug culture; moderate increase in psychological well-
being; and small increase in economic independence. No statistically significant changes were observed in frequency of arrests, in establishment of stable living arrangements, in drug related medical problems, or in the ability to relate to other people. Implications of the findings for further research and program modification are discussed. (Am. J. Public Health 68:39-43, 1978).
In 1973, the Veterans Administration (VA) began a study to measure patient improvement in its 49 drug-dependence treatment programs located throughout the country. The effectiveness of the nationwide program and of individual programs was to be inferred from the type and degree of improvement observed. This study differed in several ways from previously reported "follow-up" studies.1 It involved multiple rather than single programs; its focus was on measuring the effectiveness of total programs rather than on single modalities (such as methadone maintenance); and, although patient changes were studied, such changes were primarily of interest as indicators of treatment effectiveness rather than as measures of the course of addiction. Methodologically, this study was distinguished by its: a) emphasis on explicit treatment goal definition as a prelude to evaluation, b) the use of data collection instruments at both admission and follow-up to measure treatment goal attainment, and c) computation of follow-up time-point from admission rather than from treatment termination.
tween July and December 1973. This report focuses on a subsample of 1,655 patients who were randomly selected for follow-up.* Patients who were incarcerated at the time of follow-up were not interviewed because it seemed doubtful that private interviews could regularly be arranged for in prisons. The average age of the sample of 1,655 was 28.2 years; 57 per cent were white, 19 per cent had some education beyond high school, 32 per cent were on probation or parole, and 6: per cent had been admitted to drug dependence treatment at least once before. Heroin was rated as the primary drug of abuse for 58 per cent of the sample. A comparison of the admission characteristics of the follow-up sample with the total admission group showed no statistically significant differences Q2 test, .05 level of probability) on age, race, education, number of previous admissions for drug abuse, frequency of arrests, and self-support. The follow-up group did have more stable living arrangements (before admission), were less often referred directly from military service, and were rated as having had higher motivation to stop using drugs.
Method
Instrumentation Two fixed alternative interview schedules, the Intake Fonn and the Eleven Month Follow-up Form, were written to assess patient status relative to the following treatment goals: 1) elimination of nonprescribed drug use, 2) development of attitudes and skills conducive to being economi-
Subjects Admission data were collected on 4,946 patients admitted to all 49 VA drug dependence treatment programs beFrom the Veterans Administration, Washington, DC. Address reprint requests to Theodore W. Lorei, MSW, Veterans Administration, 810 Vermont Avenue, NW, Washington, DC 20420. This paper, submitted to the Journal March 17, 1976, was revised and accepted for publication April 5, 1977.
AJPH January, 1978, Vol. 68, No. 1
*Sixty four per cent of the 2,607 patients originally selected (15 per cent not located, 8 per cent incarcerated with private interviews not possible, 2 per cent dead, 7 per cent miscellaneous losses, 4 per cent refusals). 39
LOREI, ET AL.
cally self-supporting, 3) elimination of antisocial behavior, 4) establishment of stable living arrangements, 5) improvement of health, 6) improvement of social interactional skills, and 7) promotion of a sense of psychological well-being independent of the drug culture. Goal relevant items in both forms were identical, thereby permitting comparison of patient status at both time-points. The Intake Form also contained items for identifying data and characteristics hypothesized to be related to treatment outcome. The Eleven Month Followup Form contained items for describing length of treatment received, as well as some additional items for assessing goal status (for which there were no parallel items in the Intake Form).t
Procedure The Intake Form was completed by clinical staff; the Eleven Month Follow-up Form was completed by survey research interviewers employed by Macro Systems, Inc., and Opinion Research Corporation. Urine samples were requested at the conclusion of the follow-up interviews and $10 incentives were paid. Urinalyses were performed by The Laboratory of Chromatography, Flushing, NY.
TABLE 1-Changes in Total Drug Use Category
Used
Admission % using
Follow-up % using
tb
dc
88.1 a
69.1
-18.68*
-.46
86.0
61.7
-21.88*
-.54
81.0
32.9
-25.63*
-.63
one or more
drugs Used one or more drugs other
than alcohol Used one or more drugs other than alcohol or
marijuana
a One would expect that this entry would be 100 per cent for patients admitted to drug dependence treatment. The fact that it is not apparently reflects the fact that some patients were not using a drug "on the average of twice a week during the first four weeks" as was required to be checked "yes" on the drug items. b t's were computed using the three types of total drug use scores as the basic observations. c d is a standardized effect size measure proposed by Cohen.2 Conceptually, it is the mean of the difference scores between admission and follow-up mean divided by the standard deviation of these scores. The computational formula suggested by Cohen d = t/\/i (2.5.9) was used. In the absence of any more appropriate standards, he suggests that d values of .20, .50, and .80 indicate small, medium, and large differences, respectively.
'p < .01.
Results Functioning at Follow-up (Total Group) Drug use. The description of drug use during the four weeks prior to the follow-up interview is based on patient report. Some evidence for the validity of these data is provided by the 95 per cent agreement between patient reported drug use in the past 48 hours (a period "within reach" of laboratory tests) and urinalyses for five drugs.** According to patient report, 69 per cent were using at least one of the drugs listedtt on the average of twice a week during the four weeks prior to the follow-up interview, 62 per cent were using one or more drugs other than alcohol (to the point of intoxication), and 33 per cent were using one or more drugs other than alcohol or marijuana (Table 1). Considering each drug individually, the results appear more favorable (Table 2). The degree of improvement on each of the "total" measures in Table 1 was statistically significant and "medium" in size according to the effect size measure (d) suggested by Cohen.2 The degree of improvement for heroin use (Table 2) was close to large. The reader may, of course, determine his or her own definitions of what constitutes a small, medium, or large change. The d statistic is intended only as a guide when more appropriate standards are not
available. The use of marijuana did not change to a statistically significant extent, and the use of alcohol increased slightly (Table 2). Eighty-nine per cent of the patients said they used drugs
*Copies of the Intake and Eleven Month Follow-up Forms are available on request from the senior author. **Eighty-four per cent provided urine samples. ltSee Table 2 for list. 40
TABLE 2-Changes in Individual Drug Use Drug
Methadone (nonprescribed) Heroin Other opiates, opium preparations, and synthetics Cocaine Barbiturates Other sedatives, hypnotics and
tranquilizers Amphetamines Cannabis sativa (marijuana or hashish) Hallucinogens Alcohol (to the point of intoxication) Others Note. N
Admission % using
Follow-up % using
ta
db
11.0 55.3
3.2 16.1
-8.96* -28.42*
-.22 -.70
15.6 17.6 24.2
4.0 6.2 7.7
-1 1.73* -10.47* -14.16*
-.29 -.26 -.35
19.8 21.3
9.0 8.3
-9.20* -11.67*
-.23 -.29
49.7 10.5
52.3 2.0
1.72 -10.61*
.04 -.26
26.1
32.1 3.5
4.05* -3.15*
.10 -.08
5.7 =
1,655
a Although the percentages of 2's (using drugs, as opposed to
l's, not
using drugs) are presented for convenience, t's were computed using the di-
chotomous scores. b d values of .20, .50, and .80 indicate small, medium and large differences respectively, between admission and follow-up.
*p
< .01.
less often at follow-up than at admission, 7 per cent said they used them as often, and 4 per cent said they used them more often. Self-support. At follow-up, 60 per cent reported that they were self-supporting (the mean of 1.60 in the follow-up column of Table 3 indicates that 60 per cent were rated "2", AJPH January, 1978, Vol. 68, No. 1
EVALUATION OF DRUG DEPENDENCE TREATMENT TABLE 4-Changes in Patient Behavior Relevant to other Treatment Goals
TABLE 3-Changes in Self-Support Behavior Follow-up
Admission Item
Supported self from employment (1 = No;2 = Yes) Supported dependents, if applicable (1 = No; 2 = Yes) Hours of employment per week Weekly incomea Attended school or job training (1 = No;2 = Yes)
t
X
d
1.45
1.60
9.94*
.24
1.39
1.54
6.74*
.24
17.84 $66.37
20.93 $76.32
4.88* 3.50*
.12 .09
1.07
1.17
9.21*
.23
N= 1,655 The means for this variable were computed including patients with no income. b d values of .20, .50, and .80 indicate small, medium, and large differences, respectively. *p < .01. a
i.e., self-supporting). The average number of hours of employment per week was 21, and the average weekly income was $76. When these last two statistics were computed, for only those individuals who were employed (56 per cent), the average number of hours of employment per week was 37 and the average income was $137.* As noted above, there were goal relevant items on the Eleven Month Follow-up Form which had no parallel on the Intake Form. Responses, as they relate to each goal, are discussed in the text. Relative to self-support, 13 per cent reported receiving money from public welfare, and 10 per cent from illegal activities. Eight per cent reported volunteer activities. Antisocial behavior. At follow-up, the average number of arrests reported by patients for the previous six months was .65 (Table 4). Thirty-six per cent had been arrested at least once for something other than traffic violations. There was no statistically significant change from admission (when arrests during the previous six months were recorded) in the mean number of arrests nor in the proportion that was arrested. The follow-up percentage may be an underestimate of the true figure because patients in jail were not interviewed. (Of course, not all who were in jail were necessarily arrested in the six month interval prior to the intended follow-up date). Stable living arrangements. At follow-up, 83 per cent were rated as having "some form of stable, noninstitutional living arrangements" (Table 4).t This percentage was not different from the admission percentage to a statistically significant degree. Follow-up items (not included in the admission form) indicated that 20 per cent had lived in more than one place during the past month, 12 per cent had had nights when they *Some statistics presented in the text do not appear in tables. tAs noted above, the means of dichotomous variables are readily converted to percentages. AJPH January, 1978, Vol. 68, No. 1
Admission Follow-up X X
Item
Mean number of arrests in last six months Stable living arrangements(1 = No;2 = Yes) Physical distress and disability (1 = Not at all; ... 4 = Severely) Difficulty in getting along with people (1 = Not at all; ... 4= Very definitely) Able to feel good without drugs (1 = Not at all; ... 4 = Very definitely) Involvement in the drug culture (1 = Not involved; ... 4= Heavily involved)
t
da
.59
.65
1.24
.03
1.81
1.83
1.45
.04
1.71
1.77
2.07
.05
1.98
1.92
-1.75
-.04
2.03
2.77
22.77*
2.81
2.05
.56
-22.87* -.56
N = 1,655
ad values of .20, .50, ferences, respectively.
and .80 indicate small, medium, and large dif-
*p < .01.
did not know whether they would have a place to sleep, 22 per cent had spent at least one night in an institution, 7 per cent felt they had not had adequate shelter, 10 per cent had serious problems in getting enough to eat, and 10 per cent had problems in getting adequate clothing. These percentages were not mutually exclusive. Medical problems. At follow-up, the average rating of the extent to which the patient had been "troubled by physical distress or disability from drug-related medical problems" was 1.77 on a four-point scale on which 2.00 was defined as "somewhat."** As can be seen from Table 4, this mean did not differ from the admission mean to a statistically significant degree. At follow-up, the most frequently reported "medical difficulties" were: insomnia (45 per cent), constipation (24 per cent), and withdrawal symptoms (22 per cent). Fortyseven per cent said they had no medical problems. Interpersonal adjustment. At follow-up, the average rating of the veteran's "difficulty in getting along with people in his immediate environment, such as wife, girl friend, parents, employer, co-worker" was 1.92 on a four-point scale on which 2.00 was defined as "somewhat".#4 This mean was not significantly different from the mean of 1.98 for the admission ratings (Table 4). Twenty per cent reported that others caused them "quite a bit" or "a great deal" of difficulty. Fourteen per cent reported that they caused others "quite a bit" or "'a
**The four scale points were defined as follows: 1=not at all, 2=somewhat, 3=moderately, 4=very definitely. tlThe four scale points were defined as follows: l=not at all, 2= somewhat, 3 =moderately, 4= very definitely.
LOREI, ET AL.
great deal" of trouble. Fifteen per cent reported getting irritated with others "very easily", and 13 per cent reported having gotten into a physical fight with someone. Psychological well-being. This concept was measured by two questions at both admission and follow-up (Table 4). The follow-up mean on "ability to achieve a sense of psychological well-being apart from taking drugs and being a part of the drug culture" was 2.77 on a four-point scale on which 3.00 was defined as "moderately".t# The mean "involvement with the drug culture" was 2.05 on a four-point scale on which 2.00 was defined as "somewhat".#4 Both of these means indicated significant improvement over admission status and the effect sizes were medium.
Functioning at Follow-up (Patients Grouped by Program) The basic question was whether some programs produced better results than others. The lack of random assignment of patients to programs precluded answering this question definitively, but the analyses were intended to develop evidence regarding possible differential program effectiveness. Seven analyses of multiple covariance (BMD 04V) were performed.3 The dependent variables were follow-up status on total drug use, self-support, number of arrests, stable living arrangements, physical distress, interpersonal difficulty, and psychological well-being. This analyses of covariance yielded significant F values (p < .01) for all dependent variables except self-support. In other words, the outcomes for patient cohorts from each program differed significantly, even when adjustments had been made for initial characteristics that might have influenced those outcomes. The interprogram differences were all small with the exception of patients' reported sense of psychological well-being, which were of medium size (using the effect size standards suggested by Cohen2).
Discussion The contributions of the present study toward answering questions about program effectiveness are discussed below, with the recognition that additional studies are necessary to answer these questions definitively. This study should be regarded as a treatment program evaluation rather than a treatment type evaluation because it was concerned with the total set of treatment activities rather than with specific modalities such as methadone maintenance. The following description of treatment program evaluation is provided by Sells.5 Stated most generally, in treatment program evaluation, the "treatment" is defined, in the logician's terms, by extension; it is the treatment offered at a specific organization (agency, hospital, clinic, or treatment center) and the research is concerned with the results obtained by that organization, measured against its specific goals, in administering its particular treatment program to a given population of patients.
When the Veterans Administration's treatment program is viewed as a whole (rather than by individual treatment cen42
ters, or by modalities, such as methadone maintenance), the study results suggest its treatment was helpful in reducing drug abuse or misuse, in reducing involvement in the drug culture, in increasing a sense of psychological well-being, and in increasing self-support. These results also suggest that the treatment program was not effective in reducing the use of alcohol, distress from drug-related medical problems, frequency of arrests, interpersonal difficulty, and instability of living arrangements.
Although statistically significant improvements
were
observed in each of the areas just mentioned at some local programs, and although such improvements may have occurred for certain unidentified patient subgroups, the implications of the negative findings for the overall program should not be ignored. Treatment strategies need to be reexamined to determine why they are not more effective in the areas identified above. Also, it should be noted that, although there was statistically significant increase in the percentage of patients who were self-supporting, the large percentage who were not self-supporting at follow-up suggests that treatment efforts need to be reconsidered in this area. Utilization of the findings of this study makes it possible to define program goals more precisely and thus to state the question about effectiveness more specifically. The goals formulated to guide the present study, although more explicitly stated than those used in most studies, were quite general. For example, one of these goals was "to eliminate the nonprescribed use of drugs." But, is this realistic? Probably most program directors would feel that, even under the best conditions, some patients would continue to use drugs. The question really is: what degree of elimination of drug use is achievable in the present state of treatment technology? The results make it possible to set provisional, specific goals in the areas considered or, alternatively, to provide baselines against which to judge future treatment efforts. Because of major differences in variable definition and study design, it is almost impossible to make meaningful comparisons between these results and those of other investigators. Nevertheless, it is interesting to note, that in studying change from before to during treatment, Sells5 found a large reduction in opiate use, a finding that is similar to the large reduction in heroin use reported here. Contrary to the present results, Sells5 found a small decrease in the use of alcohol and a small increase in unemployment. Comparison of the number of arrests was not possible between the studies. In a study for the National Institute on Drug Abuse, Macro Systems, Inc.,6 found large decreases in drug use, criminal activities, and unemployment. As is apparent, there is agreement between the findings of the present study and the two studies just mentioned in that participation in treatment was associated with a large decrease in drug use. The results on the other outcome measures were not consistent across studies. From the vantage point of research, the present study had many weaknesses. The design problems have already been discussed. The instruments were crude, partly because they had to be kept brief and suitable for use by clinical staff with diverse professional backgrounds. The time period for follow-up was short, possibly accounting for the fact that AJPH January, 1978, Vol. 68, No. 1
EVALUATION OF DRUG DEPENDENCE TREATMENT
variables such as number of arrests did not decrease. From the second vantage point, that of program evaluation as practiced by some government agencies, the study appears stronger. We feel that the Veterans Administration engaged in a pioneering attempt to assess the consequences of one of its newer programs and to provide feedback to staff at all levels of responsibility. When one considers that many evaluation studies completely neglect outcome measures in favor of workload or other process measures,7 the VA effort looks impressive despite its limitations. Information gleaned for this study obviously is a prerequisite for the decision to search for and implement alternative treatment approaches.
REFERENCES 1. Mensh, I. N. Drug addiction. In C. G. Costello, Ed. Symptoms of Psychopathology. New York: Wiley, 1970. 2. Cohen, J. Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press, 1969.
3. Dixon, W. J., Ed. Biomedical Computer Programs. Los Angeles: University of California Press, 1967. 4. Campbell, D. T., & Stanley, J. C. Experimental and Quasiexperimental Designs for Research. Chicago: Rand McNally, 1963. 5. Sells, S. B., Ed. Evaluation of Treatment (Vol. 1). Cambridge, MA: Ballinger, 1974. 6. Macro Systems, Inc. Three-year Follow-up Study of Clients Enrolled in Treatment Programs in New York City (Phase III Final Report). Silver Spring, MD: Macro, 1975. 7. Hatry, H. P. Measuring the effectiveness of nondefense public programs. Operations Research, 1970, 18:772-784.
ACKNOWLEDGMENTS The authors express their indebtedness to the following: Mansell Piper, Harry McKnight, Stewart Baker, Louise King, Brian Ochs, Jack Collier, Robert Normand, Wanda McKinney (VA Central Office staff), the VA field staff who participated in the study design and data collection, and the staffs of Macro Systems, Inc., and Opinion Research Corporation.
National Qualifying Exam for Pediatric Nurse Practitioners and Associates to be Held in April The second National Qualifying Examination for Pediatric Nurse Practitioners and Associates will be administered on Friday, April 14, 1978. Jointly developed by the professions of nursing and medicine, this voluntary certification examination is sponsored by the National Board of Pediatric Nurse Practitioners and Associates. Members of the Board are the National Association of Pediatric Nurse Associates and Practitioners and the American Academy of Pediatrics. The National Qualifying Examination is a one-day written examination that evaluates entry-level competence in the pediatric nurse practitioner/associate role. Certification by the National Board of Pediatric Nurse Practitioners and Associates provides formal recognition of the nurse's qualifications to practice as a pediatric nurse practitioner/associate. Assistance in examination development, scoring and analysis, and evaluation is provided by the National Board of Medical Examiners. Applications and detailed information regarding the 1978 National Qualifying Examination should be directed to: Mary Kaye Willian, RN, PNA, Executive Director, National Board of Pediatric Nurse Practitioners and Associates, P.O. Box 1034, Evanston, IL 60204
AJPH January, 1978, Vol. 68, No. 1
43
