BEHAVIORAL AND NEURAL BIOLOGY 25, 523--534
(1979)
Noradrenaline and Conditioned Reinforcement STEPHEN T. M A S O N 1 AND TREVOR W . ROBBINS 2 The Psychological Laboratory, University of Cambridge, Downing Street, Cambridge, England Male albino Wistar rats received bilateral intracerebral injections of 8/~g of the neurotoxin 6-hydroxydopamine into the dorsal noradrenergic bundle producing severe depletion of forebrain noradrenaline. Acquisition and extinction of lever responding on a schedule of random reinforcement on two levers was studied to measure the effects of conditioned reinforcement (light flash or auditory dick) which preceded each presentation of food. The rats with lesions failed to show the usual increase in rate following reduction of reinforcement probability from .5 to •1. During extinction, the rats with noradrenaline depletion showed greater resistance to extinction in terms of time to reach an extinction criterion of no response for 2 consecutive min. No change in the degree of conditioned reinforcing value obtained by either initially neutral stimulus (light flash or auditory click) was seen. Results are discussed in terms of theories of function of the dorsal noradrenergic projection, but appear to rule out hypotheses based on conditioned reinforcement•
Numerous previous reports have indicated that the noradrenergic (NA) fibers ascending from the pontine nucleus, the locus coeruleus, and innervating forebrain areas such as cortex and hippocampus play a critical role in extinction processes (Mason, 1979). Lesion to these fibers using the selective neurotoxin 6-hydroxydopamine (6-OHDA) (Uretsky & Iversen, 1969) has been found to cause resistance to extinction in many situations when reward is withdrawn. Following the initial demonstration by Mason and Iversen (1975) in a runway task, similar effects on extinction have been seen on operant lever pressing (Mason & Iversen, 1977a; Thornton, Goudie, & Bithell, 1975), complex motor manipulative tasks (Mason & Iversen, 1977b), a go/no-go alternation paradigm (Tremmel, Morris, & Gebhart, 1977), one-way active avoidance (Ashford & Jones, 1976), ex' Present address: Division of Neurological Sciences, Department of Psychiatry, University of British Columbia, Vancouver, B.C. V6T lW5, Canada. The experiments were controlled by the ONLIsystem of online computer control developed by Dr. S. E. G. Lea and Dr. C. Crook with the support of MRC Grant G970/297/B to A. J. Watson. The technical assistance of Mrs. Julia Grey, supported by MRC Grant G975/753/N to S. D. Iversen is appreciated• 523 0163 - 1047/79/040523 - 12502.00/0 Copyright © 1979by AcademicPress, Inc. All rights of reproduction in any form reserved.
524
MASON AND ROBBINS
tinction of the exploratory response (Mason & Fibiger, 1977), and replicated in the original runway situation (Owens, Boarder, & Gray, 1977). The present experiment attempts to define the generality of the dorsal bundle extinction effect, and to identify the behavioral mechanisms which underlie it. The dorsal bundle extinction effect has been demonstrated in extinction following training under continuous reinforcement. The probability of reinforcement (Pr) following responding therefore shifts from Pr = 1.00 to pr = 0.00. This experiment explores the generality of the dorsal bundle extinction effect by training rats under an intermittent schedule where 0.00 < p~ < 1.00 and also examines the effect of reducing p~ during training to values, however, greater than Pr = 0.00. These manipulations will test whether the rate-increasing effects of noradrenaline (NA) depletion in extinction generalize to other situations in which the probability of reinforcement is reduced. Several behavioral mechanisms have been suggested to account for the enhanced responding in extinction shown after NA depletions. Some of these mechanisms such as internal inhibition (Mason & Iversen, 1977c), perseveration (Mason & Iversen, 1977d), and frustrative nonreward (Mason & Iversen, 1978) have been tested and rejected. A further possibility is that the lesion enhances the effect of conditioned reinforcement. Stimuli paired with reward may themselves acquire reinforcing properties and increase resistance to extinction (Wike, 1966). The present experiment is a choice paradigm for measuring the effects of NA depletion on conditioned reinforcement. This paradigm has been previously used successfully to measure conditioned reinforcement and the modification of it by drugs (Robbins, 1975). Conditioned reinforcers must acquire their properties by the extent to which the organism in some sense "attends" to them during training. In this experiment, a compound stimulus, consisting of a click from the magazine and a flash of light from a bulb located above the magazine tray, preceded each delivery of food. A further aim of this experiment was to investigate which components of the compound stimulus came to control responding in extinction and how NA depletion would affect this differential control. In situations where two or more stimuli are equally correlated with reinforcement, it is sometimes possible to show a negative correlation between the contribution of the two to discriminative performance (Sutherland & Holgate, 1966). This finding may demonstrate a form of stimulus processing by the organism related to a selective attention. One mechanism by which NA depletion might enhance responding in extinction has been linked with a derangement of attentional mechanisms (Mason & Iversen, 1977a, 1977c). Specifically, it has been suggested that NA depletion increases the span of attention during training so that more stimuli than normal come to control performance. Therefore, the dorsal bundle extinction effect might arise since more S-R connections are laid
NA AND CONDITIONED REINFORCEMENT
525
down during acquisition and hence more such connections have to be broken during extinction before responding ceases. This hypothesis can be tested by observing whether the relative rate of responding for the choice between the components of the compound conditioned reinforcer is reduced in rats with NA depletion. The present study thus investigates the effects of forebrain NA depletion on acquisition and extinction of a novel two-lever schedule, whether any resistance to extinction could be explained by a change in conditioned reinforcement and whether similar resistance to extinction occurs if the probability of reinforcement is changed, not to zero as occurs in extinction, but to a lower, still positive value. METHOD
Subjects Ten 6-OHDA dorsal bundle lesioned rats and 10 vehicle-injected controls were used.
Apparatus Behavioral tests were conducted in standard two-lever operant test chambers (Campden Instruments, CI-410) supplied with pellet dispensers and housed in sound attenuating chambers. The schedule was controlled and the data collected by an Online computer (Modular One, CTL, England).
Surgical Procedure Male albino rats weighing approximately 200 g and aged 2.5 months were anaesthetized with Equithesin (3 ml/kg), positioned in a stereotaxic apparatus (Kopf Instruments), and the skull exposed. The head was leveled between bregma and lambda, two holes drilled in the skull, and a 30-gauge cannula lowered bilaterally to the following coordinates taken from K6nig and Klippel (1963): 6 mm posterior from bregma, 0.8 mm lateral from the midline, and 5 mm below dura. Eight micrograms of 6-OHDA dissolved in 2 ~1 of 0.9% saline with 1 mg/ml ascorbic acid antioxidant were infused at the rate of 1 tzl/min over 2 rain. Control animals had the cannula lowered to the same position but received only saline-ascorbate infusion of the same volume. The cannula was left in place for a further I min to allow diffusion of the drug and then withdrawn and the skin sutured.
Biochemical Assay At the completion of behavioral testing a random sample of the surviving animals was sacrificed by decapitation and their brains assayed for catecholamines to confirm the extent and pattern of the depletions. Following decapitation the brain was rapidly removed from the skull and
526
MASON AND ROBBINS
dissected on ice into regions. The brain was placed with the ventral surface uppermost and a coronal cut made anterior to the olfactory tubercle. The tissue anterior to this was included in the cortex region. A coronal cut at the level of the optic chiasm was then made and from the dorsal part of this coronal section the cortex and striatum were dissected. The brain was rotated dorsal surface uppermost and a parasagittal cut along the rhinal fissure separated the amygdaloid-pyfiform complex from the cortex. The rest of the cortex and the hippocampus were blunt dissected from the remainder of the brain. The caudal part of the striatum was removed, the brain turned ventral surface uppermost and a coronal cut posterior to the mammillary bodies separated the diencephalon from the brain stem. The hypothalamus was removed from the ventral portion of this coronal section by parasagittal cuts made at the lateral border of the anterior hypothalamus. The regions were weighed and then homogenized in 0.1 N perchloric acid. The homogenates were then assayed for NA and DA by a sensitive radioenzymatic assay method modified from Cuello, Hiley, and Iversen (1973). This method is based on the conversion of catecholamines to their O-methylated derivatives in the presence of tritiated S-adenosyl methionine and the enzyme catechol-Omethlytransferase.
Behavioral Procedure Acquisition. Animals were deprived to 90% of their free-feeding weight starting 1 week after the operation. Subsequently they were fed 15 g of laboratory chow per rat immediately after the end of the testing for that day, this schedule permits a gradual increase in body weight above the initial 90% level. Water was available ad libitum. Animals were lever shaped for food reward (P. J. Noyes, 45 mg) as described elsewhere (Mason & Iversen, 1977a). After this shaping the animals were placed immediately onto a two-lever random schedule of reinforcement (Robbins, 1975). After each reinforcement, a random number routine selected which lever next delivers reinforcement with equal probability (i.e., p = 0.5), and with no constraints on the number of successive reinforcements on a given lever. The lever selected for reinforcement remained selected until reinforcement was delivered. During this time, pressing the other lever had no consequence. Presses on the selected lever delivered reinforcement initially with p = 0.5 (i.e., a random ratio schedule). This schedule has been found to produce high and equivalent response rates on both levers (Robbins, 1975). As well as the number of lever responses on the left and right levers, the number of reinforcements obtained and the latency to collect the food pellet after its delivery (measured by latency to displace the food magazine door) were recorded. Preceding each delivery of food, a flash of light occurred from the magazine lamp which lasted .5 sec and a click from the automatic feeder sounded. These were the stimuli designed to acquire conditioned reinforcing properties over the course of
NA AND CONDITIONED R E I N F O R C E M E N T
527
training. Seventeen daily sessions of 15-min duration were given on this schedule and then the animals were shifted to a lower probability of reinforcement, for pressing on " s e l e c t e d" leverp = 0.1. A further 13 days of training was given under this schedule. Since there were two levers, only one of which delivered reinforcement with a probability of .5 o r . 1 (see above) the overall probability of reinforcement per lever press was .25 or .05 respectively. Extinction. After acquisition training, in which every presentation of food was paired with light flash and click, the conditioned reinforcing properties of these stimuli were tested in extinction. Now, responding on one lever produced only one, or both, of the light and click but never gave any food pellets (primary reinforcement). The conditioned reinforcers occurred with the same probability following a lever press as had primary reinforcement (. 1). Responses on the other lever had no consequence but were recorded. It has been found in previous experiments (Robbins, 1975) that normal rats will sample both levers and their stimulus consequences and come to display a preference for the lever producing the conditioned reinforcer. The lever producing the conditioned reinforcer (CR lever) was counterbalanced in terms of left- and right-hand side, as was the order or presentation of light only, click only, or both as the CR throughout the subsequent extinction testing. Between each extinction test day was interpolated 1 day of retraining on the original two-lever reinforced schedule used at the end of acquisition; this retraining was included to retard extinction. On the fourth day of extinction testing, pressing one lever produced the light and pressing the other now produced the click (both with p = 0.1) so a direct choice preference was measured between the two conditioned reinforcers. The response rates on the CR lever and on the no-conditioned reinforcer (NCR) lever were recorded for a 15-rain session, and at 45-sec intervals throughout this period. Thus, four preference values were obtained for each rat, the preference for light vs nothing, click vs nothing, click and light vs nothing, and click vs light. These different preferences were obtained on different extinction days for different animals using a Latin square design so that the effect of extinction per se would be counterbalanced. Following this design, 5 days of reinforced retraining were given and the animals were then placed again into extinction. Lever responses now produced neither food nor the conditioned reinforcer. The session ended when no lever response had occurred for 2 consecutive min. The time to reach this extinction criterion and the number of responses emitted prior to reaching it were recorded. Extinction testing continued for 2 consecutive days.
RESULTS
Biochemical Data The pattern and extent of central catecholamine depletions produced by the intracerebral injection of 6-OHDA into the fibers of the dorsal bundle
528
MASON A N D R O B B I N S
are shown in Table 1. It confirms that severe and p e r m a n e n t destruction of noradrenergic neurons in the cortex and h i p p o c a m p u s occurred with some loss of hypothalamic N A but no change in brain dopamine (DA) in striatum, hypothalamus, or frontal cortex.
Behavioral Results Acquisition. Acquisition of the r a n d o m two lever schedule did not differ between the treated and control groups. By the last session of training with Pr = .5, both groups were earning a considerable n u m b e r of reinf o r c e m e n t s [control m e a n --- 158.5, treated mean = 154.7, t (18) = .49, NS], which they collected rapidly after delivery [control m e a n -- 1.06 sec, treated mean = 1.29, t (18) = 1.10, NS] thus indicating that both groups had learned the significance of the flash and click in indicating the presentation of food. Thus, immediately prior to transfer to a lower probability of reinforcement both groups showed equivalent performance. U p o n reduction o f p r to . 1, h o w e v e r , a difference appeared b e t w e e n treated and control groups shown in Fig. 1 with the treated animals failing to increase their response rate as rapidly as controls. On the last day of .5 training the controls had emitted a m e a n of 271 responses on the two levers together and the treated mean of 248. On the first day w h e n p r was reduced t o . 1 the controls increased their responses to a mean of 398 whereas the treated animals emitted only a mean of 208 It (18) -- 2.73, p < .05]. This difference was also reflected in the n u m b e r of reinforcements earned, with the controls gaining a mean of 42.5 (this is less than that on the last day of .5 training b e c a u s e of the fivefold increase in the average n u m b e r of responses required before delivery of a food pellet) while the treated rats
TABLE 1 Post-mortem Assay of Amines Following Dorsal Bundle 6-OHDA Lesion a
Region Noradrenaline Cortex Hippocampus Hypothalamus Cerebellum Dopamine Cortex Hypothalamus Striatum
Control (N = 6)
.149 .206 .838 .176
+_ .006 -+ .019 _+ .111 +_ .084
.093 +_ .025 .299 _+ .033 5.435 _+ .686
Treated (N = 3)
.014 .035 .583 .107
+ -+ +_ __
Percentage
.010 .018 .327 .030
9.4 17.0 69.6 60.8
.180 -4- .085 .256 _+ .098 8.237 -4- 1.131
193 85.6 152
a Values are means with standard error of the mean in micrograms of amine per gram wet weight of tissue of three randomly selected treated rats and six controls. Percentage is the percentage of control concentrations remaining in lesioned tissues.
NA AND CONDITIONED REINFORCEMENT
529
800
m.__~
o
."
I I I z
~l Im~
¢
•
500
ItIt
o
o
CONTROLS
•
TREATED
0/01 I
....................
/o
7e../ . . . . . . . . . . . . . . . . . . . . . . . . .
t
200
I
j
0.5 1;
1'7
0.1 1'
2'
;
4'
5'
6'
7'
8'
DAYS
Fro. 1. Lever responses of control and DB-lesioned rats upon transfer from a random schedule with probability of reinforcement equal to .5 to a similar schedule with lower probability of reinforcement (Pr = .1). Values are means of 10 control and 10 treated rats. Dotted horizontal line indicates .5 response rate. An asterisk indicates that control and treated rats were significantly different at the 5% level. r e c e i v e d o n l y 17.5 r e i n f o r c e m e n t s [t (18) = 3.01, p < .01]. A s i m i l a r t r e n d o c c u r r e d f o r t h e n e x t 2 d a y s (Fig. 1) b u t f a i l e d to r e a c h i n d i v i d u a l significance. A f t e r a f u r t h e r p e r i o d o f t r a i n i n g , the t w o g r o u p s h a d b e c o m e i n d i s t i n g u i s h a b l e in t e r m s o f t h e n u m b e r o f r e i n f o r c e m e n t s e a r n e d [last d a y o f p , = 0.1 s c h e d u l e , c o n t r o l m e a n = 49.4 t r e a t e d m e a n = 49.5, t (18) = .012, N S ] . T h u s , b o t h g r o u p s b e g a n t h e e x t i n c t i o n p r e f e r e n c e t e s t s r e s p o n d i n g at the s a m e l e v e l . Extinction. T h e p r e f e r e n c e f o r o n e o r o t h e r o f the c o n d i t i o n e d r e i n f o r c e r s is s h o w n in Fig. 2. T h e o r d e r o f t e s t w a s c o u n t e r b a l a n c e d f o r d i f f e r e n t a n i m a l s o v e r the 3 d a y s o f e x t i n c t i o n in t h e c a s e s o f t h e " f l a s h , " " c l i c k , " a n d " b o t h " t e s t s , b u t all a n i m a l s r e c e i v e d the choice b e t w e e n c l i c k a n d flash on D a y 4 o f e x t i n c t i o n . A m a r k e d p r e f e r e n c e for t h e l e v e r
530
MASON AND ROBBINS CLICK
FLASH
BOTH
CHOICE
300
200 w co z 0 o. cO w cc
100
C
T
iiiiiiiSi
C
T
C
T
C
T
FIG. 2. Preference for the lever producing conditioned reinforcer (flash, click, or both together) for 10 control (C) rats and 10 treated (T) rats. The vertically shaded columns show the response rate on the lever producing the conditioned reinforcer compared to that producing no stimulus (dotted column). In the case of a direct choice between click and flash (choice) the vertical lines indicate responses on the crick lever, the cross-hatched columns indicate responses on the flash lever.
which gave the click compared to the lever which had no consequence was found, and this did not differ between the control and lesioned rats. A smaller, but still significant, preference for the flash was also shown and again did not differ between the control and treated groups. The two groups were also similar in their preference for the c o m p o u n d click with flash stimulus over no consequence. These conclusions were confirmed by analysis of variance (Winer, 1962). Animals emitted significantly more responses to the CR lever than on the N R lever IF(l,18) = 30.5, p < .001] showing the adequacy of the paradigm to produce and measure conditioned reinforcement. The nature of the conditioned reinforcer was also found to be significant [F(2,36) = 10.4, p < .001] indicating that the preference for the flash was much less than that for the other two CRs. H o w e v e r , neither the group effect nor the group by CR condition interaction were significant (all Fs less than 1) indicating that the lesion had no effect on any of these preferences. Similar analysis of Day 4 choice of flash versus click indicated a significant preference for the click over the flash [F(1,18) = 14.72, p < .001] but this was not affected by the lesion since neither the group effect nor the group by CR lever interaction reached significance at the 5% level.
NA AND CONDITIONED REINFORCEMENT
531
After 5 days of reinforced retraining the response rates were again high for both groups which were performing similarly one to the other [control reinforcement = 50.9, lesioned = 48.2, t (18) = .69, NS]. The animals were then placed into extinction where no conditioned reinforcers were presented and a significant dorsal bundle extinction effect was seen. The treated animals took significantly longer to reach the extinction criterion on day one than controls [control mean = 421 sec, treated mean = 886 sec, t (18) = 3.81,p < .01]. By Day 2 of extinction testing both groups had extinguished to the same level. On neither day did the rate criterion (number of lever presses emitted prior to reaching extinction criterion) differ between the control and lesioned groups [control mean = 164, lesioned mean = 175, t (18) = .19, NS].
DISCUSSION The results reported here define more precisely the nature of the dorsal bundle extinction effect and eliminate certain behavioral explanations of it. First, the results demonstrate an enhanced resistance to extinction as seen before following NA depletion (Mason & Iversen, 1975; Thornton et al., 1975; Owens et al., 1977), but only according to a latency criterion. Training of the rats under an intermittent schedule of reinforcement evidently attenuates the normal increase in r a t e of responding shown by rats with noradrenaline depletion in extinction. This is consistent with the effect of previous extinction experience on the dorsal bundle extinction effect (Mason, 1978) and with other partially reinforced schedules (Mason & Fibiger, 1978). H o w e v e r , the rats with lesions were more persistent in extinction, taking significantly longer than controls to reach a criterion of no-responding. A possible reason for the absence of an overall increase in rate of responding during extinction can be deduced from the acquisition data. When the probability of reinforcement for lever pressing was reduced on the random schedule the control rats showed a marked increase in responding which h o w e v e r was not shown by rats with noradrenaline depletion. This apparent failure of the rats with lesions to exhibit the normal increase in responding following the reduction of reinforcement probability has been noted previously. Mason and Iversen (1977d) found that transfer from CRF (continuously reinforced responding) to DRL (differential reinforcement of low rates of responding) schedules produced a lower rate of responding in rats with NA depletion than in controls. Price, Murray and Fibiger (1977) found a possibly similar effect when transfer occurred from CRF to VI. These effects of reducing the reinforcement probability to some smaller but still positive value obviously stand in contrast to those found in extinction following CRF training where Pr becomes zero. To what extent can these opposing effects be reconciled? One possibility is that nor-
532
MASON AND ROBBINS
adrenaline depletion has two effects which are directly opposed in a situation in which extinction follows training under intermittent reinforcement. For example, noradrenaline depletion might attenuate the transient rateincreasing effects of nonreinforcement which have been labeled frustrative by some theorists (Amsel, 1958), leading to a slight decline in responding shown in initial portions of the final extinction trial. However, in addition to this effect the lesion may increase responding in the latter portion of the extinction trial by an increased resistance due perhaps to a reduction in the aversiveness of nonreward (Amsel, 1958). These ideas have been investigated more thoroughly elsewhere (Mason & Iversen, 1978). Conditioned reinforcement does not seem to contribute to any of these effects within the limits of the present experimental paradigm. Although a significant conditioned reinforcement effect was found, it failed to differ between the control and lesioned animals. This result is interesting for several reasons. Stein (1964) and Hill (1970) have suggested that the effects of conditioned reinforcement might depend on the release of noradrenaline. This hypothesis is supported by evidence that psychomotor stimulants such as pipradrol can enhance the effects of conditioned reinforcement (Hill, 1970; Robbins, 1976), including a demonstration using the present paradigm (Robbins, 1975). It might be expected from this model therefore, that noradrenaline depletion would reduce the effects of conditioned reinforcement; our prediction from the stimulus sampling model was conversely that an increase might occur, which could therefore explain the increased responding in extinction. Neither effect was found. This lack of effect might suggest that pipradrol is exercising its effect on conditioned reinforcement through another, non-noradrenergic, possibly dopaminergic system (Robbins, unpublished data). Another consideration supports the conclusion that the dorsal bundle extinction effect does not depend on a lesioned-induced change in conditioned reinforcement. An increased resistance to extinction was shown by treated rats when responding had no consequence in presenting conditioned stimuli during the final extinction trial. The only possible source of conditioned reinforcement would therefore arise as response feedback. In a CRF situation kinaesthetic and proprioreceptive feedback might be expected to acquire conditioned reinforcing properties, through their frequent pairing with reward. However, in the present experiment such kinaesthetic stimuli are paired only intermittently with reinforcement, are not predictive of it, and therefore would not be expected to acquire conditioned reinforcing properties. Yet the dorsal bundle extinction effect occurs in this situation which minimizes conditioned reinforcers. Finally, the results with conditioned reinforcement have implications for attentional theories of the function of the dorsal NA bundle. The
NA AND CONDITIONED REINFORCEMENT
533
conditioned reinforcer was a compound stimulus consisting of click and light flash. B o t h o f t h e s e stimuli a c q u i r e d s o m e c o n t r o l o v e r r e s p o n d i n g in e x t i n c t i o n , a l t h o u g h c o n t r o l w a s s t r o n g e r f o r t h e click. W h e n m e a s u r e d o n s e p a r a t e o c c a s i o n s as a c h o i c e b e t w e e n t h e t w o stimuli, t h e t r e a t e d a n i m a l s s h o w e d n o a t t e n u a t i o n o f this b i a s in f a v o r o f t h e click. T h u s , t h e y d i d n o t a t t e n d to t h e t w o s i m u l i in a m o r e d i s t r i b u t e d f a s h i o n t h a n c o n t r o l s . I t m i g h t , h o w e v e r , b e t h a t if stimuli c o u l d b e f o u n d to w h i c h t h e c o n t r o l animals did n o t a t t e n d at all (unlike the crick and flash in the p r e s e n t e x p e r i m e n t ) t h e n t h e l e s i o n e d a n i m a l s c o u l d b e s h o w n to b e s a m p l i n g t h e s e . T h u s , it m i g h t b e t h a t t h e a t t e n t i o n a l m e c h a n i s m o f d o r s a l b u n d l e f u n c t i o n a c t s , n o t in an a n a l o g f a s h i o n to i n c r e a s e t h e a m o u n t o f a t t e n t i o n o r s a l i e n c e g i v e n to a s t i m u l u s o n c e it h a s b e e n s e l e c t e d , b u t in a digital f a s h i o n to d e t e r m i n e if t h a t s t i m u l u s is s e l e c t e d at all. T h i s finding, a n d o t h e r s r e p o r t e d in this p a p e r , h e n c e d o n o t r e f u t e an a t t e n t i o n a l t h e o r y o f d o r s a l N A b u n d l e f u n c t i o n ( M a s o n & I v e r s e n , 1977a; S e g a l & B l o o m , 1976); t h e y d o h o w e v e r p l a c e c o n s t r a i n t s o n h o w s u c h a t h e o r y m i g h t b e constructed.
REFERENCES Amsel, A. (1978). The role of frustrative nonreward in noncontinuous reward situations. Psychological Bulletin, 55, 102-119. Ashford, J., & Jones, B. J. (1976). The effects of intra-amygdaloid injections of 6-hydroxydopamine on avoidance responding in rats. British Journal of Pharmacology, 56, 255-261. Cuello, A. C., Hiley, R., & Iversen, L. L. (1973). Use of catechol-O-methyl-transferase for the enzyme radiochemical assay of dopamine. Journal of Neurochemistry, 21, 13371340. Hill, R. T. (1970). Facilitation of conditioned reinforcement as a mechanism for psychomotor stimulation. In E. Costa and S. Garattini (Eds.), Amphetamine and Related Compounds, pp. 781-795. New York, Raven Press. K6nig, J. F., & Klippel, R. A. (1963). The Rat Brain, A Stereotaxic Atlas. Baltimore: Williams & Wilkins Co. Mason, S. T. (1978). Parameters of the dorsal bundle extinction effect. Previous extinction experience. Pharmacology Biochemistry and Behavior, 8, 655-659. Mason, S. T. (1979). Noradrenaline: Reward or extinction. Neuroscience and Biobehavioral Reviews, in press. Mason, S. T., & Fibiger, H. C. (1978). Noradrenaline and partial reinforcement. Journal of Comparative and Physiological Psychology, 92, 1110-1118. Mason, S. T., & Fibiger, H. C. (1977). Altered exploratory behaviour after 6-OHDA lesion to the dorsal noradrenergic bundle. Nature (London), 269, 705-706. Mason, S. T., & Iversen, S. D. (1975). Learning in the absence of forebrain noradrenaline. Nature (London), 258, 422-424. Mason, S. T., & Iversen, S. D. (1977). Effects of selective forebrain noradrenaline loss on behavioural inhibition in the rat. Journal of Comparative and Physiological Psychology, 91, 165-173. (a) Mason, S. T., & Iversen, S. D. (1977). An investigation of the role of cortical and cerebellar noradrenaline in associative motor learning. Brain Research, 134, 513-527. (b)
534
MASON AND ROBBINS
Mason, S. T., & Iversen, S. D. (1977). Reward, attention and the dorsal noradrenergic bundle. Brain Research, 150, 135-148. (c) Mason, S. T., & Iversen, S. D. (1977). Behavioural basis of the dorsal bundle extinction effect. Pharmacology Biochemistry and Behavior, 7, 373-379. (d) Mason, S. T., & Iversen, S. D. (1978). The dorsal noradrenergic bundle, extinction and non-reward. Physiology & Behavior, 21, 1043-1046. Owens, S., Boarder, M. R., & Gray, J. A. (1977). The effects of depletion of forebrain noradrenaline on runway behaviour of rats. Experimental Brain Research, 28, R22R23. Price, M. T. C., Murray, G. N., & Fibiger, H. C. (1977). Schedule dependent changes in operant responding after lesions of the dorsal tegmental noradrenergic projection. Pharmacology Biochemistry and Behavior, 6, 11-15. Robbins, T. W. (1975). The potentiation of conditioned reinforcement by psychomotor stimulant drugs. A test of Hill's hypothesis. Psychopharmacology, 45, 103-114. Robbins, T. W. (1976). Relationship between reward-enhancing and stereotypical effects of psychomotor stimulant drugs. Nature (London), 264, 57-59. Segal, M., & Bloom, F. E. (1976). The action of norepinephrine in rat hippocampus IV. The effects of locus coeruleus stimulation on evoked hippocampal unit activity. Brain Research, 107, 513-525. Stein, L. (1964). Amphetamine and neural reward mechanisms. In H. Steinberg, A. V. S. de Reuck and J. Knight (Eds.), Animal Behaviour and Drug Action, pp. 98-118. London: Churchill. Sutherland, N. S., & Holgate, V. (1966). Two-cue discrimination learning in rats. Journal of Comparative and Physiological Psychology, 61, 198-207. Thornton, E. W., Goudie, A. J., & Bithell, V. (1975). The effects of neonatal 6-hydroxydopamine induced sympathectomy on response inhibition in extinction. Life Sciences 17, 363-368. Tremmel, F., Morris, M. D., & Gebhart, G. (1977). The effect of forebrain norepinephrine loss on two measures of response suppression. Brain Research, 126, 185-188. Uretsky, N. J., & Iversen, L. L. (1969). Effects of 6-hydroxydopamine on noradrenalinecontaining neurons in the rat brain. Nature (London), 221, 557-559. Wike, E. L. (1966). Secondary Reinforcement. New York: Harper & Row. Winer, B. J. (1962). Statistical Principles in Experimental Design. New York: McGrawHill.
(1979)
Noradrenaline and Conditioned Reinforcement STEPHEN T. M A S O N 1 AND TREVOR W . ROBBINS 2 The Psychological Laboratory, University of Cambridge, Downing Street, Cambridge, England Male albino Wistar rats received bilateral intracerebral injections of 8/~g of the neurotoxin 6-hydroxydopamine into the dorsal noradrenergic bundle producing severe depletion of forebrain noradrenaline. Acquisition and extinction of lever responding on a schedule of random reinforcement on two levers was studied to measure the effects of conditioned reinforcement (light flash or auditory dick) which preceded each presentation of food. The rats with lesions failed to show the usual increase in rate following reduction of reinforcement probability from .5 to •1. During extinction, the rats with noradrenaline depletion showed greater resistance to extinction in terms of time to reach an extinction criterion of no response for 2 consecutive min. No change in the degree of conditioned reinforcing value obtained by either initially neutral stimulus (light flash or auditory click) was seen. Results are discussed in terms of theories of function of the dorsal noradrenergic projection, but appear to rule out hypotheses based on conditioned reinforcement•
Numerous previous reports have indicated that the noradrenergic (NA) fibers ascending from the pontine nucleus, the locus coeruleus, and innervating forebrain areas such as cortex and hippocampus play a critical role in extinction processes (Mason, 1979). Lesion to these fibers using the selective neurotoxin 6-hydroxydopamine (6-OHDA) (Uretsky & Iversen, 1969) has been found to cause resistance to extinction in many situations when reward is withdrawn. Following the initial demonstration by Mason and Iversen (1975) in a runway task, similar effects on extinction have been seen on operant lever pressing (Mason & Iversen, 1977a; Thornton, Goudie, & Bithell, 1975), complex motor manipulative tasks (Mason & Iversen, 1977b), a go/no-go alternation paradigm (Tremmel, Morris, & Gebhart, 1977), one-way active avoidance (Ashford & Jones, 1976), ex' Present address: Division of Neurological Sciences, Department of Psychiatry, University of British Columbia, Vancouver, B.C. V6T lW5, Canada. The experiments were controlled by the ONLIsystem of online computer control developed by Dr. S. E. G. Lea and Dr. C. Crook with the support of MRC Grant G970/297/B to A. J. Watson. The technical assistance of Mrs. Julia Grey, supported by MRC Grant G975/753/N to S. D. Iversen is appreciated• 523 0163 - 1047/79/040523 - 12502.00/0 Copyright © 1979by AcademicPress, Inc. All rights of reproduction in any form reserved.
524
MASON AND ROBBINS
tinction of the exploratory response (Mason & Fibiger, 1977), and replicated in the original runway situation (Owens, Boarder, & Gray, 1977). The present experiment attempts to define the generality of the dorsal bundle extinction effect, and to identify the behavioral mechanisms which underlie it. The dorsal bundle extinction effect has been demonstrated in extinction following training under continuous reinforcement. The probability of reinforcement (Pr) following responding therefore shifts from Pr = 1.00 to pr = 0.00. This experiment explores the generality of the dorsal bundle extinction effect by training rats under an intermittent schedule where 0.00 < p~ < 1.00 and also examines the effect of reducing p~ during training to values, however, greater than Pr = 0.00. These manipulations will test whether the rate-increasing effects of noradrenaline (NA) depletion in extinction generalize to other situations in which the probability of reinforcement is reduced. Several behavioral mechanisms have been suggested to account for the enhanced responding in extinction shown after NA depletions. Some of these mechanisms such as internal inhibition (Mason & Iversen, 1977c), perseveration (Mason & Iversen, 1977d), and frustrative nonreward (Mason & Iversen, 1978) have been tested and rejected. A further possibility is that the lesion enhances the effect of conditioned reinforcement. Stimuli paired with reward may themselves acquire reinforcing properties and increase resistance to extinction (Wike, 1966). The present experiment is a choice paradigm for measuring the effects of NA depletion on conditioned reinforcement. This paradigm has been previously used successfully to measure conditioned reinforcement and the modification of it by drugs (Robbins, 1975). Conditioned reinforcers must acquire their properties by the extent to which the organism in some sense "attends" to them during training. In this experiment, a compound stimulus, consisting of a click from the magazine and a flash of light from a bulb located above the magazine tray, preceded each delivery of food. A further aim of this experiment was to investigate which components of the compound stimulus came to control responding in extinction and how NA depletion would affect this differential control. In situations where two or more stimuli are equally correlated with reinforcement, it is sometimes possible to show a negative correlation between the contribution of the two to discriminative performance (Sutherland & Holgate, 1966). This finding may demonstrate a form of stimulus processing by the organism related to a selective attention. One mechanism by which NA depletion might enhance responding in extinction has been linked with a derangement of attentional mechanisms (Mason & Iversen, 1977a, 1977c). Specifically, it has been suggested that NA depletion increases the span of attention during training so that more stimuli than normal come to control performance. Therefore, the dorsal bundle extinction effect might arise since more S-R connections are laid
NA AND CONDITIONED REINFORCEMENT
525
down during acquisition and hence more such connections have to be broken during extinction before responding ceases. This hypothesis can be tested by observing whether the relative rate of responding for the choice between the components of the compound conditioned reinforcer is reduced in rats with NA depletion. The present study thus investigates the effects of forebrain NA depletion on acquisition and extinction of a novel two-lever schedule, whether any resistance to extinction could be explained by a change in conditioned reinforcement and whether similar resistance to extinction occurs if the probability of reinforcement is changed, not to zero as occurs in extinction, but to a lower, still positive value. METHOD
Subjects Ten 6-OHDA dorsal bundle lesioned rats and 10 vehicle-injected controls were used.
Apparatus Behavioral tests were conducted in standard two-lever operant test chambers (Campden Instruments, CI-410) supplied with pellet dispensers and housed in sound attenuating chambers. The schedule was controlled and the data collected by an Online computer (Modular One, CTL, England).
Surgical Procedure Male albino rats weighing approximately 200 g and aged 2.5 months were anaesthetized with Equithesin (3 ml/kg), positioned in a stereotaxic apparatus (Kopf Instruments), and the skull exposed. The head was leveled between bregma and lambda, two holes drilled in the skull, and a 30-gauge cannula lowered bilaterally to the following coordinates taken from K6nig and Klippel (1963): 6 mm posterior from bregma, 0.8 mm lateral from the midline, and 5 mm below dura. Eight micrograms of 6-OHDA dissolved in 2 ~1 of 0.9% saline with 1 mg/ml ascorbic acid antioxidant were infused at the rate of 1 tzl/min over 2 rain. Control animals had the cannula lowered to the same position but received only saline-ascorbate infusion of the same volume. The cannula was left in place for a further I min to allow diffusion of the drug and then withdrawn and the skin sutured.
Biochemical Assay At the completion of behavioral testing a random sample of the surviving animals was sacrificed by decapitation and their brains assayed for catecholamines to confirm the extent and pattern of the depletions. Following decapitation the brain was rapidly removed from the skull and
526
MASON AND ROBBINS
dissected on ice into regions. The brain was placed with the ventral surface uppermost and a coronal cut made anterior to the olfactory tubercle. The tissue anterior to this was included in the cortex region. A coronal cut at the level of the optic chiasm was then made and from the dorsal part of this coronal section the cortex and striatum were dissected. The brain was rotated dorsal surface uppermost and a parasagittal cut along the rhinal fissure separated the amygdaloid-pyfiform complex from the cortex. The rest of the cortex and the hippocampus were blunt dissected from the remainder of the brain. The caudal part of the striatum was removed, the brain turned ventral surface uppermost and a coronal cut posterior to the mammillary bodies separated the diencephalon from the brain stem. The hypothalamus was removed from the ventral portion of this coronal section by parasagittal cuts made at the lateral border of the anterior hypothalamus. The regions were weighed and then homogenized in 0.1 N perchloric acid. The homogenates were then assayed for NA and DA by a sensitive radioenzymatic assay method modified from Cuello, Hiley, and Iversen (1973). This method is based on the conversion of catecholamines to their O-methylated derivatives in the presence of tritiated S-adenosyl methionine and the enzyme catechol-Omethlytransferase.
Behavioral Procedure Acquisition. Animals were deprived to 90% of their free-feeding weight starting 1 week after the operation. Subsequently they were fed 15 g of laboratory chow per rat immediately after the end of the testing for that day, this schedule permits a gradual increase in body weight above the initial 90% level. Water was available ad libitum. Animals were lever shaped for food reward (P. J. Noyes, 45 mg) as described elsewhere (Mason & Iversen, 1977a). After this shaping the animals were placed immediately onto a two-lever random schedule of reinforcement (Robbins, 1975). After each reinforcement, a random number routine selected which lever next delivers reinforcement with equal probability (i.e., p = 0.5), and with no constraints on the number of successive reinforcements on a given lever. The lever selected for reinforcement remained selected until reinforcement was delivered. During this time, pressing the other lever had no consequence. Presses on the selected lever delivered reinforcement initially with p = 0.5 (i.e., a random ratio schedule). This schedule has been found to produce high and equivalent response rates on both levers (Robbins, 1975). As well as the number of lever responses on the left and right levers, the number of reinforcements obtained and the latency to collect the food pellet after its delivery (measured by latency to displace the food magazine door) were recorded. Preceding each delivery of food, a flash of light occurred from the magazine lamp which lasted .5 sec and a click from the automatic feeder sounded. These were the stimuli designed to acquire conditioned reinforcing properties over the course of
NA AND CONDITIONED R E I N F O R C E M E N T
527
training. Seventeen daily sessions of 15-min duration were given on this schedule and then the animals were shifted to a lower probability of reinforcement, for pressing on " s e l e c t e d" leverp = 0.1. A further 13 days of training was given under this schedule. Since there were two levers, only one of which delivered reinforcement with a probability of .5 o r . 1 (see above) the overall probability of reinforcement per lever press was .25 or .05 respectively. Extinction. After acquisition training, in which every presentation of food was paired with light flash and click, the conditioned reinforcing properties of these stimuli were tested in extinction. Now, responding on one lever produced only one, or both, of the light and click but never gave any food pellets (primary reinforcement). The conditioned reinforcers occurred with the same probability following a lever press as had primary reinforcement (. 1). Responses on the other lever had no consequence but were recorded. It has been found in previous experiments (Robbins, 1975) that normal rats will sample both levers and their stimulus consequences and come to display a preference for the lever producing the conditioned reinforcer. The lever producing the conditioned reinforcer (CR lever) was counterbalanced in terms of left- and right-hand side, as was the order or presentation of light only, click only, or both as the CR throughout the subsequent extinction testing. Between each extinction test day was interpolated 1 day of retraining on the original two-lever reinforced schedule used at the end of acquisition; this retraining was included to retard extinction. On the fourth day of extinction testing, pressing one lever produced the light and pressing the other now produced the click (both with p = 0.1) so a direct choice preference was measured between the two conditioned reinforcers. The response rates on the CR lever and on the no-conditioned reinforcer (NCR) lever were recorded for a 15-rain session, and at 45-sec intervals throughout this period. Thus, four preference values were obtained for each rat, the preference for light vs nothing, click vs nothing, click and light vs nothing, and click vs light. These different preferences were obtained on different extinction days for different animals using a Latin square design so that the effect of extinction per se would be counterbalanced. Following this design, 5 days of reinforced retraining were given and the animals were then placed again into extinction. Lever responses now produced neither food nor the conditioned reinforcer. The session ended when no lever response had occurred for 2 consecutive min. The time to reach this extinction criterion and the number of responses emitted prior to reaching it were recorded. Extinction testing continued for 2 consecutive days.
RESULTS
Biochemical Data The pattern and extent of central catecholamine depletions produced by the intracerebral injection of 6-OHDA into the fibers of the dorsal bundle
528
MASON A N D R O B B I N S
are shown in Table 1. It confirms that severe and p e r m a n e n t destruction of noradrenergic neurons in the cortex and h i p p o c a m p u s occurred with some loss of hypothalamic N A but no change in brain dopamine (DA) in striatum, hypothalamus, or frontal cortex.
Behavioral Results Acquisition. Acquisition of the r a n d o m two lever schedule did not differ between the treated and control groups. By the last session of training with Pr = .5, both groups were earning a considerable n u m b e r of reinf o r c e m e n t s [control m e a n --- 158.5, treated mean = 154.7, t (18) = .49, NS], which they collected rapidly after delivery [control m e a n -- 1.06 sec, treated mean = 1.29, t (18) = 1.10, NS] thus indicating that both groups had learned the significance of the flash and click in indicating the presentation of food. Thus, immediately prior to transfer to a lower probability of reinforcement both groups showed equivalent performance. U p o n reduction o f p r to . 1, h o w e v e r , a difference appeared b e t w e e n treated and control groups shown in Fig. 1 with the treated animals failing to increase their response rate as rapidly as controls. On the last day of .5 training the controls had emitted a m e a n of 271 responses on the two levers together and the treated mean of 248. On the first day w h e n p r was reduced t o . 1 the controls increased their responses to a mean of 398 whereas the treated animals emitted only a mean of 208 It (18) -- 2.73, p < .05]. This difference was also reflected in the n u m b e r of reinforcements earned, with the controls gaining a mean of 42.5 (this is less than that on the last day of .5 training b e c a u s e of the fivefold increase in the average n u m b e r of responses required before delivery of a food pellet) while the treated rats
TABLE 1 Post-mortem Assay of Amines Following Dorsal Bundle 6-OHDA Lesion a
Region Noradrenaline Cortex Hippocampus Hypothalamus Cerebellum Dopamine Cortex Hypothalamus Striatum
Control (N = 6)
.149 .206 .838 .176
+_ .006 -+ .019 _+ .111 +_ .084
.093 +_ .025 .299 _+ .033 5.435 _+ .686
Treated (N = 3)
.014 .035 .583 .107
+ -+ +_ __
Percentage
.010 .018 .327 .030
9.4 17.0 69.6 60.8
.180 -4- .085 .256 _+ .098 8.237 -4- 1.131
193 85.6 152
a Values are means with standard error of the mean in micrograms of amine per gram wet weight of tissue of three randomly selected treated rats and six controls. Percentage is the percentage of control concentrations remaining in lesioned tissues.
NA AND CONDITIONED REINFORCEMENT
529
800
m.__~
o
."
I I I z
~l Im~
¢
•
500
ItIt
o
o
CONTROLS
•
TREATED
0/01 I
....................
/o
7e../ . . . . . . . . . . . . . . . . . . . . . . . . .
t
200
I
j
0.5 1;
1'7
0.1 1'
2'
;
4'
5'
6'
7'
8'
DAYS
Fro. 1. Lever responses of control and DB-lesioned rats upon transfer from a random schedule with probability of reinforcement equal to .5 to a similar schedule with lower probability of reinforcement (Pr = .1). Values are means of 10 control and 10 treated rats. Dotted horizontal line indicates .5 response rate. An asterisk indicates that control and treated rats were significantly different at the 5% level. r e c e i v e d o n l y 17.5 r e i n f o r c e m e n t s [t (18) = 3.01, p < .01]. A s i m i l a r t r e n d o c c u r r e d f o r t h e n e x t 2 d a y s (Fig. 1) b u t f a i l e d to r e a c h i n d i v i d u a l significance. A f t e r a f u r t h e r p e r i o d o f t r a i n i n g , the t w o g r o u p s h a d b e c o m e i n d i s t i n g u i s h a b l e in t e r m s o f t h e n u m b e r o f r e i n f o r c e m e n t s e a r n e d [last d a y o f p , = 0.1 s c h e d u l e , c o n t r o l m e a n = 49.4 t r e a t e d m e a n = 49.5, t (18) = .012, N S ] . T h u s , b o t h g r o u p s b e g a n t h e e x t i n c t i o n p r e f e r e n c e t e s t s r e s p o n d i n g at the s a m e l e v e l . Extinction. T h e p r e f e r e n c e f o r o n e o r o t h e r o f the c o n d i t i o n e d r e i n f o r c e r s is s h o w n in Fig. 2. T h e o r d e r o f t e s t w a s c o u n t e r b a l a n c e d f o r d i f f e r e n t a n i m a l s o v e r the 3 d a y s o f e x t i n c t i o n in t h e c a s e s o f t h e " f l a s h , " " c l i c k , " a n d " b o t h " t e s t s , b u t all a n i m a l s r e c e i v e d the choice b e t w e e n c l i c k a n d flash on D a y 4 o f e x t i n c t i o n . A m a r k e d p r e f e r e n c e for t h e l e v e r
530
MASON AND ROBBINS CLICK
FLASH
BOTH
CHOICE
300
200 w co z 0 o. cO w cc
100
C
T
iiiiiiiSi
C
T
C
T
C
T
FIG. 2. Preference for the lever producing conditioned reinforcer (flash, click, or both together) for 10 control (C) rats and 10 treated (T) rats. The vertically shaded columns show the response rate on the lever producing the conditioned reinforcer compared to that producing no stimulus (dotted column). In the case of a direct choice between click and flash (choice) the vertical lines indicate responses on the crick lever, the cross-hatched columns indicate responses on the flash lever.
which gave the click compared to the lever which had no consequence was found, and this did not differ between the control and lesioned rats. A smaller, but still significant, preference for the flash was also shown and again did not differ between the control and treated groups. The two groups were also similar in their preference for the c o m p o u n d click with flash stimulus over no consequence. These conclusions were confirmed by analysis of variance (Winer, 1962). Animals emitted significantly more responses to the CR lever than on the N R lever IF(l,18) = 30.5, p < .001] showing the adequacy of the paradigm to produce and measure conditioned reinforcement. The nature of the conditioned reinforcer was also found to be significant [F(2,36) = 10.4, p < .001] indicating that the preference for the flash was much less than that for the other two CRs. H o w e v e r , neither the group effect nor the group by CR condition interaction were significant (all Fs less than 1) indicating that the lesion had no effect on any of these preferences. Similar analysis of Day 4 choice of flash versus click indicated a significant preference for the click over the flash [F(1,18) = 14.72, p < .001] but this was not affected by the lesion since neither the group effect nor the group by CR lever interaction reached significance at the 5% level.
NA AND CONDITIONED REINFORCEMENT
531
After 5 days of reinforced retraining the response rates were again high for both groups which were performing similarly one to the other [control reinforcement = 50.9, lesioned = 48.2, t (18) = .69, NS]. The animals were then placed into extinction where no conditioned reinforcers were presented and a significant dorsal bundle extinction effect was seen. The treated animals took significantly longer to reach the extinction criterion on day one than controls [control mean = 421 sec, treated mean = 886 sec, t (18) = 3.81,p < .01]. By Day 2 of extinction testing both groups had extinguished to the same level. On neither day did the rate criterion (number of lever presses emitted prior to reaching extinction criterion) differ between the control and lesioned groups [control mean = 164, lesioned mean = 175, t (18) = .19, NS].
DISCUSSION The results reported here define more precisely the nature of the dorsal bundle extinction effect and eliminate certain behavioral explanations of it. First, the results demonstrate an enhanced resistance to extinction as seen before following NA depletion (Mason & Iversen, 1975; Thornton et al., 1975; Owens et al., 1977), but only according to a latency criterion. Training of the rats under an intermittent schedule of reinforcement evidently attenuates the normal increase in r a t e of responding shown by rats with noradrenaline depletion in extinction. This is consistent with the effect of previous extinction experience on the dorsal bundle extinction effect (Mason, 1978) and with other partially reinforced schedules (Mason & Fibiger, 1978). H o w e v e r , the rats with lesions were more persistent in extinction, taking significantly longer than controls to reach a criterion of no-responding. A possible reason for the absence of an overall increase in rate of responding during extinction can be deduced from the acquisition data. When the probability of reinforcement for lever pressing was reduced on the random schedule the control rats showed a marked increase in responding which h o w e v e r was not shown by rats with noradrenaline depletion. This apparent failure of the rats with lesions to exhibit the normal increase in responding following the reduction of reinforcement probability has been noted previously. Mason and Iversen (1977d) found that transfer from CRF (continuously reinforced responding) to DRL (differential reinforcement of low rates of responding) schedules produced a lower rate of responding in rats with NA depletion than in controls. Price, Murray and Fibiger (1977) found a possibly similar effect when transfer occurred from CRF to VI. These effects of reducing the reinforcement probability to some smaller but still positive value obviously stand in contrast to those found in extinction following CRF training where Pr becomes zero. To what extent can these opposing effects be reconciled? One possibility is that nor-
532
MASON AND ROBBINS
adrenaline depletion has two effects which are directly opposed in a situation in which extinction follows training under intermittent reinforcement. For example, noradrenaline depletion might attenuate the transient rateincreasing effects of nonreinforcement which have been labeled frustrative by some theorists (Amsel, 1958), leading to a slight decline in responding shown in initial portions of the final extinction trial. However, in addition to this effect the lesion may increase responding in the latter portion of the extinction trial by an increased resistance due perhaps to a reduction in the aversiveness of nonreward (Amsel, 1958). These ideas have been investigated more thoroughly elsewhere (Mason & Iversen, 1978). Conditioned reinforcement does not seem to contribute to any of these effects within the limits of the present experimental paradigm. Although a significant conditioned reinforcement effect was found, it failed to differ between the control and lesioned animals. This result is interesting for several reasons. Stein (1964) and Hill (1970) have suggested that the effects of conditioned reinforcement might depend on the release of noradrenaline. This hypothesis is supported by evidence that psychomotor stimulants such as pipradrol can enhance the effects of conditioned reinforcement (Hill, 1970; Robbins, 1976), including a demonstration using the present paradigm (Robbins, 1975). It might be expected from this model therefore, that noradrenaline depletion would reduce the effects of conditioned reinforcement; our prediction from the stimulus sampling model was conversely that an increase might occur, which could therefore explain the increased responding in extinction. Neither effect was found. This lack of effect might suggest that pipradrol is exercising its effect on conditioned reinforcement through another, non-noradrenergic, possibly dopaminergic system (Robbins, unpublished data). Another consideration supports the conclusion that the dorsal bundle extinction effect does not depend on a lesioned-induced change in conditioned reinforcement. An increased resistance to extinction was shown by treated rats when responding had no consequence in presenting conditioned stimuli during the final extinction trial. The only possible source of conditioned reinforcement would therefore arise as response feedback. In a CRF situation kinaesthetic and proprioreceptive feedback might be expected to acquire conditioned reinforcing properties, through their frequent pairing with reward. However, in the present experiment such kinaesthetic stimuli are paired only intermittently with reinforcement, are not predictive of it, and therefore would not be expected to acquire conditioned reinforcing properties. Yet the dorsal bundle extinction effect occurs in this situation which minimizes conditioned reinforcers. Finally, the results with conditioned reinforcement have implications for attentional theories of the function of the dorsal NA bundle. The
NA AND CONDITIONED REINFORCEMENT
533
conditioned reinforcer was a compound stimulus consisting of click and light flash. B o t h o f t h e s e stimuli a c q u i r e d s o m e c o n t r o l o v e r r e s p o n d i n g in e x t i n c t i o n , a l t h o u g h c o n t r o l w a s s t r o n g e r f o r t h e click. W h e n m e a s u r e d o n s e p a r a t e o c c a s i o n s as a c h o i c e b e t w e e n t h e t w o stimuli, t h e t r e a t e d a n i m a l s s h o w e d n o a t t e n u a t i o n o f this b i a s in f a v o r o f t h e click. T h u s , t h e y d i d n o t a t t e n d to t h e t w o s i m u l i in a m o r e d i s t r i b u t e d f a s h i o n t h a n c o n t r o l s . I t m i g h t , h o w e v e r , b e t h a t if stimuli c o u l d b e f o u n d to w h i c h t h e c o n t r o l animals did n o t a t t e n d at all (unlike the crick and flash in the p r e s e n t e x p e r i m e n t ) t h e n t h e l e s i o n e d a n i m a l s c o u l d b e s h o w n to b e s a m p l i n g t h e s e . T h u s , it m i g h t b e t h a t t h e a t t e n t i o n a l m e c h a n i s m o f d o r s a l b u n d l e f u n c t i o n a c t s , n o t in an a n a l o g f a s h i o n to i n c r e a s e t h e a m o u n t o f a t t e n t i o n o r s a l i e n c e g i v e n to a s t i m u l u s o n c e it h a s b e e n s e l e c t e d , b u t in a digital f a s h i o n to d e t e r m i n e if t h a t s t i m u l u s is s e l e c t e d at all. T h i s finding, a n d o t h e r s r e p o r t e d in this p a p e r , h e n c e d o n o t r e f u t e an a t t e n t i o n a l t h e o r y o f d o r s a l N A b u n d l e f u n c t i o n ( M a s o n & I v e r s e n , 1977a; S e g a l & B l o o m , 1976); t h e y d o h o w e v e r p l a c e c o n s t r a i n t s o n h o w s u c h a t h e o r y m i g h t b e constructed.
REFERENCES Amsel, A. (1978). The role of frustrative nonreward in noncontinuous reward situations. Psychological Bulletin, 55, 102-119. Ashford, J., & Jones, B. J. (1976). The effects of intra-amygdaloid injections of 6-hydroxydopamine on avoidance responding in rats. British Journal of Pharmacology, 56, 255-261. Cuello, A. C., Hiley, R., & Iversen, L. L. (1973). Use of catechol-O-methyl-transferase for the enzyme radiochemical assay of dopamine. Journal of Neurochemistry, 21, 13371340. Hill, R. T. (1970). Facilitation of conditioned reinforcement as a mechanism for psychomotor stimulation. In E. Costa and S. Garattini (Eds.), Amphetamine and Related Compounds, pp. 781-795. New York, Raven Press. K6nig, J. F., & Klippel, R. A. (1963). The Rat Brain, A Stereotaxic Atlas. Baltimore: Williams & Wilkins Co. Mason, S. T. (1978). Parameters of the dorsal bundle extinction effect. Previous extinction experience. Pharmacology Biochemistry and Behavior, 8, 655-659. Mason, S. T. (1979). Noradrenaline: Reward or extinction. Neuroscience and Biobehavioral Reviews, in press. Mason, S. T., & Fibiger, H. C. (1978). Noradrenaline and partial reinforcement. Journal of Comparative and Physiological Psychology, 92, 1110-1118. Mason, S. T., & Fibiger, H. C. (1977). Altered exploratory behaviour after 6-OHDA lesion to the dorsal noradrenergic bundle. Nature (London), 269, 705-706. Mason, S. T., & Iversen, S. D. (1975). Learning in the absence of forebrain noradrenaline. Nature (London), 258, 422-424. Mason, S. T., & Iversen, S. D. (1977). Effects of selective forebrain noradrenaline loss on behavioural inhibition in the rat. Journal of Comparative and Physiological Psychology, 91, 165-173. (a) Mason, S. T., & Iversen, S. D. (1977). An investigation of the role of cortical and cerebellar noradrenaline in associative motor learning. Brain Research, 134, 513-527. (b)
534
MASON AND ROBBINS
Mason, S. T., & Iversen, S. D. (1977). Reward, attention and the dorsal noradrenergic bundle. Brain Research, 150, 135-148. (c) Mason, S. T., & Iversen, S. D. (1977). Behavioural basis of the dorsal bundle extinction effect. Pharmacology Biochemistry and Behavior, 7, 373-379. (d) Mason, S. T., & Iversen, S. D. (1978). The dorsal noradrenergic bundle, extinction and non-reward. Physiology & Behavior, 21, 1043-1046. Owens, S., Boarder, M. R., & Gray, J. A. (1977). The effects of depletion of forebrain noradrenaline on runway behaviour of rats. Experimental Brain Research, 28, R22R23. Price, M. T. C., Murray, G. N., & Fibiger, H. C. (1977). Schedule dependent changes in operant responding after lesions of the dorsal tegmental noradrenergic projection. Pharmacology Biochemistry and Behavior, 6, 11-15. Robbins, T. W. (1975). The potentiation of conditioned reinforcement by psychomotor stimulant drugs. A test of Hill's hypothesis. Psychopharmacology, 45, 103-114. Robbins, T. W. (1976). Relationship between reward-enhancing and stereotypical effects of psychomotor stimulant drugs. Nature (London), 264, 57-59. Segal, M., & Bloom, F. E. (1976). The action of norepinephrine in rat hippocampus IV. The effects of locus coeruleus stimulation on evoked hippocampal unit activity. Brain Research, 107, 513-525. Stein, L. (1964). Amphetamine and neural reward mechanisms. In H. Steinberg, A. V. S. de Reuck and J. Knight (Eds.), Animal Behaviour and Drug Action, pp. 98-118. London: Churchill. Sutherland, N. S., & Holgate, V. (1966). Two-cue discrimination learning in rats. Journal of Comparative and Physiological Psychology, 61, 198-207. Thornton, E. W., Goudie, A. J., & Bithell, V. (1975). The effects of neonatal 6-hydroxydopamine induced sympathectomy on response inhibition in extinction. Life Sciences 17, 363-368. Tremmel, F., Morris, M. D., & Gebhart, G. (1977). The effect of forebrain norepinephrine loss on two measures of response suppression. Brain Research, 126, 185-188. Uretsky, N. J., & Iversen, L. L. (1969). Effects of 6-hydroxydopamine on noradrenalinecontaining neurons in the rat brain. Nature (London), 221, 557-559. Wike, E. L. (1966). Secondary Reinforcement. New York: Harper & Row. Winer, B. J. (1962). Statistical Principles in Experimental Design. New York: McGrawHill.
