Neuroscience 265 (2014) 291–301

NORADRENERGIC MODULATION OF VICARIOUS TRIAL-AND-ERROR BEHAVIOR DURING A SPATIAL DECISION-MAKING TASK IN RATS S. AMEMIYA, a,b T. NOJI, a N. KUBOTA, a T. NISHIJIMA a AND I. KITA a*

INTRODUCTION When facing a decision-making situation, we deliberately explore possible alternatives to determine the optimal option before taking action. Rats also show exploratory behavior at choice points. The exploratory behavior is called vicarious trial-and-error behavior (VTE). This VTE is typically expressed as a pause and looking back and forth from one option to others before making a decision. VTE is thought to relate to deliberative decision-making (Tolman, 1938, 1939, 1948; Johnson et al., 2007). Several studies reported that the frequency of VTE increased in choice situations with high task demand, such as difficult choices, the early phases of learning, after changing task contingency, and during reward-delay discounting (Muenzinger, 1938; Tolman, 1938, 1948; Griesbach et al., 1998; Johnson and Redish, 2007; Blumenthal et al., 2011; Papale et al., 2012) and that VTE highly correlated with the level of performance and learning efficiency (Muenzinger, 1938; Hu et al., 2006). Furthermore, electrophysiological studies have reported that VTE is associated with neuronal activity in the hippocampus and striatum that can be related to representation of future paths and expected reward, respectively (Johnson and Redish, 2007; van der Meer, 2009), suggesting that VTE is involved in expectation and evaluation of outcomes. Thus, it is possible that VTE is associated with a deliberative process when a valuable choice is unknown. It has been suggested that noradrenaline neurons in the locus coeruleus (i.e., the LC-NA system) are important in maintaining arousal and a state of vigilance (Foote et al., 1980; Aston-Jones and Bloom, 1981a,b), and more recent studies have further proposed that the LC-NA system modulates attention, behavioral flexibility, and exploration, which contribute to choosing adaptive behavior (Aston-Jones et al., 1994; Aston-Jones et al., 1999; Usher, 1999; Aston-Jones and Cohen, 2005; Yu and Dayan, 2005). For example, the discharge rate of LC-NA neurons and the release of NA in the prefrontal cortex increased in response to changes of reward contingency (Aston-Jones et al., 1997; Dalley et al., 2001), and the state of high LC-NA activity facilitated exploratory responses, resulting in gathering new information and subsequently determining the correct choice in a cognitive task (Aston-Jones et al., 1994; Aston-Jones et al., 1999; Usher, 1999; Aston-Jones and Cohen, 2005; Yu and Dayan, 2005). Furthermore, pharmacological enhancement of NA release promoted

a

Department of Human Health Sciences, Tokyo Metropolitan University, Minamiohsawa, Hachioji, Tokyo 192-0397, Japan b Research Fellow of the Japan Society for the Promotion of Science, 5-3-1 Kojimachi, Chiyoda-ku, Tokyo 102-0083, Japan

Abstract—Deliberation between possible options before making a decision is crucial to responding with an optimal choice. However, the neural mechanisms regulating this deliberative decision-making process are still unclear. Recent studies have proposed that the locus coeruleus-noradrenaline (LC-NA) system plays a role in attention, behavioral flexibility, and exploration, which contribute to the search for an optimal choice under uncertain situations. In the present study, we examined whether the LC-NA system relates to the deliberative process in a T-maze spatial decision-making task in rats. To quantify deliberation in rats, we recorded vicarious trial-and-error behavior (VTE), which is considered to reflect a deliberative process exploring optimal choices. In experiment 1, we manipulated the difficulty of choice by varying the amount of reward pellets between the two maze arms (0 vs. 4, 1 vs. 3, 2 vs. 2). A difficulty-dependent increase in VTE was accompanied by a reduction of choice bias toward the high reward arm and an increase in time required to select one of the two arms in the more difficult manipulation. In addition, the increase of c-Fos-positive NA neurons in the LC depended on the task difficulty and the amount of c-Fos expression in LC-NA neurons positively correlated with the occurrence of VTE. In experiment 2, we inhibited LC-NA activity by injection of clonidine, an agonist of the alpha2 autoreceptor, during a decision-making task (1 vs. 3). The clonidine injection suppressed occurrence of VTE in the early phase of the task and subsequently impaired a valuable choice later in the task. These results suggest that the LC-NA system regulates the deliberative process during decision-making. Ó 2014 IBRO. Published by Elsevier Ltd. All rights reserved.

Key words: noradrenaline, locus coeruleus, vicarious trialand-error, deliberative decision-making, c-Fos immunochemistry, clonidine.

*Corresponding author. Address: Department of Human Health Sciences, Tokyo Metropolitan University, 1-1 Minamiohsawa, Hachioji, Tokyo 192-0397, Japan. Tel: +81-42-677-1111; fax: +81-42-677-2961. E-mail address: [email protected] (I. Kita). Abbreviations: DAB, diaminobenzidine; HR, high reward arm; LC, locus coeruleus; NA, noradrenaline; PBS, Phosphate-buffered saline; TH, tyrosine hydroxylase; VTE, vicarious trial-and-error behavior. http://dx.doi.org/10.1016/j.neuroscience.2014.01.031 0306-4522/Ó 2014 IBRO. Published by Elsevier Ltd. All rights reserved. 291

292

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

acquisition of a new contingency after a contingency switch (Devauges and Sara, 1990; Lapiz and Morilak, 2006; Seu et al., 2008), whereas pharmacological depletion of NA impaired the same task (Tait et al., 2007; McGaughy et al., 2008). Taken together, we hypothesize that the LC-NA system is crucial in the search for an appropriate choice. In the present study, we investigated whether the LC-NA system is involved in the deliberative process to search for the optimal choice in decision-making, using VTE as a behavioral expression of this deliberative process. In the first experiment, we examined the relationship between the activity of LC-NA neurons and the appearance of VTE during a T-maze spatial decision-making task with varying levels of difficulty. In the second experiment, we examined effects of suppression of NA transmission using clonidine, an agonist of the alpha2 autoreceptor (van Veldhuizen et al., 1993; Berridge and Abercrombie, 1999; Pudovkina et al., 2001), on the same T-maze spatial decision-making task.

EXPERIMENTAL PROCEDURES Animals Thirty nine male Wistar rats (Sankyo Labo Service Corporation Inc., Tokyo, Japan), approximately 10 weeks old at the start of the experiment, were used. The rats were housed in groups of 3 at a room temperature of 24 °C and a 12-h light:12-h dark cycle (lights on at 08:00 am) with ad libitum access to water. The rats were food restricted to maintain at 85% of their free-feeding body weight throughout the experiment. Animals were handled for several days before starting pretest training to habituate them to the experimenter. The behavioral test was conducted within the first six hours of the dark phase of the light cycle. All experimental procedures were carried out in accordance with the European Communities Council Directive of November 24, 1986 (86/609/EEC) and were approved by the Animal Experimentation Ethics Committee of the Tokyo Metropolitan University. Every effort was made to minimize animal suffering and the number of animals used. Apparatus The apparatus was a black-painted wooden continuous Tmaze, 90  90  1 cm, with 5 acrylic doors 30 cm in height from the surface (Fig. 1A). The maze consisted of a T-shaped runway with return runways, and reward sites on each return runway. The maze was placed in a dim test room at a height of 75 cm from the floor, surrounded by a black curtain. All experiments were performed with the experimenter standing behind the curtain. Rats performed maze running without any interference throughout the experiment. Pretest training During the first few days of pretest training, rats were habituated to the T-maze. For habituation, all doors

Fig. 1. The spatial decision-making task in a continuous T-maze. (A) Diagram of T-maze apparatus. The maze consists of a T-shaped runway, return runways, and five doors. Three areas were defined for behavioral analysis: (1) the start area, (2) the choice area, and (3) the transit area. (B) Examples of head position in choice area during VTE trial or non-VTE trial. (i) and (ii) VTE in choice area. (iii) VTE before reaching the choice point. (iv) No VTE. Black lines show the position of front doors.

were opened, with food pellets (45 mg/pellet, F0021-J, Bio-serv, Frenchtown, NJ, USA) placed around the T-maze, and rats freely explored and ate during a 30-min period. In the days following habituation, rats were trained to be proficient at running and getting food pellets in the T-maze. Rats were placed at the start area and then began to run the maze when both the center door and the front door at the T-shaped point were opened. As the rats reached a reward site and acquired a pellet, the back door on that side was opened and the rats returned to the start area spontaneously and obtained a pellet in the start area. The running was alternately repeated (i.e., opening the left runway was followed by opening of the right runway) and continued for 30 min or until the rat had reached the criterion of 40 runs in 30 min per day. After rats achieved this criterion on two consecutive days, they were given the experimental T-maze spatial choice task. Experiment 1 Maze test. In the maze test, the rats were assigned randomly to three groups (n = 9 each). Each group was only tested on one of the three reward conditions (between-subject design). On the first day, rats ran the maze when both arms had one food pellet (1vs1). On

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

the second day, we varied the task difficulty by manipulating the difference in amount of reward pellets between the two. The conditions were a) no pellet vs. 4 pellets (0vs4), b) 1 pellet vs. 3 pellets (1vs3) and c) 2 pellets vs. 2 pellets (2vs2). The arm with the larger number of pellets (i.e., 3 and 4 pellets) was defined as the high reward arm (HR). The side of the HR was counterbalanced (right or left) among subjects within each condition and was constant throughout the maze test. In the 2vs2 condition, the left arm was defined as the HR for expressing and processing the data. To assess improvement of task performance, choice toward HR was deemed to be a correct choice. Each test session consisted of 40 trials. The trials were begun by simultaneously opening the center door and the front doors to allow the rat to choose either arm. After reaching the reward site, the back door of the side where the rat ate opened and then the rat went back to the start area. After the second day’s test, rats were returned to their home cages in a familiar, dark room. Behavioral data. The behavioral data were scored from videos recorded at 30 Hz by a video camera mounted above the center of the maze. The percent of choice of the HR (HR choice), the absolute value of choice bias (Absolute bias), the number of pellets acquired during the test, time spent in the choice area (Choice time) and in the transit area (Transit time), and the number of VTE and of VTE trials were recorded. Absolute bias was calculated by |50 – HR choice| to evaluate inter-individual side preferences. Time was calculated as the number of video frames. VTE was defined as in previous studies (Hu et al., 1997; Johnson et al., 2007): (1) head movements from one side to the other side across the midline of the rat’s body at the T-shaped choice point; (2) lateral head movements away from and returning to the rat’s direction of travel before the choice point; (3) pausing within the choice area (Fig. 1B). Repeated head movements crossing the midline at the choice point, lateral head movements before the choice point, and/or multiple pauses in a trial were counted as separate VTE. The VTE was classified and counted from videos recorded by the same experimenter blinded to the reward condition. Each trial where VTE was observed was counted toward the number of VTE trials, regardless of the number of VTE within the trial. Immunohistochemistry. To quantify neuronal activity during the maze task, the rats were deeply anesthetized 90 min after the end of the maze test (Vann et al., 2000) with sodium pentobarbital (50 mg/kg, i.p.) and perfused transcardially with heparin solution (1000 U/l, 0.9% saline), followed by ice-cooled 4% paraformaldehyde, 0.1% glutaraldehyde, and 0.2% picric acid in 0.1 M phosphate-buffered saline (PBS, pH 7.4). The brains were removed and post-fixed in the same fixative without glutaraldehyde for 24 h at 4 °C. The brains were then cryoprotected in a phosphate-buffered 30% sucrose solution with 0.1% sodium azide for 24–48 h. The brains were then frozen and cut in the coronal

293

plane (6 series of 40-lm-thick sections) on a microtome and collected in 0.1 M PBS with 0.1% sodium azide. Immunohistochemical visualization of c-Fos and NA neurons was carried out on free-floating sections using antibody and avidin–biotin–peroxidase methods as previously described (Amemiya et al., 2010; Kubota et al., 2012). The immunohistochemical staining of the different groups was performed together to prevent staining variations across staining sessions. After blocking of endogenous peroxidase and pre-incubation in 10% normal horse serum, the sections were incubated for 16 h at room temperature in primary rabbit polyclonal anti-Fos antiserum (sc-52, Santa Cruz Biotechnology Inc., Santa Cruz, CA, USA) diluted 1:600 in 0.1 M PBS with 0.1% Triton X-100 (PBS-TX). After the sections were rinsed three times for 10 min in PBSTX, they were further incubated with secondary biotinylated donkey anti-rabbit IgG (AP182B, Chemicon, Temecula, CA, 1:800) for 120 min at room temperature, rinsed three times for 10 min in PBS-TX, and finally treated with avidin–biotin–peroxidase complex (Vectastain ABC peroxidase kit, Vector Lab Inc., Burlingame, CA, USA), diluted 1:400, for 2–3 h. The sections were reacted for peroxidase activity in a solution consisting of nickel ammonium sulfate, 0.02% 3,30 -diaminobenzidine (DAB) in 0.1 M Tris–HCl buffer (pH 7.6), and 0.01% H2O2 for 20 min. Immunoreactivity for c-Fos was localized to the cell nuclei and appeared as a dark gray-black stain. For dual immunostaining for NA neurons, sections containing the LC were sequentially incubated in rabbit anti-tyrosine hydroxylase (TH) polyclonal antibody (AB152, Chemicon, Temecula, CA, USA) diluted 1:2000, and the sections were further incubated with secondary biotinylated donkey anti-rabbit IgG (AP182B, Chemicon, Temecula, CA, USA), diluted 1:800. Avidin–biotin–peroxidase complex was visualized with DAB in 0.1 M Tris–HCl buffer (pH 7.6) without nickel ammonium sulfate. TH immunoreactivity was localized to the cell cytoplasm and visible as a brown stain. Sections were then washed in 0.01 M PBS, mounted on gelatin-coated glass slides, air-dried, dehydrated in graded alcohols, cleared in xylene, and cover-slipped with Permount mounting medium (Fisher Scientific, Hanover Park, IL, USA). Cell counts and quantification. Immunoreactive cells were observed with an Olympus BH-2 microscope equipped with a camera (ELMO, CN42H). A quantitative analysis was performed on two sections containing the LC (sections between AP 9.6 and 10.1 mm from Bregma and 200 lm apart to avoid possible double counting), which were standardized according to the atlas of rat brain (Paxinos and Watson, 2005). The total numbers of TH-Fos double-labeled cells were counted within the areas of the LC. These double-labeled cells were identified by their shape and color (overlapping between brown-colored noradrenergic cell body and back-colored c-Fos-positive nuclei) (Fig. 4B). The counting was manually conducted while sections were viewed live under 400 magnification on a monitor connecting the light microscope. The sections that

294

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

included the LC were horizontally divided 80 lm each on the monitor. The observing field was horizontally moved along each division and the double-labeled cells were counted inside the field. In order to calculate the density of the TH-Fos double-labeled cells in the LC area, we also measured the size of the LC in the same sections in which the double-labeled cells were counted. The size measurement was manually executed by surrounding the whole area of the brown-colored LC on 100 magnification images and calculating it using Image J software (http://rsbweb.nih.gov/ij/). The measurer was blinded to the reward condition. The double-labeled cell data were expressed by the density of TH-Fos double-labeled cells to LC area per section.

condition as a within-subject factor  blocks as a repeated measure] in experiment 2 followed by multiple T-tests with Bonferroni correction for post hoc analyses. For the numbers of immunoreactive cells, a one-way ANOVA was performed to evaluate the effects of reward conditions on neuronal activation with subsequent multiple T-tests with Bonferroni correction for post hoc analyses. To determine the relationships between behavioral data and TH-Fos double-labeled cells, Pearson’s correlations were calculated. All analyses adopted a probability value of 0.05 as the level of significance. All data are presented as the mean ± S.E.

Experiment 2

Experiment 1

Maze test. A within-subject design was used in the second experiment (n = 12); all rats were tested on all drug conditions (3 drug conditions described below). In the maze test, the reward condition was 1 pellet vs. 3 pellets and the arm with 3 pellets was defined as the HR. All procedures (test sessions and recording of data during the test) were the same as experiment 1 except for injection of saline or the drug 30 min before the maze test. After completion of the test session, rats received a rest period of at least two days and then received training of the alternative choice again until reaching the criterion (40 runs in 30 min) in 2 consecutive days to exclude likely drug effects and memory traces of the previous reward condition. On the day following the second day after reaching the pretest training criterion, rats were tested with the next drug condition and the reward conditions were reversed from those of the previous test (i.e., in the first test the right arm had three pellets, and then in the second test the left arm had three pellets). Analysis of the behavioral data was the same as in experiment 1.

Behavioral data in pretest training. To make sure there were no differences among the three groups prior to the maze test, we analyzed the behavioral data from the first test day (1vs1). There were no differences in any measure of task performance in pretest training [F(2, 24) = 0.23, p = 0.76 for HR choice; F(2, 24) = 0.16, p = 0.85 for choice time; F(2, 24) = 0.61, p = 0.55 for transit time; F(2, 24) = 0.88, p = 0.43 for number of VTE; F(2, 24) = 0.469, p = 0.63 for number of VTE trials].

Pharmacological manipulation. Clonidine hydrochloride (Sigma), an alpha2 autoreceptor agonist, was dissolved in saline and the dosages used were 20 or 50 lg/kg. Saline vehicle was used as a control. On the test days, one of two doses of drug or saline vehicle was intraperitoneally injected 30 min before the test. The doses and waiting time after the injection were chosen based on previous studies (Sara et al., 1995; Jentsch and Anzivino, 2004; Lapiz and Morilak, 2006). Each rat received all three drug conditions; the order was counterbalanced among the rats. Statistics The difference of HR choice from chance level was tested using a chi-square test. To assess temporal changes in the behavioral data, the maze test was divided into 4 blocks of 10 consecutive trials. The behavioral data were compared using a two-way repeated-measures ANOVA [reward condition as a between-subject factor  blocks as a repeated measure] in experiment 1, and a two-way repeated-measures ANOVA [drug

RESULTS

HR choice. Both the 0vs4 and the 1vs3 conditions showed bias toward the HR arm from chance level (chisquare test, p < 0.01) and the 2vs2 did not (chi-square test, p = 0.10). HR choice increased as the difference of reward size between the two options increased. A two-way ANOVA revealed significant effects on HR choice of the reward condition [F(2, 24) = 31.09, p < 0.01], the block [F(3, 72) = 17.58, p < 0.01], and the condition  block interaction [F(6, 72) = 6.25, p < 0.01] (Fig. 2A). Post hoc testing revealed that the 2vs2 showed less bias toward HR than the 0vs4 and the 1vs3, and the 1vs3 showed less bias toward HR than the 0vs4 throughout the session (p < 0.05). The group differences were present within each block. The 2vs2 showed less bias toward HR in all four blocks compared to the 0vs4 and in blocks 2–4 compared to the 1vs3 (p < 0.05). Furthermore, the 1vs3 showed less bias toward HR in blocks 2–4 compared to the 0vs4 (p < 0.05). There was a sequential change within each reward condition. Compared to block 1, the 0vs4 showed increased HR choice in blocks 2–4 (p < 0.05), and the 1vs3 showed increased HR choice in blocks 3 and 4 (p < 0.05). Absolute bias showed the similar trends with HR choice. A two-way ANOVA revealed significant effects on absolute bias of the reward condition [F(2, 24) = 22.51, p < 0.01], the block [F(3, 72) = 22.54, p < 0.01], and the condition  block interaction [F(6, 72) = 4.46, p < 0.01]. The 2vs2 showed less absolute bias than the 0vs4 and the 1vs3, and the 1vs3 showed less absolute bias than the 0vs4 throughout the session (p < 0.05). The group differences were present within each block. The 2vs2 and the 1vs3 showed less absolute bias in blocks 2–4 compared to the 0vs4 (p < 0.05). Furthermore, the 2vs2

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

295

Fig. 2. Behavioral index in each reward condition. (A) HR choice (±S.E.) over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/ block) (right panel). (B) Choice time (±S.E.) over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). (C) Transit time (±S.E.) over 40 trials (left panel), and sequential analysis of HR choice over 4 blocks (10 trials/block) (right panel). ⁄p < 0.05, vs. 0vs4; # p < 0.05, vs. 1vs3;  p < 0.05, vs. first block.

showed less absolute bias in block 4 compared to the 1vs3 (p < 0.05). Number of pellets. A two-way ANOVA revealed significant effects on the number of pellets of the reward condition [F(2, 24) = 81.45, p < 0.01], the block [F(3, 72) = 37.70, p < 0.01], and the condition  block interaction [F(6, 72) = 17.58, p < 0.01]. The 2vs2 earned fewer pellets than the 0vs4 and the 1vs3 (p < 0.05), and the 1vs3 earned fewer pellets than the 0vs4 throughout the session (p < 0.05). Within each block, the 2vs2 and the 1vs3 earned fewer pellets in all blocks compared to the 0vs4 (p < 0.05), and the 2vs2 also earned fewer pellets in blocks 3 and 4 compared to the 1vs3 (p < 0.05). Within each reward condition, the 0vs4 showed an increase in the number of pellets in blocks 2–4 (p < 0.05) compared to block 1, and the 1vs3 showed an increase in block 4 compared to block 1 (p < 0.05). Choice time and transit time. A two-way ANOVA revealed significant effects on choice time of the reward condition [F(2, 24) = 4.00, p < 0.05] and the block [F(3, 72) = 6.23, p < 0.01], but not the condition  block interaction [F(6, 72) = 1.57, p = 0.16] (Fig. 2B). Choice time in the 2vs2 was prolonged compared to the 0vs4 throughout the session (p < 0.05). No effects were found on transit time of the reward condition [F(2, 24) = 0.023, p = 0.97], the block [F(3, 72) = 1.31, p = 0.27], or the condition  block interaction [F(6, 72) = 0.78, p = 0.58] (Fig. 2C). Number of VTE and VTE trials. A two-way ANOVA revealed significant effects on the number of VTE of the reward condition [F(2,24) = 13.05, p < 0.01] and the block [F(3, 72) = 16.42, p < 0.01], although there was no significant condition  block interaction [F(6, 72) = 0.92, p = 0.48] (Fig. 3A). The 2vs2 showed more VTE than the 0vs4 and the 1vs3 through the session (p < 0.05). A two-way ANOVA revealed significant effects on the number of VTE trials of the reward condition [F(2, 24) = 5.02, p < 0.05] and the block [F(3, 72) = 8.11, p < 0.01], as well as the condition  block interaction [F(6, 72) = 2.34, p < 0.05] (Fig. 3B). The 2vs2 showed more VTE trials than the 0vs4 throughout the session

(p < 0.05). Within each block, the 2vs2 expressed more VTE trials in blocks 2 and 4 compared to the 0vs4 (p < 0.05). Within each reward condition, the 0vs4 showed a decrease in VTE trials in blocks 2 and 4 compared to block 1 (p < 0.05), and the 1vs3 showed a decrease in block 4 compared to block 1 (p < 0.05).

Immunohistochemical analysis We performed double-labeling immunohistochemistry for c-Fos and TH in the LC after behavioral tests under the different reward conditions (Fig. 4A, B). One sample in the 0vs4 group was excluded from this analysis because of a technical problem in the preparation of brain slices. A one-way ANOVA revealed that the reward condition had a main effect on the density of THFos double-labeled cells in the LC [F(2, 23) = 8.36, p < 0.01]; the density of TH-Fos double-labeled cells in the 2vs2 was higher than in the 0vs4 (p < 0.05) (Fig. 4C).

Correlative analyses. To further analyze the relationships between neural and behavioral data, we performed correlative analysis between each pair of data. Results of these analyses are shown in Table 1. Additionally, to demonstrate more strictly the link between the LC-NA system and decision uncertainty, we examined correlative analysis using absolute bias instead of HR choice (Table 1). Furthermore, we performed correlative analysis of absolute bias in each reward condition so as not to be confounded by the likely effects of difference in ratio of reward magnitude among the conditions. The analysis showed correlation coefficients between absolute bias and the density of TH-Fos double-labeled cells (0vs4: r = 0.37, p = 0.37, 1vs3: r = 0.00, p = 0.99, 2vs2: r = 0.15, p = 70.00), the number of VTE (0vs4: r = 0.23, p = 0.56, 1vs3: r = 0.40, p = 0.29, 2vs2: r = 0.05, p = 0.90), the number of VTE trials (0vs4: r = 0.39, p = 0.31, 1vs3: r = 0.37, p = 0.32, 2vs2: r = 0.18, p = 0.64), choice time (0vs4: r = 0.14, p = 0.71, 1vs3: r = 0.18, p = 0.65, 2vs2: r = 0.40, p = 0.28), although sample size in each condition might not be sufficient for reliable statistical results. To supplement the results of correlative analysis from small sample size, scatter plots are also showed (Fig. 5).

296

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

Fig. 3. The number of vicarious trial-and-error behaviors (VTE) and of VTE trials in each reward condition. (A) Total number (±S.E.) of VTE over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). (B) Total number (±S.E.) of VTE trials over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). ⁄p < 0.05, vs. 0vs4; #p < 0.05, vs. 1vs3;  p < 0.05, vs. first block.

[F(2, 20) = 4.03, p < 0.05], the block [F(3, 30) = 12.09, p < 0.01], and the condition  block interaction [F(6, 60) = 2.48, p < 0.05] (Fig. 6A). The 50-lg dose showed lower HR choice than the control throughout the session (p < 0.05). Within each block, the 50-lg dose showed lower HR choice in block 4 compared to the control (p < 0.05). Within each drug condition, the control showed increased HR choice in blocks 3 and 4 compared to block 1 (p < 0.05). The 20-lg dose also increased HR choice in block 4 compared to block 1 (p < 0.05), whereas the 50-lg dose did not bias HR choice.

Fig. 4. Immunostaining of c-Fos and noradrenergic neurons in locus coeruleus (LC). (A) LC was located along the fourth ventricle (4 V). Black rectangle shows the area magnified in (B). (B) c-Fos and tyrosine hydroxylase (TH) double-labeled cells, indicated by arrowheads. (C) Density (±S.E.) of c-Fos and TH double-labeled cells in the LC in each reward condition. Scale bar = 200 lm (A) and 20 lm (B). ⁄p < 0.05, vs. 0vs4.

Choice time and transit time. A two-way ANOVA revealed a significant effect on choice time of the drug condition [F(2, 20) = 17.12, p < 0.01], but not the block [F(3, 30) = 0.97, p = 0.42] or the condition  block interaction [F(6, 60) = 1.13, p = 0.36] (Fig. 6B). The 20lg and the 50-lg doses prolonged choice time compared to the control throughout the session (p < 0.01). There were no effects on transit time of drug condition [F(2, 20) = 3.111, p = 0.07], the block [F(3, 30) = 2.30, p = 0.10], or the condition  block interaction [F(6, 60) = 0.64, p = 0.70] (Fig. 6C).

Experiment 2 Data from eleven rats were statistically analyzed. The data of one rat were discarded because the rat did not complete the maze test on a drug injection day. HR choice. A two-way ANOVA revealed significant effects on HR choice of the drug condition

Number of VTE and VTE trials. A two-way ANOVA revealed significant effects on the number of VTE of the drug condition [F(2, 20) = 5.13, p < 0.05], the block [F(3, 30) = 3.36, p < 0.05], and the condition  block interaction [F(6, 60) = 2.83, p < 0.05] (Fig. 7A). The 50-lg dose showed fewer VTE than the control throughout the session (p < 0.05). Within each

Table 1. Correlations between neuronal and behavioral measures

Density of TH-Fos double-labeled cells Number of VTE Number of VTE trials Choice time Absolute bias

Number of TH-Fos double-labeled cells

Number of VTE

Number of VTE trials

Choice time

Absolute bias

1.00

0.65**

0.44**

0.54**

0.50*

– – – –

1.00 – – –

0.87** 1.00 – –

0.69** 0.66** 1.00 –

0.64** 0.53** –0.36# 1.00

Matrix of Pearson’s correlation coefficient between the number of TH-Fos double-labeled cells, the number of VTE, the number of VTE trials, choice time, and absolute bias. ** p < 0.01. * p < 0.05. # p = 0.06.

297

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

showed fewer VTE trials than the control throughout the session (p < 0.05). Within each block, the 20-lg dose showed fewer VTE trials in block 1 (p < 0.05), and the 50-lg dose showed fewer in blocks 1 and 3 (p < 0.01) compared to the control. Within each drug condition, only the control group showed a decrease in the number of VTE trials in blocks 2–4 compared to block 1 (p < 0.05).

DISCUSSION

Fig. 5. Scatter plots between absolute bias and the density of THFos double-labeled cells (A), the number of VTE (B), the number of VTE trials (C), or choice time (D).

block, the 50-lg dose showed fewer VTE in blocks 1 and 3 compared to the control (p < 0.05). Within each drug condition, only the control showed a decrease in the number of VTE in blocks 3 and 4 compared to in block 1 (p < 0.05). A two-way ANOVA revealed significant effects on the number of VTE trials of the drug condition [F(2, 20) = 7.85, p < 0.01], the block [F(3, 30) = 3.84, p < 0.05], and the condition  block interaction [F(6, 60) = 2.90, p < 0.05] (Fig. 7B). The 50-lg dose

In the present study, we examined the involvement of the LC-NA system in decision-making, using VTE as a behavioral index of the deliberative process. Experiment 1 showed that VTE increased depending on the difficulty of discrimination of reward size in the spatial decisionmaking task. In addition, we found the density of c-Fospositive NA neurons in the LC depended on the task difficulty; additionally, c-Fos expression in LC-NA neurons and the occurrence of VTE were positively correlated. Experiment 2 showed that the injection of clonidine, which inhibited LC-NA activity and NA release, suppressed the increase of VTE in the early phase of the task observed in the control group. Clonidine also inhibited improvement of task performance later in the task. In addition, the injection of clonidine prolonged choice time without affecting running speed, suggesting clonidine causes a delay of commitment to a decision rather an overall slowing or motor effect. These results suggest that LC-NA system is involved in a deliberative process during decisionmaking.

Fig. 6. Behavioral index in each drug condition. (A) HR choice (±S.E.) over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/ block) (right panel). (B) Choice time (±S.E.) over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). (C) Transit time (±S.E.) over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). ⁄p < 0.05, vs. control;  p < 0.05, vs. first block.

Fig. 7. The number of vicarious trial-and-error behavior (VTE) and of VTE trials in each drug condition. (A) Total number (±S.E.) of VTE over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). (B) Total number (±S.E.) of VTE trials over 40 trials (left panel) and sequential analysis over 4 blocks (10 trials/block) (right panel). ⁄p < 0.05, vs. control;  p < 0.05, vs. first block.

298

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

According with previous studies (Tolman, 1939, 1948), our results showed a difficulty-dependent increase of VTE. In our experiment, we set the difficulty of choice by manipulating size of reward of options, suggesting VTE relates to evaluation of relative value between options. In the previous studies, the difficulty of choice was set by manipulating the brightness of visual stimuli in visual discrimination tasks (Tolman, 1939, 1948), suggesting VTE relates to visual processing. In addition, other previous studies have reported that VTE occurred in various choice situations: in discrimination of other sensory stimuli, such as odor and sound (Hu et al., 1997; Griesbach et al., 1998; Johnson and Redish, 2007), in spatial discrimination (Bett et al., 2012), and in reward-delay discounting (Papale et al., 2012). These studies suggest that VTE is nonspecifically induced in choice situations regardless of how the options are differentiated. Furthermore, previous studies and our study showed that VTE increased when rats face a novel choice situation and a contingency change, and decreased when rats acquire the choice situation (Muenzinger, 1938; Tolman, 1948; Griesbach et al., 1998; Blumenthal et al., 2011; Bett et al., 2012), indicating that VTE is especially needed when there is a deficiency in information and comprehension of the choice situation. In favor of the notion, our results in experiment 2 showed that the injection of clonidine dose-dependently decreased VTE, especially in the first block, and this decrease of VTE may have caused the corresponding dose-dependent impairment of task performance (slower bias toward HR in 20-lg dose than in control, and no bias toward HR in 50-lg dose), suggesting VTE has beneficial effects on information gathering and comprehension of the situation. Moreover, it has been reported that VTE requires the hippocampus (Hu and Amsel, 1995; Hu et al., 2006), which is a critical brain region for learning experiences and using learned knowledge for future choices (Eichenbaum, 2004; Johnson and Redish, 2007; Gupta et al., 2012). Taken together, the evidence suggests VTE is associated with processes to collect and utilize information required for determination of the optimal choice in a given situation. Previous studies have proposed that the LC-NA system is crucial in attentional modulation and exploration under novel and uncertain choice situations (Aston-Jones and Cohen, 2005; Yu and Dayan, 2005; Sara and Bouret, 2012). A change of reward contingency activated LC-NA neurons and increased release of NA in the prefrontal cortex (Aston-Jones et al., 1997; Dalley et al., 2001) and pharmacological enhancement of NA release facilitated the acquisition of a new contingency (Devauges and Sara, 1990; Lapiz and Morilak, 2006; Seu et al., 2008), suggesting that the LC-NA system is activated in response to contextual uncertainty and facilitates resolving the uncertain context to take a suitable choice. This notion is supported by our results. Experiment 1 showed that the LC-NA system was activated in accordance with the level of difficulty of choice based on uncertainty of the better option. Experiment 2 showed that inhibition of

LC-NA activity impaired the bias toward a valuable choice in the same task as experiment 1, indicating the necessity of the LC-NA system in determining the better option under uncertainty. These results suggest that the LC-NA system regulates decision-making processes in proportion to the uncertainty of the current context. In previous studies, the activity and function of the LC-NA system has been well matched with VTE. Activity in the LC-NA system increases in new choice situations and decreases after acquisition of the situation (AstonJones et al., 1997; Dalley et al., 2001). Increased activity in the LC-NA system led to suitable choices (Devauges and Sara, 1990; Lapiz and Morilak, 2006; Seu et al., 2008). Similar to the LC-NA activity, the number of VTE increased in a new choice situation and decreased after acquisition of the situation (Muenzinger, 1938; Tolman, 1938; Tolman, 1948; Griesbach et al., 1998; Johnson and Redish, 2007; Blumenthal et al., 2011). The occurrence of VTE facilitated a suitable choice (Muenzinger, 1938; Hu et al., 2006). Combining the findings from the previous studies, it is suggested that the LC-NA system relates to the occurrence of VTE. However, there has been no evidence directly showing a relationship between VTE and LC-NA activity until now. In our current experiments, we found that the activity of LC-NA neurons positively correlated with the occurrence of VTE. Furthermore, the inhibition of LC-NA activity suppressed the occurrence of VTE in the early phase of the task. Our results suggest that the LC-NA system modulates VTE when an evaluation of the environment is needed in a search for a suitable choice. A series of studies have suggested that VTE is implicated in a vicarious search of options using cognitive functions, such as formation and retrieval of memory, attentional control, exploration, and reward expectation (Tolman, 1939, 1948; Johnson et al., 2007, 2012), thus our results may imply that these cognitive functions are modulated by the LC-NA system. There are many literatures reporting NA functions during action selection involving rewarding outcomes and it is suggested that NA facilitates formation and retrieval of memory, attentional control, exploration, and reward expectation (Sara and Devauges, 1989; Devauges and Sara, 1990, 1991; Haapalinna et al., 1998; Bouret and Sara, 2004). Furthermore, VTE is attributed to the hippocampus, which plays a role in memory, attention, exploration, and expectation (Hu and Amsel, 1995; Hu et al., 2006; Johnson and Redish, 2007; Bett et al., 2012). It is reported that hippocampal theta and gamma oscillations, which are cognitive process-related oscillatory activities observed during VTE (Johnson and Redish, 2007), are modulated by LC activity (Berridge and Foote, 1991; Brown, 2005; Lemon et al., 2009; Walling et al., 2011). These studies may indicate that the LC-NA system relates to VTE through regulating cognitive functions during the VTE process. In spite of decreasing occurrence of VTE (i.e., less deliberation), the injection of clonidine, at either dosage, caused the choice time to be prolonged, rather than a prompt or impulsive choice. Because it is thought that required time for response selection when facing a

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

choice situation primarily indicates the time needed for collection and accumulation of sufficient information and evidence to commit to a decision (Hanes and Schall, 1996; Resulaj et al., 2009; Domenech and Dreher, 2010; Brunton et al., 2013), the resultant prolongation of the choice time may indicate that VTE was insufficient to gather enough evidence to decide. However, although the prolongation of choice time was not different between the drug conditions, the 20-lg dose showed less of a decrease in the number of VTE than the 50-lg dose, and the 20-lg, but not 50-lg, dose showed improvement on the task performance, which might indicate the lower dose allowed for the accumulation of information and evidence for a decision more than the 50-lg dose. Thus, it might be possible that clonidine also delayed a commitment of decision by affecting a process independent of VTE. Previous studies showed that an execution of choice response followed an activation of LC neurons in discrimination tasks, suggesting a role of the LC-NA neuronal activation in committing a decision and executing a behavioral response (Bouret and Sara, 2004; Clayton, 2004; Rajkowski et al., 2004). It has also been reported that injection of clonidine inhibits LC neuronal discharge (Berridge and Foote, 1991; Berridge et al., 1993; Bouret et al., 2003). Thus, the prolongation of choice time observed in experiment 2 might indicate that the injection of clonidine hindered LC-NA activity relating to not only VTE but also to the commitment of decision. Furthermore, the difference of dose–response to clonidine between VTE and choice time might imply that the LC-NA system modulates VTE and the commitment of decision via different neural mechanisms. For example, VTE is attributed to brain regions associated with exploration and flexible behavior, such as the hippocampus (Johnson et al., 2012), while it is thought that the commitment of decision can be attributed to brain regions associated with expectation of value and value-based responses, such as striatum and amygdala (Baxter and Murray, 2002; O’Doherty et al., 2004; Rangel et al., 2008), and the different modulation of these brain regions by the LC-NA system may cause the discrepant response pattern between VTE and choice time against the doses of clonidine in experiment 2. Because alpha2 adrenergic receptors mainly exist on presynaptic membranes, soma, and dendrite of noradrenergic neurons, and act as an autoreceptor (Starke, 2001), which suppresses LC-NA activity, we used clonidine for the purpose of suppressing LC-NA activity in experiment 2. Previous studies have reported that injection of clonidine decreases NA release in various brain regions and neuronal discharge in LC (De Sarro et al., 1987; Berridge and Foote, 1991; Berridge et al., 1993; van Veldhuizen et al., 1993; Berridge and Abercrombie, 1999; Kawahara et al., 1999; Bouret et al., 2003; Ferna´ndez-Pastor et al., 2005). In addition, previous studies using several doses of clonidine have reported that clonidine affects metabolism of NA in LC and ECoG in a dose-dependent manner (De Sarro et al., 1987; Quintin et al., 1987). Furthermore, clonidine suppressed activation of LC-NA induced by

299

experimental manipulations activating LC-NA (Quintin et al., 1987; Berridge and Foote, 1991). For example, cFos expression in LC in response to injection of drugs was suppressed by pre-administration of clonidine (Li and Dampney, 1995; Rasmussen et al., 1995). Thus, it may be interpreted that the injection of clonidine suppressed LC-NA activity, or c-Fos induction in LC-NA neurons, in experiment 2. However, alpha2 adrenergic receptors locate postsynaptically as well (Arnsten et al., 1996; Aoki et al., 1998), and, for instance, suppress excitatory synaptic transmission and neuronal firing (Ma et al., 2004; Ji et al., 2007). Therefore, experiment 2 could not preclude that our results are attributed, to some extent, to the postsynaptic effects of clonidine. Moreover, as well as for alpha2 receptors, clonidine also has an affinity for imidazole receptors (Reis and Piletz, 1997). It has been reported that systemic injection of imidazole before a maze task has no acute effect on acquisition of learning (Roussinov and Yonkov, 1976), suggesting that the injection of clonidine is less likely to be attributed to our results via imidazole receptors. Taken together, to specify the mechanism of the effects of clonidine on VTE and decision-making, it is required to examine activity of LC-NA or on-going synaptic mechanisms during the task. Although, in experiment 1, we prepared the three different choice conditions to reflect choice difficulty by varying amounts of rewards between the two options, the chosen discriminations also differ in terms of the magnitude of reward obtained on both a trial-by-trial and cumulative basis as the rats learn the task. It may not exclude the possibility that cognitive factors, such as motivation, reward expectation, and saliency, which are included in situations regardless of difficulty, drove observed behavior and c-Fos expression of LC-NA neurons in experiment 1. Previous studies reported that c-Fos expression in LC-NA neurons increased with food presentation to food-restricted rats (Valdes et al., 2005), but did not increase with enhancement of satiation induced by administration of the anorectic agent (Schreihofer et al., 1997), suggesting motivational saliency of reward was associated with LC-NA activity (Ventura et al., 2008). In experiment 1, rats in the 0vs4 condition experienced some trials without reward (i.e., fewer reward presentations than the other condition) and the total number of pellets increased with the quantity of reward of HR. However, running speed did not differ among conditions, suggesting that motivation and saliency of reward is less likely to affect the results of behavior and c-Fos expression in LC-NA neurons in this experiment.

CONCLUSIONS In conclusion, experiment 1 showed that the correlative activation of LC-NA neurons with the occurrence of VTE during the decision-making task. Experiment 2 showed that interference with the LC-NA activity by injection of clonidine suppressed the occurrence of VTE in the early phase of the task, with subsequent impairment of task performance. Our findings suggest that the LC-NA

300

S. Amemiya et al. / Neuroscience 265 (2014) 291–301

system regulates cognitive processes associated with VTE, and this proposition offers a beachhead for future investigation of a role of the LC-NA system in deliberative decision-making. Acknowledgment—The present study was supported by a Grantin-Aid for JSPS Fellows from Japan Society for the Promotion of Science (11J06508).

REFERENCES Amemiya S, Yanagita S, Suzuki S, Kubota N, Motoki C, Otsuka T, Nishijima T, Kita I (2010) Differential effects of background noise of various intensities on neuronal activation associated with arousal and stress response in a maze task. Physiol Behav 99:521–528. Aoki C, Venkatesan C, Go CG, Forman R, Kurose H (1998) Cellular and subcellular sites for noradrenergic action in the monkey dorsolateral prefrontal cortex as revealed by the immunocytochemical localization of noradrenergic receptors and axons. Cereb Cortex 8:269–277. Arnsten AF, Steere JC, Hunt RD (1996) The contribution of alpha 2noradrenergic mechanisms of prefrontal cortical cognitive function. Potential significance for attention-deficit hyperactivity disorder. Arch Gen Psychiatry 53:448–455. Aston-Jones G, Bloom FE (1981a) Activity of norepinephrinecontaining locus coeruleus neurons in behaving rats anticipates fluctuations in the sleep-waking cycle. J Neurosci 1:876–886. Aston-Jones G, Bloom FE (1981b) Norepinephrine-containing locus coeruleus neurons in behaving rats exhibit pronounced responses to non-noxious environmental stimuli. J Neurosci 1:887–900. Aston-Jones G, Cohen JD (2005) An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu Rev Neurosci 28:403–450. Aston-Jones G, Rajkowski J, Cohen J (1999) Role of locus coeruleus in attention and behavioral flexibility. Biol Psychiatry 46: 1309–1320. Aston-Jones G, Rajkowski J, Kubiak P (1997) Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 80:697–715. Aston-Jones G, Rajkowski J, Kubiak P, Alexinsky T (1994) Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task. J Neurosci 14:4467–4480. Baxter MG, Murray EA (2002) The amygdala and reward. Nat Rev Neurosci 3:563–573. Berridge CW, Abercrombie ED (1999) Relationship between locus coeruleus discharge rates and rates of norepinephrine release within neocortex as assessed by in vivo microdialysis. Neuroscience 93:1263–1270. Berridge CW, Foote SL (1991) Effects of locus coeruleus activation on electroencephalographic activity in neocortex and hippocampus. J Neurosci 11:3135–3145. Berridge CW, Page ME, Valentino RJ, Foote SL (1993) Effects of locus coeruleus inactivation on electroencephalographic activity in neocortex and hippocampus. Neuroscience 55:381–393. Bett D, Allison E, Murdoch LH, Kaefer K, Wood ER, Dudchenko PA (2012) The neural substrates of deliberative decision making: contrasting effects of hippocampus lesions on performance and vicarious trial-and-error behavior in a spatial memory task and a visual discrimination task. Front Behav Neurosci 6:70. Blumenthal A, Steiner A, Seeland K, David Redish A (2011) Effects of pharmacological manipulations of NMDA-receptors on deliberation in the Multiple-T task. Neurobiol Learn Mem 95: 376–384. Bouret S, Duvel A, Onat S, Sara SJ (2003) Phasic activation of locus ceruleus neurons by the central nucleus of the amygdala. J Neurosci 23:3491–3497.

Bouret S, Sara SJ (2004) Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur J Neurosci 20:791–802. Brown RAM (2005) Locus ceruleus activation suppresses feedforward interneurons and reduces electroencephalogram frequencies while it enhances frequencies in rat dentate gyrus. J Neurosci 25:1985–1991. Brunton BW, Botvinick MM, Brody CD (2013) Rats and humans can optimally accumulate evidence for decision-making. Science 340: 95–98. Clayton EC (2004) Phasic activation of monkey locus ceruleus neurons by simple decisions in a forced-choice task. J Neurosci 24:9914–9920. Dalley JW, McGaughy J, O’Connell MT, Cardinal RN, Levita L, Robbins TW (2001) Distinct changes in cortical acetylcholine and noradrenaline efflux during contingent and noncontingent performance of a visual attentional task. J Neurosci 21:4908–4914. De Sarro GB, Ascioti C, Froio F, Libri V, Nistico G (1987) Evidence that locus coeruleus is the site where clonidine and drugs acting at alpha 1- and alpha 2-adrenoceptors affect sleep and arousal mechanisms. Br J Pharmacol 90:675–685. Devauges V, Sara SJ (1990) Activation of the noradrenergic system facilitates an attentional shift in the rat. Behav Brain Res 39:19–28. Devauges V, Sara SJ (1991) Memory retrieval enhancement by locus coeruleus stimulation: evidence for mediation by beta-receptors. Behav Brain Res 43:93–97. Domenech P, Dreher JC (2010) Decision threshold modulation in the human brain. J Neurosci 30:14305–14317. Eichenbaum H (2004) Hippocampus: cognitive processes and neural representations that underlie declarative memory. Neuron 44:109–120. Ferna´ndez-Pastor B, Mateo Y, Go´mez-Urquijo S, Javier Meana J (2005) Characterization of noradrenaline release in the locus coeruleus of freely moving awake rats by in vivo microdialysis. Psychopharmacology 180:570–579. Foote SL, Aston-Jones G, Bloom FE (1980) Impulse activity of locus coeruleus neurons in awake rats and monkeys is a function of sensory stimulation and arousal. Proc Natl Acad Sci U S A 77:3033–3037. Griesbach GS, Hu D, Amsel A (1998) Effects of MK-801 on vicarious trial-and-error and reversal of olfactory discrimination learning in weanling rats. Behav Brain Res 97:29–38. Gupta AS, van der Meer MA, Touretzky DS, Redish AD (2012) Segmentation of spatial experience by hippocampal theta sequences. Nat Neurosci 15:1032–1039. Haapalinna A, Sirvio J, Lammintausta R (1998) Facilitation of cognitive functions by a specific alpha2-adrenoceptor antagonist, atipamezole. Eur J Pharmacol 347:29–40. Hanes DP, Schall JD (1996) Neural control of voluntary movement initiation. Science 274:427–430. Hu D, Amsel A (1995) A simple test of the vicarious trial-and-error hypothesis of hippocampal function. Proc Natl Acad Sci U S A 92:5506–5509. Hu D, Griesbach G, Amsel A (1997) Development of vicarious trial-and-error behavior in odor discrimination learning in the rat: relation to hippocampal function? Behav Brain Res 86:67–70. Hu D, Xu X, Gonzalez-Lima F (2006) Vicarious trial-and-error behavior and hippocampal cytochrome oxidase activity during Y-maze discrimination learning in the rat. Int J Neurosci 116:265–280. Jentsch JD, Anzivino L (2004) A low dose of the alpha2 agonist clonidine ameliorates the visual attention and spatial working memory deficits produced by phencyclidine administration to rats. Psychopharmacology 175. Ji X-H, Ji J-Z, Zhang H, Li B-M (2007) Stimulation of a2adrenoceptors suppresses excitatory synaptic transmission in the medial prefrontal cortex of rat. Neuropsychopharmacology 33:2263–2271.

S. Amemiya et al. / Neuroscience 265 (2014) 291–301 Johnson A, Redish AD (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27:12176–12189. Johnson A, van der Meer MAA, Redish AD (2007) Integrating hippocampus and striatum in decision-making. Curr Opin Neurobiol 17:692–697. Johnson A, Varberg Z, Benhardus J, Maahs A, Schrater P (2012) The hippocampus and exploration: dynamically evolving behavior and neural representations. Front Hum Neurosci 6:216. Kawahara Y, Kawahara H, Westerink BH (1999) Tonic regulation of the activity of noradrenergic neurons in the locus coeruleus of the conscious rat studied by dual-probe microdialysis. Brain Res 823:42–48. Kubota N, Amemiya S, Motoki C, Otsuka T, Nishijima T, Kita I (2012) Corticotropin-releasing factor antagonist reduces activation of noradrenalin and serotonin neurons in the locus coeruleus and dorsal raphe in the arousal response accompanied by yawning behavior in rats. Neurosci Res 72:316–323. Lapiz M, Morilak D (2006) Noradrenergic modulation of cognitive function in rat medial prefrontal cortex as measured by attentional set shifting capability. Neuroscience 137:1039–1049. Lemon N, Aydin-Abidin S, Funke K, Manahan-Vaughan D (2009) Locus coeruleus activation facilitates memory encoding and induces hippocampal LTD that depends on b-adrenergic receptor activation. Cereb Cortex 19:2827–2837. Li YW, Dampney RA (1995) Clonidine and rilmenidine suppress hypotension-induced Fos expression in the lower brainstem of the conscious rabbit. Neuroscience 66:391–402. Ma D, Rajakumaraswamy N, Maze M (2004) Alpha2-Adrenoceptor agonists: shedding light on neuroprotection? Br Med Bull 71:77–92. McGaughy J, Ross R, Eichenbaum H (2008) Noradrenergic, but not cholinergic, deafferentation of prefrontal cortex impairs attentional set-shifting. Neuroscience 153:63–71. Muenzinger KF (1938) Vicarious trial and error at a point of choice: I. A general survey of its relation to learning efficiency. J Genet Psychol 53:75–86. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454. Papale AE, Stott JJ, Powell NJ, Regier PS, Redish AD (2012) Interactions between deliberation and delay-discounting in rats. Cogn Affect Behav Neurosci 12:513–526. Paxinos G, Watson C (2005) The rat brain: in stereotaxic coordinates. 5th ed. Elsevier. Pudovkina OL, Kawahara Y, de Vries J, Westerink BH (2001) The release of noradrenaline in the locus coeruleus and prefrontal cortex studied with dual-probe microdialysis. Brain Res 906:38–45. Quintin L, Ghignone M, Pujol JF (1987) The ability of the alpha 2-adrenergic agonist clonidine to suppress central noradrenergic hyperactivity secondary to hemodynamic or environmental stimuli. J Cardiovasc Pharmacol 10(Suppl 12):S128–S134. Rajkowski J, Majczynski H, Clayton E, Aston-Jones G (2004) Activation of monkey locus coeruleus neurons varies with difficulty and performance in a target detection task. J Neurophysiol 92:361–371. Rangel A, Camerer C, Montague PR (2008) A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci 9:545–556. Rasmussen K, Brodsky M, Inturrisi CE (1995) NMDA antagonists and clonidine block c-fos expression during morphine withdrawal. Synapse 20:68–74.

301

Reis DJ, Piletz JE (1997) The imidazoline receptor in control of blood pressure by clonidine and allied drugs. Am J Physiol 273: R1569–R1571. Resulaj A, Kiani R, Wolpert DM, Shadlen MN (2009) Changes of mind in decision-making. Nature 461:263–266. Roussinov K, Yonkov D (1976) Effect of imidazole, papaverine and histamine on learning and memory in albino rats. Acta Physiol Pharmacol Bulg 2:49–57. Sara SJ, Bouret S (2012) Orienting and reorienting: the locus coeruleus mediates cognition through arousal. Neuron 76:130–141. Sara SJ, Devauges V (1989) Idazoxan, an alpha-2 antagonist, facilitates memory retrieval in the rat. Behav Neural Biol 51: 401–411. Sara SJ, Dyon-Laurent C, Herve A (1995) Novelty seeking behavior in the rat is dependent upon the integrity of the noradrenergic system. Brain Res Cogn Brain Res 2:181–187. Schreihofer DA, Cameron JL, Verbalis JG, Rinaman L (1997) Cholecystokinin induces Fos expression in catecholaminergic neurons of the macaque monkey caudal medulla. Brain Res 770:37–44. Seu E, Lang A, Rivera RJ, Jentsch JD (2008) Inhibition of the norepinephrine transporter improves behavioral flexibility in rats and monkeys. Psychopharmacology 202:505–519. Starke K (2001) Presynaptic autoreceptors in the third decade: focus on alpha2-adrenoceptors. J Neurochem 78:685–693. Tait DS, Brown VJ, Farovik A, Theobald DE, Dalley JW, Robbins TW (2007) Lesions of the dorsal noradrenergic bundle impair attentional set-shifting in the rat. Eur J Neurosci 25:3719–3724. Tolman EC (1938) The determiners of behavior at a choice point. Psychol Rev 45:1–41. Tolman EC (1939) Prediction of vicarious trial and error by means of the schematic sowbug. Psychol Rev 46:318–336. Tolman EC (1948) Cognitive maps in rats and men. Psychol Rev 55:189–208. Usher M (1999) The role of locus coeruleus in the regulation of cognitive performance. Science 283:549–554. Valdes JL, Farias P, Ocampo-Garces A, Cortes N, Seron-Ferre M, Torrealba F (2005) Arousal and differential Fos expression in histaminergic neurons of the ascending arousal system during a feeding-related motivated behaviour. Eur J Neurosci 21: 1931–1942. van der Meer MAA (2009) Covert expectation-of-reward in rat ventral striatum at decision points. Front Integr Neurosci 3. van Veldhuizen MJ, Feenstra MG, Heinsbroek RP, Boer GJ (1993) In vivo microdialysis of noradrenaline overflow: effects of alpha-adrenoceptor agonists and antagonists measured by cumulative concentration-response curves. Br J Pharmacol 109:655–660. Vann SD, Brown MW, Erichsen JT, Aggleton JP (2000) Fos imaging reveals differential patterns of hippocampal and parahippocampal subfield activation in rats in response to different spatial memory tests. J Neurosci 20:2711–2718. Ventura R, Latagliata EC, Morrone C, La Mela I, Puglisi-Allegra S (2008) Prefrontal norepinephrine determines attribution of ‘‘high’’ motivational salience. PLoS One 3:e3044. Walling SG, Brown RAM, Milway JS, Earle AG, Harley CW (2011) Selective tuning of hippocampal oscillations by phasic locus coeruleus activation in awake male rats. Hippocampus 21: 1250–1262. Yu AJ, Dayan P (2005) Uncertainty, neuromodulation, and attention. Neuron 46:681–692.

(Accepted 17 January 2014) (Available online 28 January 2014)