HIPPOCAMPUS 24:516–527 (2014)

Two Separate, But Interacting, Neural Systems for Familiarity and Novelty Detection: A Dual-Route Mechanism Alexandros Kafkas* and Daniela Montaldi

ABSTRACT: It has long been assumed that familiarity- and noveltyrelated processes fall on a single continuum drawing on the same cognitive and neural mechanisms. The possibility that familiarity and novelty processing involve distinct neural networks was explored in a functional magnetic resonance imaging study (fMRI), in which familiarity and novelty judgments were made in contexts emphasizing either familiarity or novelty decisions. Parametrically modulated BOLD responses to familiarity and novelty strength were isolated in two separate, nonoverlapping brain networks. The novelty system involved brain regions along the ventral visual stream, the hippocampus, and the perirhinal and parahippocampal cortices. The familiarity system, on the other hand, involved the dorsomedial thalamic nucleus, and regions within the medial prefrontal cortex and the medial and lateral parietal cortex. Convergence of the two networks, treating familiarity and novelty as a single continuum was only found in a fronto-parietal network. Finally, the orbitomedial prefrontal cortex was found to be sensitive to reported strength/confidence, irrespective of stimulus’ familiarity or novelty. This pattern of results suggests a dual-route mechanism supported by the existence of two distinct but interacting functional systems for familiarity and novelty. Overall, these findings challenge current assumptions regarding the neural systems that support the processing of novel and familiar information, and have important implications for research into C 2014 Wiley Periodicals, Inc. the neural bases of recognition memory. V KEY WORDS:

familiarity; novelty; fMRI; medial temporal lobe; thalamus

INTRODUCTION Our outstanding ability to discriminate what is old and what is new in the environment constitutes a central aspect of human cognition and underlies much of our accurate recognition memory. Familiarity memory is assumed to provide a relatively fast route to judge the previous occurrence of a stimulus (Brown and Aggleton, 2001), but the distinct roles of the familiarity and novelty signals in supporting this ability have not been systematically studied. The question we investigate here is whether the neural systems supporting familiarity and novelty detection are predominantly overlapping or distinct. Answering this question is

Human Memory and Amnesia Laboratory, School of Psychological Sciences, University of Manchester, United Kingdom Additional Supporting Information may be found in the online version of this article *Correspondence to: Dr. Alex Kafkas, School of Psychological Sciences, University of Manchester, Manchester, M13 9PL. E-mail: [email protected] Accepted for publication 9 January 2014. DOI 10.1002/hipo.22241 Published online 16 January 2014 in Wiley Online Library (wileyonlinelibrary.com). C 2014 WILEY PERIODICALS, INC. V

critical to our evaluation and understanding of previously proposed memory systems as well as to theories explaining behavioral phenomena such as novelty seeking. Two potential mechanisms might underlie our ability to discriminate between the old and the new. The first one, which we will refer to as the unitary-route mechanism, suggests that neural signals responsible for detecting and coding the novelty of sensory information also provide the means for coding the relativefamiliarity of old stimuli. This mechanism is consistent with the commonly held view that familiarity and novelty detection processes can be seen as mirror images of each other. A key prediction of this view, which underlies the majority of memory research to date, is that brain regions commonly supporting recognition memory (e.g., medial temporal lobe; MTL) should show patterns of BOLD activity reflecting the full continuum from strong novelty to strong familiarity. A second, potential mechanism, however, suggests that neurally distinctive familiarity and novelty signals converge elsewhere to promote accurate retrieval and guide encoding. We will refer to this as the dual-route mechanism. The single continuum mechanism fits comfortably with the behavioral outcomes of old/new decisions and the standard use of the term familiarity to encompass both novelty and familiarity detection. However, we believe that the dual-route mechanism may provide a more accurate account of the complex neural mechanisms involved in discriminating between old and new events. Furthermore, it stresses that the classification of a stimulus as old or new takes into account both the signal generated by the re-activation of an existing representation (i.e., familiarity), and that generated by novelty detection. Novelty neurons, neuronal populations that selectively respond to the initial occurrence of an event (Fahy et al., 1993; Xiang and Brown, 1998; Rutishauser et al., 2006; Rutishauser et al., 2008) and familiarity/recency neurons, neurons that are specifically sensitive to recurrence of events (Xiang and Brown, 1998; Rolls et al., 2005; Rutishauser et al., 2006) have been identified in the monkey’s temporal lobe. Similar findings have also been reported in human subjects in single-neuron (Viskontas et al., 2006) and neuroimaging studies (Tulving et al., 1996; Yassa and Stark, 2008). This evidence supports the idea that neuronal populations responsible for familiarity and

FAMILIARITY AND NOVELTY DETECTION PROCESSES novelty detection may not be entirely overlapping, however, to date, there has been no clear model proposed to explain how, and where, familiarity and novelty signals might collaborate to support recognition memory and to what extent they originate from distinct or overlapping brain structures. This issue is directly investigated using functional Magnetic Resonance Imaging (fMRI). A parametric event-related fMRI study explored the neural bases of familiarity and novelty by correlating regional brain activity with variations in the reported strength of familiarity and novelty. We expected that familiarity-selective regions would show either increased or decreased parametric responses to familiarity strength (i.e., strength rating provided in response to the presentation of familiar stimuli), while novelty-selective regions would show increased or decreased parametric responses to novelty strength (i.e., strength rating provided in response to the presentation of novel stimuli). To further constrain this regional selectivity to familiarity or novelty processing, exclusive masks were applied (see Methods below). This allowed the discrimination between those brain regions particularly sensitive to modulations of novelty and those particularly sensitive to modulations of familiarity, independent of whether these modulations were characterized by increases or decreases in activity. In contrast to traditional recognition memory tasks, a paradigm was developed in which familiarity and novelty strength was rated under two separate conditions/contexts one emphasizing familiarity (familiarity task; FT) and the other, novelty (novelty task; NT) detection. This design ensured that the emphasis was on familiarity or novelty detection and that in each of these contexts/conditions participants would engage the appropriate detection system.

MATERIALS AND METHODS Participants A total of 19 right-handed participants gave informed consent and participated in the study. From these, one participant was excluded from the analysis due to excessive movement during scanning and another due to a technical problem affecting the recording of behavioral responses. Data from the remaining participants (N 5 17; 7 male; mean age 5 22.6 years; SD 5 64.2 years) were analyzed and are reported below. Remuneration was given at a rate of £20 per session. The study received ethical approval from the National Research Ethics Service (North West-GM South).

Stimulus Materials A total of 318 stimuli (18 practice stimuli) depicting manmade and natural objects were used. The stimuli were randomly allocated to two lists corresponding to the two memory conditions (FT and NT). Allocation to the FT and NT conditions was counterbalanced across participants. Furthermore,

517

stimuli within each list were pseudorandomly chosen to serve as old (presented at encoding) and new stimuli, with the critical exception being that while 110 stimuli (about 70%) in the FT condition had been previously presented at study, only 40 stimuli (about 30%) in the NT condition had been.

Procedure and Design The experiment consisted of two phases: a study phase and a recognition phase. During study, a perceptual matching-tosample task, similar to that described in previous studies (Montaldi et al., 2006; Kafkas and Montaldi, 2011, 2012), was completed. In this task, participants are presented with three closely identical images of a stimulus and are asked to decide which of the two lower images matches the target image presented above. Either the size (slightly smaller or bigger than the original image) or the orientation (turned slightly towards the right or left) of the nontarget was slightly modified. The location (left or right) of the matched stimulus was randomly assigned. One hundred and fifty trials were presented in this way and participants had 4s to identify the matched image. After completion of the encoding phase, subjects were instructed on the use of familiarity and novelty detection ratings, each to be used in the FT and NT conditions respectively and were also carefully trained to discriminate between instances of familiarity and recollection (see Kafkas and Montaldi, 2012). Two practice blocks, one outside the scanner and another in the scanner (during T1 structural image acquisition), were completed before the main retrieval task. Scanning took place during this retrieval session. During this phase, two tasks were presented to participants. In the familiarity task (FT), participants were asked to rate the familiarity of the stimuli using a choice between three levels of increasing strength (F1: weak, F2: moderate, F3: strong familiarity), along with one choice for new (foil), and another for recollected items. In the novelty task (NT), participants were asked to rate their feelings of novelty for each stimulus by selecting one of three levels of novelty strength (N1: weak, N2: moderate, N3: strong novelty) along with a choice for familiar and another for recollected stimuli. Critically, in the FT condition 70% of the stimuli from the encoding phase were presented along with 30% new trials. In the NT condition 30% of the items from the study phase were randomly presented along with 70% of new images. Thus, equal numbers of target stimuli (either familiar or novel) were presented in the two tasks, ensuring that the emphasis in each condition was maintained selectively either on familiarity, or on novelty detection, as required. A mixed event-related/block design was implemented by presenting alternating blocks of familiarity (FT) and novelty (NT) conditions. Each of these blocks lasted for 270 s followed by a 20 s fixation cross during which participants were instructed to focus on the center of the screen. In total, six task blocks (e.g., FT-NT-FT-NT-FT-NT) were presented to each participant, over two sessions (three task blocks in each session). Each session started and ended with a fixation block. A visual cue was presented centrally for 2.5 s, before the beginning of each Hippocampus

518

KAFKAS AND MONTALDI

TABLE 1. Proportion of Trials, Performance, and Response Times in the fMRI Experiment Proportion of trials FT F1 F2 F3 M CR FA NT N1 N2 N3 M CR FA

Hits-FAs

RT (ms)

0.28 0.27 0.29 0.12 0.55 0.37

(0.15) (0.09) (0.09) (0.08) (0.21) (0.06)

0.06 (0.12) 0.15 (0.12) 0.26 (0.08)

1906 1801 1542 1855 1714 1772

(262) (291) (234) (277) (220) (298)

0.23 0.27 0.42 0.08 0.38 0.44

(0.15) (0.11) (0.16) (0.06) (0.19) (0.26)

0.06 (0.17) 0.12 (0.12) 0.30 (0.15)

1990 1897 1632 1915 1679 1893

(246) (233) (225) (298) (193) (263)

Numbers in the parentheses are standard deviations. FT, familiarity task; NT, novelty task; FAs, false alarms; RT, response time.

block providing the instruction “how FAMILIAR are these stimuli?” for the FT condition, “how NOVEL are these stimuli?” for the NT condition or “Fixate” for the intermediate fixation task. Within each FT or NT block a total of 50 stimuli were presented, each for 3 s, intermixed with 10 3 s fixation trials (implicit baseline), pseudorandomly allocated across the block with the only restriction being that no two fixation trials were presented consecutively. Each stimulus and fixation trial was followed by a 1 s fixation cross. Participants used both hands to indicate their responses via two MR compatible button boxes. The responding hand for the target choices (three buttons) and the additional choices (two buttons) was counterbalanced across participants. Finally, half of the participants started with an FT block and half with an NT block.

fMRI Acquisition and Analyses A 3T MRI scanner (Philips, Achieva) was used for scanning. A gradient echo-planar imaging (EPI) sequence was used to obtain functional images using the blood oxygenation level dependent (BOLD) contrast. A total of 726 (363 per session) volumes were acquired (TR 5 2.5 s; TE 5 35 ms). Each volume consisted of 40 slices (matrix size: 96 3 96; voxel size: 2.5 3 2.5 3 3.5 mm3) and the acquisition was positioned parallel to the AC-PC line and covered the whole brain. T1weighted images (matrix size: 256 3 256, number of slices: 180, voxel size: 1 mm isotropic) were acquired before the functional run. Special foam wedges and soft pads were used to stabilize the participant’s head and restrain head motion during scanning. Earplugs were also provided to minimize noise from the MRI scanner. Data were preprocessed and analyzed using SPM8 (Statistical Parametric Mapping, Wellcome Department of Cognitive NeuHippocampus

rology, University College London, London, UK; Available at: http://www.fil.ion.ucl.ac.uk/spm/). A six-parameter rigid body transformation was used to realign the EPI data to the first image and reslice them using sinc interpolation in space. The resliced images were slice-time corrected to the middle slice and spatially normalized to the Montreal Neurological Institute (MNI) EPI template. An isotropic 8mm full width half maximum (FWHM) Gaussian kernel was used for smoothing. A sufficient number of trials (>8 trials) were produced for all event-types (see proportions for each response category in Table 1). As the focus of this study was to explore the neural bases of familiarity and novelty, as contributors to recognition memory, spontaneously recollected events were discarded from the current analyses (and are reported elsewhere, Kafkas and Montaldi, 2012). Event-related responses were modelled separately for each participant using a canonical hemodynamic response function (Friston et al., 1998). Regressors of no interest (e.g., cue screens preceding each block, trials with no response etc.) were modeled separately. Furthermore, the six movement parameters, in each of the two sessions of the experiment, were modeled as covariates, to capture residual variance in the time series that can be explained by movement artifacts. Modulations of brain activity by familiarity and novelty strength were explored in two parametric models, for each participant, using an orthogonal polynomial expansion of the first order exploring linear effects (B€ uchel et al., 1998). Nonlinear polynomial quadratic effects (positive and negative) were also modeled to capture residual effects not explained by the linear function. These produced no additional activation for novelty, while very few areas showed positive (but not negative) quadratic familiarity effects (see Supporting Information Table 1). The sparsity of nonlinear BOLD effects across familiarity or novelty strength (either increasing or decreasing) underlines the fact that the majority of the variance of the BOLD response is explained by a linear function. Moreover, as familiarity and novelty strength are ordinal scales, the linear model does not assume equal contribution to the BOLD signal for each of the successive levels of familiarity or novelty. Instead, it assumes a pattern of increased or decreased responses across the scale that can be fitted by a linear function. Therefore, we prefer to characterize these effects as monotonic as opposed to linear. This is also reflected in the plots of the parameter estimates in the areas that show significant parametric modulations (see Figures below). For the parametric analyses, the onsets of familiarity and novelty responses for the two tasks (FT and NT) were specified separately for each participant in the first level analysis, while the strength rating that accompanied each response was entered as a parametric covariate (convolved with the stimulus-specific HRF) using the standard parametric modulation option in SPM. The resulting parametric modulation regressors then tested for an interaction between familiarity or novelty responses and the reported strength. The two parametric models consisted of four levels of strength for familiarity (M, F1, F2, F3) and four levels for novelty (M, N1, N2, N3). Misses

FAMILIARITY AND NOVELTY DETECTION PROCESSES (M) in the two conditions (Old reported as New in the FT task and New reported as Old in the NT task) were used as the zero level of familiarity and novelty respectively. It is worth noting that similar parametric analyses with only three levels of strength for familiarity and novelty (i.e., without misses) produced very similar results and therefore are not reported separately. We reasoned that familiarity-specific regions should respond to variations in the subjective rating of familiar stimuli either by increasing or decreasing their activity. Similarly, noveltyspecific regions would respond to novel stimuli either by increasing or decreasing their activity in response to novelty strength. Therefore, for each participant four t-contrasts (i.e., positive modulation by F-strength, negative modulation by Fstrength, positive modulation by N-strength, and negative modulation by N-strength) were specified to identify brain regions in which activity was positively or negatively modulated by familiarity (F) and/or novelty (N) strength. Finally, at the group (random-effects) level, the t-contrasts for the negative and positive parametric modulations for familiarity and novelty were entered into a one-sample t-test in SPM, as standard. This analysis allowed the differentiation between brain areas that respond to stimulus familiarity and those that respond to stimulus novelty, by either increasing or decreasing their activity across F- or N-strength. Moreover, to further constrain the identification of familiarity- and novelty-sensitive areas, the results of each parametric analysis were exclusively masked with familiarity and novelty activations. Specifically, modulations in brain regions that showed F-strength increases or decreases were exclusively masked with brain regions showing N-strength decreases or increases. Similarly, N-strength increases and decreases in activity were exclusively masked with areas showing F-strength decreases and increases. This enabled us to isolate those regions which specifically code for stimulus familiarity—but not novelty, or stimulus novelty—but not familiarity. On the other hand, three conjunction analyses (CA) were used to reveal brain regions that respond to modulations of both familiarity and novelty strength. The conjunction null hypothesis for the two parametric contrasts (F and N) was tested in SPM to show regions of common activation (Friston et al., 2005). The first CA explored the areas whose activity reflects relativefamiliarity (i.e., the full N3 to F3 continuum). This contrast includes areas that showed increased activity to F-strength and decreased activity to N-strength. It should be noted here that a region may respond to the relative-familiarity continuum, without necessarily showing a continuously increasing BOLD signal across all levels of F and N. Rather it is the directionality of the effects, underpinned by the combination of both the increased activity across F-strength and the decreased activity across Nstrength that characterizes the shared sensitivity to relative familiarity (see for example Figs. 3d and 3e). The second CA explored the areas that respond to stimulus relative-novelty (i.e., the full F3 to N3 continuum) and includes areas that showed concurrent decreased activity to F-strength and increased activity to Nstrength. Finally, overlapping activation and deactivation patterns

519

produced in response to reported familiarity and novelty strength were explored in the last CA (i.e., the 1–3 continuum for both novelty and familiarity). This last analysis therefore reveals areas that are sensitive to the reported strength/confidence of the response, irrespective of the familiarity or novelty status of a stimulus. Activations surviving a cluster-wise P < 0.05 significance level (cluster FWE corrected) are reported unless otherwise stated. The MTL regions are reported as significant when the activation survived a FWE small volume correction for the volume of the MTL or when 8 or more contiguous voxels were active at P < 0.001. This level was established as being equivalent to a corrected probability level of P < 0.05 for the volume of the MTL using a Monte Carlo simulation (implemented in AlphaSim tool in AFNI). Finally, memory performance and response time (RT) data were analyzed using a two-way repeated measures analyses of variance (ANOVA) with task (familiarity vs. novelty) and recognition strength (three levels of familiarity and novelty) as the within-subject factors. A significance level of P < 0.05 was adopted for the analysis of the behavioral data.

RESULTS Behavioral Results Participants accurately recognized old and new stimuli in familiarity (FT) and novelty (NT) tasks with a mean accuracy of 0.76 6 0.08 (SD) and 0.77 6 0.06 (SD) respectively. Behavioral data from the two experiments are summarized in Table 1. As expected, discrimination performance (hits-false alarms) increased as participants reported increased familiarity or novelty strength (F2,32 5 21.57, P < 0.001), while participants performed equally well on both tasks as revealed by the nonsignificant effect of task (F < 1). Finally, in both FT and NT, shorter response times (RT) were produced for the stronger than for the moderate or weaker familiarity and novelty responses (F2,32 5 41.10, P < 0.001).

Neuroimaging Results Consistent with the hypothesis of this study, several brain regions showed selective sensitivity to either familiarity or novelty processing, while a smaller number demonstrated sensitivity to both; showing either a relative-familiarity effect (activity along the N3 to F3 continuum) or sensitivity to confidence of response (1–3 continuum) irrespective of the novelty or familiarity status of the stimulus.

Relative-Familiarity Effects: The Novelty– Familiarity Continuum First, to identify areas sensitive to the full novelty-familiarity continuum, a conjunction analysis (testing the conjunction null; see Friston et al., 2005) was performed on the parametric Hippocampus

520

KAFKAS AND MONTALDI

FIGURE 1. Novelty-specific effects in the hippocampus (a), the parahippocampal cortex (b), and the perirhinal cortex (PRC; c). The three regions show increased activity with increased novelty strength. Error bars represent s.e.m.

familiarity and novelty brain responses (see Methods). Critically, no MTL region (even at reduced thresholds) was found to show activity that was modulated along the N3 to F3 or F3 to N3 continua. However, relative-familiarity effects (N3 to F3 activation increases) were found in the bilateral inferior parietal lobule (BA 40; MNI peak 5 -46, 260, 41 and 50, 237, 45; Fig. 2d) extended to the angular gyrus (BA 39), in the left middle frontal gyrus (BA 8/6; MNI peak 5 -38, 3, 55; Figs. 3d,e) and the left superior medial frontal gyrus (BA 8; MNI peak 5 -6, 28, 38; Fig. 4g). The topography and activation parameters of these regions, in relation to familiarity-specific and novelty-specific regions, are shown in Figures 2, 3, and 4 (see below). However, relative-novelty effects across the full Hippocampus

continuum (F3 to N3 activation increases) were not found anywhere in the brain even at a considerably lower threshold (P < 0.01).

Novelty-Specific Responses A network of brain structures running along the ventral visual stream leading to, and including, the MTL structures (Fig. 1 and Table 2) showed selective sensitivity to novelty strength (N0 to N3). Specifically, the bilateral middle occipital gyrus (BA 18/19), the bilateral fusiform gyrus (BA 37/19), the bilateral parahippocampal cortex (BA 36; Fig. 1b), bilateral hippocampus (Fig. 1a), and bilateral PRC (BA 35; Fig. 1c)

FAMILIARITY AND NOVELTY DETECTION PROCESSES

521

TABLE 2. Brain Regions Monotonically Modulated by Novelty Strength Side

Region

Monotonic increases with novelty (N0 to N3) R Hippocampus L Hippocampus L Perirhinal/parahippocampal cortex R Perirhinal cortex R Parahippocampal cortex L Middle temporal gyrus L Middle temporal pole L Fusiform gyrus R Fusiform gyrus R Middle occipital gyrus L Middle occipital gyrus R Orbitomedial frontal gyrus Monotonic decreases with novelty (N3 to N0) L Inferior parietal lobule R Inferior parietal lobule L Superior medial frontal gyrus L Middle frontal gyrus L Inferior frontal gyrus R Precuneus L Precuneus L Globus pallidus R Caudate nucleus

Voxels

89 32 48 8 68 129 253 101 160 275 65 433 125 356 323 231 36 34 7 18

BA

MNI x y z

Z

34/35/36 35 36 21 38 37/19 37 18/19 18/19 10/11

27 212 219 221 210 219 223 27 229 30 210 236 32 230 215 253 25 226 248 8 226 221 245 219 22 237 215 37 287 13 226 2100 13 2 58 28

5.23 4.29 3.55 3.14 3.91 4.72 3.74 4.6 4.13 4.28 4.28 3.56

238 255 41 40 242 41 (8 21 52 (46 16 38 251 33 20 12 265 45 28 272 48 213 3 3 20 11 13

4.72 3.76 4.63 4.34 4.12 3.65 3.66 3.37* 3.09*

40/39 40/39 8/6 6/9 10/11/46 7 7

Parametric activations are reported as significant at P < 0.05, FWE-corrected at the cluster level, except *P < 0.001, uncorrected.

demonstrated monotonic increases in activity as a function of novelty strength. Regions which showed decreased activity as a function of novelty strength (Table 2 lower panel) included, the bilateral inferior parietal lobule (BA 40; Figs. 2a,c), the left inferior (BA 10/11/46) and middle (BA 10/46) frontal gyri (Figs. 3a,c), the bilateral precuneus (BA 7; Fig. 4a) and the left medial/superior frontal gyrus (BA 6/8/32; Fig. 4b).

Familiarity-Specific Responses Familiarity-specific monotonic increases in brain activity with increased strength (F0 to F3) were not found in the MTL even at a lowered threshold, but instead, were found in frontal, parietal, and sub-cortical regions (see Table 3). Specifically, familiarity responses were found in the left thalamus (mostly dorsomedial) (Fig. 5), the bilateral inferior parietal lobe including the angular gyrus (BA 39/40; Figs. 2a,b), the superior medial frontal gyrus (BA 8/9; Fig. 4d), the left middle frontal gyrus (BA 6/8; Figs. 3a,b) and the left precuneus (BA 7; Fig. 4c). Decreases in brain activity with increasing familiarity were only found in the left inferior occipital lobe (P < 0.001, cluster >5 voxels).

Response to Confidence To isolate areas responding to strength along the confidence continuum irrespective of the type of stimulus (familiar or novel), a conjunction analysis of weak to strong familiarity and

novelty was performed. This identified activity in the bilateral orbitomedial PFC (BA 10/11; MNI peak 5 251, 16, 28; Fig. 4f ), an area at the temporoparietal junction (BA 39; MNI peak 5 248, 272, 24) and a cluster in the left retrosplenial cortex (BA 23; MNI peak 5 23, 252, 27; Fig. 4e). Overall, the imaging data strongly suggest the existence of two separate systems for familiarity and novelty detection and evaluation that converge (as shown by the conjunction analyses) in frontal and parietal regions. The implications of the findings are discussed below.

DISCUSSION The results from the experiment reported here suggest that familiarity and novelty judgments draw on dedicated but critically interacting and complementary neural systems. While familiarity and novelty detection draw upon brain signals produced by (but not limited to) areas previously associated with recognition memory, it is particularly notable that no region of the MTL was found to activate in response to the modulation of the full novelty2familiarity continuum, i.e., to relative2familiarity (even at a reduced threshold). Instead, brain regions in the MTL and the thalamus, were found to be selectively sensitive either to novelty or to familiarity detection respectively. Only fronto-parietal brain areas Hippocampus

522

KAFKAS AND MONTALDI

FIGURE 2. Activity in lateral parietal cortex. (a) Familiarityspecific (red), novelty-specific (yellow) and shared (blue) parametric activity patterns in the lateral parietal cortex. (b) Parameter estimates in left angular gyrus (peak MNI 5 253, 267, 38) showing increased activity with increased familiarity strength. (c) Parameter estimates in the left inferior parietal lobe (peak

MNI 5 238, 255, 41) showing decreased activity with increased novelty strength. (d) Parameter estimates in a left inferior parietal lobe area (peak MNI 5 246, 260, 41) treating familiarity and novelty as a continuum showing a relative-familiarity effect (i.e., increased activity from N3 to F3). Error bars represent s.e.m.

were found to respond to the full novelty2familiarity continuum, with tightly adjacent fronto-parietal regions showing selectivity for either novelty or familiarity detection (see Figs. 2–4). Yet another, quite distinctive, set of fronto-parietal regions (see Figs. 4e,f) responded to variations in response strength irrespective of stimulus status (familiar or novel), reflecting, therefore, general response confidence.

between old and new stimuli (see e.g., Brown and Xiang, 1998) suggests that processing novel information may, as a result of its environmental importance, be subjected to a rapid, highly specialized novelty evaluation system. Current thinking assumes that novelty judgments are served by the same system that serves familiarity judgments. However, the current fMRI data suggest instead that novelty-specific signals (for visual stimuli) are computed in a network of brain regions along the ventral visual stream, consisting of the middle occipital areas (BA 18/19), the fusiform gyri (BA 19/37), leading to the parahippocampal and perirhinal cortices (PRC) and ultimately to the hippocampus (see Fig. 1). This novelty-specific network includes selective fronto-parietal regions (Figs. 2 and 3) responding only to changes in novelty strength.

Distinct Novelty and Familiarity Systems The early detection of novelty constitutes an evolutionary advantage ensuring fast and accurate evaluation of novel information, its potential salience and the possible threat it may present. Interestingly, the rapidity of the discrimination Hippocampus

FAMILIARITY AND NOVELTY DETECTION PROCESSES

523

FIGURE 3. Activity in lateral prefrontal cortex. (a) Familiarity-specific (red), novelty-specific (yellow) and shared (blue) parametric activity patterns in the lateral prefrontal cortex. (b) Parameter estimates in left middle frontal gyrus (peak MNI 5 236, 8, 55) showing increased activity with increased familiarity strength. (c) Parameter estimates in the left middle

frontal gyrus (peak MNI 5 246, 16, 38) showing decreased activity with increased novelty strength. (d and e) Parameter estimates in two left middle frontal gyrus areas (d peak MNI 5 251, 23, 27; e peak MNI 5 238, 3, 55) treating familiarity and novelty as a continuum showing a relative-familiarity effect. Error bars represent s.e.m.

In contrast, familiarity-specific signals were found in brain regions including, in particular, the dorsomedial (Dm) thalamus where activity increased with increased familiarity strength (Fig. 5). We have previously reported an increase in Dm acitvation with increasing familiarity strength (Montaldi et al., 2006) and a selective contribution of the Dm-thalamus to strong familiarity when contrasted to equally strong recollection (Kafkas and Montaldi, 2012). The increased sensitivity of the Dmthalamus to familiarity strength in the current study, as well as in our previous studies, is strongly consistent with it playing a

specialized role in recognition memory particularly in the processing of familiarity (Aggleton and Brown, 1999; Van Der Werf et al., 2003; Aggleton et al., 2010). Furthermore, activity in Dm-thalamus has been found to interact directly with responses in neocortical and more specifically frontal regions for old compared to CR (new) responses (Staudigl et al., 2012). Indeed, the same frontal familiarity-specific activation pattern was also found, in the current study, in selective regions of the dorsolateral PFC (BA 8/9; Fig. 3), the superior medial frontal gyrus (BA 8/9) and the anterior cingulate (BA 32; see Hippocampus

524

KAFKAS AND MONTALDI

FIGURE 4. Activity in the medial prefrontal and medial parietal cortices for familiarity-specific (red), novelty-specific (yellow) and shared (blue) regions. Parameter estimates in the left precuneus (a; peak MNI 5 28, 272, 48) and the left superior medial frontal gyrus (b; peak MNI 5 28, 21, 52) showing decreased activity with increased novelty strength. Parameter estimates in the left precuneus (c; peak MNI 5 23, 265, 34) and the left superior medial frontal gyrus (d; peak MNI 5 28, 51, 41) showing increased activity with

increased familiarity strength. Parameter estimates in the left posterior cingulate gyrus/retrosplenial cortex (e; peak MNI 5 23, 252, 27) and the orbitomedial PFC (f; peak left MNI 5 251, 16, 28) responding to familiarity and novelty confidence. Parameter estimates in the left superior medial frontal gyrus (g; peak MNI 5 26, 28, 38) treating familiarity and novelty as a continuum showing a relativefamiliarity effect (i.e., increased activity from N3 to F3). Error bars represent s.e.m.

Fig. 4), as well as in the left angular gyrus, (Fig. 2) and the precuneus (Fig. 4). Taken together, these sets of data strongly suggest that two distinct neural systems exist; a novelty-specific and a familiarity-specific processing system, whose signals only converge in common fronto-parietal regions where the activity is parametrically modulated by both novelty and familiarity. Our identification of two separate but interacting novelty and familiarity brain mechanisms suggests that our memory system employs familiarity and novelty signals heuristically, depending on the context in which a stimulus is encountered. This organization allows the system not only to be rapid and accurate in its processing of the relative familiarity of a stimulus, but also, and crucially, to be flexible. Such flexibility ensures that the detection of novelty or familiarity is interpreted and acted on by the brain in a way that appropriately

promotes retrieval and guides further encoding depending on contextual demands. Exactly what contextual factors modulate these two systems remains to be established. It is important to note that both the PRC and the hippocampus emerged as parts of the novelty-specific network. This is consistent with previously reported selective response of MTL structures to new versus old stimuli, rather than the opposite (Johnson et al., 2008; Suzuki et al., 2011). There is increasing evidence to suggest that the hippocampus serves a different functional role from the PRC within this network, probably supporting the active encoding of novel information into long-term memory (Tulving et al., 1996; Eldridge et al., 2005; Strange et al., 2005; Fernandez and Tendolkar, 2006; Staresina et al., 2012). Of particular note is the finding that activity in the PRC not only failed to show monotonic changes

Hippocampus

FAMILIARITY AND NOVELTY DETECTION PROCESSES

525

TABLE 3. Brain Regions Monotonically Modulated by Familiarity Strength Side

Region

Monotonic increases with familiarity (F0 to F3) L Angular gyrus R Angular gyrus R Supramarginal gyrus L Superior medial frontal gyrus L Middle frontal gyrus L&R Middle cingulate gyrus L Anterior Cingulate gyrus R Caudate nucleus L Precuneus L Thalamus R Insula Monotonic decreases with familiarity (F3 to F0) L Inferior occipital lobe

Voxels

BA

MNI x y z

Z-value

426 83 40 343 170 63 14 17 207 18 51

39/40 39/40 40 8/9 6/8/9 23 32

253 267 38 45 255 34 52 240 41 28 51 41 236 8 55 2 225 34 26 41 20 15 6 17 23 265 34 213 210 3 50 8 25

5.39 3.95 3.47 4.54 3.92 3.88 3.8 3.93* 3.91 3.83 3.77*

12

19

233 272 25

3.99*

7

Parametric activations are reported as significant at P < 0.05, FWE-corrected at the cluster level, except *P < 0.001, uncorrected.

in activity corresponding to the familiarity2novelty continuum, but unlike previous fMRI studies (Henson et al., 2003; Gonsalves et al., 2005; Montaldi et al., 2006), it failed to show decreased activity with increasing familiarity strength. It might therefore be argued that the PRC supports familiarity memory, indirectly, by providing a novelty signal, which, as shown in this study, is sensitive to the degree of subjective novelty attributed to a new stimulus. This study is the only study, the authors are aware of, that requires participants to actively distinguish between familiarity and novelty detection by engaging explicitly and exclusively in novelty and familiarity detection at different times. It is possible therefore that in the familiarity condition, our participants attended very carefully to feelings of familiarity and attention to novelty was minimized; something that is not controlled in other studies. This may also explain the failure of some previous studies (Eldridge et al., 2005; Yonelinas et al., 2005) to show a familiarity effect in the PRC, as it is likely that a novelty signal is not triggered in the PRC when highly familiar stimuli are used; determined either pre-experimentally (i.e., very common stimuli) or within an experiment (e.g., deep encoding procedures). Indeed, consistent with the neurophsyiological literature (Xiang and Brown, 1998; Viskontas et al., 2006) the reduced BOLD response found with repetition in fMRI is normally conceptualized as a signal reduction relative to that generated for correct rejections or misses, i.e., new, or perceived as new, stimuli (Henson et al., 2003; Gonsalves et al., 2005; Staresina et al., 2012) and therefore is more consistent with the role of PRC in novelty, rather than in familiarity, detection. Relatedly, Xiang and Brown (1998) found no change in activity in monkey PRC neurons for stimuli that became increasingly familiar (appeared more than twice), which is consistent with the lack of relative familiarity responses in the human PRC found in the present study. However, it is also possible that a full explanation of the function of the PRC is that regions within it

carry out contrasting roles and perform different computations with respect to stimulus familiarity and novelty, depending on the contextual demands of the task and the inherent novelty or familiarity of the stimuli (see also Kafkas and Montaldi, 2012 for the discussion of a selective role of PRC in strong familiarity vs. equally strong recollection).

Convergent Memory Evaluation Network As noted above, familiarity-specific parametric effects were isolated in prefrontal and parietal regions, showing increased activity with increasing familiarity strength. Critically, novelty signals were also isolated in different prefrontal and parietal regions in the left middle and inferior frontal areas (BA 9/10/ 11/46), encompassing dorsolateral and ventrolateral prefrontal areas and in the left and right inferior parietal lobe areas (BA 40), extending into the superior parietal cortex (BA 7) and the bilateral precuneus (Figs. 2 and 4). The response patterns in these novelty-sensitive regions within the PFC and the parietal cortex took the form of decreased responsiveness to highly novel stimuli (N3) and increased activity to weakly novel stimuli (N1). Therefore, despite the distinctiveness of the familiarity-specific and novelty-specific PFC and parietal areas on the basis of the status of stimulus they code (novel or familiar), the direction of their response to different levels of familiarity and novelty strength reveals that both these frontoparietal networks provide information regarding the relativefamiliarity of a stimulus. This is accomplished in the familiarity-sensitive areas (e.g., dorsolateral and medial PFC) by increasing activity to highly familiar stimuli, while in the novelty-sensitive areas (e.g., lateral and superior medial PFC) by decreasing the activity for the strongly novel stimuli. In addition, the conjunction analysis revealed a set of frontoparietal regions within the lateral and medial aspects of PFC and inferior parietal cortex, in close proximity to the selective Hippocampus

526

KAFKAS AND MONTALDI medial aspect of PFC, was found to be sensitive to reported strength/confidence, irrespective of whether familiarity or novelty processes were involved (Fig. 4f ). Consistent with this, the orbitomedial PFC has previously been implicated in successful object recognition (Meunier et al., 1997) and is associated with response monitoring (Tsujimoto et al., 2009) and the regulation of choice and goal-directed behavior (Petrides, 2007; Izquierdo and Murray, 2010; De Martino et al., 2013).

CONCLUSIONS

FIGURE 5. Familiarity-specific effects in the left thalamus (peak MNI 5 213, 210, 3). The parameter estimates show the increased activity in the thalamus (including the dorsomedial aspect) with increased familiarity strength. Error bars represent s.e.m. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

familiarity and novelty regions, which respond to both novelty and familiarity and therefore produce a unitary relativefamiliarity signal showing monotonic changes in activity corresponding to the familiarity-novelty continuum (N3 to F3; areas in blue in Figs. 2–4). Taken together, these findings strongly suggest that distinct familiarity and novelty mechanisms, produce signals that converge in fronto-parietal areas where a rating of the relative-familiarity of a stimulus is computed. Despite previous suggestions (Rugg et al., 2003; Herron et al., 2004; Achim and Lepage, 2005), the existence of such mechanisms within the PFC indicates that the PFC does not simply support post-retrieval checking and decisional processes but, instead, plays a central role in the evaluation of recognition memory. Interestingly, consistent with the effects reported here, the primate medial PFC neurons have previously been found to respond to familiar rather than novel stimuli (Xiang and Brown, 2004) and neuropsychological patients with damage to the medial PFC have been identified as manifesting impaired ability to judge the familiarity of verbal material (Schnyer et al., 2004). The findings are also consistent with a recent proposal regarding the potential role of the medial PFC in supporting retrieval (and encoding) of information that is congruent with preexisting knowledge, or schemas (van Kesteren et al., 2010; van Kesteren et al., 2012). Importantly, the present fMRI data demonstrate that the role of the medial PFC in supporting recognition is certainly not homogeneous. Indeed this study suggests that the medial PFC areas and specifically the anterior cingulate cortex (BA 32) and the superior medial PFC (BA 9/10), constitute parts of the familiarity network responsible for coding the degree that the presented visual stimulus matches existing representations. On the other hand, the orbitomedial PFC (BA 10/11) may be a candidate area for supporting a decisional role, irrespective of memory status. This area, located ventrally to the Hippocampus

Overall, the data presented here strongly support the hypothesis that familiarity and novelty signals are computed in separate, nonoverlapping, but interacting, brain systems. This is consistent with a dual-route, rather than a unitary detection mechanism. The results suggest that the components of the dual-route mechanism are primarily the MTL and selective fronto-parietal regions for novelty detection and the Dm-thalamus and selective fronto-parietal regions for familiarity detection. However, we do not propose that these two systems work in isolation; indeed, we suggest that the success of the dual-route mechanism in detecting novelty and familiarity and critically, in computing relative-familiarity, is dependent on their convergence on, and interaction with, a further set of specialized frontoparietal regions. Therefore, a mechanism involving a network of interacting parietal and prefrontal signals, which receive distinct inputs from the MTL and thalamic structures, generates a computation of the relative-familiarity of a stimulus, and therefore underlies our efficient detection of familiarity and novelty in an ever-changing environment. Exactly how this mechanism is modulated by environmental demands, expectations and contexts remains to be explored, but the existence of this kind of mechanism is essential in order to explain the effects of these types of factors on our everyday experiences of novelty and familiarity.

REFERENCES Achim AM, Lepage M. 2005. Dorsolateral prefrontal cortex involvement in memory post-retrieval monitoring revealed in both item and associative recognition tests. NeuroImage 24:1113–1121. Aggleton JP, Brown MW. 1999. Episodic memory, amnesia, and the hippocampal, anterior thalamic axis. Behav Brain Sci 22:425–444. Aggleton JP, O’Mara SM, Vann SD, Wright NF, Tsanov M, Erichsen JT. 2010. Hippocampal–anterior thalamic pathways for memory: Uncovering a network of direct and indirect actions. Eur J Neurosci 31:2292–2307. Brown MW, Aggleton JP. 2001. Recognition memory: What are the roles of the perirhinal cortex and hippocampus? Nat Rev Neurosci 2:51–61. Brown MW, Xiang JZ. 1998. Recognition memory: Neuronal substrates of the judgement of prior occurrence. Prog Neurobiol 55: 149–189. B€ uchel C, Holmes AP, Rees G, Friston KJ. 1998. Characterizing stimulus-response functions using nonlinear regressors in parametric fMRI experiments. NeuroImage 8:140–148.

FAMILIARITY AND NOVELTY DETECTION PROCESSES De Martino B, Fleming SM, Garrett N, Dolan RJ. 2013. Confidence in value-based choice. Nat Neurosci 16:105–110. Eldridge LL, Engel SA, Zeineh MM, Bookheimer SY, Knowlton BJ. 2005. A dissociation of encoding and retrieval processes in the human hippocampus. J Neurosci 25:3280–3286. Fahy FL, Riches LI, Brown MW. 1993. Neuronal activity related to visual recognition memory: Long-term memory and the encoding of recency and familiarity information in the primate anterior and medial inferior temporal and rhinal cortex. Exp Brain Res 96:457– 472. Fernandez G, Tendolkar I. 2006. The rhinal cortex: ‘gatekeeper’ of the declarative memory system. Trends Cogn Sci 10:358–362. Friston KJ, Fletcher P, Josephs O, Holmes A, Rugg MD, Turner R. 1998. Event-related fMRI: Characterizing differential responses. NeuroImage 7:30–40. Friston KJ, Penny WD, Glaser DE. 2005. Conjunction revisited. NeuroImage 25:661–667. Gonsalves BD, Kahn I, Curran T, Norman KA, Wagner AD. 2005. Memory strength and repetition suppression: multimodal imaging of medial temporal cortical contributions to recognition. Neuron 47:751–761. Johnson JD, Muftuler, LT, Rugg MD. 2008. Multiple repetitions reveal functionally and anatomically distinct patterns of hippocampal activity during continuous recognition memory. Hippocampus 18:975–980. Henson RNA, Cansino S, Herron JE, Robb WGK, Rugg MD. 2003. A familiarity signal in human anterior medial temporal cortex? Hippocampus 13:301–304. Herron JE, Henson RNA, Rugg MD. 2004. Probability effects on the neural correlates of retrieval success: An fMRI study. NeuroImage 21:302–310. Izquierdo A, Murray EA. 2010. Functional interaction of medial mediodorsal thalamic nucleus but not nucleus accumbens with amygdala and orbital prefrontal cortex is essential for adaptive response selection after reinforcer devaluation. J Neurosci 30:661–669. Kafkas A, Montaldi D. 2011. Recognition memory strength is predicted by pupillary responses at encoding while fixation patterns distinguish recollection from familiarity. Quarterly J Exp Psychol 64:1971–1989. Kafkas A, Montaldi D. 2012. Familiarity and recollection produce distinct eye movement, pupil and medial temporal lobe responses when memory strength is matched. Neuropsychologia 50:3080– 3093. Meunier M, Bachevalier J, Mishkin M. 1997. Effects of orbital frontal and anterior cingulate lesions on object and spatial memory in rhesus monkeys. Neuropsychologia 35:999–1015. Montaldi D, Spencer TJ, Roberts N, Mayes AR. 2006. The neural system that mediates familiarity memory. Hippocampus 16:504– 520. Petrides M. 2007. The orbitofrontal cortex: Novelty, deviation from expectation, and memory. In: Schoenbaum G, Gottfried JA, Murray EA, Ramus SJ, editors. Linking Affect to Action: Critical Contributions of the Orbitofrontal Cortex. Oxford: Blackwell Publishing. pp 33–53. Rolls ET, Franco L, Stringer SM. 2005. The perirhinal cortex and long-term familiarity memory. Quarterly J Exp Psychol Sect B 58: 234–245.

527

Rugg MD, Henson RNA, Robb WGK. 2003. Neural correlates of retrieval processing in the prefrontal cortex during recognition and exclusion tasks. Neuropsychologia 41:40–52. Rutishauser U, Mamelak AN, Schuman EM. 2006. Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron 49:805–813. Rutishauser U, Schuman EM, Mamelak AN. 2008. Activity of human hippocampal and amygdala neurons during retrieval of declarative memories. Proc Natl Acad Sci USA 105:329–334. Schnyer DM, Verfaellie M, Alexander MP, LaFleche G, Nicholls L, Kaszniak AW. 2004. A role for right medial prefrontal cortex in accurate feeling-of-knowing judgments: Evidence from patients with lesions to frontal cortex. Neuropsychologia 42:957–966. Staresina BP, Fell J, Do Lam ATA, Axmacher N, Henson RN. 2012. Memory signals are temporally dissociated in and across human hippocampus and perirhinal cortex. Nat Neurosci 15:1167–1173. Staudigl T, Zaehle T, Voges J, Hanslmayr S, Esslinger C, Hinrichs H, Schmitt FC, Heinze H-J, Richardson-Klavehn A. 2012. Memory signals from the thalamus: Early thalamocortical phase synchronization entrains gamma oscillations during long-term memory retrieval. Neuropsychologia 50:3519–3527. Strange BA, Hurlemann R, Duggins A, Heinze H-J, Dolan RJ. 2005. Dissociating intentional learning from relative novelty responses in the medial temporal lobe. NeuroImage 25:51–62. Suzuki M, Johnson JD, Rugg MD. 2011. Decrements in hippocampal activity with item repetition during continuous recognition: An fMRI study. J Cogn Neurosci 23:1522–1532. Tsujimoto S, Genovesio A, Wise SP. 2009. Monkey orbitofrontal cortex encodes response choices near feedback time. J Neurosci 29: 2569–2574. Tulving E, Markowitsch HJ, Craik FIM, Habib R, Houle S. 1996. Novelty and familiarity activations in PET studies of memory encoding and retrieval. Cerebral Cortex 6:71–79. Van Der Werf YD, Jolles J, Witter MP, Uylings HBM. 2003. Contributions of thalamic nuclei to declarative memory functioning. Cortex 39(4-5):1047–1062. van Kesteren MTR, Rijpkema M, Ruiter DJ, Fernandez G. 2010. Retrieval of associative information congruent with prior knowledge is related to increased medial prefrontal activity and connectivity. J Neurosci 30:15888–15894. van Kesteren MTR, Ruiter DJ, Fernandez G, Henson RN. 2012. How schema and novelty augment memory formation. Trends Neurosci 35:211–219. Viskontas IV, Knowlton BJ, Steinmetz PN, Fried I. 2006. Differences in mnemonic processing by neurons in the human hippocampus and parahippocampal regions. J Cogn Neurosci 18:1654–1662. Xiang JZ, Brown MW. 1998. Differential neuronal encoding of novelty, familiarity and recency in regions of the anterior temporal lobe. Neuropharmacology 37(4-5):657–676. Xiang JZ, Brown MW. 2004. Neuronal responses related to long-term recognition memory processes in prefrontal cortex. Neuron 42: 817–829. Yassa MA, Stark CEL. 2008. Multiple signals of recognition memory in the medial temporal lobe. Hippocampus 18:945–954. Yonelinas AP, Otten LJ, Shaw KN, Rugg MD. 2005. Separating the brain regions involved in recollection and familiarity in recognition memory. J Neurosci 25:3002–3008.

Hippocampus