Brain Advance Access published March 31, 2015 doi:10.1093/brain/awv084
BRAIN 2015: Page 1 of 3
| e1
LETTER TO THE EDITOR Differential functions of ventral and dorsal striatum Yashar Zeighami1 and Ahmed A. Moustafa2 1 McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, 3801 University Street, Montreal, QC, Canada 2 School of Social Sciences and Psychology and Marcs Institute for Brain and Behaviour, University of Western Sydney, Sydney, New South Wales, Australia Correspondence to: Dr Ahmed Moustafa, School of Social Sciences and Psychology and Marcs Institute for Brain and Behaviour, University of Western Sydney, Sydney, NSW, Australia E-mail: [email protected]
Vo et al. also analysed learning and performance processes using computational models. Regression modelling of current choice based on the previous choice and the history of rewards revealed the following: in action-value learning tasks, both Patient XG and healthy controls demonstrated normal patterns of reinforcement learning (i.e. exponential decline in weight of past trials). In contrast, Patient XG (compared to controls) did not show this pattern in stimulus-value tasks. In fact, the regression weights were not significantly different from zero, which indicated lack of learning. Consistent with this, a three-parameter reinforcement learning model explained choice behaviour better than the null model for controls in both tasks. For Patient XG, the model explained choice behaviour better for the action-value tasks, but could not explain the behaviour in stimulus-value tasks. These results address two different aspects of reinforcement learning: (i) the anatomical dissociation of the ventral striatum and dorsal striatum in learning; and (ii) the behavioural dissociation between stimulus-value versus actionvalue learning. These two aspects are addressed separately hereafter. Studies investigating functional segregation of the striatum suggest that the ventral striatum is more involved in goal-directed learning, whereas the dorsal striatum is more involved in action related habitual learning (Balleine et al., 2007; Balleine and O’Doherty, 2010; Redgrave et al.,
ß The Author (2015). Published by Oxford University Press on behalf of the Guarantors of Brain. All rights reserved. For Permissions, please email: [email protected]
Downloaded from by guest on June 6, 2015
Sir, The case study by Vo and colleagues (2014) aims to address the differential roles of ventral versus dorsal striatum in learning, specifically, whether they are essential for learning or simply involved in it. The authors reported a dissociation between action-value (based on the outcomes, values will be assigned to actions) and stimulus-value learning (values will be associated with the stimuli), and how impairment of the dorsal striatum will affect each of these processes. To achieve this, Vo and colleagues tested a patient (known as XG) who has bilateral damage to the dorsal striatum, while the ventral striatum including the nucleus accumbens is spared. To compare XG’s performance in different tasks with a healthy population statistically, the researchers tested 11 matched control subjects. Among the seven reinforcement learning tasks employed, three could be solved only by learning stimulus-values, one only by learning action-values, and the remaining tasks, with either strategy. Surprisingly, they found that Patient XG was able to learn all the tasks involving action-value learning and his performance resembled those of healthy controls. However, he was impaired at learning the tasks which could only be learned using stimulus values; his performance in those tasks was significantly poorer than controls and was no different from random.
e2
| BRAIN 2015: Page 2 of 3
the potential alterations in cortico-striatal connections in Patient XG may elaborate on this knowledge gap. Various theoretical models suggesting dissociation between stimulus- and action-value learning have been proposed. However, existing models contradict findings reported by Vo et al. For example, Piray and colleagues (2014) provided evidence for dissociation in patients with Parkinson’s disease ON/OFF dopamine medication, those with impulse control disorders and controls using an Actor/Critic model. Their findings suggest that the ventral striatum plays a role in stimulus-value learning while the dorsal striatum in action-value learning whereas Vo et al. suggest dorsal striatum being necessary for stimulus-value learning process and not involved in action-value learning. Differences in findings might be due to the structure of the task used in the study by Vo and colleagues. In actionvalue learning tasks, Patient XG was presented with a single state (trial type) whereas the stimulus-value learning tasks consisted of multiple states. Impairment in the latter task may therefore reflect the involvement of the dorsal striatum in state representation, and its interaction with working memory and not necessarily in learning stimulusvalue learning. Future study designs should introduce multiple state action-learning tasks and examine whether the impairment occurs in state representation or stimulus-value learning.
References Balleine BW, Delgado MR, Hikosaka O. The role of the dorsal striatum in reward and decision-making. J Neurosci 2007; 27: 8161–5. Balleine BW, O’Doherty JP. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010; 35: 48–69. Camille N, Tsuchida A, Fellows LK. Double dissociation of stimulusvalue and action-value learning in humans with orbitofrontal or anterior cingulate cortex damage. J Neurosci 2011; 31: 15048–52. Everitt BJ, Robbins TW. From the ventral to the dorsal striatum: devolving views of their roles in drug addiction. Neurosci Biobehav Rev 2013; 37 (9 Pt A): 1946–54. Garrison J, Erdeniz B, Done J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci Biobehav Rev 2013; 37: 1297–310. Haber SN, Calzavara R. The cortico-basal ganglia integrative network: the role of the thalamus. Brain Res Bull 2009; 78: 69–74. Joel D, Niv Y, Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 2002; 15: 535–47. Niv Y. Reinforcement learning in the brain. J Math Psychol 2009; 53: 139–54. O’Doherty , Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science (New York, NY) 2004; 304: 452–4. Piray P, Zeighami Y, Bahrami F, Eissa AM, Hewedi DH, Moustafa AA. Impulse control disorders in Parkinson’s disease are associated with dysfunction in stimulus valuation but not action valuation. J Neurosci 2014; 34: 7814–24. Porrino LJ, Lyons D, Smith HR, Daunais JB, Nader MA. Cocaine selfadministration produces a progressive involvement of limbic, association, and sensorimotor striatal domains. J Neurosci 2004; 24: 3554–62.
Downloaded from by guest on June 6, 2015
2010); this is consistent with findings in anatomical connectivity of the striatum. Moving in the ventral-dorsal axis, the cortico-striatal connectivity changes from limbic/cognitive areas (ventromedial prefrontal cortex, orbitofrontal cortex) towards associative (dorsal anterior cingulate cortex, dorsal prefrontal cortex) and finally motor areas (supplementary motor area) (Postuma and Dagher, 2006; Haber and Calzavara, 2009). These theoretical findings have been validated in rodent addiction studies where goal-directed behaviour subserved by the ventral striatum shifts towards habitual behaviour by the dorsal striatum as addiction progresses (Porrino et al., 2004; Everitt and Robbins, 2013). These findings can explain Patient XG’s spared performance in tasks involving action-value learning, but fails to address the impaired stimulus-value learning. Stimulus- and action-value learning dissociations have also been studied in cortical areas. Orbitofrontal cortex lesions disrupt stimulus-value but not action-value learning, whereas dorsal anterior cingulate lesions show the opposite effects (Camille et al., 2011). Based on these findings and the previously discussed anatomical connections between dorsal striatum and dorsal anterior cingulate cortex, dorsal striatum impairment in Patient XG should result in action-value learning impairment while stimulus-value learning remains intact, i.e. the exact opposite of what was observed by Vo et al. Using functional and structural connectivity analysis and functional MRI during task performance in Patient XG can enhance our knowledge about the cortico-striatal connectivity and the brain structures involved in each learning process and clarify the source of this contradiction. In the striatum, stimulus- and action-value learning dissociation has been studied using conditioning paradigms including Pavlovian (stimulus-outcome association) and instrumental (stimulus-action-outcome association) as well as Actor/Critic models. The Actor/Critic model of the striatum suggests that ventral striatum plays a similar role to the ‘critic’ by calculating the prediction errors, and dorsal striatum is similar to the ‘actor’ that is involved in action selection (Joel et al., 2002; Niv, 2009). Consistent with these results, O’Doherty et al. (2004) found that dorsal striatum is involved in instrumental conditioning while ventral striatum is involved in both instrumental and Pavlovian conditioning. Schonberg and colleagues also showed a selective impairment of prediction error-related activity in dorsolateral but not ventral striatum in patients with Parkinson’s disease, confirming dissociation between the two areas (Schonberg et al., 2010). In contrast, a metaanalysis by Garrison and colleagues found that the ventral striatum was chiefly responsible for instrumental condition specific activity, whereas mainly cortical areas were identified in Pavlovian condition (Garrison et al., 2013). These findings suggest that the ventral striatum may be responsible for intact action-value learning but does not explain the impaired stimulus-value learning in Patient XG suffering from bilateral dorsal striatum damage. Investigations of
Letter to the Editor
Letter to the Editor Postuma RB, Dagher A. Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cereb Cortex 2006; 16: 1508–21. Redgrave P, Rodriguez M, Smith Y, Rodriguez-Oroz MC, Lehericy S, Bergman H, et al. Goal-directed and habitual control in the basal ganglia: implications for Parkinson’s disease. Nat Rev 2010; 11: 760–72.
BRAIN 2015: Page 3 of 3
| e3
Schonberg T, O’Doherty JP, Joel D, Inzelberg R, Segev Y, Daw ND. Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson’s disease patients: evidence from a model-based fMRI study. Neuroimage 2010; 49: 772–81. Vo K, Rutledge RB, Chatterjee A, Kable JW. Dorsal striatum is necessary for stimulus-value but not action-value learning in humans. Brain 2014; 137 (Pt 12): 3129–35.
Downloaded from by guest on June 6, 2015
BRAIN 2015: Page 1 of 3
| e1
LETTER TO THE EDITOR Differential functions of ventral and dorsal striatum Yashar Zeighami1 and Ahmed A. Moustafa2 1 McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, 3801 University Street, Montreal, QC, Canada 2 School of Social Sciences and Psychology and Marcs Institute for Brain and Behaviour, University of Western Sydney, Sydney, New South Wales, Australia Correspondence to: Dr Ahmed Moustafa, School of Social Sciences and Psychology and Marcs Institute for Brain and Behaviour, University of Western Sydney, Sydney, NSW, Australia E-mail: [email protected]
Vo et al. also analysed learning and performance processes using computational models. Regression modelling of current choice based on the previous choice and the history of rewards revealed the following: in action-value learning tasks, both Patient XG and healthy controls demonstrated normal patterns of reinforcement learning (i.e. exponential decline in weight of past trials). In contrast, Patient XG (compared to controls) did not show this pattern in stimulus-value tasks. In fact, the regression weights were not significantly different from zero, which indicated lack of learning. Consistent with this, a three-parameter reinforcement learning model explained choice behaviour better than the null model for controls in both tasks. For Patient XG, the model explained choice behaviour better for the action-value tasks, but could not explain the behaviour in stimulus-value tasks. These results address two different aspects of reinforcement learning: (i) the anatomical dissociation of the ventral striatum and dorsal striatum in learning; and (ii) the behavioural dissociation between stimulus-value versus actionvalue learning. These two aspects are addressed separately hereafter. Studies investigating functional segregation of the striatum suggest that the ventral striatum is more involved in goal-directed learning, whereas the dorsal striatum is more involved in action related habitual learning (Balleine et al., 2007; Balleine and O’Doherty, 2010; Redgrave et al.,
ß The Author (2015). Published by Oxford University Press on behalf of the Guarantors of Brain. All rights reserved. For Permissions, please email: [email protected]
Downloaded from by guest on June 6, 2015
Sir, The case study by Vo and colleagues (2014) aims to address the differential roles of ventral versus dorsal striatum in learning, specifically, whether they are essential for learning or simply involved in it. The authors reported a dissociation between action-value (based on the outcomes, values will be assigned to actions) and stimulus-value learning (values will be associated with the stimuli), and how impairment of the dorsal striatum will affect each of these processes. To achieve this, Vo and colleagues tested a patient (known as XG) who has bilateral damage to the dorsal striatum, while the ventral striatum including the nucleus accumbens is spared. To compare XG’s performance in different tasks with a healthy population statistically, the researchers tested 11 matched control subjects. Among the seven reinforcement learning tasks employed, three could be solved only by learning stimulus-values, one only by learning action-values, and the remaining tasks, with either strategy. Surprisingly, they found that Patient XG was able to learn all the tasks involving action-value learning and his performance resembled those of healthy controls. However, he was impaired at learning the tasks which could only be learned using stimulus values; his performance in those tasks was significantly poorer than controls and was no different from random.
e2
| BRAIN 2015: Page 2 of 3
the potential alterations in cortico-striatal connections in Patient XG may elaborate on this knowledge gap. Various theoretical models suggesting dissociation between stimulus- and action-value learning have been proposed. However, existing models contradict findings reported by Vo et al. For example, Piray and colleagues (2014) provided evidence for dissociation in patients with Parkinson’s disease ON/OFF dopamine medication, those with impulse control disorders and controls using an Actor/Critic model. Their findings suggest that the ventral striatum plays a role in stimulus-value learning while the dorsal striatum in action-value learning whereas Vo et al. suggest dorsal striatum being necessary for stimulus-value learning process and not involved in action-value learning. Differences in findings might be due to the structure of the task used in the study by Vo and colleagues. In actionvalue learning tasks, Patient XG was presented with a single state (trial type) whereas the stimulus-value learning tasks consisted of multiple states. Impairment in the latter task may therefore reflect the involvement of the dorsal striatum in state representation, and its interaction with working memory and not necessarily in learning stimulusvalue learning. Future study designs should introduce multiple state action-learning tasks and examine whether the impairment occurs in state representation or stimulus-value learning.
References Balleine BW, Delgado MR, Hikosaka O. The role of the dorsal striatum in reward and decision-making. J Neurosci 2007; 27: 8161–5. Balleine BW, O’Doherty JP. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010; 35: 48–69. Camille N, Tsuchida A, Fellows LK. Double dissociation of stimulusvalue and action-value learning in humans with orbitofrontal or anterior cingulate cortex damage. J Neurosci 2011; 31: 15048–52. Everitt BJ, Robbins TW. From the ventral to the dorsal striatum: devolving views of their roles in drug addiction. Neurosci Biobehav Rev 2013; 37 (9 Pt A): 1946–54. Garrison J, Erdeniz B, Done J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci Biobehav Rev 2013; 37: 1297–310. Haber SN, Calzavara R. The cortico-basal ganglia integrative network: the role of the thalamus. Brain Res Bull 2009; 78: 69–74. Joel D, Niv Y, Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 2002; 15: 535–47. Niv Y. Reinforcement learning in the brain. J Math Psychol 2009; 53: 139–54. O’Doherty , Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science (New York, NY) 2004; 304: 452–4. Piray P, Zeighami Y, Bahrami F, Eissa AM, Hewedi DH, Moustafa AA. Impulse control disorders in Parkinson’s disease are associated with dysfunction in stimulus valuation but not action valuation. J Neurosci 2014; 34: 7814–24. Porrino LJ, Lyons D, Smith HR, Daunais JB, Nader MA. Cocaine selfadministration produces a progressive involvement of limbic, association, and sensorimotor striatal domains. J Neurosci 2004; 24: 3554–62.
Downloaded from by guest on June 6, 2015
2010); this is consistent with findings in anatomical connectivity of the striatum. Moving in the ventral-dorsal axis, the cortico-striatal connectivity changes from limbic/cognitive areas (ventromedial prefrontal cortex, orbitofrontal cortex) towards associative (dorsal anterior cingulate cortex, dorsal prefrontal cortex) and finally motor areas (supplementary motor area) (Postuma and Dagher, 2006; Haber and Calzavara, 2009). These theoretical findings have been validated in rodent addiction studies where goal-directed behaviour subserved by the ventral striatum shifts towards habitual behaviour by the dorsal striatum as addiction progresses (Porrino et al., 2004; Everitt and Robbins, 2013). These findings can explain Patient XG’s spared performance in tasks involving action-value learning, but fails to address the impaired stimulus-value learning. Stimulus- and action-value learning dissociations have also been studied in cortical areas. Orbitofrontal cortex lesions disrupt stimulus-value but not action-value learning, whereas dorsal anterior cingulate lesions show the opposite effects (Camille et al., 2011). Based on these findings and the previously discussed anatomical connections between dorsal striatum and dorsal anterior cingulate cortex, dorsal striatum impairment in Patient XG should result in action-value learning impairment while stimulus-value learning remains intact, i.e. the exact opposite of what was observed by Vo et al. Using functional and structural connectivity analysis and functional MRI during task performance in Patient XG can enhance our knowledge about the cortico-striatal connectivity and the brain structures involved in each learning process and clarify the source of this contradiction. In the striatum, stimulus- and action-value learning dissociation has been studied using conditioning paradigms including Pavlovian (stimulus-outcome association) and instrumental (stimulus-action-outcome association) as well as Actor/Critic models. The Actor/Critic model of the striatum suggests that ventral striatum plays a similar role to the ‘critic’ by calculating the prediction errors, and dorsal striatum is similar to the ‘actor’ that is involved in action selection (Joel et al., 2002; Niv, 2009). Consistent with these results, O’Doherty et al. (2004) found that dorsal striatum is involved in instrumental conditioning while ventral striatum is involved in both instrumental and Pavlovian conditioning. Schonberg and colleagues also showed a selective impairment of prediction error-related activity in dorsolateral but not ventral striatum in patients with Parkinson’s disease, confirming dissociation between the two areas (Schonberg et al., 2010). In contrast, a metaanalysis by Garrison and colleagues found that the ventral striatum was chiefly responsible for instrumental condition specific activity, whereas mainly cortical areas were identified in Pavlovian condition (Garrison et al., 2013). These findings suggest that the ventral striatum may be responsible for intact action-value learning but does not explain the impaired stimulus-value learning in Patient XG suffering from bilateral dorsal striatum damage. Investigations of
Letter to the Editor
Letter to the Editor Postuma RB, Dagher A. Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cereb Cortex 2006; 16: 1508–21. Redgrave P, Rodriguez M, Smith Y, Rodriguez-Oroz MC, Lehericy S, Bergman H, et al. Goal-directed and habitual control in the basal ganglia: implications for Parkinson’s disease. Nat Rev 2010; 11: 760–72.
BRAIN 2015: Page 3 of 3
| e3
Schonberg T, O’Doherty JP, Joel D, Inzelberg R, Segev Y, Daw ND. Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson’s disease patients: evidence from a model-based fMRI study. Neuroimage 2010; 49: 772–81. Vo K, Rutledge RB, Chatterjee A, Kable JW. Dorsal striatum is necessary for stimulus-value but not action-value learning in humans. Brain 2014; 137 (Pt 12): 3129–35.
Downloaded from by guest on June 6, 2015
