Bruno Averbeck, Ph.D.
The section on Learning and Decision making studies the neural circuitry that underlies reinforcement learning. Reinforcement learning (RL) is the behavioral process of learning to make advantageous choices. While some preferences are innate, many are learned over time. How do we learn what we like and what we want to avoid? The lab uses a combination of experiments in in-vivo model systems, human participants including patients and computational modeling. We examine several facets of the learning problem including learning from gains vs. losses, learning to select rewarding actions vs. learning to select rewarding objects, and the explore-exploit trade-off. The explore-exploit trade-off describes a fundamental problem in learning. Should you try every restaurant when visiting a new city, or explore a small set of them and then return to your favorite several times?
Standard models of RL assume that dopamine neurons code reward prediction errors (RPEs; the difference between the size of the reward received and the reward that was expected following a choice). These RPEs are then communicated to the basal ganglia, specifically the striatum, because of its substantial dopamine innervation. This dopamine signal drives learning in frontal-striatal and amygdala-striatal circuits, such that choices that have previously been rewarded lead to larger neural responses in the striatum, and choices that have previously not been rewarded (or have been punished) lead to smaller responses. Thus, the striatal neurons come to represent the values of choices. They signal a high-value choice with higher activity and this higher activity drives decision processes. These models often mention a potential role for the amygdala, without formally incorporating it. They further suggest a general role for the ventral-striatum (VS) in representing values of decisions, whether they are decisions about actions or decisions about objects and independent of whether values are related to reward magnitude or probability.
In contrast to the standard model, we have recently shown that the amygdala has a larger role in RL than the VS (Costa VD et al., Neuron, 2016). In addition, the role of the VS depends strongly on the reward environment. When rewards are predictable, the VS has almost no role in learning whereas when rewards are less predictable the VS plays a larger role. This data outlines a more specific role for the VS in RL than is attributed to it by current models. Given that the VS has been implicated in depression, particularly adolescent depression, this delineation of the contribution of the VS to normal behavior may help inform hypotheses about the mechanisms and circuitry underlying depression.
Dr. Averbeck obtained a B.S. in Electrical Engineering from the University of Minnesota in 1994, worked in industry for three years, then returned to Minnesota to complete a doctorate in neuroscience under Dr. Apostolos Georgopoulos. Awarded a Ph.D. 2001, with a dissertation on Neural Mechanisms of Copying Geometrical Shapes . For his postdoctoral studies, Dr. Averbeck joined the laboratory Dr. Daeyeol Lee at the University of Rochester, where he studied neural mechanisms underlying sequential learning, coding of vocalizations, and population coding. In 2006, as a Senior Lecturer at University College London, he began using neuroimaging of human study participants to investigate the role of frontal-striatal circuits in learning. Dr. Averbeck joined the NIMH Intramural Research Program as a Principal Investigator in 2009. A tenured member of the faculty since 2016, he is Chief of the Section on Learning and Decision Making.
Averbeck BB (2022). Pruning recurrent neural networks replicates adolescent changes in working memory and reinforcement learning. Proc Natl Acad Sci U S A 119, e2121331119. https://doi.org/10.1073/pnas.2121331119. [Pubmed Link ]
Tang H, Costa VD, Bartolo R, Averbeck BB (2022). Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior. Cell Rep 38, 110198. https://doi.org/10.1016/j.celrep.2021.110198. [Pubmed Link ]
Averbeck BB, Murray EA (2020). Hypothalamic Interactions with Large-Scale Neural Circuits Underlying Reinforcement Learning and Motivated Behavior. Trends Neurosci 43, 681-694. https://doi.org/10.1016/j.tins.2020.06.006. [Pubmed Link ]
Bartolo R, Averbeck BB (2020). Prefrontal Cortex Predicts State Switches during Reversal Learning. Neuron 106, 1044-1054.e4. https://doi.org/10.1016/j.neuron.2020.03.024. [Pubmed Link ]
Costa VD, Mitz AR, Averbeck BB (2019). Subcortical Substrates of Explore-Exploit Decisions in Primates. Neuron 103, 533-545.e5. https://doi.org/10.1016/j.neuron.2019.05.017. [Pubmed Link ]