Positive Valence Systems: Workshop Proceedings

June 19–21, 2011
Rockville, Maryland

Introduction

The Research Domain Criteria (RDoC) project is designed to implement Strategy 1.4 of the NIMH Strategic Plan: “Develop, for research purposes, new ways of classifying mental disorders based on dimensions of observable behavior and neurobiological measures.” NIMH intends RDoC to serve as a research framework encouraging new approaches to research on mental disorders, in which fundamental dimensions that cut across traditional disorder categories are used as the basis for grouping patients in clinical studies. RDoC represents an inherently translational approach, considering psychopathology in terms of dysregulation and dysfunction in fundamental aspects of behavior as established through basic neuroscience and behavioral science research. The major RDoC framework consists of a matrix where the rows represent specified functional Constructs, concepts summarizing data about a specified functional dimension of behavior, characterized in aggregate by the genes, molecules, circuits, etc., responsible for it. Constructs are in turn grouped into higher-level Domains of functioning, reflecting contemporary knowledge about major systems of cognition, motivation, and social behavior. In its present form, there are five Domains in the RDoC matrix: Negative Valence Systems, Positive Valence Systems, Cognitive Systems, Systems for Social Processes, and Arousal/Regulatory Systems. The matrix columns specify Units of Analysis used to study the Constructs, and include genes, molecules, cells, circuits, physiology (e.g., heart rate or event-related potentials), behavior, and self-reports. The matrix also has a separate column to specify well-validated paradigms used in studying each Construct.

The RDoC matrix is being developed to serve as a heuristic, subject to change with scientific advances from the field. To “build the matrix,” NIMH is bringing together leading experts to coalesce and articulate the state of knowledge for each of the five domains. Six meetings are planned: this workshop, focused on the Positive Valence Systems (PVS) Domain, was the third in the series.

For detailed information about RDoC and the updated matrix, please see the RDoC web page. To comment on any aspect of the proceedings, send email to the following: rdoc@mail.nih.gov.

Workshop Proceedings

This workshop on the Positive Valence Systems (PVS) Domain was convened to define the Constructs to be included within the Domain, enumerate what is known about the Units of Analysis for the Constructs, list questions that remain unanswered, and outline potential avenues of research that will answer these questions. The goals of this workshop were to: 1) arrive at a set of Constructs in the PVS Domain and an agreed-upon definition for each, incorporating how the field views each Construct and how it is distinguished from other similar Constructs; and, 2) provide an annotated listing (based on current knowledge) of the elements that would populate the RDoC matrix with respect to the genes, molecules, cells, circuits, physiology, and self-reports comprising each PVS Construct, as well as identify promising and reliable behavioral tasks that can be used to assess function within a Construct. The entries in the various Units of Analysis may be considered as priority elements for classifying research participants in clinical research grant applications.

Preliminary Discussion

The NIMH RDoC working group had initially proposed three draft Constructs within the PVS Domain for consideration during the workshop: Reward Seeking, Consummatory Behavior, and Reward/Habit Learning. Workshop members were invited to evaluate, modify, and define the Constructs, or to consider new Constructs if warranted.

Based on each individual’s scientific expertise, the workshop participants were assigned to one of two “construct groups”: (1) Reward Seeking and Consummatory Behavior, moderated by Diego Pizzagalli, PhD; and (2) Reward/Habit Learning, moderated by Ann Graybiel, PhD. Each group was initially tasked with evaluating its assigned draft Construct(s) to determine which constructs were sufficient and which aspects might benefit from revision—e.g., by refining the nature of the Constructs, adding additional Constructs, etc. Each Construct group was subdivided into two parallel breakout groups to facilitate discussion and encourage exploration of divergent opinions; Kelly Klump, PhD and Todd Gould, MD moderated two of the smaller, initial breakout groups. Following breakout group meetings, the Construct groups (and then the entire group) reassembled for further discussion and refinement of the products as necessary.

As a result of this process, the workshop participants generated a total of five Constructs under the PVS Domain. The definitions of these Constructs are provided below, followed by a summary of the discussions of the two Construct groups.

Construct definitions

1. Approach motivation: A multi-faceted Construct involving mechanisms/processes that regulate the direction and maintenance of approach behavior influenced by pre-existing tendencies, learning, memory, stimulus characteristics, and deprivation states. Approach behavior can be directed toward innate or acquired cues (i.e., unconditioned vs. learned stimuli), implicit or explicit goals; it can consist of goal-directed or Pavlovian conditioned responses. Component processes include reward valuation, effort valuation/willingness to work, expectancy/reward prediction error, and action selection/decision making.
   1. Reward valuation. Processes by which the probability and benefits of a prospective outcome are computed and calibrated by reference to external information, social context (e.g., group input, counterfactual comparisons), and/or prior experience. This calibration is influenced by pre-existing biases, learning, memory, stimulus characteristics, and deprivation states. Reward valuation may involve the assignment of incentive salience to stimuli.
   2. Effort valuation/Willingness to work. Processes by which the cost(s) of obtaining an outcome is computed; tendency to overcome response costs to obtain a reinforcer.
   3. Expectancy/Reward prediction error. A state triggered by exposure to internal or external stimuli, experiences or contexts that predict the possibility of reward. Reward expectation can alter the experience of an outcome and can influence the use of resources (e.g., cognitive resources).
   4. Action selection/Preference-based decision making. Processes involving an evaluation of costs/benefits and occurring in the context of multiple potential choices being available for decision-making.
2. Initial responsiveness to reward attainment: Mechanisms/processes associated with hedonic responses—as reflected in subjective experiences, behavioral responses, and/or engagement of the neural systems to a positive reinforcer—and culmination of reward seeking.
3. Sustained/Longer-term responsiveness to reward attainment: Mechanisms/processes associated with the termination of reward seeking, e.g., satisfaction, satiation, regulation of consummatory behavior.
4. Reward Learning: A process by which organisms acquire information about stimuli, actions, and contexts that predict positive outcomes, and by which behavior is modified when a novel reward occurs or outcomes are better than expected. Reward learning is a type of reinforcement learning, and similar processes may be involved in learning related to negative reinforcement.
5. Habit: Sequential, repetitive, motor, or cognitive behaviors elicited by external or internal triggers that, once initiated, can go to completion without constant conscious oversight. Habits can be adaptive by virtue of freeing up cognitive resources. Habit formation is a frequent consequence of reward learning, but its expression can become resistant to changes in outcome value. Related behaviors could be pathological expression of a process that under normal circumstances subserves adaptive goals.

Summary of Construct Group Deliberations

The material in the following sections is intended to provide background and context for the final Construct definitions as provided above. Workshop participants discussed a variety of considerations and perspectives, and the resulting set of Constructs and their definitions emerged.

Reward Seeking and Consummatory Behavior Group

Construct and Definition Development

In the two breakout groups and as a full Construct group, participants discussed the distinctions among many different theoretical concepts and behaviors that have been the focus of basic and clinical research, including consummatory behavior, satiety, reward seeking, reward anticipation, the “wanting” versus “liking” distinction, incentive motivation, and incentive salience. The group discussed the temporal and theoretical relationships between these concepts, the availability of methods for measuring them in human and non-human animals, and various ways in which they could be incorporated into a set of RDoC Constructs.

Participants discussed whether motivation (or a specific type of motivation) might be a suitable Construct for the Positive Valence Systems Domain or whether, like arousal, motivation is a cross-cutting concept that is not central to positive valence specifically. This discussion reflected recent findings in the literature that many relevant brain areas that have traditionally been viewed as involved primarily with either negative valence or positive valence (such as the amygdala and nucleus accumbens, respectively) are now known to contribute to both types of motivational processes. Participants decided that “Approach” was a common element in the Constructs generated by the two breakout groups; therefore, within the context of the Positive Valence Systems Domain, the group discussed specifying “Approach Motivation” as a Construct. The group discussed a broad interpretation of approach-related behaviors that would include those that are directed toward innately rewarding stimuli, or toward stimuli that have been associated with reward through learning. It was also noted that approach motivation may be conscious or unconscious, and is not necessarily limited to achievement of declarative goals. Appetitive Motivation was also discussed, but there was concern that this term emphasized the biological drives necessary for sustaining homeostasis, whereas a less exclusive Construct such as Approach Motivation would include the disposition to seek out other types of reinforcers (e.g., social interactions, sensation seeking). It was noted that not all forms of approach are positive (for example, approaching someone in order to harm them) and that similarities have been found in EEG activity associated with approach and with anger; therefore, an area for possible future research would be to investigate the dissociations among different types of positive and negative approach behaviors.

The proposed “Reward Seeking” Construct was discussed and a definition was drafted, but it was noted that reward seeking consisted of various cognitive and behavioral processes that might be better captured by breaking down the elements into subcomponents under the Construct of Approach Motivation.

There was an extensive discussion of valuation and the differences between valuation and other concepts, such as reward expectancy. Participants voiced different opinions about whether valuation would be more appropriately organized as a component under the more general approach motivation Construct. There was a proposal for reward expectancy to be considered part of approach motivation, but also that reward expectancy be kept distinct from valuation. Ultimately, it was agreed that valuation would be included as one of a few subcomponents of approach motivation, although it was noted that valuation (as well as willingness to work for reward) is also important in learning and decision making. There was some agreement that broad Constructs with components or subcomponents would provide an appropriate balance between comprehensiveness and specificity for shaping the direction of future research, such that investigators could identify specific, dissociable subcomponents that are amenable to future study.

The group discussed the temporal changes in behavior and motivation that occur with initial receipt of a reward or positive outcome, compared to those that occur with subsequent rewards/positive outcomes; separate Constructs for initial reward receipt and sustained reward were defined. In this context, the group discussed consummatory behavior. The important distinction between “consumption” (referring to eating, drinking, ingesting, or otherwise using a resource) and “consummation” (referring to completion or achievement) was noted. There was some agreement that consummatory behavior is potentially an important element in the “behavior” column of the matrix, but is not a satisfactory Construct on its own. Consummatory behavior can be driven by a variety of different motivations, and not all consummatory acts are hedonic. Similarly, hedonic impact is not derived only from consummation. Moreover, there can be pathological dissociation between attainment of rewards/reinforcement and hedonic impact. The group wanted a Construct that would capture the organism’s ability to experience reward in a given context and/or the hedonic impact that accompanies a reduction in the distance between a desired state and current state (given that reward magnitude can be a relative measure). Various potential concepts, including reward receipt, gain, satisfaction and satiation, and hedonic capacity were discussed. Focusing on responses associated with receipt of rewarding outcomes, the group agreed to include “Initial responsiveness to reward attainment” as a Construct at this stage of RDoC, with the intention of capturing the positive hedonic impact of reward, and dissociating this Construct from the achievement of physiological homeostasis that results from an organism meeting its basic biological needs.

The group discussed whether there are dissociable neural circuits for reward taking (i.e., passive receipt of reward or receipt following a minimal response), consummation, and/or reward seeking (i.e., the exertion of effort toward achieving a reward). There was interest in including ideas about effort, based on recent work about the role of dopamine in reward seeking. It was suggested that this aspect could be captured by a Construct related to the termination of approach. It was noted that approach might be terminated due to a variety of factors, including satiety and reward devaluation. The group discussed the dissociation between hedonic impact and satisfaction or satiety (e.g., an animal receives sucrose and has hedonic impact, but not satiety, or a human receives a reward, and his/her subsequent effort increases), and thus considered the possibility of a separate Construct for satiety; however, there was some uncertainty about whether there is an identifiable neural circuit for satiety. These various ideas informed the decision to create the Effort Valuation/Willingness to Work subcomponent of the Approach Motivation Construct and the “Sustained/longer-term responsiveness to reward attainment” Construct. There was agreement that “sustained” is intended to refer to responses that are more prolonged than immediate (i.e., differentiating the behaviors and responses subsequent to the first exposure to a hedonic stimulus, which are shaped by different factors and circuits than are those involved in the initial exposure), but not “long term.” The group noted the need for more research on the interface between initial and sustained responses to hedonic/rewarding stimuli and outcomes.

In the course of defining the Constructs and beginning to populate the matrix, the workshop participants discussed the importance of being able to translate the Constructs and elements into research focused on clinical populations and clinical phenomena. Participants observed that the RDoC matrix will need to address the problems that patients experience, both for current research issues and for the eventual clinical utility to which RDoC aspires. While the RDoC framework encourages clarity and specificity, the participants recognized that, clinical presentations are complicated and change over time. The example of anhedonia was discussed. In terms of the PVS Constructs, anhedonia can be thought of as a series of decisions; however, different patients may be anhedonic for different reasons and may arrive at the state of anhedonia via different behavioral, cognitive and/or neurophysiological “paths.” Similarly, reduced cortical thickness is observed in individuals with various psychiatric illnesses, but this may result from disparate neurodevelopmental and neurophysiological causes. The effects of sex and age/development are also important factors to consider with regard to each of the Constructs, and should be considered in study design. The group acknowledged the tension between optimizing the immediate clinical relevance of the PVS Constructs, versus going beyond the current thinking on psychopathology, which has not proven as generative for advancing scientific progress as might be hoped.

Populating the Elements within the Units of Analysis in the RDoC Matrix

It should be noted that due to the time required to refine Construct definitions, the Reward Seeking and Consummatory Behavior Construct group filled in the matrix elements with limited time for discussion and refinement. Thus, the draft matrix includes only those elements that had substantial agreement within the group, acknowledging that there may be elements that were overlooked due to time constraints.

With regard to entries in the gene column for approach motivation and its associated components, there was agreement among the Construct group that there was insufficient basis to identify specific genes for the individual subcomponents of approach motivation. Genes that were identified as promising for distinguishing among the subcomponents were DARPP-32 (dopamine- and cyclic-AMP-regulated phosphoprotein of molecular weight 32000), COMT (catechol-O-methyltransferase), NARP (neuronal activity regulated pentraxin), TITF1 (thyroid transcription factor 1), CB1 (cannabinoid receptor type 1) and the various epigenetic factors affecting the expression of these genes.

Orexin was not included in the matrix, although there is some evidence to support the involvement of orexin in reward valuation, and the Construct group identified this as a topic in need of further study.

With regard to the Expectancy/Reward Prediction Error component, glutamate co-release from ventral tegmental area (VTA) neurons was not included in the matrix, but was identified as a topic in need of further study.

As a general suggestion, it was noted that informant reports can be useful in addition to the self-report tools identified in the matrix.

Reward/Habit Learning Group

Construct and Definition Development

Given its initial charge and suggested draft Constructs, this group decided to define Constructs of Reward Learning and Habit/Habit Formation.

A significant amount of discussion revolved around Reward Learning as a component of Reinforcement Learning: Should the definition be expanded to include positive and negative reinforcement learning? Is there sufficient scientific evidence to split reinforcement learning into two, separately valenced RDoC Constructs? While there were somewhat divergent opinions regarding the evidence for and against different brain circuits being involved in positive and negative reinforcement learning, the participants decided to proceed with definition development of Reward Learning (consistent with the Positive Valence System Domain), but with the assurance that this question would be adequately represented in the Workshop Proceedings.

The group also discussed whether a reward has to be a positive outcome relative to a neutral outcome, or whether an outcome that is less negative than expected could be considered a reward. For example, if an individual is expecting a fine of $75, but instead receives a fine of $25, should that outcome be considered positive and therefore applicable within Reward Learning? This issue is included in the definition, which includes the phrase “when these outcomes are better than expected,” that allows for less-negative-than-expected outcomes and the resulting behavior modifications to be included under Reward Learning.

The group discussed the necessity of prediction or anticipated outcome in reward learning; in this light, it was unclear how Reward Learning would include behaviors that are reinforced even when the individual does not have a preexisting expectation or prediction regarding the outcome of a specific behavior (for example, self-injurious behavior in individuals with autism spectrum disorder). There was broad agreement on the importance of this issue; however the discussants did not reach agreement about how to include this aspect in the Construct definition.

In defining Habit/Habit Formation, the group emphasized the characteristic formation of a habit through repetition, and that performance of a habit is “semi-automatic,” not requiring full attention or cognitive awareness to direct the behavioral pattern. Additionally, habits are often goal-independent, rather than dependent on a reinforcer to maintain expression of a behavior, and are relatively insensitive to devaluation of a reinforcer. “Stereotypy” was not included in the definition of a habit, although the participants discussed that stereotypy could be a pathological behavioral expression involving a circuit that under normal circumstances/conditions would typically sub-serve habits.

In addition to these two Constructs, the group discussed a definition for Expectancy or Anticipation of Reward, including the importance of such a Construct to explain, e.g., the placebo effect. When the two Construct groups reconvened together, this concept was included as a subcomponent under the Construct of Approach Motivation.

The participants acknowledged overarching themes that would be applicable to the Constructs of Reward Learning and Habit/Habit Formation, including evolutionary adaptability, developmental considerations, and sex differences.

Populating the Elements within the Units of Analysis in the RDoC Matrix

When populating the elements within the Units of Analysis in the RDoC Matrix for Reward Learning and Habit/Habit Formation, the group attempted to be selective in their inclusion of elements, restricting their choices to those elements with the most scientific support within each defined Construct. The specificity of any one element for a defined Construct may be relatively low; however, greater specificity could be attained by examining multiple elements across different Units of Analysis (for example, for Reward Learning: epigenetic changes in medium spiny neurons of the striatum during Pavlovian conditioning). The group also noted the need for developing new self-report tools that would be directly relevant for assessing the RDoC Construct definitions developed during this PVS Workshop. A potentially powerful consideration for tool/assay development is the inclusion of mobile technology platforms.

Over-arching and open questions identified at the workshop:

• To what degree do these positive valence system processes work in similar or different ways regarding social versus non-social rewards?
• How dissociable are the systems that support positive and negative reinforcement learning?
• How does one best describe the relationship between positive and negative valence?
• Is the neural and cognitive representation of an outcome that is less adverse than predicted akin to a positive outcome?
• How do modulatory neurotransmitters influence positive valence systems? (It is anticipated that this will be discussed in the Arousal Domain Workshop.)
• How will the field best reconcile a behaviorist approach to motivation with a judgment/decision-making framework for motivation?

NIMH encourages comments on any aspect of the workshop and the proceedings outlined here. Please send comments to: rdoc@mail.nih.gov.

Participants

• David Axelson, MD, University of Pittsburgh School of Medicine
• Deanna M. Barch, PhD, Washington University
• Hans Breiter, MD, Northwestern University Feinberg School of Medicine and Massachusetts General Hospital
• William A. Carlezon Jr., PhD, Harvard Medical School
• William T. Carpenter, MD, University of Maryland School of Medicine
• Cameron S. Carter, MD, University of California, Davis
• Michael First, MD, Columbia University
• Jean A. Frazier, MD, University of Massachusetts
• Todd Gould, MD, University of Maryland School of Medicine
• Ann M. Graybiel, PhD, Massachusetts Institute of Technology
• Suzanne N. Haber, PhD, University of Rochester School of Medicine and Dentistry
• Peter Holland, PhD, Johns Hopkins University
• Yasmin Hurd, PhD, Mount Sinai School of Medicine
• Steven E. Hyman, MD, Harvard University
• Sheri L. Johnson, PhD, University of California, Berkeley
• Pamela K. Keel, PhD, Florida State University
• Kelly L. Klump, PhD, FAED, Michigan State University
• Ellen Leibenluft, MD, National Institute of Mental Health
• Marco Leyton, PhD, McGill University
• James Lock, MD, Ph.D, Stanford University
• Athina Markou, PhD, University of California, San Diego
• Melvin G. McInnis, MD, FRCPscyh, University of Michigan
• Betsy Murray, PhD, National Institute of Mental Health
• Rajesh Narendran, MD, University of Pittsburgh
• Saleem M. Nicola, PhD, Albert Einstein College of Medicine
• Martin P. Paulus, MD, University of California, San Diego
• Godfrey Pearlson, MD, Yale University
• Mary L. Phillips, MD, University of Pittsburgh
• Diego A. Pizzagalli, PhD, Harvard Medical School
• Natalie Rasgon, MD, PhD, Stanford University
• John Salamone, PhD, University of Connecticut
• C. Daniel Salzman, MD, PhD, Columbia University
• W. Kyle Simmons, PhD, Laureate Institute for Brain Research
• Gregory Paul Strauss, PhD, University of Maryland School of Medicine
• Mark J. Thomas, PhD, University of Minnesota
• Michael T. Treadway, MA, Vanderbilt University
• Madhukar H. Trivedi, MD, University of Texas, Southwestern Medical Center at Dallas
• B. Timothy Walsh MD, Columbia University
• James Andrew Waltz, PhD, University of Maryland School of Medicine
• Stephen Wonderlich, PhD, University of North Dakota School of Medicine & Health Sciences
• Eric Youngstrom, PhD, University of North Carolina at Chapel Hill
• Xiaoxi Zhuang, PhD, The University of Chicago
• Nancy Zucker, PhD, Duke University Medical Center

NIMH RDoC Working Group Members

• Bruce Cuthbert, PhD
• Marjorie Garvey, MB, BCh
• Robert Heinssen, PhD
• Michael Kozak, PhD
• Sarah Morris, PhD
• Daniel Pine, MD
• Kevin Quinn, PhD
• Charles Sanislow, PhD, Wesleyan University
• Rebecca Steiner, PhD
• Phil Wang, MD, DrPH

NIH Staff in attendance

• Kathy Anderson, PhD
• Shelli Avenevoli, PhD
• Susan Borja, PhD
• Alice Luo Clayton, PhD
• Rebecca Del Carmen, PhD (NICHD/NIMH)
• Jovier Evans, PhD
• Holly Garriock, PhD
• Mi Hillefors, MD, PhD
• Scherri Jacobsen, BA
• Douglas Meinecke, PhD
• Peter Muehrer, PhD
• George Niederehe, PhD
• Molly Oliveri, PhD
• Michelle Pearson
• Andrew Rossi, PhD
• Mercedes Rubio, PhD
• Janine Simmons, MD, PhD
• Jane Steinberg, PhD
• Ann Wagner, PhD
• Julia Zehr, PhD

1a. Approach motivation: Reward valuation

1b. Approach motivation: Effort valuation/Willingness to work

1c. Approach motivation: Expectancy/Reward prediction error

1d. Approach motivation: Action selection/Preference-based decision making

2. Initial responsiveness to reward attainment

3. Sustained/Longer-term responsiveness to reward attainment

4. Reward Learning

5. Habit