Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model

Hot Off the Press – May 5, 2023

Published in Nature Neuroscience by Yuji Takahashi, Thomas Stalnaker, Lauren Mueller, and Geoffrey Schoenbaum of the NIDA IRP Behavioral Neurophysiology Neuroscience Section.

Summary

Dopamine neurons signal reward prediction errors – critical teaching signals that are broadcast throughout the brain to undergird associative learning.  Current models applied to understand these signals generally assume these errors are calculated as deviations from a single predictive stream.   Yet in much of life, upcoming events – particularly outcomes – are multifaceted.  Even a single outcome is defined by multiple, potentially dissociable features – its timing, location, quality and quantity, and even the internal significance and desirability attached to these aspects – and often we expect more than a single outcome in a given situation.   Is the dopamine system capable of tracking predictions about outcomes at a more detailed level to support efficient learning in situations where there are independent outcomes, or is it truly acting on a single stream of predictions, such that its operation might interfere with learning under more complex settings?    Here the Langdon (NIMH) and Schoenbaum (NIDA) labs teamed up to address this question.  The spiking activity of dopamine neurons was recorded under conditions in which individual cues predicted multiple rewards with different timings and flavors, and during recording the timing of some of the rewards was shifted, with and without changes in flavor, to induce reward prediction errors.   The results – supported by computational modeling – show that the dopamine neurons can separately track and update multiple independent reward predictions.  Notably this was true both for rewards that were truly independent – that is rewards that were both present and moved independently within a single trial – as well as for rewards whose independence was more ambiguous or depended on the internal belief of the subject.

Publication Information

@article{pmid37081296,

title = {Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model},

author = {Yuji K Takahashi and Thomas A Stalnaker and Lauren E Mueller and Sevan K Harootonian and Angela J Langdon and Geoffrey Schoenbaum},

url = {https://pubmed.ncbi.nlm.nih.gov/37081296/},

doi = {10.1038/s41593-023-01310-x},

issn = {1546-1726},

year  = {2023},

date = {2023-04-01},

urldate = {2023-04-01},

journal = {Nat Neurosci},

abstract = {Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}