Deep Reinforcement Learning and its Neuroscientific Implications

In plain words

Reinforcement learning is a way to teach a system by trial and error: it tries actions, gets a reward or nothing back, and gradually prefers the actions that pay off. Deep learning is the technique of stacking many layers of simple learned units, which has been very good at recognising patterns. Combining the two lets a single learned network decide what to do and also build its own internal picture of the situation, shaped by what earns reward.

This review argues that the combination is useful to brain scientists, not just to engineers. Earlier neuroscience work used deep networks to explain what the brain sees, and reward-learning theories to explain how animals decide. The combined system links the two, so it offers a way to study how reward shapes the way the brain represents the world. The authors walk through the history, show a few early studies, and list open questions and limits.

TL;DR

• A review, not a method: it defines deep RL as RL where the representations are learned by a deep network rather than designed by hand §2.3.
• It traces deep RL from a 1990s backgammon player to DQN and later systems, and names reward sparsity and over-fitting to the training task as two problems that learned representations must address §5.1.
• Early neuroscience uses are a recurrent network that matches monkey orbitofrontal recordings, meta-RL that explains dopamine and prefrontal findings, and distributional RL that fits mouse dopamine data §4 fig. 2 fig. 3 fig. 4.
• Six topics for next-step research are listed, plus caveats on sample efficiency, long-term credit assignment and biological plausibility of backpropagation §5 §6.
• Relevant for an engineer who wants background on deep RL, or an agent that needs the review's framing. Not a source of hyperparameters or benchmark numbers.

Problem

The review addresses a gap. Deep networks trained with supervised learning have modelled neural activity, and reinforcement learning has explained dopamine and decision-making, but the two were mostly used apart. Neuroscience had little guidance on how representations are formed through reward §3.

For the engineering side, the review notes that earlier RL needed tabular or linear state representations, which do not generalise, and that RL with neural networks was unstable before DQN §2.1 §2.3.

Method

This is a narrative review. It has no algorithm, equations, tables or experiments of its own. Its "method" is the conceptual frame it gives:

1. Definition. Deep RL is any system that solves an RL problem (maximises long-term reward) with representations learned by a deep network §2.3.
2. Components. Deep learning trains weights by backpropagation to fit an input-output mapping; RL needs exploration, has sensory-motor feedback, and faces non-stationary data §2.2.
3. Integration. Reward-driven learning shapes internal representations, and those representations then support decisions. The authors argue this produces effects that neither deep learning nor RL shows alone §3.
4. Stabilisation. DQN made deep RL reliable mainly through mechanisms that reduce non-stationarity, including experience replay §2.3.

The paper's sections are unnumbered in the source. The numbers used here count them in order: 1 Introduction; 2 An introduction to Deep RL (2.1 Reinforcement learning, 2.2 Deep learning, 2.3 Deep reinforcement learning); 3 Deep RL and Neuroscience; 4 Vanguard studies; 5 Topics for next-step research (5.1 Representation learning, 5.2 Model-based RL, 5.3 Memory, 5.4 Exploration, 5.5 Cognitive control and action hierarchies, 5.6 Social cognition); 6 Challenges and caveats; 7 Conclusion.

Experience replay is the one mechanism the review describes in enough detail to sketch. The pseudocode below follows the prose only.

# Experience replay as described for DQN  [§2.3]
# Store past transitions and re-present them in random order.

def replay_step(buffer, network, new_transition):
    # Store the newest (state, action, reward, next state) tuple  [§2.3]
    buffer.add(new_transition)
    # Draw past transitions at random, like supervised sampling  [§2.3]
    batch = buffer.sample_random()
    # Update the network from the sampled transitions  [§2.3]
    return network.update(batch)

Other mechanisms the review names only at the level of ideas: auxiliary self-supervised and prediction-learning objectives §5.1, intrinsic rewards for novelty and uncertainty §5.4, and hierarchical policies that act at several time-scales §5.5.

Hyperparameters

Not applicable. The review gives no hyperparameters. The one set of numbers in the text is the setup of the meta-RL figure:

These are the figure's stated settings for an illustration, not a recommended training recipe. No network sizes, optimiser or learning rates are stated.

Results

No new results. The review summarises earlier studies. The following are the authors' descriptions, not measurements taken here.

Paper vs code

No official code. A GitHub search by title words and author lab returned no repositories. The paper contains no code or project link.

Pitfalls

• Review, not method. Statements such as "deep RL is much more than the sum of its parts" are the authors' framing. Treat them as hypotheses, not findings §3.
• Mixed claims. The review covers settled facts (DQN, TD-Gammon), reported studies, and proposals with no data. Check the cited source before relying on one.
• Dated. The review is from 2020. Later deep RL systems are not covered.
• Figure numbers. The source does not print figure numbers in order, so they follow the order of figure environments in the LaTeX (1 deep RL overview, 2 neuroscience examples, 3 meta-RL, 4 distributional RL, 5 representation t-SNE). Confirm against the published PDF.
• Section numbers. The paper's sections are unnumbered; the numbering in this card is the count given in the Method section.
• Figures not reproduced. Figure images are not used here. Their content is described only in words, and the captions credit third-party sources.
• Biological plausibility. The authors note that backpropagation may not be implemented in biological circuits, and that energy efficiency and inheritance of knowledge are not natural questions in engineering-oriented deep RL §6.

Lineage

• Builds on: the RL framework §2.1, DQN §2.3, and the earlier computational neuroscience of dopamine and reward prediction error §3.
• Compared against: not applicable.
• Superseded by: not known from content in this wiki.