---
kind: concept
slug: self-play
title: Self-play
summary: >-
  Training an agent by playing copies of itself, which can yield a strong partner for those copies but not necessarily for a stranger. Covers plain self-play, other-play (random symmetry relabelling), search-guided self-play (AlphaGo Zero style) and self-play as a curriculum source. Three papers; results are author-reported.
tags: [reinforcement-learning, multi-agent, coordination]
papers: ["2003.02979", "2610.17391", "2003.04960"]
related: [multi-agent-systems, domain-randomization, reinforcement-learning-evaluation, monte-carlo-tree-search, policy-gradient, value-iteration, curriculum-learning]
status: draft
updated_at: 2026-10-19
---

## Overview

Self-play trains a policy against itself, or against copies of itself, so the
learner never needs a partner from outside its own training run. In a fully
cooperative game with shared reward, the baseline objective maximises the
return of one joint policy pair [arxiv:2003.02979 §4] [eq. 1]. The paper
argues that this objective lets the pair settle on arbitrary conventions that
only work together [arxiv:2003.02979 §3] [arxiv:2003.02979 §4].

In games with a win or loss outcome, self-play is the source of the training
data. AlphaGo Zero starts from random weights and learns by playing itself,
with a search guided by the current network [arxiv:2610.17391 app. B]. AlphaGo
also refines its policy network by reinforcement learning from self-play, after
supervised training on expert games [arxiv:2610.17391 §7.2.1] [arxiv:2610.17391 app. B].

A survey of curriculum learning for reinforcement learning also lists self-play
as one way a curriculum arises: agents that interact in the same environment
shape the sequence of experience they meet [arxiv:2003.04960 §4.2.2].

The invariant that the variants share is the game itself: the rules and the
outcome of each game are fixed. What differs is either what the learner must
cope with, such as a partner that labels states differently
[arxiv:2003.02979 §5], or how the training targets are produced, such as by a
search [arxiv:2610.17391 app. B].

## Variants

### Plain self-play

Two copies of the same learner (or a pair learned together) play each other,
and the update only uses the pair's own return. In the paper's Hanabi
baseline, the SAD method reaches a self-play score of 23.97 ± 0.04 but a
cross-play score of 2.52 ± 0.34 with independently trained SAD agents
[arxiv:2003.02979 tab. 1]. Card: [Other-Play for Zero-Shot Coordination](/papers/2003.02979).

### Other-play

Each agent is trained against a partner whose inputs and outputs are
relabelled by a symmetry drawn uniformly from the game's symmetry set, with a
new draw for each agent and each episode [arxiv:2003.02979 §4] [arxiv:2003.02979 §5].
The environment is not relabelled; only the agent's view of it is
[arxiv:2003.02979 §5]. It sits on top of any self-play algorithm; the paper
applies it to SAD and to SAD + AUX [arxiv:2003.02979 §6.3]. Card: [Other-Play for Zero-Shot Coordination](/papers/2003.02979).

In Hanabi, the symmetry class used is colour permutation, which holds only when
colours carry no side information [arxiv:2003.02979 §6.2]. Reported
cross-play rises to 15.32 ± 0.65 for SAD + OP and 22.07 ± 0.11 for
SAD + AUX + OP, with self-play scores near 24 in all rows [arxiv:2003.02979 tab. 1].
These are author-reported results from one game.

### Search-guided self-play (AlphaGo Zero style)

Each training move is chosen by a Monte Carlo tree search that uses the
current network to guide it. The search's visit distribution becomes the policy
target for that position, and the game's final result (-1 or +1) becomes the
value target for every position in the game [arxiv:2610.17391 app. B]. The
network is trained to minimise squared value error, cross-entropy to the search
distribution, and an L2 weight penalty [arxiv:2610.17391 app. B]. The book
gives the run's scale as 25,000 self-play games per iteration and 1,600
simulations per move [arxiv:2610.17391 app. B]. Its networks are described
with one residual tower and two heads, and no rollout policy
[arxiv:2610.17391 app. B]. The book reports that this network outperformed the
original AlphaGo after 36 hours of training [arxiv:2610.17391 §7.2.4]. Card:
[Learning to Play: Reinforcement Learning and Games](/papers/2610.17391).

The book applies the same search-and-evaluate architecture to Chess and Shogi as
well, using a 20-block residual network [arxiv:2610.17391 §7.3.1]. It notes that
the trained networks do not transfer between games [arxiv:2610.17391 §7.3.1].

### Self-play policy refinement (AlphaGo)

AlphaGo's selection policy is first trained by supervised learning on 160,000
expert games, then refined by policy-gradient reinforcement learning on
self-play [arxiv:2610.17391 app. B] [arxiv:2610.17391 §7.2.1]. The book reports
that the refined policy, used without search, won 85% of games against Pachi,
an open-source MCTS program, while the supervised-only policy won 11%
[arxiv:2610.17391 app. B]. The value network is trained on 30 million
self-play positions, which the book says were chosen to be uncorrelated
[arxiv:2610.17391 app. B]. Card: [Learning to Play: Reinforcement Learning and Games](/papers/2610.17391).

### Self-play as a curriculum (co-learning)

The survey files self-play under sequencing methods, as co-learning: a
curriculum emerges from agents that act in the same environment, and the
sequence of tasks is not designed by hand [arxiv:2003.04960 §4.2.2]. Its table
lists co-learning methods that transfer policies between stages, with an
automatic sequencer and adaptive ordering in every row [arxiv:2003.04960 tab. 2].
The survey runs no experiments of its own, so the table classifies published
work and does not compare results [arxiv:2003.04960 tab. 2]. Card:
[Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](/papers/2003.04960).

## When to use which

| Situation | Use | Why |
|---|---|---|
| The agent only ever plays copies of itself, in a cooperative game | Plain self-play | The paper's baseline reaches high self-play scores on its own [arxiv:2003.02979 tab. 1]. OP is motivated by unseen partners, and the paper's gains are shown on cross-play [arxiv:2003.02979 tab. 1]. |
| The partner is unseen at test time and the game has known symmetries you can encode | Other-play | Reported cross-play gains with independently trained agents, and gains with human players in one study [arxiv:2003.02979 tab. 1] [arxiv:2003.02979 §6.5]. |
| Symmetries are unknown or only partly known | Neither as stated | The paper assumes the symmetry set is given and names finding it as future work [arxiv:2003.02979 §8]. A wrong or incomplete set is not tested. |
| The game has side information that breaks the symmetry | Re-derive the symmetry set before using OP | Colour permutation is valid only without side information [arxiv:2003.02979 §6.2]. |
| A board game you can simulate, with a win or loss outcome, and no expert records | Search-guided self-play | The book's AlphaGo Zero starts from random weights and uses no human data [arxiv:2610.17391 app. B]. Its evidence is for Go, with a search of 1,600 simulations per move [arxiv:2610.17391 app. B]. |
| Expert game records exist and a search-free policy is needed | AlphaGo-style: supervised first, then self-play RL | The supervised stage comes first [arxiv:2610.17391 §7.2.1]. The book reports 85% wins against Pachi for the RL-trained policy and 11% for the supervised-only one, both without search [arxiv:2610.17391 app. B]. |
| Training should follow an easy-to-hard order over a shared environment, and you do not want to design intermediate tasks | Self-play as co-learning | The survey classes co-learning as a curriculum that emerges from interacting agents [arxiv:2003.04960 §4.2.2] [arxiv:2003.04960 tab. 2]. It reports no measured gains, so check the benefit on your own task. |
| Imperfect-information game such as Poker | No self-play method given | The book treats it as a separate challenge [arxiv:2610.17391 §8.2]. The OP evidence is from one partially observed cooperative game [arxiv:2003.02979 §6.2]. |
| Compute or evaluation budget is tight | Plain self-play, or check cost first | The OP runs use two simulation GPUs instead of one and 12 models per method for cross-play [arxiv:2003.02979 §6.3]. The search-guided runs need a search per move [arxiv:2610.17391 app. B]. |

## Implementation sketch

OP changes only how each episode is run. The base update is unchanged
[arxiv:2003.02979 §5]. The sampling step is the part to get right.

```pseudocode
# Other-play episode loop  [arxiv:2003.02979 §5]
# Input:  env, symmetry set Phi, one self-play learner per agent
# Output: trained policies

def train_other_play(env, agents, symmetries):
    # Repeat until the budget is spent  [arxiv:2003.02979 §5]
    while training:
        # Draw an independent symmetry for each agent  [arxiv:2003.02979 §5]
        relabellings = [sample_uniform(symmetries) for agent in agents]
        # Each agent sees and acts through its own relabelling;
        # the underlying game is unchanged  [arxiv:2003.02979 §5]
        trajectories = run_episode(env, agents, relabellings)
        # Base self-play update, e.g. SAD in Hanabi  [arxiv:2003.02979 §6.3]
        for agent, trajectory in zip(agents, trajectories):
            update(agent, trajectory)
    return agents
```

The paper does not state how the gradient treats the relabelled trajectories,
so check the base method's update before implementing [arxiv:2003.02979 §5].

Search-guided self-play replaces the episode loop with search-based move choice
and changes the targets, as in the book's AlphaGo Zero loop
[arxiv:2610.17391 app. B].

```pseudocode
# Search-guided self-play loop (AlphaGo Zero style)  [arxiv:2610.17391 app. B]
# Input:  game simulator, network with policy and value heads, random init
# Output: trained network

def train_search_guided(game, network, iterations, games_per_iter):
    # Repeat for each training iteration  [arxiv:2610.17391 app. B]
    for _ in range(iterations):
        examples = []
        # Generate a batch of self-play games  [arxiv:2610.17391 app. B]
        for _ in range(games_per_iter):
            state, history = game.initial_state(), []
            while not game.is_over(state):
                # Search guided by the current network  [arxiv:2610.17391 app. B]
                probs = mcts(state, network, simulations=1600)
                history.append((state, probs))
                state = game.apply(state, sample_move(probs))
            # Outcome z in {-1, +1}, the same for every position  [arxiv:2610.17391 app. B]
            z = game.outcome(state)
            examples += [(s, p, z) for s, p in history]
        # Fit value to z and policy to search probabilities, with L2
        # regularisation  [arxiv:2610.17391 app. B]
        train(network, examples)
    return network
```

## Contradictions

- **Value target convention, within one source.** The book describes AlphaGo's
  value network as a single tanh output read as a probability of winning, and
  describes AlphaGo Zero as training its value against an outcome in {-1, +1}.
  The book does not reconcile the two [arxiv:2610.17391 app. B].
- **Different tests, no direct disagreement.** The OP paper argues that
  self-play partners fail with strangers [arxiv:2003.02979 §3]. The book's
  self-play results are measured against fixed programs, such as Pachi, Stockfish
  and earlier AlphaGo versions [arxiv:2610.17391 app. B] [arxiv:2610.17391 §7.3.1],
  not against independently trained self-play partners. Neither source
  tests the other's setting, so the results are not comparable.

## Papers

- [Other-Play for Zero-Shot Coordination](/papers/2003.02979): proposes
  training self-play agents under random symmetry relabellings of the game and
  reports cross-play and human-study gains in Hanabi.
- [Learning to Play: Reinforcement Learning and Games](/papers/2610.17391): a
  textbook covering self-play with search through the AlphaGo, AlphaGo Zero and
  AlphaZero case studies, with the reported settings and results.
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](/papers/2003.04960):
  a survey that places co-learning by self-playing agents among curriculum
  sequencing methods, with a table of published work. It has no experiments.