Self-play

Overview

Self-play trains a policy against itself, or against copies of itself, so the learner never needs a partner from outside its own training run. In a fully cooperative game with shared reward, the baseline objective maximises the return of one joint policy pair 2003.02979 §4 eq. 1. The paper argues that this objective lets the pair settle on arbitrary conventions that only work together 2003.02979 §3 2003.02979 §4.

In games with a win or loss outcome, self-play is the source of the training data. AlphaGo Zero starts from random weights and learns by playing itself, with a search guided by the current network 2610.17391 app. B. AlphaGo also refines its policy network by reinforcement learning from self-play, after supervised training on expert games 2610.17391 §7.2.1 2610.17391 app. B.

A survey of curriculum learning for reinforcement learning also lists self-play as one way a curriculum arises: agents that interact in the same environment shape the sequence of experience they meet 2003.04960 §4.2.2.

The invariant that the variants share is the game itself: the rules and the outcome of each game are fixed. What differs is either what the learner must cope with, such as a partner that labels states differently 2003.02979 §5, or how the training targets are produced, such as by a search 2610.17391 app. B.

Variants

Plain self-play

Two copies of the same learner (or a pair learned together) play each other, and the update only uses the pair's own return. In the paper's Hanabi baseline, the SAD method reaches a self-play score of 23.97 ± 0.04 but a cross-play score of 2.52 ± 0.34 with independently trained SAD agents 2003.02979 tab. 1. Card: Other-Play for Zero-Shot Coordination.

Other-play

Each agent is trained against a partner whose inputs and outputs are relabelled by a symmetry drawn uniformly from the game's symmetry set, with a new draw for each agent and each episode 2003.02979 §4 2003.02979 §5. The environment is not relabelled; only the agent's view of it is 2003.02979 §5. It sits on top of any self-play algorithm; the paper applies it to SAD and to SAD + AUX 2003.02979 §6.3. Card: Other-Play for Zero-Shot Coordination.

In Hanabi, the symmetry class used is colour permutation, which holds only when colours carry no side information 2003.02979 §6.2. Reported cross-play rises to 15.32 ± 0.65 for SAD + OP and 22.07 ± 0.11 for SAD + AUX + OP, with self-play scores near 24 in all rows 2003.02979 tab. 1. These are author-reported results from one game.

Search-guided self-play (AlphaGo Zero style)

Each training move is chosen by a Monte Carlo tree search that uses the current network to guide it. The search's visit distribution becomes the policy target for that position, and the game's final result (-1 or +1) becomes the value target for every position in the game 2610.17391 app. B. The network is trained to minimise squared value error, cross-entropy to the search distribution, and an L2 weight penalty 2610.17391 app. B. The book gives the run's scale as 25,000 self-play games per iteration and 1,600 simulations per move 2610.17391 app. B. Its networks are described with one residual tower and two heads, and no rollout policy 2610.17391 app. B. The book reports that this network outperformed the original AlphaGo after 36 hours of training 2610.17391 §7.2.4. Card: Learning to Play: Reinforcement Learning and Games.

The book applies the same search-and-evaluate architecture to Chess and Shogi as well, using a 20-block residual network 2610.17391 §7.3.1. It notes that the trained networks do not transfer between games 2610.17391 §7.3.1.

Self-play policy refinement (AlphaGo)

AlphaGo's selection policy is first trained by supervised learning on 160,000 expert games, then refined by policy-gradient reinforcement learning on self-play 2610.17391 app. B 2610.17391 §7.2.1. The book reports that the refined policy, used without search, won 85% of games against Pachi, an open-source MCTS program, while the supervised-only policy won 11% 2610.17391 app. B. The value network is trained on 30 million self-play positions, which the book says were chosen to be uncorrelated 2610.17391 app. B. Card: Learning to Play: Reinforcement Learning and Games.

Self-play as a curriculum (co-learning)

The survey files self-play under sequencing methods, as co-learning: a curriculum emerges from agents that act in the same environment, and the sequence of tasks is not designed by hand 2003.04960 §4.2.2. Its table lists co-learning methods that transfer policies between stages, with an automatic sequencer and adaptive ordering in every row 2003.04960 tab. 2. The survey runs no experiments of its own, so the table classifies published work and does not compare results 2003.04960 tab. 2. Card: Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey.

When to use which

Implementation sketch

OP changes only how each episode is run. The base update is unchanged 2003.02979 §5. The sampling step is the part to get right.

# Other-play episode loop  [arxiv:2003.02979 §5]
# Input:  env, symmetry set Phi, one self-play learner per agent
# Output: trained policies

def train_other_play(env, agents, symmetries):
    # Repeat until the budget is spent  [arxiv:2003.02979 §5]
    while training:
        # Draw an independent symmetry for each agent  [arxiv:2003.02979 §5]
        relabellings = [sample_uniform(symmetries) for agent in agents]
        # Each agent sees and acts through its own relabelling;
        # the underlying game is unchanged  [arxiv:2003.02979 §5]
        trajectories = run_episode(env, agents, relabellings)
        # Base self-play update, e.g. SAD in Hanabi  [arxiv:2003.02979 §6.3]
        for agent, trajectory in zip(agents, trajectories):
            update(agent, trajectory)
    return agents

The paper does not state how the gradient treats the relabelled trajectories, so check the base method's update before implementing 2003.02979 §5.

Search-guided self-play replaces the episode loop with search-based move choice and changes the targets, as in the book's AlphaGo Zero loop 2610.17391 app. B.

# Search-guided self-play loop (AlphaGo Zero style)  [arxiv:2610.17391 app. B]
# Input:  game simulator, network with policy and value heads, random init
# Output: trained network

def train_search_guided(game, network, iterations, games_per_iter):
    # Repeat for each training iteration  [arxiv:2610.17391 app. B]
    for _ in range(iterations):
        examples = []
        # Generate a batch of self-play games  [arxiv:2610.17391 app. B]
        for _ in range(games_per_iter):
            state, history = game.initial_state(), []
            while not game.is_over(state):
                # Search guided by the current network  [arxiv:2610.17391 app. B]
                probs = mcts(state, network, simulations=1600)
                history.append((state, probs))
                state = game.apply(state, sample_move(probs))
            # Outcome z in {-1, +1}, the same for every position  [arxiv:2610.17391 app. B]
            z = game.outcome(state)
            examples += [(s, p, z) for s, p in history]
        # Fit value to z and policy to search probabilities, with L2
        # regularisation  [arxiv:2610.17391 app. B]
        train(network, examples)
    return network

Contradictions

• Value target convention, within one source. The book describes AlphaGo's value network as a single tanh output read as a probability of winning, and describes AlphaGo Zero as training its value against an outcome in {-1, +1}. The book does not reconcile the two 2610.17391 app. B.
• Different tests, no direct disagreement. The OP paper argues that self-play partners fail with strangers 2003.02979 §3. The book's self-play results are measured against fixed programs, such as Pachi, Stockfish and earlier AlphaGo versions 2610.17391 app. B 2610.17391 §7.3.1, not against independently trained self-play partners. Neither source tests the other's setting, so the results are not comparable.

Papers

• Other-Play for Zero-Shot Coordination: proposes training self-play agents under random symmetry relabellings of the game and reports cross-play and human-study gains in Hanabi.
• Learning to Play: Reinforcement Learning and Games: a textbook covering self-play with search through the AlphaGo, AlphaGo Zero and AlphaZero case studies, with the reported settings and results.
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey: a survey that places co-learning by self-playing agents among curriculum sequencing methods, with a table of published work. It has no experiments.