MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence

In plain words

Most reinforcement-learning research environments are built for a handful of agents. This paper introduces MAgent, a platform for the other extreme: a large map where hundreds, thousands or even a million simple agents learn at the same time. Each agent sees a small window of the map and a few numbers about itself, then chooses how to move or attack.

The platform is meant less for building one clever agent and more for watching a crowd. Rules that give rewards for events, such as one agent attacking another, are written in a small description language in Python. The demo shows three such worlds: predators hunting prey, agents competing for food, and two armies fighting. Agents learn these behaviours from scratch, and the point is to observe the group patterns that emerge. The paper does not measure those patterns; it shows them as pictures and examples.

TL;DR

MAgent is a gridworld platform for many-agent reinforcement learning, with a C++ engine, a Python interface, a reward rule language and three baseline learners (DQN, DRQN, A2C). The paper claims it can host up to one million agents on one GPU server [abstract] but reports no measurements. It is a demo paper: useful as a simulator to build on, not as evidence for any algorithm.

Problem

Existing RL platforms (ALE, OpenAI Gym/Universe, Malmo, ELF, SC2LE) are, in the authors' words, designed for no more than dozens of agents § Introduction. Most multi-agent RL algorithms are also limited to dozens of agents, so the authors argue there is a need for a platform for large populations § Introduction. The paper does not give quantitative evidence for this gap.

Method

The platform has four parts. The paper describes each in prose without equations, algorithms or tables.

1. Gridworld engine. A large gridworld is the base environment. Each agent has a rectangular body, a local view and optional global information. Actions are moving, turning and attacking. A C++ engine runs the simulation, and heterogeneous agent types can be registered. The state space, action space and agent attributes are configurable § Gridworld Fundamentals.

2. Reward description language. In the Python interface, users define agent symbols and events. When the boolean expression of an event is true, the listed receivers get the listed reward values. Events can combine with and, or and not § Reward Description Language. The paper's example is the pursuit rule: a predator attacking a prey gives the predator +1 and the prey -1 § Reward Description Language. The repository's pursuit config uses the same rule pursuit.py.

3. Scale through parameter sharing and ID embedding. The abstract says the platform scales through network sharing and ID embedding § The MAgent Platform. The baseline section says each agent's unique ID is turned into an embedding that is trained § Baseline Algorithms. How this is done is covered in Pitfalls.

4. Baseline learners. Parameter-sharing DQN, DRQN and A2C are implemented. The authors found DQN performed best in their settings and used it for their experiments § Baseline Algorithms.

The demo environments are shown in fig. 1, five panels in total: Pursuit (1) and (2), Gathering, and Battle (1) and (2). The panel captions name the three running examples fig. 1.

Pseudocode for the training loop, reconstructed from the paper's description and the repository's battle example (not in the paper itself):

# Parameter-sharing training, one network per side  [§ Baseline Algorithms]
# Input:  env, a list of agent-type handles (one per side), one model per handle
# Output: trained models

def train_rounds(env, handles, models, num_rounds):
    for round_index in range(num_rounds):
        # 1. Exploration rate decays with the round number
        #    [code: examples/train_battle.py#L217]
        eps = piecewise_decay(round_index, [0, 700, 1400], [1, 0.2, 0.05])
        # 2. Every agent of a side acts with that side's network
        #    [§ Baseline Algorithms]
        # 3. Store transitions and update each side's network
        #    [code: python/magent/builtin/tf_model/dqn.py#L277-L346]
        # play_a_round is the battle example's loop
        #    [code: examples/train_battle.py#L43]
        play_a_round(env, handles, models, eps)
    return models

DQN target used by the repository's default learner (double Q-learning variant, not described in the paper) dqn.py:

# Bootstrapped target for one stored transition  [code: dqn.py#L233-L248]
# Inputs: reward, terminal flag, next-state views and features (batched)

def dqn_target(reward, terminal, next_obs, gamma):
    # 1. Choose the next action with the online network  [code: dqn.py#L237-L240]
    best_next = argmax(online_q(next_obs), axis=1)
    # 2. Score it with the target network  [code: dqn.py#L240]
    next_value = target_q(next_obs)[arange, best_next]
    # 3. Terminal transitions keep only the reward  [code: dqn.py#L246]
    return where(terminal, reward, reward + gamma * next_value)

Hyperparameters

The paper states none of the values below. The table lists the defaults in the repository's DQN class and in its battle example. These are code values, not values the paper reports.

Results

The paper is a demo paper and reports no quantitative results. The table lists the claims and what they rest on.

Author-reported, qualitative only. The repository has a 1M-agent test script (random actors, no learning) test_1m.py test_1m.py.

Paper vs code

• Agent ID embedding. Paper: each agent's unique ID gets a trained embedding § Baseline Algorithms. Code: the ID is written as a fixed binary code, with the bits of the integer ID, and the embedding is not learned in the engine GridWorld.h. The learned part is a dense layer that reads these bits in the network dqn.py.
• Parameter sharing. Paper: parameter-sharing DQN § Baseline Algorithms. Code: the battle example makes one model per side, so both armies in a battle use two networks, each shared within its side train_battle.py.
• Learner variant. Paper names DQN only. Code defaults to double Q-learning and a dueling head dqn.py dqn.py.
• Attack penalty. Paper's reward language example adds only the +1/-1 rule § Reward Description Language. Code gives each predator a penalty of -0.2 on its type, and the engine applies it on every attack, including attacks on empty cells pursuit.py GridWorld.cc GridWorld.cc.
• A2C in the battle example. Paper lists A2C as a baseline § Baseline Algorithms. The battle example raises NotImplementedError for A2C train_battle.py. The example points to another script for A2C.
• Reward rule terminal option. The code's add_reward_rule has a terminal argument that ends the episode when the event fires gridworld.py. The paper does not mention it.

Pitfalls

• Section numbers are not shown. The source sets secnumdepth to 0 [magent.tex: \setcounter{secnumdepth}{0}], so the compiled paper has no section numbers. The citations on this card use section titles instead (for example § Baseline Algorithms).
• "Trained ID embedding" is ambiguous. The engine feeds a fixed binary code of each agent's ID as part of the feature vector; a learned dense layer reads it. An implementer following the paper literally may train a lookup table.
• Abstract versus body. The abstract names communication languages, leadership and altruism as phenomena the platform enables. The body describes only local cooperation, competition and the hybrid of both § Live and Interactive Part. The paper does not show those three behaviours.
• Scale is not measured. The million-agent claim has no hardware, model, map size or throughput in the paper [abstract]. The repository's test uses random actors, not learning agents test_1m.py.
• Training schedule is not in the paper. Batch size, target update interval, exploration decay and replay size are only in the battle example train_battle.py and train_battle.py. Treat them as one working setup, not as the paper's settings.

Lineage

• Builds on: DQN, DRQN and A2C are used as baselines without citation to their original papers in the source § Baseline Algorithms.
• Compared against: ALE, OpenAI Gym/Universe, Malmo, ELF and SC2LE are named as existing platforms that do not scale to many agents § Introduction. No quantitative comparison is given.
• Superseded by: not known from content in this wiki.