Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

In plain words

Learning a hard skill often goes better when the lessons come in a sensible order, from simpler material to harder material. This paper is about applying that idea to reinforcement learning agents, which learn by trying actions and collecting rewards. A "curriculum" here is the order in which an agent meets its experience, either as single moments from a task or as a sequence of easier and harder tasks.

The authors give one shared vocabulary for these methods. Building a curriculum has three parts: choosing the intermediate tasks, deciding their order, and carrying useful knowledge from one task to the next. They then sort existing work by what each method assumes and what it changes, and they list open questions for future research. The paper runs no new experiments; it is a map of the field.

TL;DR

• Curriculum learning for RL has three parts: task generation, sequencing, and transfer learning §3.2.
• The paper defines a curriculum as a directed acyclic graph over experience, with a simplified sequence form and task-level forms §3.1.
• Existing methods are classified along seven dimensions, and tables place specific papers on them §3.4 tab. 1 tab. 2 tab. 3.
• It is a survey: no experiments, no code, and no numbers of its own. Its open problems are listed in §6.

Problem

Reinforcement learning often needs a large number of interactions with the environment, which can be too expensive in realistic settings §1. Transfer learning can reuse what was learned on one task for the next, harder task, and a sequence of such tasks forms a curriculum §1. The paper's aim is to give the field a common foundation and terminology, because curriculum methods have been developed separately in robotics, multi-agent systems, human-robot interaction and intrinsic motivation §1.

The paper also states that the term "curriculum" is used inconsistently in the literature, so it fixes definitions before comparing methods §1 §3.1.

Method

The paper is a framework and a survey, so its "method" is a set of definitions and a classification scheme.

Curriculum definition §3.1. Each task is a Markov decision process. A curriculum is a directed acyclic graph whose vertices are associated with sets of transition samples from the tasks, and whose edges say which vertex should be trained before another. All paths end in a single sink vertex, the target task. Three simplifications are named: a single-task curriculum (samples from the target task only), a task-level curriculum (each vertex is one whole task), and a sequence curriculum (every vertex has at most one predecessor and one successor). Curricula can be built online, adding edges from learning progress, or offline, before training §3.1.

# Curriculum as a DAG over experience  [§3.1, curriculum definition]
# A vertex holds a subset of transition samples drawn from the tasks.

def curriculum(vertices, edges, samples_of, target):
    # edges: pairs (before, after); `before` is trained first  [§3.1]
    # samples_of: maps each vertex to its set of (s, a, r, s') samples
    # target: the single sink vertex; every path must end there  [§3.1]
    assert is_acyclic(vertices, edges)
    assert all_paths_end_at(target, edges)
    return (vertices, edges, samples_of, target)

# Task-level form: samples_of maps each vertex to one whole task  [§3.1]
# Sequence form: each vertex has at most one predecessor and one successor  [§3.1]

Curriculum learning, three parts §3.2.

1. Task generation: create the set of intermediate tasks, either fixed in advance or generated while the agent learns.
2. Sequencing: choose the ordering of samples or tasks, which means choosing the edges of the graph. This is the focus of the survey.
3. Transfer learning: move reusable knowledge from one task to the next.

Evaluation §2.3 §3.3. The paper adapts transfer-learning metrics. Time to threshold measures how much faster a target performance level is reached with transfer. Asymptotic performance compares final performance. Jumpstart measures initial gain. Total reward compares accumulated reward up to a fixed point. The paper distinguishes weak transfer, where time spent on intermediate tasks is a sunk cost, from strong transfer, where that time is counted §3.3 fig. 3.

Seven classification dimensions §3.4. Each method is labelled on seven axes. Task generation asks whether intermediate tasks are only the target, made automatically, or designed by domain experts or naive users. Representation asks whether the curriculum is a single-task ordering, a sequence or a graph. Transfer method names the knowledge carried over, such as policies, a value function, a task model, partial policies or a shaping reward, or says that none is carried. The sequencer axis uses the same actors as task generation. Adaptivity asks whether the order is fixed before training or changes during it. The evaluation metric is one of the four metrics above. The application area is toy, simulated or real robotics, video games, or other.

Sequencing methods, ordered by how far they change the task §4.2. The paper arranges sequencing approaches on a spectrum, from reordering samples of the final task without altering the MDP to defining new intermediate tasks:

• Sample sequencing: reorder experience from the target task, e.g. prioritised replay and hindsight replay §4.2.1 tab. 2.
• Co-learning: a curriculum emerges from interacting or self-playing agents in the same environment §4.2.2 tab. 2.
• Reward and initial/terminal state changes: intermediate tasks differ only in reward or in start and end state distributions §4.2.3 tab. 2.
• No restrictions: intermediate tasks may differ from the target in any way §4.2.4 tab. 2.
• Human-in-the-loop: people sequence tasks by hand §4.2.5 tab. 2 fig. 5.

Task generation §4.1 tab. 1. Known methods assume a parameterised domain whose parameters define tasks, and then apply simplification, promising initialisation or mistake-based rewinding to make intermediate tasks §4.1. Object-oriented generation and Powerplay-style invention are also covered §4.1.

Knowledge transfer §4.3 tab. 3. Transferred knowledge is classed as low level (policies, value functions, task models, training instances) or high level (partial policies, options, skills, shaping rewards, subtask definitions) §4.3. tab. 3 places papers by transfer method.

Structures of curricula fig. 4. Figure 4 shows a linear sequence in a grid world and a directed acyclic graph in a block-based game, both taken from prior work fig. 4.

Hyperparameters

Not applicable. The paper trains no models and states no hyperparameters. Choices it leaves to the reader, such as when to stop training on a vertex, are discussed without values: training to convergence or for a fixed number of episodes or epochs §3.1.

Results

Author-reported. This is a survey with no experiments of its own. Its "results" are the classifications in its tables, which place published work on the seven dimensions §3.4.

Each table marks, in bold, the evaluation metrics that show strong transfer tab. 1 tab. 2 tab. 3.

Paper vs code

No official code.

I searched the arXiv metadata, the paper's abstract and its LaTeX source (no project page, no repository URL in the text) and a GitHub repository search for the survey's title keywords and first author, which returned no results. The linked material in the wider literature is paper lists, not implementations, so none is recorded here.

Pitfalls

• Classifications rest on experiments, not on claims. The transfer-method value is what each paper actually transferred in its experiments §3.4. Papers that describe other transfer types are placed by their reported results, so a card that quotes a method's motivation may disagree with the table.
• Adaptivity has a value outside the stated set. The dimension lists static and adaptive §3.4, but one row in tab. 2 is marked "both" for adaptivity. Treat it as a mixed case, not as a clean label.
• Metric naming is not uniform. Section 2.3 uses the metric name "jump start" in the prose, while §3.4 writes "jumpstart"; tables use both forms tab. 3.
• Strong and weak transfer change the comparison. A bolded metric in a table indicates strong transfer, which counts the cost of building and training on the curriculum. Asymptotic gains imply strong transfer §3.3 tab. 1.
• Cost of building the curriculum is often hidden. Human-designed curricula make construction time hard to measure, and automated methods often treat it as sunk §3.3.
• Several classifications are judgement calls. Human-designed and automatically generated tasks are separated by a line the paper describes only in words §3.4 §4.1.
• The tables are a snapshot. The text does not describe inclusion criteria for the surveyed papers §1. It was published in 2020, so later work is not covered.
• Several rows are commented out in the source. The LaTeX contains disabled rows and notes. Only the rows printed in the compiled tables are reported here.

Lineage

• Builds on: transfer learning for reinforcement learning, cited as Taylor (2009) in §1 and §2.3; curriculum learning for supervised learning by Bengio et al., cited in §1 and §4.2.1.
• Builds on, in the classifications: Narvekar et al. on autonomous curricula, shown in fig. 4 and tab. 2.
• Compared against: the paper compares its dimensions with the existing literature; it names no competing survey.
• Points to: the open problems in §6 concern automatic task creation, transferring different knowledge types, reusing curricula across agents, combining task generation with sequencing, theory, and principles of curriculum design.
• Superseded by: not known from this wiki.