---
kind: concept
slug: hierarchical-reinforcement-learning
title: Hierarchical reinforcement learning
summary: >-
  A high-level policy sets a goal or subtask and a low-level policy acts under it, so decisions span several steps.
  Eleven cards so far, from text-GAN managers and robot subtask planners to humanoid, bipedal, navigation, dexterous
  and offline-maze control.
tags: [reinforcement-learning, hierarchical-policy, text-generation, robotics, red-teaming, humanoid-control, locomotion, navigation, dexterous-manipulation, offline-reinforcement-learning, diffusion, loco-manipulation]
papers: ["1709.08624", "2405.18418", "2508.04451", "2512.12993", "2603.03596", "2402.12498", "2306.06766", "2411.04005", "2411.07760", "2411.12982", "2509.20322"]
related: [policy-gradient, generative-adversarial-network, adversarial-text-generation, goal-conditioned-reinforcement-learning, task-decomposition, vision-language-action-model, red-teaming, rlhf, model-based-reinforcement-learning, model-predictive-control, world-model, proximal-policy-optimization, variational-autoencoder, sim-to-real-transfer, imitation-learning, offline-reinforcement-learning, vector-quantization, diffusion-models, reward-design, digital-twin, motion-retargeting, on-policy-distillation]
status: draft
updated_at: 2026-10-31
---

## Overview

A hierarchical policy has two levels. A high-level policy emits a goal or a subtask, and a low-level policy chooses
the actions under it, such as the next word, a robot action chunk or the next utterance in a dialogue. How often the
high level acts is a design choice. The LeakGAN manager emits a new goal at every word [arxiv:1709.08624 §Hierarchical
Structure of G]. The high level of the red-teaming card picks a guide once per dialogue turn [arxiv:2508.04451 §4.2].
The MEM card does not state how often its high-level policy updates [arxiv:2603.03596 §3.1]. The humanoid puppeteer
plans a new command at every control step, since both levels run at the same frequency [arxiv:2405.18418 §4.1]. The
bipedal walker's high level acts at 33 Hz, while its low level runs at 1 kHz [arxiv:2512.12993 §III.C] [arxiv:2512.12993
§III.F].

The navigation card's mid-level network outputs a new pixel waypoint for each decision, and its worker repeats that
waypoint over two environment steps [arxiv:2402.12498 §4.2] [arxiv:2402.12498 §4.3].

How the high level is trained is where the variants differ. LeakGAN's manager is trained with a policy gradient, and
its reward is the change in discriminator features over the next few steps, so the manager is trained to align its
goal with that change [arxiv:1709.08624 §Training of G]. The worker gets a separate intrinsic reward, the cosine
between the feature change and the recent goals [arxiv:1709.08624 §Training of G]. The red-teaming high level is
trained with PPO on a one-step critic built from the target's reply to the utterance [arxiv:2508.04451 §4.3]
[arxiv:2508.04451 eq. 1]. MEM's high-level policy is trained to predict its next language memory from the previous
memory, the current observation and the task. Its training labels are written by an off-the-shelf LLM, and the card
gives no reward or RL objective for the high level [arxiv:2603.03596 §3.2]. The puppeteer is not trained with a
policy gradient at all: it is a planner that is trained from scratch for each task on the task reward, and its
low-level tracker is fixed [arxiv:2405.18418 §3.2]. The bipedal walker's high level is a PPO policy whose reward is
a tracking reward on the foot and speed targets it sets [arxiv:2512.12993 §III.E] [arxiv:2512.12993 §III.F].

Several of the newer cards do not train the high level with a policy gradient. FeudalNav's mid-level network is trained on
human point-clicks and no RL is used at all [arxiv:2402.12498 §4.2] [arxiv:2402.12498 §1]. The object-centric planner
is trained by behaviour cloning on human wrist motion, and PPO trains only the finger controller [arxiv:2411.04005 §4.1]
[arxiv:2411.04005 §4.2]. QPHIL's planner predicts the next landmark token by teacher forcing, and its two low-level
policies are trained offline with IQL [arxiv:2411.07760 §4.2] [arxiv:2411.07760 §4.3]. HDP's Guider is a conditional
diffusion model trained with a denoising loss on demonstrated contacts [arxiv:2411.12982 §3.2]. PIRL-WIN is the one new
card that trains its high level with PPO, while its local policy is behaviour-cloned and then frozen
[arxiv:2306.06766 §IV-B]. VisualMimic also uses teacher and student policies at both levels. Its high level, a keypoint
generator, is trained with PPO on object state and then distilled into a depth-only student with DAgger [arxiv:2509.20322
§3.2]. Its low level, a keypoint tracker, is distilled the same way from a PPO-trained teacher [arxiv:2509.20322 §3.1].

Eleven cards cover the concept so far. LeakGAN applies the hierarchy to long text generation. MEM applies a subtask-level
hierarchy with a memory to robot manipulation over minutes. The red-teaming card applies a two-level PPO scheme to a
multi-turn attacker that is scored by a harm classifier. The puppeteer applies a planning-based high level to whole-body
humanoid motion from pixels, and the bipedal walker applies a PPO high level to terrain-aware walking. FeudalNav applies
a three-level, RL-free navigator to image goals. PIRL-WIN applies a PPO waypoint policy with physics-shaped rewards to
wireless indoor navigation. The object-centric card applies a planner and a PPO finger controller to dexterous
manipulation, QPHIL applies discrete landmark planning to offline maze navigation, and HDP applies a contact-predicting
diffusion model above a diffusion actor, and VisualMimic applies a distilled keypoint generator and tracker to
humanoid loco-manipulation with depth input. The hierarchical framework LeakGAN builds on is named in its lineage notes, but its content is not used here.

## Variants

### Manager–worker over discriminator features (LeakGAN)

A convolutional discriminator reads the prefix generated so far and produces a feature vector. A manager LSTM maps
that vector to a unit-length goal, and a goal embedding is formed from the sum of the last c goals, with c = 4 in the
paper's experiments [arxiv:1709.08624 §Hierarchical Structure of G] [arxiv:1709.08624 §Experiment]. A worker LSTM reads
the current word and outputs a matrix with one row per vocabulary word. Its scores are the row-by-goal products,
divided by a temperature, and a softmax gives the next-word distribution [arxiv:1709.08624 §Hierarchical Structure of G].

Training runs in stages. The discriminator is first trained to separate real from generated text, and the generator
is pre-trained by maximum likelihood [arxiv:1709.08624 §Training of G]. The adversarial loop then alternates
generator updates with discriminator updates. The manager is updated with a policy gradient weighted by a Monte
Carlo estimate of the expected sentence reward. The worker is updated with a REINFORCE-style loss weighted by the
intrinsic reward [arxiv:1709.08624 §Training of G]. The paper also interleaves one supervised generator epoch after
every fifteen adversarial epochs [arxiv:1709.08624 §Interleaved Training]. Card: [Long Text Generation via Adversarial Training with Leaked Information](/papers/1709.08624).

### Subtask policy with language memory (MEM)

The policy is factored into a low-level policy that outputs an action chunk and a high-level policy that outputs the
next subtask and an updated language memory [arxiv:2603.03596 §3.1]. The high level sees the task goal in text, its
own previous memory and the current observation. Because it predicts the new memory from the old one, it decides when
and how to update the memory [arxiv:2603.03596 §3.2]. The low level sees the last K dense frames, and the value of K
is not stated [arxiv:2603.03596 §3.1].

Memory labels come from an LLM, which is given the subtask instructions with a success flag for each. It is asked to
keep only what still matters for the future, so object attributes are reduced to a count and a location. A failed
attempt leaves the memory unchanged until the subtask succeeds [arxiv:2603.03596 §3.2] [arxiv:2603.03596 §4.1].

A separate short-horizon video encoder gives the backbone a view of recent camera frames. Every fourth layer attends
across time, and tokens of past frames are dropped in the upper layers, so the backbone sees one frame's worth of
tokens [arxiv:2603.03596 §3.3]. The encoder adds no learnable parameters and equals the single-image ViT when K = 1
[arxiv:2603.03596 §3.3]. The authors build this on pi0.6, which is initialised from a Gemma3-4B VLM and has an 860M
parameter action expert [arxiv:2603.03596 §3.4]. Card: [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](/papers/2603.03596).

### Two-level PPO over dialogue turns (red-teaming attacker)

The state is the conversation so far. Each turn, the high-level policy reads the history and a target goal, and it
outputs a guide. The low-level policy then writes the utterance one token at a time under that guide. The target
model answers, and the answer is appended to the history [arxiv:2508.04451 §4.1] [arxiv:2508.04451 §4.2]. The
high-level guide is either a topic plus a persona from a fixed template pool, or a short free-text guide from an LLM
[arxiv:2508.04451 §4.3] [arxiv:2508.04451 app. F].

Both levels are trained with PPO [arxiv:2508.04451 §4.3]. The reward for a turn is the harm gain: the classifier's
"unsafe" probability for the target's reply, minus the same probability for the utterance [arxiv:2508.04451 §4.2]. The
low-level turn reward adds a cosine similarity between reference-model embeddings of the guide and the utterance. The
authors state this term keeps the low level on its guide and stops it from overfitting one utterance
[arxiv:2508.04451 eq. 2]. Credit inside an utterance is assigned by masking single tokens and pairs of its top-attended
tokens, and measuring the drop in the turn reward when a mask is applied [arxiv:2508.04451 eq. 3] [arxiv:2508.04451
eq. 4]. The paper leaves the reward for tokens outside the top-attended set undefined [arxiv:2508.04451 eq. 4].

The authors' ablation on HarmBench, ASR@5 with Llama-3.1-8B, gives 75.2 with both levels trained and 31.2 with no
high-level policy. Freezing the high level gives 47.0, and freezing the low level gives 40.7 [arxiv:2508.04451 tab. 3].
Card: [There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning](/papers/2508.04451).

### Puppeteer over a frozen whole-body tracker (humanoid)

Two agents are trained separately, and both are world-model planners [arxiv:2405.18418 §3]. The low level is a
tracker. Its input is the proprioceptive state and a 15-value command, which holds the relative 3D positions of five
end-effectors taken from a reference motion. It outputs a sequence of joint actions for a 56-joint humanoid
[arxiv:2405.18418 §3.1] [arxiv:2405.18418 app. E]. One tracker is pretrained on offline rollouts for 836 motion-capture
clips, with offline and online data mixed in each batch [arxiv:2405.18418 §3.1] [arxiv:2405.18418 §4.1].

The high level, called the puppeteer, sees the same proprioceptive state and a 64x64 RGB image from a third-person
camera. Its action space is the tracker's command space, and it never drives joints directly [arxiv:2405.18418 §3.2].
The tracker is frozen while the puppeteer learns, and both act at the same frequency [arxiv:2405.18418 §3.2]
[arxiv:2405.18418 §4.1]. The puppeteer is trained from scratch for each downstream task, using only the task reward
[arxiv:2405.18418 §3.2] [arxiv:2405.18418 §4.1]. Planning, not a policy gradient, produces its actions: both levels
choose actions by model predictive control with a learned policy prior as a warm start [arxiv:2405.18418 §2]. The
paper also adds a learned termination head, which the planner uses to predict early episode ends
[arxiv:2405.18418 §3.3]. Card: [Hierarchical World Models as Visual Whole-Body Humanoid Controllers](/papers/2405.18418).

### Hierarchical PPO over a terrain latent (bipedal walker)

The terrain is compressed first. A convolutional VAE maps a 20x40 local height map to a 16-value latent
[arxiv:2512.12993 §II.B]. The high level is an MLP policy trained with PPO. Its input is a reduced-order state built
from the base's LIP-like state, the speed error, the swing foot, the latent and the last action [arxiv:2512.12993
§III.B] [arxiv:2512.12993 eq. 5]. Its output is six task-space values: the swing foot's landing position, a swing
clearance and two speed offsets [arxiv:2512.12993 §III.C] [arxiv:2512.12993 eq. 6].

The low level is not learned. A task-space inverse dynamics problem, solved as a quadratic program, tracks the
foot and base targets that the high level sets, and it runs at 1 kHz [arxiv:2512.12993 §III.D] [arxiv:2512.12993
§III.F]. The reward is a weighted sum of tracking terms for speed and swing-foot targets, plus an action-smoothness
term, each of Gaussian form [arxiv:2512.12993 eq. 11] [arxiv:2512.12993 §III.E]. The authors also train a second
encoder that predicts the same latent from two depth cameras, pulling it toward the height-map latent
[arxiv:2512.12993 §IV.C]. The paper also tests history-aware encoders that stack several past height maps
[arxiv:2512.12993 §II.D]. Card: [Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](/papers/2512.12993).

### Manager–worker over a learned place memory, without RL (FeudalNav)

FeudalNav is a three-level navigator for the image-goal task, and it is trained without reinforcement learning. The top
level keeps a latent memory map. Its features come from a self-supervised clustering model, and a new view joins the
nearest cluster center when the distance is below a threshold, otherwise it starts a new center
[arxiv:2402.12498 §4.1] [arxiv:2402.12498 §5.2]. The mid-level network is a fine-tuned ResNet-18 that takes the RGBD
observation and the memory map, and it predicts a pixel to move toward [arxiv:2402.12498 §4.2]. The worker maps that
pixel to a turn, a step, or a turn and a step, on a fixed image grid. It then re-locates the waypoint and repeats it for
a second environment step [arxiv:2402.12498 §4.3]. When a keypoint match with the goal image is confident enough, the
match replaces the waypoint [arxiv:2402.12498 §4.2]. The paper argues that a learned memory can stand in for a metric
graph, and that the hierarchy eases learning from little data [arxiv:2402.12498 §1]. Card:
[Feudal Networks for Visual Navigation](/papers/2402.12498).

Author-reported on Gibson with the NRNS protocol, the curved-path success average is 40.50 for the stacked variant,
against 23.97 for NRNS without noise [arxiv:2402.12498 tab. 2]. On straight paths the stacked variant has higher success
than NRNS, but a lower SPL [arxiv:2402.12498 tab. 2] [arxiv:2402.12498 §6.2]. The evaluation is in simulation only
[arxiv:2402.12498 §8].

### Waypoint policy trained with PPO over a physics-shaped reward (PIRL-WIN)

A global policy trained with PPO sits above a fixed planner and a local visual policy [arxiv:2306.06766 §IV]. From a
radio input built from the five strongest paths, the global policy outputs an angle and a link-state estimate. A goal is
placed along that angle, at a distance set by the link-state estimate, and a Fast Marching planner on the map turns it
into a short-term target [arxiv:2306.06766 §IV-A]. The local policy, a recurrent network on a pretrained ResNet-18, is
trained by behaviour cloning and then kept fixed [arxiv:2306.06766 §IV-B]. The reward adds three radio-physics terms to
a distance cost: a monotonicity term on the link state, a term that follows the angle of arrival, and an SNR-gradient
term in the weakest link state [arxiv:2306.06766 §III-B]. The paper says the shaped reward is not potential-based, so
the optimal policy under it differs from the shortest path [arxiv:2306.06766 §III-B]. Card:
[Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](/papers/2306.06766).

Author-reported, on six unseen Gibson maps, the normalised path length in the 2+-NLOS category is 2.46 to 2.82, against
4.50 to 5.14 for end-to-end RL [arxiv:2306.06766 tab. 3]. On maps P and T, 2+-NLOS success is 1.00, against 0.40 to 0.45
for end-to-end RL and 0.85 to 0.90 for a wireless-assisted heuristic [arxiv:2306.06766 tab. 4]. The paper's ± values
are not defined [arxiv:2306.06766 tab. 3].

### Behaviour-cloned planner over a PPO finger controller (object-centric dexterous manipulation)

The task is defined by a sequence of object poses. A transformer planner, trained by behaviour cloning on human wrist
motion from ARCTIC, outputs a 6-DoF pose for each wrist at each of ten steps [arxiv:2411.04005 §4.1]. A PPO policy
outputs finger joint actions, plus a wrist residual bounded to ±4 cm and ±0.5 rad that corrects the planned wrist pose
[arxiv:2411.04005 §4.2]. Its reward is an exponential of a weighted sum of rotation, translation and joint-angle gaps
[arxiv:2411.04005 §4.2] [arxiv:2411.04005 app. B]. Successful rollouts are fed back into the planner's training data over
several rounds [arxiv:2411.04005 §4.3] [arxiv:2411.04005 alg. 1]. Card:
[Object-Centric Dexterous Manipulation from Human Motion Data](/papers/2411.04005).

Author-reported. In simulation, one policy per object completes 83.8% of trained trajectories with the feedback loop,
against 59.6% for the ablated variant [arxiv:2411.04005 tab. 4]. On six real objects, one policy per object trained on
one goal trajectory completes 64.3% to 100%, against 4.5% to 56.1% for vanilla RL [arxiv:2411.04005 tab. 2].

### Discrete landmark plan with offline IQL policies (QPHIL)

QPHIL is a hierarchical method for offline goal reaching. A VQ-VAE maps each state to one of k learned landmark tokens,
trained with reconstruction, commitment and a temporal triplet loss [arxiv:2411.07760 §4.1]. A transformer plans a
sequence of landmark tokens from the goal token and the compressed history, trained by teacher forcing
[arxiv:2411.07760 §4.2]. Two IQL-trained policies execute the plan: a landmark policy moves the agent through the
planned tokens, and a goal policy takes over for the final approach [arxiv:2411.07760 §4.3]. An optional stitching step
splices recorded trajectories that visit the same landmarks, which the authors offer as a cheaper alternative to
fine-tuning the planner with offline RL [arxiv:2411.07760 §4.2]. The method assumes the low-level policy can reach any
state inside a landmark from any other state in it [arxiv:2411.07760 §4.2]. Card:
[Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](/papers/2411.07760).

Author-reported. On AntMaze-Ultra diverse, QPHIL without augmentation reaches 70.3 ± 6.9, against 52.6 ± 8.7 for HIQL in
its "w/o repr." setting [arxiv:2411.07760 tab. 1]. On the new AntMaze-Extreme map, the augmented variant reaches 50.0 on
play and 39.5 on diverse, against 22.8 and 21.9 for the same HIQL setting [arxiv:2411.07760 fig. 5].

### Diffusion contact guider over a diffusion actor (HDP)

Two conditional diffusion models run in sequence. The Guider predicts the next contact target, which is each finger's
position in the object frame plus a flag for whether it touches, from the recent observations [arxiv:2411.12982 §3.2].
The Actor denoises a 16-step action sequence conditioned on the observation and that contact [arxiv:2411.12982 tab. 8].
Its loss adds a Q-value term, weighted by η = 0.001, on an action denoised in one step [arxiv:2411.12982 §4.4]
[arxiv:2411.12982 tab. 9]. The critic is trained on demonstrated transitions, with a reward for reaching the target within
a short window [arxiv:2411.12982 §3.2]. Training targets come from the demonstrations: each demonstration is cut where
finger contact changes, and phases with similar object poses are merged [arxiv:2411.12982 §4.1]. The method needs an
object point cloud, placed in the world frame from the object pose for rigid objects [arxiv:2411.12982 §4.2]. Card:
[Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance](/papers/2411.12982).

Author-reported. On the real UR5e Move-T task, stage-2 success is 0.60 to 0.75 for HDP and 0.30 to 0.45 for Diffusion
Policy, depending on the dataset [arxiv:2411.12982 tab. 5]. On cloth unfolding over 50 trials, success is 0.76 against
0.60 [arxiv:2411.12982 tab. 7]. The simulation comparison uses a smaller Diffusion Policy baseline that the authors
reproduced, so it is not a comparison with published numbers [arxiv:2411.12982 §6.2] [arxiv:2411.12982 tab. 2].

### Keypoint generator over a distilled keypoint tracker (VisualMimic)

The low level is a task-agnostic keypoint tracker. Its input is an 18-value command, made of the root position error
and the offsets of five keypoints (head, two hands, two feet) from the root, and it outputs joint actions
[arxiv:2509.20322 §3.1] [arxiv:2509.20322 eq. 1]. The high level, the keypoint generator, produces that command. It
reads egocentric depth and proprioception, and it is retrained for each task [arxiv:2509.20322 §3] [arxiv:2509.20322
§3.2].

Both levels are trained as teacher and student. The tracker teacher sees the reference motion for the next 2 seconds
and privileged proprioception, and it is trained with PPO. Its student sees only proprioception and the command, and is
distilled with DAgger [arxiv:2509.20322 §3.1]. The generator teacher sees object state and is trained with PPO on a
task reward built from approach, forward-progress, force and two regularising terms. Its depth student is distilled
with DAgger [arxiv:2509.20322 §3.2] [arxiv:2509.20322 eq. 4] [arxiv:2509.20322 eq. 9]. The paper gives no weights for
these reward terms [arxiv:2509.20322 §3.2].

Two stabilisers are added. While the tracker trains, each command value is multiplied by a factor drawn from U(0.5,
1.5). The generator's output is clipped to a band of μ ± 1.64σ, estimated from human motion [arxiv:2509.20322 §3.3].
The paper reports that removing either one hurts training [arxiv:2509.20322 §4.5]. Card: [Visual Humanoid
Loco-Manipulation via Motion Tracking and Generation](/papers/2509.20322).

Author-reported, in simulation, the vision-based student's Push Box distance over a one-minute rollout is 37 ± 28 m,
against 152 ± 36 m for the state-based teacher and 2 ± 0 m for a student with no vision [arxiv:2509.20322 tab. 2]. Without command noise, the vision
student's Push Box distance falls to 2 ± 1 [arxiv:2509.20322 tab. 3]. Real-robot results on a Unitree G1 are
qualitative, with no success rates [arxiv:2509.20322 §4.2].

## When to use which

Compare the variants by what the high level outputs and what data or signal trains it. Each row names the card that
supports it.

| Situation | Use | Why |
|---|---|---|
| Long generated sequences, where a single end-of-sequence reward is too sparse | Manager–worker over discriminator features | The paper's central claim is that per-step guidance helps beyond about 20 words [arxiv:1709.08624 §Introduction]. Its BLEU-2 gain over SeqGAN on WMT News is 0.956 against 0.859 [arxiv:1709.08624 tab. 2] |
| Short text, such as four-line poems | Likely still usable, but the gain is smaller | The paper reports BLEU-2 0.881 against 0.812 for RankGAN on poems [arxiv:1709.08624 tab. 4], a narrower margin than on long text |
| Need a fixed, non-learned reward | Not the LeakGAN variant | The goals track features of a discriminator that keeps training. The paper's appendix reports that replacing the hierarchy with a fully connected layer made training numerically unstable [arxiv:1709.08624 app. Discussions] |
| Tight compute budget, text GAN | Plan for the cost of Monte Carlo estimates | Q is estimated by Monte Carlo search [arxiv:1709.08624 §Training of G]. The paper reports no training-time figures |
| Tight compute budget, robot policy | Plan for the video encoder's cost | The factorised attention costs O(Kn² + nK²) per layer, with no constants given [arxiv:2603.03596 §3.3]. Latency is shown only as a figure [arxiv:2603.03596 fig. 2] |
| Robot task of several minutes whose progress can be written as text | Subtask policy with language memory | The authors report tasks of up to about 15 minutes [arxiv:2603.03596 §4.1]. The text memory carries progress, and the video encoder carries recent visual detail [arxiv:2603.03596 §3.1] |
| Task where a subtask is often retried | Subtask policy, with failed attempts excluded from memory | Naive concatenated instructions work significantly worse than learned summaries [arxiv:2603.03596 §4.1]. Retries are a train-inference shift for naive memory [arxiv:2603.03596 §4.1] |
| Multi-turn interaction where the outcome shows only after a whole conversation | Two-level PPO over dialogue turns | The authors frame the reward as sparse and delayed over long dialogues [arxiv:2508.04451 §4]. Against the baselines they ran, their method has the highest ASR@5 in each of the four settings of tab. 1. On Llama-8B context-aware, the gap is 62.5 against 41.0 for X-Teaming [arxiv:2508.04451 tab. 1] |
| Myopic target that sees only the latest message | Expect a smaller margin | On Llama-8B myopic, WildTeaming reaches 65.0 against 75.2 for this method. The authors report larger gains in the context-aware setting [arxiv:2508.04451 tab. 1] [arxiv:2508.04451 §5.1] |
| Training with a different harm classifier | Re-measure, because results move with the classifier | With LlamaGuard 3 8B the training run gives 75.2, with LlamaGuard 2 8B 69.4, and with ShieldGemma 2B 62.2 [arxiv:2508.04451 tab. 3] |
| Tight compute budget, two-level PPO | Plan for the token-credit cost | The authors say the credit step multiplies the cost of plain fine-tuning by n·k, with k = 5 in the experiments [arxiv:2508.04451 app. C]. The 8B LoRA run takes about 10 h on two H100s [arxiv:2508.04451 app. E] |
| Whole-body humanoid from pixels, with human-like motion as a goal | Puppeteer over a frozen tracker | The authors report that the puppeteer matches TD-MPC2 return on 8 tasks except stairs, and that 97.8% of 510 human judgements preferred its motion [arxiv:2405.18418 §4.2] [arxiv:2405.18418 tab. 6]. This is a simulated humanoid, and the human study covers five visual tasks only |
| Reusing one controller across many tasks | Puppeteer over a frozen tracker | The tracker is pretrained once and frozen, and only the puppeteer is retrained per task [arxiv:2405.18418 §3.2] [arxiv:2405.18418 §4.1]. The authors report that the puppeteer works on gaps up to 1.2 m, three times the training range [arxiv:2405.18418 fig. 9] |
| Tight compute budget, humanoid | Plan for the pretraining and planning cost | The card gives about 12 days of tracker training and about 4 days of puppeteer training on one RTX 3090 [arxiv:2405.18418 §4.1]. Inference takes 88.2 ms per step for the puppeteer against 50.8 ms for TD-MPC2 [arxiv:2405.18418 tab. 3] |
| Bipedal walking over stairs and steps, with a short observation and few samples | Hierarchical PPO over a terrain latent | The authors report above 80% success on every tested terrain in simulation, and about 30 million training samples against 60 million for the end-to-end baselines they tabulate [arxiv:2512.12993 §IV.E] [arxiv:2512.12993 tab. 1]. The baselines use other robots, so the comparison is across setups |
| Image-goal navigation with no odometry or metric map, no RL budget, and a source of human clicks | Manager–worker over a learned place memory | The authors' system learns from human clicks and needs no odometry or metric map [arxiv:2402.12498 §1] [arxiv:2402.12498 §3]. On curved paths the stacked variant reaches 40.50 success against 23.97 for NRNS without noise [arxiv:2402.12498 tab. 2]. On straight paths its SPL is lower than NRNS's, which the authors attribute to broader exploration [arxiv:2402.12498 tab. 2] [arxiv:2402.12498 §6.2] |
| Radio-guided search, with a digital twin of the space to train in, and a need for transfer to unseen layouts | PPO waypoint policy with physics-shaped reward | The authors report a 2+-NLOS path length of 2.46 to 2.82, against 4.50 to 5.14 for end-to-end RL, on six unseen maps [arxiv:2306.06766 tab. 3] [arxiv:2306.06766 §V]. The shaped reward is not potential-based, so its optimum is not the shortest path [arxiv:2306.06766 §III-B] |
| Offline demonstrations of long goal-reaching tasks in mazes, with budget for a transformer and two IQL policies | Discrete landmark plan with IQL policies | The authors report their strongest results on the largest mazes, 70.3 on AntMaze-Ultra diverse [arxiv:2411.07760 §5.2] [arxiv:2411.07760 tab. 1]. On large diverse, HIQL in its "w/ repr." setting scores higher, 88.2 against 80.25 and 80.5 [arxiv:2411.07760 tab. 1]. Each training run takes about 10 GPU-hours on V100s [arxiv:2411.07760 app. B.1] |
| Human wrist-motion data exists, the robot's hands differ from human hands, and a simulator is available | Behaviour-cloned planner over PPO finger control | The authors argue that wrist motion transfers across embodiments better than finger motion [arxiv:2411.04005 §1]. Completion on trained single-object trajectories is 83.8% in simulation, against 59.6% for the ablated variant [arxiv:2411.04005 tab. 4]. The planner and the feedback loop are not in the release, so both must be written [arxiv:2411.04005 §Paper vs code] |
| Contact-rich imitation where a single diffusion policy is not enough, with object poses available to define contacts | Diffusion Guider over diffusion Actor | The authors argue that Diffusion Policy is weaker on contact-rich tasks [arxiv:2411.12982 §1]. On the real Move-T task they report stage-2 success of 0.60 to 0.75, against 0.30 to 0.45 for Diffusion Policy [arxiv:2411.12982 tab. 5]. The simulation baseline was reduced and reproduced, so simulation gaps are within-paper only [arxiv:2411.12982 §6.2] |
| Humanoid loco-manipulation from depth, where one trained low-level tracker should serve many tasks | Keypoint generator over a distilled tracker | Only the generator is retrained per task [arxiv:2509.20322 §3]. In simulation the depth student reaches 135 ± 6 m on Kick Ball against 189 ± 3 m for the teacher, and 37 ± 28 m on Push Box against 152 ± 36 m [arxiv:2509.20322 tab. 2]. The real-robot evidence is qualitative [arxiv:2509.20322 §4.2] |
| Hardware deployment is the goal | Expect to add the hardware step | The paper's policies are tested in simulation only, and hardware deployment is listed as future work [arxiv:2512.12993 §V]. The puppeteer card reports no real-robot result [arxiv:2405.18418 §4] |
| Need a quick reimplementation of LeakGAN | Budget extra effort | The authors' code targets an old TensorFlow and Python 2, and differs from the paper in several settings; see the LeakGAN card's Paper vs code section |
| Need a reimplementation of MEM | Expect to rebuild the data pipeline | No official code is known. The LLM used for labels, its prompt and the optimiser are not stated [arxiv:2603.03596 §3.2] |
| Need a reimplementation of the red-teaming method | Expect to rebuild it from the paper | The only code link is an anonymised mirror that could not be checked, so the card records no code [arxiv:2508.04451 app. A] |
| Need a reimplementation of the puppeteer | Use the official code, and check it against the paper | The code is MIT-licensed and linked from the paper. It differs in several places, such as the termination head's input and the planner's iteration count; read the card's Paper vs code section [arxiv:2405.18418 §Paper vs code] |
| Need a reimplementation of the bipedal walker | Expect to rebuild it from the paper | No official code is known [arxiv:2512.12993 §V]. Several training values, such as the VAE beta and the TSID gains, are not stated [arxiv:2512.12993 eq. 4] [arxiv:2512.12993 §III.D] |
| Need a reimplementation of FeudalNav | Expect to rebuild the pipeline from the paper | No official code is known. The optimiser, learning rate and the feature scale behind the memory threshold are not stated [arxiv:2402.12498 §5.2] |
| Need a reimplementation of PIRL-WIN | Use the official MIT code with its differences in mind | The release's reward has no link-state or SNR term, and its policy receives the dataset's link-state label [arxiv:2306.06766 §Paper vs code] |
| Need a reimplementation of the object-centric method | Expect to write the planner and the feedback loop | The release has the RL environment and training scripts, but not the planner or the loop [arxiv:2411.04005 §Paper vs code] |
| Need a reimplementation of QPHIL | Expect to rebuild it from the paper | The appendix promises a public repository, and none is cited on the card, because the only candidate found is not usable under our licensing rules [arxiv:2411.07760 app. B.1] |
| Need a reimplementation of VisualMimic | Expect to write the training pipeline | The release has a MuJoCo sim2sim demo and four checkpoints. The tracker training code, the high-level training code and the real-robot pipeline are not released [code: README.md]. The repository has no LICENSE, so only paths are cited [arxiv:2509.20322 §Paper vs code] |
| Need a reimplementation of HDP | Use the paper, and check the code's differences | The repository has no license file, so only paths are cited. Its critic and noise augmentation differ from the paper's equations [arxiv:2411.12982 §Paper vs code] |

## Implementation sketch

Our own pseudocode for the common two-level loop. The comments name the paper section each step follows. The
LeakGAN generation loop, which the variant above uses, follows it. The red-teaming variant fits the same loop, with
its high level choosing a guide per turn and its low level writing the utterance under that guide
[arxiv:2508.04451 §4.2]. The puppeteer fits it with a planned high level and a frozen low level, and the bipedal walker
fits it with a PPO high level and a fixed low level [arxiv:2405.18418 §3.2] [arxiv:2512.12993 §III.C]. VisualMimic fits
it with a keypoint generator as the high level and a distilled keypoint tracker as the low level [arxiv:2509.20322 §3.1]
[arxiv:2509.20322 §3.2]. FeudalNav fits it with a waypoint network as the high level and a
fixed image-grid rule as the low level [arxiv:2402.12498 §4.2] [arxiv:2402.12498 §4.3]. PIRL-WIN fits it with a PPO
global policy and a frozen, behaviour-cloned local policy [arxiv:2306.06766 §IV-B].

```pseudocode
# Two-level loop, one step at a time  [arxiv:2603.03596 §3.1]
# High level picks a subgoal or subtask; low level acts under it

def hierarchical_loop(high, low, goal, steps):
    # Start value: LeakGAN uses zeros; MEM does not state one [arxiv:2603.03596 §3.2]
    high_state = initial_state
    for t in range(steps):
        obs = observe()
        # High level writes the subtask and its updated memory  [arxiv:2603.03596 §3.1] [§3.2]
        subtask, high_state = high(obs, high_state, goal)
        # Low level emits actions under the subtask  [arxiv:2603.03596 §3.1]
        actions = low(obs, subtask)
        execute(actions)
```

```pseudocode
# Generate one sequence [arxiv:1709.08624 §Hierarchical Structure of G]
# T words, feature size d, goal size k, vocabulary size V

def generate(discriminator, manager, worker, T, c, alpha):
    # Both recurrent states start at zero [arxiv:1709.08624 app. Pseudo Code]
    manager_state = zeros(hidden_size)
    worker_state = zeros(hidden_size)
    word = START_TOKEN
    tokens, goals = [], []
    for t in range(T):
        # Leaked feature of the prefix so far [arxiv:1709.08624 §Leaked Features from D]
        feature = discriminator.features(tokens)     # (d,)
        # Manager maps the feature to a unit goal [arxiv:1709.08624 §Hierarchical Structure of G]
        raw_goal, manager_state = manager(feature, manager_state)
        goals.append(raw_goal / norm(raw_goal))      # (k,)
        # Goal embedding from the last c goals [arxiv:1709.08624 §Hierarchical Structure of G]
        # Fewer than c goals exist at the start: not stated
        goal_embedding = W_psi @ sum(goals[-c:])     # (k,)
        # Worker encodes the word [arxiv:1709.08624 §Hierarchical Structure of G]
        output_matrix, worker_state = worker(word, worker_state)  # (V, k)
        # Score words by row dot goal, then soften [arxiv:1709.08624 §Hierarchical Structure of G]
        probs = softmax((output_matrix @ goal_embedding) / alpha)  # (V,)
        word = sample(probs)
        tokens.append(word)
    return tokens
```

The two LeakGAN training rules differ only in what is weighted. The manager's loss weights each goal's alignment with
the feature change over the next c steps by the estimated reward; the worker's loss weights the log-probability of
each chosen word by the mean cosine between recent feature changes and recent goals [arxiv:1709.08624 §Training of G].

## Contradictions

No disagreement between the papers. They report different tasks and metrics, so their numbers cannot be compared
directly, and each card's results are author-reported. The red-teaming paper's baselines were reproduced by its
authors in one shared setting [arxiv:2508.04451 §5], so its baseline numbers are not the original papers' numbers.
The humanoid and bipedal results are in simulation and use different robots and baselines, so they do not bear on
each other or on the text and robot-manipulation cards. The navigation and manipulation cards are in the same position:
they use different simulators, metrics (normalised path length, SPL, completion, stage success) and baselines.

Within LeakGAN, the paper's appendix formula list numbers its equations differently from the main text, so equation
citations must name which list they come from (see the LeakGAN card's Pitfalls section). The authors' code differs
from the paper in several settings, including the temperature convention, the reward rescaling constants and the
discriminator's convolution windows. These are discrepancies between paper and code, not between papers. Within MEM,
the post-training memory horizon is 54 seconds [arxiv:2603.03596 §3.4] but up to one minute [arxiv:2603.03596 §4.3].
The MEM card has no code to check against.

Within the red-teaming paper, the base model is named as Llama-3-8B-Instruct in its experiments section but as
Llama-3.1-8B-Instruct in its appendix and tables [arxiv:2508.04451 §5] [arxiv:2508.04451 app. D.1]. Some cells of
tab. 2 and tab. 4 also differ for the same method and target, such as LLM-HL on Llama-8B HarmBench, 95.0 against 97.0
[arxiv:2508.04451 tab. 2] [arxiv:2508.04451 tab. 4]. The paper does not explain these gaps.

Within the humanoid paper, the human-study shares differ between text and table. The text gives 4% for equal and 0%
for the baseline, while the table gives 1.8% and 0.4% [arxiv:2405.18418 §4.2] [arxiv:2405.18418 tab. 6]. The card
uses the table's counts. Separately, the paper's tracking-success threshold does not appear in its released code, so
its tracker success rates may not match the code's success flag [arxiv:2405.18418 app. B] [arxiv:2405.18418 §Paper vs code].

Within the bipedal paper, Table 1 and the text disagree on which end-to-end baselines take a full height map and which
take sampled heights [arxiv:2512.12993 tab. 1] [arxiv:2512.12993 §IV.B]. Check the text before citing either.

Within FeudalNav, the single-view straight-path average is 45.73 success and 37.69 SPL in tab. 3, but 47.57 and 38.41 in
tab. 2. The card treats tab. 2 as consistent [arxiv:2402.12498 tab. 2] [arxiv:2402.12498 tab. 3]. The text's curved-path
gains of 69% in success and 27% in SPL come from two different variants: the stacked variant's success and the
single-view SPL [arxiv:2402.12498 §6.2] [arxiv:2402.12498 tab. 2].

Within PIRL-WIN, the released code differs from the paper's reward and feeds the dataset's link-state label to its policy,
so the paper's numbers cannot be attributed to the released code without re-running it [arxiv:2306.06766 §III-B]
[arxiv:2306.06766 §Paper vs code].

Within QPHIL, the text says the method outperforms all others on larger maps, but the table shows HIQL in its "w/ repr."
setting ahead on large diverse [arxiv:2411.07760 §5.2] [arxiv:2411.07760 tab. 1]. The seed text says eight seeds, but the
list has seven values [arxiv:2411.07760 §5.1] [arxiv:2411.07760 §8]. The one-sample re-planning numbers match neither
QPHIL row of the main results [arxiv:2411.07760 app. D] [arxiv:2411.07760 tab. 1]. The non-extreme baselines are copied
from the HIQL paper, while the extreme-map baselines were tuned again, so the tuning effort differs [arxiv:2411.07760 app. A].

Within the object-centric paper, the 47.3% average advantage over vanilla RL in the text has no stated averaging, and a
mean over its own table gives a different gap [arxiv:2411.04005 §5.2] [arxiv:2411.04005 tab. 3]. The text's claim of more
than a 50% improvement does not hold for Microwave, at 100 against 60.6 [arxiv:2411.04005 §5.4] [arxiv:2411.04005 tab. 2].
The augmentation ranges have no stated units, while the appendix says centimetres and degrees [arxiv:2411.04005 app. A].

Within VisualMimic, §4.1 pairs the 3.8 kg box with a push distance of 37 m per minute. The 3.8 kg figure is the real-world
box [arxiv:2509.20322 §1], but the 37 m figure matches the simulated 4 kg Push Box row [arxiv:2509.20322 §4.1]
[arxiv:2509.20322 tab. 2]. The paper does not say which box the figure refers to.

Within HDP, the text gives an 84.1% average success gain for Move-T, but a simple mean of the stage-2 success rates in
tab. 5 gives 81.8%, and the paper does not say how 84.1% is computed [arxiv:2411.12982 §7.1] [arxiv:2411.12982 tab. 5].
The 20.8% headline is not derived in the text. The card's recomputation shows that an unweighted mean of the simulation
improvements in tab. 2 and tab. 3 together with the two real-robot gains reproduces it [arxiv:2411.12982 tab. 2]
[arxiv:2411.12982 tab. 3] [arxiv:2411.12982 §7.2].

## Papers

- [Long Text Generation via Adversarial Training with Leaked Information](/papers/1709.08624): a manager–worker generator for text GANs that steers each word by the discriminator's intermediate features.
- [Hierarchical World Models as Visual Whole-Body Humanoid Controllers](/papers/2405.18418): a visual puppeteer planner that drives a frozen, MoCap-pretrained humanoid tracker with end-effector commands.
- [There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning](/papers/2508.04451): a two-level PPO attacker for multi-turn dialogue, where the high level picks a guide per turn and token-level credit is assigned by masking.
- [Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](/papers/2512.12993): a PPO high level over a 16-value terrain latent that sets foot and speed targets for a quadratic-program controller on a bipedal robot.
- [Feudal Networks for Visual Navigation](/papers/2402.12498): a three-level image-goal navigator with a latent memory map and a waypoint network, trained from human clicks without RL.
- [Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](/papers/2306.06766): a PPO global policy over a fixed planner and local controller, with radio-physics terms in the reward.
- [Object-Centric Dexterous Manipulation from Human Motion Data](/papers/2411.04005): a behaviour-cloned wrist planner, a PPO finger controller with a bounded wrist residual, and a feedback loop over successful rollouts.
- [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](/papers/2411.07760): offline goal reaching with discrete landmark planning from a VQ-VAE and two IQL policies.
- [Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance](/papers/2411.12982): a diffusion Guider that predicts the next contact, and a diffusion Actor with a Q-value term.
- [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](/papers/2603.03596): a robot policy with a short-horizon video encoder and a long-horizon text memory written by its high-level policy.
- [VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation](/papers/2509.20322): a depth-based keypoint generator over a general keypoint tracker, both distilled from state-based teachers, for a humanoid.