Hierarchical reinforcement learning

Overview

A hierarchical policy has two levels. A high-level policy emits a goal or a subtask, and a low-level policy chooses the actions under it, such as the next word, a robot action chunk or the next utterance in a dialogue. How often the high level acts is a design choice. The LeakGAN manager emits a new goal at every word [arxiv:1709.08624 §Hierarchical Structure of G]. The high level of the red-teaming card picks a guide once per dialogue turn 2508.04451 §4.2. The MEM card does not state how often its high-level policy updates 2603.03596 §3.1. The humanoid puppeteer plans a new command at every control step, since both levels run at the same frequency 2405.18418 §4.1. The bipedal walker's high level acts at 33 Hz, while its low level runs at 1 kHz 2512.12993 §III.C [arxiv:2512.12993 §III.F].

The navigation card's mid-level network outputs a new pixel waypoint for each decision, and its worker repeats that waypoint over two environment steps 2402.12498 §4.2 2402.12498 §4.3.

How the high level is trained is where the variants differ. LeakGAN's manager is trained with a policy gradient, and its reward is the change in discriminator features over the next few steps, so the manager is trained to align its goal with that change 1709.08624 §Training of G. The worker gets a separate intrinsic reward, the cosine between the feature change and the recent goals 1709.08624 §Training of G. The red-teaming high level is trained with PPO on a one-step critic built from the target's reply to the utterance 2508.04451 §4.3 2508.04451 eq. 1. MEM's high-level policy is trained to predict its next language memory from the previous memory, the current observation and the task. Its training labels are written by an off-the-shelf LLM, and the card gives no reward or RL objective for the high level 2603.03596 §3.2. The puppeteer is not trained with a policy gradient at all: it is a planner that is trained from scratch for each task on the task reward, and its low-level tracker is fixed 2405.18418 §3.2. The bipedal walker's high level is a PPO policy whose reward is a tracking reward on the foot and speed targets it sets 2512.12993 §III.E 2512.12993 §III.F.

Several of the newer cards do not train the high level with a policy gradient. FeudalNav's mid-level network is trained on human point-clicks and no RL is used at all 2402.12498 §4.2 2402.12498 §1. The object-centric planner is trained by behaviour cloning on human wrist motion, and PPO trains only the finger controller 2411.04005 §4.1 2411.04005 §4.2. QPHIL's planner predicts the next landmark token by teacher forcing, and its two low-level policies are trained offline with IQL 2411.07760 §4.2 2411.07760 §4.3. HDP's Guider is a conditional diffusion model trained with a denoising loss on demonstrated contacts 2411.12982 §3.2. PIRL-WIN is the one new card that trains its high level with PPO, while its local policy is behaviour-cloned and then frozen 2306.06766 §IV-B. VisualMimic also uses teacher and student policies at both levels. Its high level, a keypoint generator, is trained with PPO on object state and then distilled into a depth-only student with DAgger [arxiv:2509.20322 §3.2]. Its low level, a keypoint tracker, is distilled the same way from a PPO-trained teacher 2509.20322 §3.1.

Eleven cards cover the concept so far. LeakGAN applies the hierarchy to long text generation. MEM applies a subtask-level hierarchy with a memory to robot manipulation over minutes. The red-teaming card applies a two-level PPO scheme to a multi-turn attacker that is scored by a harm classifier. The puppeteer applies a planning-based high level to whole-body humanoid motion from pixels, and the bipedal walker applies a PPO high level to terrain-aware walking. FeudalNav applies a three-level, RL-free navigator to image goals. PIRL-WIN applies a PPO waypoint policy with physics-shaped rewards to wireless indoor navigation. The object-centric card applies a planner and a PPO finger controller to dexterous manipulation, QPHIL applies discrete landmark planning to offline maze navigation, and HDP applies a contact-predicting diffusion model above a diffusion actor, and VisualMimic applies a distilled keypoint generator and tracker to humanoid loco-manipulation with depth input. The hierarchical framework LeakGAN builds on is named in its lineage notes, but its content is not used here.

Variants

Manager–worker over discriminator features (LeakGAN)

A convolutional discriminator reads the prefix generated so far and produces a feature vector. A manager LSTM maps that vector to a unit-length goal, and a goal embedding is formed from the sum of the last c goals, with c = 4 in the paper's experiments 1709.08624 §Hierarchical Structure of G 1709.08624 §Experiment. A worker LSTM reads the current word and outputs a matrix with one row per vocabulary word. Its scores are the row-by-goal products, divided by a temperature, and a softmax gives the next-word distribution 1709.08624 §Hierarchical Structure of G.

Training runs in stages. The discriminator is first trained to separate real from generated text, and the generator is pre-trained by maximum likelihood 1709.08624 §Training of G. The adversarial loop then alternates generator updates with discriminator updates. The manager is updated with a policy gradient weighted by a Monte Carlo estimate of the expected sentence reward. The worker is updated with a REINFORCE-style loss weighted by the intrinsic reward 1709.08624 §Training of G. The paper also interleaves one supervised generator epoch after every fifteen adversarial epochs 1709.08624 §Interleaved Training. Card: Long Text Generation via Adversarial Training with Leaked Information.

Subtask policy with language memory (MEM)

The policy is factored into a low-level policy that outputs an action chunk and a high-level policy that outputs the next subtask and an updated language memory 2603.03596 §3.1. The high level sees the task goal in text, its own previous memory and the current observation. Because it predicts the new memory from the old one, it decides when and how to update the memory 2603.03596 §3.2. The low level sees the last K dense frames, and the value of K is not stated 2603.03596 §3.1.

Memory labels come from an LLM, which is given the subtask instructions with a success flag for each. It is asked to keep only what still matters for the future, so object attributes are reduced to a count and a location. A failed attempt leaves the memory unchanged until the subtask succeeds 2603.03596 §3.2 2603.03596 §4.1.

A separate short-horizon video encoder gives the backbone a view of recent camera frames. Every fourth layer attends across time, and tokens of past frames are dropped in the upper layers, so the backbone sees one frame's worth of tokens 2603.03596 §3.3. The encoder adds no learnable parameters and equals the single-image ViT when K = 1 2603.03596 §3.3. The authors build this on pi0.6, which is initialised from a Gemma3-4B VLM and has an 860M parameter action expert 2603.03596 §3.4. Card: MEM: Multi-Scale Embodied Memory for Vision Language Action Models.

Two-level PPO over dialogue turns (red-teaming attacker)

The state is the conversation so far. Each turn, the high-level policy reads the history and a target goal, and it outputs a guide. The low-level policy then writes the utterance one token at a time under that guide. The target model answers, and the answer is appended to the history 2508.04451 §4.1 2508.04451 §4.2. The high-level guide is either a topic plus a persona from a fixed template pool, or a short free-text guide from an LLM 2508.04451 §4.3 2508.04451 app. F.

Both levels are trained with PPO 2508.04451 §4.3. The reward for a turn is the harm gain: the classifier's "unsafe" probability for the target's reply, minus the same probability for the utterance 2508.04451 §4.2. The low-level turn reward adds a cosine similarity between reference-model embeddings of the guide and the utterance. The authors state this term keeps the low level on its guide and stops it from overfitting one utterance 2508.04451 eq. 2. Credit inside an utterance is assigned by masking single tokens and pairs of its top-attended tokens, and measuring the drop in the turn reward when a mask is applied 2508.04451 eq. 3 [arxiv:2508.04451 eq. 4]. The paper leaves the reward for tokens outside the top-attended set undefined 2508.04451 eq. 4.

The authors' ablation on HarmBench, ASR@5 with Llama-3.1-8B, gives 75.2 with both levels trained and 31.2 with no high-level policy. Freezing the high level gives 47.0, and freezing the low level gives 40.7 2508.04451 tab. 3. Card: There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning.

Puppeteer over a frozen whole-body tracker (humanoid)

Two agents are trained separately, and both are world-model planners 2405.18418 §3. The low level is a tracker. Its input is the proprioceptive state and a 15-value command, which holds the relative 3D positions of five end-effectors taken from a reference motion. It outputs a sequence of joint actions for a 56-joint humanoid 2405.18418 §3.1 2405.18418 app. E. One tracker is pretrained on offline rollouts for 836 motion-capture clips, with offline and online data mixed in each batch 2405.18418 §3.1 2405.18418 §4.1.

The high level, called the puppeteer, sees the same proprioceptive state and a 64x64 RGB image from a third-person camera. Its action space is the tracker's command space, and it never drives joints directly 2405.18418 §3.2. The tracker is frozen while the puppeteer learns, and both act at the same frequency 2405.18418 §3.2 2405.18418 §4.1. The puppeteer is trained from scratch for each downstream task, using only the task reward 2405.18418 §3.2 2405.18418 §4.1. Planning, not a policy gradient, produces its actions: both levels choose actions by model predictive control with a learned policy prior as a warm start 2405.18418 §2. The paper also adds a learned termination head, which the planner uses to predict early episode ends 2405.18418 §3.3. Card: Hierarchical World Models as Visual Whole-Body Humanoid Controllers.

Hierarchical PPO over a terrain latent (bipedal walker)

The terrain is compressed first. A convolutional VAE maps a 20x40 local height map to a 16-value latent 2512.12993 §II.B. The high level is an MLP policy trained with PPO. Its input is a reduced-order state built from the base's LIP-like state, the speed error, the swing foot, the latent and the last action [arxiv:2512.12993 §III.B] 2512.12993 eq. 5. Its output is six task-space values: the swing foot's landing position, a swing clearance and two speed offsets 2512.12993 §III.C 2512.12993 eq. 6.

The low level is not learned. A task-space inverse dynamics problem, solved as a quadratic program, tracks the foot and base targets that the high level sets, and it runs at 1 kHz 2512.12993 §III.D [arxiv:2512.12993 §III.F]. The reward is a weighted sum of tracking terms for speed and swing-foot targets, plus an action-smoothness term, each of Gaussian form 2512.12993 eq. 11 2512.12993 §III.E. The authors also train a second encoder that predicts the same latent from two depth cameras, pulling it toward the height-map latent 2512.12993 §IV.C. The paper also tests history-aware encoders that stack several past height maps 2512.12993 §II.D. Card: Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations.

Manager–worker over a learned place memory, without RL (FeudalNav)

FeudalNav is a three-level navigator for the image-goal task, and it is trained without reinforcement learning. The top level keeps a latent memory map. Its features come from a self-supervised clustering model, and a new view joins the nearest cluster center when the distance is below a threshold, otherwise it starts a new center 2402.12498 §4.1 2402.12498 §5.2. The mid-level network is a fine-tuned ResNet-18 that takes the RGBD observation and the memory map, and it predicts a pixel to move toward 2402.12498 §4.2. The worker maps that pixel to a turn, a step, or a turn and a step, on a fixed image grid. It then re-locates the waypoint and repeats it for a second environment step 2402.12498 §4.3. When a keypoint match with the goal image is confident enough, the match replaces the waypoint 2402.12498 §4.2. The paper argues that a learned memory can stand in for a metric graph, and that the hierarchy eases learning from little data 2402.12498 §1. Card: Feudal Networks for Visual Navigation.

Author-reported on Gibson with the NRNS protocol, the curved-path success average is 40.50 for the stacked variant, against 23.97 for NRNS without noise 2402.12498 tab. 2. On straight paths the stacked variant has higher success than NRNS, but a lower SPL 2402.12498 tab. 2 2402.12498 §6.2. The evaluation is in simulation only 2402.12498 §8.

Waypoint policy trained with PPO over a physics-shaped reward (PIRL-WIN)

A global policy trained with PPO sits above a fixed planner and a local visual policy 2306.06766 §IV. From a radio input built from the five strongest paths, the global policy outputs an angle and a link-state estimate. A goal is placed along that angle, at a distance set by the link-state estimate, and a Fast Marching planner on the map turns it into a short-term target 2306.06766 §IV-A. The local policy, a recurrent network on a pretrained ResNet-18, is trained by behaviour cloning and then kept fixed 2306.06766 §IV-B. The reward adds three radio-physics terms to a distance cost: a monotonicity term on the link state, a term that follows the angle of arrival, and an SNR-gradient term in the weakest link state 2306.06766 §III-B. The paper says the shaped reward is not potential-based, so the optimal policy under it differs from the shortest path 2306.06766 §III-B. Card: Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning.

Author-reported, on six unseen Gibson maps, the normalised path length in the 2+-NLOS category is 2.46 to 2.82, against 4.50 to 5.14 for end-to-end RL 2306.06766 tab. 3. On maps P and T, 2+-NLOS success is 1.00, against 0.40 to 0.45 for end-to-end RL and 0.85 to 0.90 for a wireless-assisted heuristic 2306.06766 tab. 4. The paper's ± values are not defined 2306.06766 tab. 3.

Behaviour-cloned planner over a PPO finger controller (object-centric dexterous manipulation)

The task is defined by a sequence of object poses. A transformer planner, trained by behaviour cloning on human wrist motion from ARCTIC, outputs a 6-DoF pose for each wrist at each of ten steps 2411.04005 §4.1. A PPO policy outputs finger joint actions, plus a wrist residual bounded to ±4 cm and ±0.5 rad that corrects the planned wrist pose 2411.04005 §4.2. Its reward is an exponential of a weighted sum of rotation, translation and joint-angle gaps 2411.04005 §4.2 2411.04005 app. B. Successful rollouts are fed back into the planner's training data over several rounds 2411.04005 §4.3 2411.04005 alg. 1. Card: Object-Centric Dexterous Manipulation from Human Motion Data.

Author-reported. In simulation, one policy per object completes 83.8% of trained trajectories with the feedback loop, against 59.6% for the ablated variant 2411.04005 tab. 4. On six real objects, one policy per object trained on one goal trajectory completes 64.3% to 100%, against 4.5% to 56.1% for vanilla RL 2411.04005 tab. 2.

Discrete landmark plan with offline IQL policies (QPHIL)

QPHIL is a hierarchical method for offline goal reaching. A VQ-VAE maps each state to one of k learned landmark tokens, trained with reconstruction, commitment and a temporal triplet loss 2411.07760 §4.1. A transformer plans a sequence of landmark tokens from the goal token and the compressed history, trained by teacher forcing 2411.07760 §4.2. Two IQL-trained policies execute the plan: a landmark policy moves the agent through the planned tokens, and a goal policy takes over for the final approach 2411.07760 §4.3. An optional stitching step splices recorded trajectories that visit the same landmarks, which the authors offer as a cheaper alternative to fine-tuning the planner with offline RL 2411.07760 §4.2. The method assumes the low-level policy can reach any state inside a landmark from any other state in it 2411.07760 §4.2. Card: Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning.

Author-reported. On AntMaze-Ultra diverse, QPHIL without augmentation reaches 70.3 ± 6.9, against 52.6 ± 8.7 for HIQL in its "w/o repr." setting 2411.07760 tab. 1. On the new AntMaze-Extreme map, the augmented variant reaches 50.0 on play and 39.5 on diverse, against 22.8 and 21.9 for the same HIQL setting 2411.07760 fig. 5.

Diffusion contact guider over a diffusion actor (HDP)

Two conditional diffusion models run in sequence. The Guider predicts the next contact target, which is each finger's position in the object frame plus a flag for whether it touches, from the recent observations 2411.12982 §3.2. The Actor denoises a 16-step action sequence conditioned on the observation and that contact 2411.12982 tab. 8. Its loss adds a Q-value term, weighted by η = 0.001, on an action denoised in one step 2411.12982 §4.4 2411.12982 tab. 9. The critic is trained on demonstrated transitions, with a reward for reaching the target within a short window 2411.12982 §3.2. Training targets come from the demonstrations: each demonstration is cut where finger contact changes, and phases with similar object poses are merged 2411.12982 §4.1. The method needs an object point cloud, placed in the world frame from the object pose for rigid objects 2411.12982 §4.2. Card: Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance.

Author-reported. On the real UR5e Move-T task, stage-2 success is 0.60 to 0.75 for HDP and 0.30 to 0.45 for Diffusion Policy, depending on the dataset 2411.12982 tab. 5. On cloth unfolding over 50 trials, success is 0.76 against 0.60 2411.12982 tab. 7. The simulation comparison uses a smaller Diffusion Policy baseline that the authors reproduced, so it is not a comparison with published numbers 2411.12982 §6.2 2411.12982 tab. 2.

Keypoint generator over a distilled keypoint tracker (VisualMimic)

The low level is a task-agnostic keypoint tracker. Its input is an 18-value command, made of the root position error and the offsets of five keypoints (head, two hands, two feet) from the root, and it outputs joint actions 2509.20322 §3.1 2509.20322 eq. 1. The high level, the keypoint generator, produces that command. It reads egocentric depth and proprioception, and it is retrained for each task 2509.20322 §3 [arxiv:2509.20322 §3.2].

Both levels are trained as teacher and student. The tracker teacher sees the reference motion for the next 2 seconds and privileged proprioception, and it is trained with PPO. Its student sees only proprioception and the command, and is distilled with DAgger 2509.20322 §3.1. The generator teacher sees object state and is trained with PPO on a task reward built from approach, forward-progress, force and two regularising terms. Its depth student is distilled with DAgger 2509.20322 §3.2 2509.20322 eq. 4 2509.20322 eq. 9. The paper gives no weights for these reward terms 2509.20322 §3.2.

Two stabilisers are added. While the tracker trains, each command value is multiplied by a factor drawn from U(0.5, 1.5). The generator's output is clipped to a band of μ ± 1.64σ, estimated from human motion 2509.20322 §3.3. The paper reports that removing either one hurts training 2509.20322 §4.5. Card: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation.

Author-reported, in simulation, the vision-based student's Push Box distance over a one-minute rollout is 37 ± 28 m, against 152 ± 36 m for the state-based teacher and 2 ± 0 m for a student with no vision 2509.20322 tab. 2. Without command noise, the vision student's Push Box distance falls to 2 ± 1 2509.20322 tab. 3. Real-robot results on a Unitree G1 are qualitative, with no success rates 2509.20322 §4.2.

When to use which

Compare the variants by what the high level outputs and what data or signal trains it. Each row names the card that supports it.

Implementation sketch

Our own pseudocode for the common two-level loop. The comments name the paper section each step follows. The LeakGAN generation loop, which the variant above uses, follows it. The red-teaming variant fits the same loop, with its high level choosing a guide per turn and its low level writing the utterance under that guide 2508.04451 §4.2. The puppeteer fits it with a planned high level and a frozen low level, and the bipedal walker fits it with a PPO high level and a fixed low level 2405.18418 §3.2 2512.12993 §III.C. VisualMimic fits it with a keypoint generator as the high level and a distilled keypoint tracker as the low level 2509.20322 §3.1 2509.20322 §3.2. FeudalNav fits it with a waypoint network as the high level and a fixed image-grid rule as the low level 2402.12498 §4.2 2402.12498 §4.3. PIRL-WIN fits it with a PPO global policy and a frozen, behaviour-cloned local policy 2306.06766 §IV-B.

# Two-level loop, one step at a time  [arxiv:2603.03596 §3.1]
# High level picks a subgoal or subtask; low level acts under it

def hierarchical_loop(high, low, goal, steps):
    # Start value: LeakGAN uses zeros; MEM does not state one [arxiv:2603.03596 §3.2]
    high_state = initial_state
    for t in range(steps):
        obs = observe()
        # High level writes the subtask and its updated memory  [arxiv:2603.03596 §3.1] [§3.2]
        subtask, high_state = high(obs, high_state, goal)
        # Low level emits actions under the subtask  [arxiv:2603.03596 §3.1]
        actions = low(obs, subtask)
        execute(actions)

# Generate one sequence [arxiv:1709.08624 §Hierarchical Structure of G]
# T words, feature size d, goal size k, vocabulary size V

def generate(discriminator, manager, worker, T, c, alpha):
    # Both recurrent states start at zero [arxiv:1709.08624 app. Pseudo Code]
    manager_state = zeros(hidden_size)
    worker_state = zeros(hidden_size)
    word = START_TOKEN
    tokens, goals = [], []
    for t in range(T):
        # Leaked feature of the prefix so far [arxiv:1709.08624 §Leaked Features from D]
        feature = discriminator.features(tokens)     # (d,)
        # Manager maps the feature to a unit goal [arxiv:1709.08624 §Hierarchical Structure of G]
        raw_goal, manager_state = manager(feature, manager_state)
        goals.append(raw_goal / norm(raw_goal))      # (k,)
        # Goal embedding from the last c goals [arxiv:1709.08624 §Hierarchical Structure of G]
        # Fewer than c goals exist at the start: not stated
        goal_embedding = W_psi @ sum(goals[-c:])     # (k,)
        # Worker encodes the word [arxiv:1709.08624 §Hierarchical Structure of G]
        output_matrix, worker_state = worker(word, worker_state)  # (V, k)
        # Score words by row dot goal, then soften [arxiv:1709.08624 §Hierarchical Structure of G]
        probs = softmax((output_matrix @ goal_embedding) / alpha)  # (V,)
        word = sample(probs)
        tokens.append(word)
    return tokens

The two LeakGAN training rules differ only in what is weighted. The manager's loss weights each goal's alignment with the feature change over the next c steps by the estimated reward; the worker's loss weights the log-probability of each chosen word by the mean cosine between recent feature changes and recent goals 1709.08624 §Training of G.

Contradictions

No disagreement between the papers. They report different tasks and metrics, so their numbers cannot be compared directly, and each card's results are author-reported. The red-teaming paper's baselines were reproduced by its authors in one shared setting 2508.04451 §5, so its baseline numbers are not the original papers' numbers. The humanoid and bipedal results are in simulation and use different robots and baselines, so they do not bear on each other or on the text and robot-manipulation cards. The navigation and manipulation cards are in the same position: they use different simulators, metrics (normalised path length, SPL, completion, stage success) and baselines.

Within LeakGAN, the paper's appendix formula list numbers its equations differently from the main text, so equation citations must name which list they come from (see the LeakGAN card's Pitfalls section). The authors' code differs from the paper in several settings, including the temperature convention, the reward rescaling constants and the discriminator's convolution windows. These are discrepancies between paper and code, not between papers. Within MEM, the post-training memory horizon is 54 seconds 2603.03596 §3.4 but up to one minute 2603.03596 §4.3. The MEM card has no code to check against.

Within the red-teaming paper, the base model is named as Llama-3-8B-Instruct in its experiments section but as Llama-3.1-8B-Instruct in its appendix and tables 2508.04451 §5 2508.04451 app. D.1. Some cells of tab. 2 and tab. 4 also differ for the same method and target, such as LLM-HL on Llama-8B HarmBench, 95.0 against 97.0 2508.04451 tab. 2 2508.04451 tab. 4. The paper does not explain these gaps.

Within the humanoid paper, the human-study shares differ between text and table. The text gives 4% for equal and 0% for the baseline, while the table gives 1.8% and 0.4% 2405.18418 §4.2 2405.18418 tab. 6. The card uses the table's counts. Separately, the paper's tracking-success threshold does not appear in its released code, so its tracker success rates may not match the code's success flag 2405.18418 app. B 2405.18418 §Paper vs code.

Within the bipedal paper, Table 1 and the text disagree on which end-to-end baselines take a full height map and which take sampled heights 2512.12993 tab. 1 2512.12993 §IV.B. Check the text before citing either.

Within FeudalNav, the single-view straight-path average is 45.73 success and 37.69 SPL in tab. 3, but 47.57 and 38.41 in tab. 2. The card treats tab. 2 as consistent 2402.12498 tab. 2 2402.12498 tab. 3. The text's curved-path gains of 69% in success and 27% in SPL come from two different variants: the stacked variant's success and the single-view SPL 2402.12498 §6.2 2402.12498 tab. 2.

Within PIRL-WIN, the released code differs from the paper's reward and feeds the dataset's link-state label to its policy, so the paper's numbers cannot be attributed to the released code without re-running it 2306.06766 §III-B 2306.06766 §Paper vs code.

Within QPHIL, the text says the method outperforms all others on larger maps, but the table shows HIQL in its "w/ repr." setting ahead on large diverse 2411.07760 §5.2 2411.07760 tab. 1. The seed text says eight seeds, but the list has seven values 2411.07760 §5.1 2411.07760 §8. The one-sample re-planning numbers match neither QPHIL row of the main results 2411.07760 app. D 2411.07760 tab. 1. The non-extreme baselines are copied from the HIQL paper, while the extreme-map baselines were tuned again, so the tuning effort differs 2411.07760 app. A.

Within the object-centric paper, the 47.3% average advantage over vanilla RL in the text has no stated averaging, and a mean over its own table gives a different gap 2411.04005 §5.2 2411.04005 tab. 3. The text's claim of more than a 50% improvement does not hold for Microwave, at 100 against 60.6 2411.04005 §5.4 2411.04005 tab. 2. The augmentation ranges have no stated units, while the appendix says centimetres and degrees 2411.04005 app. A.

Within VisualMimic, §4.1 pairs the 3.8 kg box with a push distance of 37 m per minute. The 3.8 kg figure is the real-world box 2509.20322 §1, but the 37 m figure matches the simulated 4 kg Push Box row 2509.20322 §4.1 2509.20322 tab. 2. The paper does not say which box the figure refers to.

Within HDP, the text gives an 84.1% average success gain for Move-T, but a simple mean of the stage-2 success rates in tab. 5 gives 81.8%, and the paper does not say how 84.1% is computed 2411.12982 §7.1 2411.12982 tab. 5. The 20.8% headline is not derived in the text. The card's recomputation shows that an unweighted mean of the simulation improvements in tab. 2 and tab. 3 together with the two real-robot gains reproduces it 2411.12982 tab. 2 2411.12982 tab. 3 2411.12982 §7.2.

Papers

• Long Text Generation via Adversarial Training with Leaked Information: a manager–worker generator for text GANs that steers each word by the discriminator's intermediate features.
• Hierarchical World Models as Visual Whole-Body Humanoid Controllers: a visual puppeteer planner that drives a frozen, MoCap-pretrained humanoid tracker with end-effector commands.
• There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning: a two-level PPO attacker for multi-turn dialogue, where the high level picks a guide per turn and token-level credit is assigned by masking.
• Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations: a PPO high level over a 16-value terrain latent that sets foot and speed targets for a quadratic-program controller on a bipedal robot.
• Feudal Networks for Visual Navigation: a three-level image-goal navigator with a latent memory map and a waypoint network, trained from human clicks without RL.
• Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning: a PPO global policy over a fixed planner and local controller, with radio-physics terms in the reward.
• Object-Centric Dexterous Manipulation from Human Motion Data: a behaviour-cloned wrist planner, a PPO finger controller with a bounded wrist residual, and a feedback loop over successful rollouts.
• Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning: offline goal reaching with discrete landmark planning from a VQ-VAE and two IQL policies.
• Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance: a diffusion Guider that predicts the next contact, and a diffusion Actor with a Q-value term.
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models: a robot policy with a short-horizon video encoder and a long-horizon text memory written by its high-level policy.
• VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation: a depth-based keypoint generator over a general keypoint tracker, both distilled from state-based teachers, for a humanoid.