---
kind: concept
slug: hindsight-relabeling
title: Hindsight relabeling
summary: >-
  Turning failed or suboptimal rollouts into training samples that show how to recover. Hindsight Planner rewrites the task text from structured arguments and has an LLM reflect on the failure with the expert rollout in view. Author-reported: removing it lowers ALFRED valid SR by 2.08 (seen) and 2.68 (unseen). Only one card so far.
tags: [hindsight, embodied-ai, llm-planning, reinforcement-learning]
papers: ["2412.19562"]
related: [experience-replay, goal-conditioned-reinforcement-learning, llm-planning, in-context-learning, few-shot-example-selection]
status: draft
updated_at: 2026-10-25
---

## Overview

A learner that fails a task still produces a trajectory. Hindsight relabeling
reuses that trajectory as training data by rewriting it so that it matches a
task description the learner can learn from. The authors of the
Hindsight Planner describe earlier hindsight methods, such as HER, as changing
the statistic the learner is trained on, and they argue that their rewrite
keeps that statistic unchanged [arxiv:2412.19562 §2] [arxiv:2412.19562 §4.3].
The card does not describe HER itself, so the original definition is in the
HER paper, not on this page.

The shared core, as the card describes it, has three parts: a failed
rollout, a reference rollout that shows a correct solution, and a rewrite
that turns the failed rollout into samples about how to recover from its
wrong step [arxiv:2412.19562 §4.3] [arxiv:2412.19562 alg. 3]. Every variant
has to answer what the rewrite is conditioned on, and which samples are
kept for which learner.

## Variants

### LLM-written hindsight samples (Hindsight Planner)

The task text is first rewritten from annotated PDDL-style arguments, using a
prompt generator [arxiv:2412.19562 alg. 3 l.7] [arxiv:2412.19562 §5.1]. For
each suboptimal rollout, the LLM writes a short reflection, given the
ground-truth rollout. It then completes the rollout, guided by that
reflection. A separate call writes a critic sample from the same rollout
[arxiv:2412.19562 §4.3] [arxiv:2412.19562 alg. 3 l.8-10]. The result is a
pool of relabeled samples. The actor that is prompted with ground-truth
samples and the actor that is prompted with relabeled samples are kept apart
at deployment [arxiv:2412.19562 §4.3]. Card: [Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following](/papers/2412.19562).

The relabeled samples are used as in-context examples, not as weight
updates. The card reports no gradient training for the LLM [arxiv:2412.19562 §5.1].

## When to use which

The card supports only one variant, so the table compares the setting it was
tested in with settings it was not tested in.

| Situation | Use | Why |
|---|---|---|
| Expert solutions for training tasks, and an agent that can be run on them | Hindsight relabeling with an LLM rewrite | The authors report a 2.08 (seen) and 2.68 (unseen) valid SR drop when the method is removed [arxiv:2412.19562 §5.3] [tab. 4]. |
| No ground-truth rollout, or no structured task annotation | Not covered by this card | The rewrite reads both the reference rollout and the annotated arguments [arxiv:2412.19562 alg. 3 l.5, l.7]. |
| Learner is a gradient-trained RL policy with a replay buffer | See [experience replay](/concepts/experience-replay) | This card studies an in-context LLM planner, not gradient-based replay. |
| Need to know per-sample cost or latency | Not decidable from this card | The number of LLM calls per sample and the token cost are not reported [arxiv:2412.19562 §4.3]. |
| Domains other than household instruction following | Not decidable from this card | Only ALFRED with GPT-4 Turbo is tested [arxiv:2412.19562 §5.1]. |

## Implementation sketch

Language-agnostic pseudocode of the LLM variant. The LLM and the rewrite step
are passed in; the card does not give their prompts beyond the templates in
its appendix [arxiv:2412.19562 app. B].

```pseudocode
# Relabel one failed rollout into training samples  [arxiv:2412.19562 alg. 3]
# Input:  failed rollout, reference rollout, task, LLM
# Output: one actor sample and one critic sample

def relabel(failed, reference, task, llm):
    # Rewrite the task text from its structured arguments  [§4.3] [§5.1]
    new_task = rewrite_task(task.pddl_args)
    # Reflect on the failure, with the reference rollout in view  [§4.3]
    reflection = llm.reflect(new_task, failed, reference)
    # Complete the rollout, guided by the reflection  [alg. 3 l.9]
    actor_sample = llm.complete(new_task, failed, reference, reflection)
    # Write a critic sample from the same rollout  [alg. 3 l.10]
    critic_sample = llm.score(new_task, failed, reference)
    return [actor_sample, critic_sample]
```

Where variants differ: only the LLM variant is covered here. Whether the
reflection step is needed, and how many samples to keep per rollout, is
stated for this card only. The paper keeps one actor and one critic sample per
rollout [arxiv:2412.19562 alg. 3].

## Contradictions

Within one paper, the text and the table disagree about where the method
helps. The text says the gain is largest on long tasks such as Cool and Heat
[arxiv:2412.19562 §5.3]. Table 3 shows the large gap on Cool (21.43 with the
method, 10.31 without it), but nearly no gap on Heat (7.48 with, 7.47
without) [arxiv:2412.19562 tab. 3]. Only one card cites this concept, so there
is no cross-paper disagreement to report.

## Papers

- [Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following](/papers/2412.19562): an LLM planner that relabels failed ALFRED rollouts with hindsight prompts so a second actor learns to recover.