Hindsight relabeling

Overview

A learner that fails a task still produces a trajectory. Hindsight relabeling reuses that trajectory as training data by rewriting it so that it matches a task description the learner can learn from. The authors of the Hindsight Planner describe earlier hindsight methods, such as HER, as changing the statistic the learner is trained on, and they argue that their rewrite keeps that statistic unchanged 2412.19562 §2 2412.19562 §4.3. The card does not describe HER itself, so the original definition is in the HER paper, not on this page.

The shared core, as the card describes it, has three parts: a failed rollout, a reference rollout that shows a correct solution, and a rewrite that turns the failed rollout into samples about how to recover from its wrong step 2412.19562 §4.3 2412.19562 alg. 3. Every variant has to answer what the rewrite is conditioned on, and which samples are kept for which learner.

Variants

LLM-written hindsight samples (Hindsight Planner)

The task text is first rewritten from annotated PDDL-style arguments, using a prompt generator 2412.19562 alg. 3 l.7 2412.19562 §5.1. For each suboptimal rollout, the LLM writes a short reflection, given the ground-truth rollout. It then completes the rollout, guided by that reflection. A separate call writes a critic sample from the same rollout 2412.19562 §4.3 2412.19562 alg. 3 l.8-10. The result is a pool of relabeled samples. The actor that is prompted with ground-truth samples and the actor that is prompted with relabeled samples are kept apart at deployment 2412.19562 §4.3. Card: Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following.

The relabeled samples are used as in-context examples, not as weight updates. The card reports no gradient training for the LLM 2412.19562 §5.1.

When to use which

The card supports only one variant, so the table compares the setting it was tested in with settings it was not tested in.

Implementation sketch

Language-agnostic pseudocode of the LLM variant. The LLM and the rewrite step are passed in; the card does not give their prompts beyond the templates in its appendix 2412.19562 app. B.

# Relabel one failed rollout into training samples  [arxiv:2412.19562 alg. 3]
# Input:  failed rollout, reference rollout, task, LLM
# Output: one actor sample and one critic sample

def relabel(failed, reference, task, llm):
    # Rewrite the task text from its structured arguments  [§4.3] [§5.1]
    new_task = rewrite_task(task.pddl_args)
    # Reflect on the failure, with the reference rollout in view  [§4.3]
    reflection = llm.reflect(new_task, failed, reference)
    # Complete the rollout, guided by the reflection  [alg. 3 l.9]
    actor_sample = llm.complete(new_task, failed, reference, reflection)
    # Write a critic sample from the same rollout  [alg. 3 l.10]
    critic_sample = llm.score(new_task, failed, reference)
    return [actor_sample, critic_sample]

Where variants differ: only the LLM variant is covered here. Whether the reflection step is needed, and how many samples to keep per rollout, is stated for this card only. The paper keeps one actor and one critic sample per rollout 2412.19562 alg. 3.

Contradictions

Within one paper, the text and the table disagree about where the method helps. The text says the gain is largest on long tasks such as Cool and Heat 2412.19562 §5.3. Table 3 shows the large gap on Cool (21.43 with the method, 10.31 without it), but nearly no gap on Heat (7.48 with, 7.47 without) 2412.19562 tab. 3. Only one card cites this concept, so there is no cross-paper disagreement to report.

Papers

• Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following: an LLM planner that relabels failed ALFRED rollouts with hindsight prompts so a second actor learns to recover.