Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following

In plain words

A household robot in a simulated house gets a sentence such as "put a warm apple in the fridge". It sees the scene only one view at a time, and it has to work out a long chain of steps: find the object, open the appliance, heat it, put it away. This paper builds a planner that uses a large language model for those decisions and that does not need a big training set. It learns from a small number of solved examples in the prompt.

The planner checks what it can see and guesses the parts of the task it cannot see yet, such as which container is needed. It then imagines several short futures, scores them, and carries out only the first step of the best one. After each step it looks again and plans anew, so a wrong move does not end the task.

The authors' main idea is to learn from mistakes that happened during practice. The planner's own failed attempts are rewritten, with the help of the language model and the correct solution, into attempts that succeed. A second planner is then shown these rewritten attempts, so it knows how to recover from a wrong move. The paper reports that this closed loop, with few examples, reaches results that come close to a fully supervised agent.

TL;DR

• Few-shot LLM planner for ALFRED. It uses about 140 solved trajectories and no gradient training of the LLM §5.1.
• Each step predicts hidden PDDL arguments, then runs an actor-critic beam search over sub-goals §4.4 §4.5 alg. 1.
• Hindsight prompts relabel suboptimal rollouts with the help of ground-truth rollouts, so one actor can recover from errors §4.3 alg. 3.
• Author-reported: test-seen SR 25.51 and test-unseen SR 18.77, against LLM-Planner at 15.33 and 13.41 tab. 1.
• Only ALFRED and GPT-4 Turbo are tested. Low-level control is reused from HLSM §5.1.

Problem

Embodied instruction following on ALFRED needs long-horizon plans, and the reward arrives only at the end of a task §1. The authors argue that few-shot LLM planners that imitate expert trajectories are not robust. A wrong action can leave the planner in a state it never saw in its examples, and the task then fails §1 fig. 1. The planner also sees only an egocentric view, so the task's hidden state, such as the required container or the device to toggle, is unknown §1 §4.4. Under a few-shot assumption there are not enough trajectories to cover these cases §1.

The paper frames the task as a POMDP, with observations that are partial and latent state that is hidden §1 §3.1 eq. 1 eq. 2.

Method

Overview §4.1. At each time step the planner gets an object list from observation functions, adds the adapter's predicted PDDL arguments to form the state, and asks an actor-critic planner (LLM-PL) for a sub-goal. A frozen BERT maps the output to the legal sub-goal space. A low-level controller (Low-PL) executes the sub-goal. When the sub-goal finishes or fails, the planner replans alg. 1 fig. 3.

Prompts and retrieval §4.2. Each component prompt starts with a short description of the task and of the LLM's role. A frozen BERT encodes the task description. The K nearest examples by Euclidean distance are copied into the prompt as in-context examples, as in LLM-Planner §4.2 §5.1. The prompt for the adapter and the critic follows the template in app. B.

Adaptation module §4.4. The adapter is an LLM. At each step, before reasoning, it predicts seven PDDL-style arguments from the observed object list: the target object, the parent receptacle, the movable receptacle, the toggle target, the object state (clean, heat or cool), whether the object is sliced, and whether two objects are handled §4.4. The object list grows as the agent explores §4.4. Predictions are not made once at the start of the task, as in FILM, but at every step §2.

Task planner §4.5. Two actors and one critic are used. Actor_gt gets in-context examples from the ground-truth pool; Actor_hind gets examples from the hindsight pool. Each actor proposes half of the W candidate actions at deployment app. A. The critic scores candidate rollouts. A generator psi produces the next state for each action. Beam search keeps the B best rollouts per depth and returns the first action of the best rollout as the sub-goal §4.5 alg. 2.

# Hindsight Planner, one replanning call  [alg. 1] [§4.1]
# Input:  instruction, observed objects, sub-goal history
# Output: next sub-goal

def plan_sub_goal(instruction, observed, history):
    # Predict the hidden PDDL arguments from what is seen  [§4.4]
    pddl_args = adapter(instruction, observed)
    # Combine the partial view and the predicted arguments  [§4.1]
    state = (observed, pddl_args)
    # Search over future rollouts; return the first action  [§4.5] [alg. 2]
    sub_goal = beam_search(state, history)
    return sub_goal

# Beam search over rollouts  [alg. 2] [§4.5]
# Input:  actors, critic, generator, state, B, W, U
# Output: first action of the best rollout

def beam_search(state, actors, critic, generator, B, W, U):
    # Start from the current state only  [alg. 2 l.12]
    frontier = [[state]]
    # Expand each rollout by one action per depth  [alg. 2 l.15]
    for depth in range(U + 1):
        candidates = []
        for rollout in frontier:
            # Actors split the W proposals between them  [alg. 2 l.16-18]
            proposals = [
                actor.propose(rollout, W / len(actors))
                for actor in actors
            ]
            for action in flatten(proposals):
                # The generator gives the next state  [alg. 2 l.19]
                next_state = generator(rollout[-1], action)
                extended = rollout + [action, next_state]
                # The critic scores the extended rollout  [alg. 2 l.20]
                value = critic(extended)
                candidates.append((value, extended))
        # Keep the B highest-valued rollouts  [alg. 2 l.21]
        frontier = [r for _, r in top_k(candidates, B)]
    # Pick the best final rollout; return its first action  [alg. 2 l.23-24]
    best = max(frontier, key=critic)
    return best[1]

Hindsight method §4.3. During collection, the agent runs on training tasks and produces suboptimal rollouts. The task text is first rewritten from its PDDL arguments by a prompt generator phi alg. 3 §5.1. The LLM then writes a reflection (Think) on the rollout, given the ground-truth rollout, and completes the rollout guided by that reflection. A separate LLM call writes a critic sample from the same rollout §4.3 alg. 3 app. B. At deployment, Actor_gt is prompted with ground-truth samples, while Actor_hind and the critic are prompted with relabeled samples §4.3. The authors say the rewrite lets the planner recover from a wrong action that has already been taken §4.3 fig. 2 fig. 3.

The authors argue that earlier hindsight methods, such as HER, change the distribution of the information statistic, while this method keeps it unchanged §2 §4.3 eq. 4 eq. 5 eq. 6. Section 3 defines the information-matching objective used for this argument §3.2 eq. 3.

# Build the hindsight pool from suboptimal rollouts  [alg. 3] [§4.3]
# Input:  ground-truth pool, prompt generator phi, LLM
# Output: ground-truth pool plus relabeled samples

def build_hindsight_pool(gt_pool, agent, phi, llm):
    # Relabeled samples start empty  [alg. 3 l.2]
    hind_pool = []
    for sample in gt_pool:
        # Ground-truth rollout, task and PDDL arguments  [alg. 3 l.5]
        gt_rollout, task, pddl = unpack(sample)
        # Suboptimal rollout; the task itself is not an example  [alg. 3 l.6]
        traj = agent.run(task, sample, gt_pool_without(sample))
        # Rewrite the task from its PDDL arguments  [alg. 3 l.7] [§5.1]
        relabeled_task = phi(pddl)
        # Reflect on the suboptimal rollout  [alg. 3 l.8] [§4.3]
        think = llm(relabeled_task, traj, gt_rollout)
        # Complete the rollout, guided by the reflection  [alg. 3 l.9]
        actor_sample = llm(relabeled_task, traj, gt_rollout, think)
        # Write a critic sample from the same rollout  [alg. 3 l.10]
        critic_sample = llm(relabeled_task, traj, gt_rollout)
        hind_pool.append(actor_sample)
        hind_pool.append(critic_sample)
    return gt_pool + hind_pool

Hyperparameters

Results

Author-reported. Setup: ALFRED, 140 few-shot trajectories, GPT-4 Turbo as target LLM, HLSM low-level controller §5.1. Metrics: Success Rate (SR) and Goal Condition (GC) §5.1. "full" rows use the full training set; "few" rows use few examples; "low inst." uses step-by-step instructions tab. 1.

Table 1. Comparison with state-of-the-art, test splits tab. 1.

Table 2. Same low-level controller (HLSM) tab. 2.

Table 3. Ablation by task type, Valid Seen SR tab. 3 (labelled tab:5 in the source).

Table 4. Ablation on Valid Seen and Valid Unseen tab. 4 (labelled tab:3 in the source).

Table 5. Average sub-goal length per task type tab. 5 (labelled tab:4 in the source).

Headline claims. The authors report test-seen SR 25.51 and test-unseen SR 18.77 §1 tab. 1. They report +10.18 and +5.36 SR over LLM-Planner on the two test splits §5.2. With the same low-level controller, the method "nearly matches and even surpasses" the full-data HLSM on some metrics §5.2 tab. 2. Removing the hindsight method lowers Valid Seen SR by 2.08 and Valid Unseen SR by 2.68 §5.3 tab. 4. Removing the adapter lowers them by 2.44 and 4.01 §5.3 tab. 4. The authors say the hindsight method helps most in long tasks such as Cool and Heat §5.3 tab. 3 tab. 5.

Paper vs code

No official code.

Search: the paper and its arXiv metadata have no code link. Author-owned GitHub repositories: yangyuxiao-sjtu/hindsight_planner_page is a project page (GitHub Pages template) with no planner code. Repository search for "hindsight planner" and "hindsight ALFRED planner" found no implementation from the authors.

Pitfalls

• Beam-search sizes are ambiguous. The text says "a depth and width of 2" §5.1. The algorithm uses B, W and U alg. 2. It is not stated which of these is 2, nor the value of W. Deployment gives each actor W/2 proposals app. A, which is one action per actor if W = 2. Treat the sizes as unknown until the authors confirm.
• Dataset size is inconsistent. Section 5.1 says ALFRED has "25k annotations", then says the few examples come from a "21k training set" §5.1.
• The headline percentages do not follow from Table 1. The introduction claims a 60% and 39% improvement over the previous baseline §1. Table 1 gives 25.51 / 15.33 (+66%) and 18.77 / 13.41 (+40%) against LLM-Planner. The source does not say which baseline the stated figures use.
• Relabeling uses ground-truth PDDL arguments, deployment uses predicted ones. During collection, the task is rewritten from the annotated PDDL arguments alg. 3 l.5, l.7. At test time the adapter must predict them, and errors there propagate to the planner. The paper does not measure how much the adapter's errors hurt.
• Hindsight samples depend on ground truth. The reflection and completion are written with the ground-truth rollout visible alg. 3 l.8-10. The relabeled data therefore encode the expert's solution. The paper's argument for this is theoretical §4.3, not empirical.
• The "theoretical superiority" claim is an argument, not a theorem. The contributions list says the method is "theoretically superior" §1. The sources contain no theorem or proof. Equations 4 to 6 state the argument as an inequality, and the text reads it as a sketch §4.3 eq. 4 eq. 5 eq. 6.
• Hidden per-step cost. Each planning call runs the adapter, two actors, a generator and a critic at each beam depth. Several candidates are scored per depth alg. 2. The number of LLM calls per sub-goal, the token cost, and the latency are not reported.
• Replanning rule is partly unstated. "Failed" is used as a sub-goal state alg. 1, but how failure is detected is not defined.
• Output parsing is not specified. Actor outputs must be action strings, and BERT maps them to the legal space §4.1. The mapping rule is not described. The critic must end with "the value is a/b", which the prompt asks for app. B, but how the value is parsed is not stated.
• Temperature for relabeling calls is unknown. The stated temperature 0 applies to the target LLM §5.1. The paper does not say whether relabeling and adapter calls use the same setting.
• Baselines are not matched. Table 1 mixes full-data training, low-instruction variants and few-shot planners. Only Table 2 uses the same low-level controller for all methods.
• Results come from one benchmark and one LLM. Only ALFRED and GPT-4 Turbo are reported §5.1. Generalization is not shown.
• Figures are qualitative. Figure 1 is an illustration and Figure 2 compares the methods on one example fig. 1 fig. 2. Figure 3 walks through one task fig. 3. None gives a number, so all quantitative claims rest on the tables.

Lineage

• Builds on: RAFA, the actor-critic LLM planning framework whose tree search and critic this planner adapts §2 §4.5. LLM-Planner, whose kNN retrieval of in-context examples the planner uses §4.2. HLSM, whose navigation, perception and low-level control modules are reused unchanged §5.1. Hindsight methods such as HER and the information-matching view of Furuta et al. §2 §3.2.
• Compared against: HiTUT, HLSM, FILM, MCR-Agent, LLM-Planner (with and without low-level instructions), and Socratic-Planner tab. 1 tab. 2.
• Superseded by: not known from content already in this wiki.