LLM planning

Overview

A language model used as a planner turns a goal and an observation into a sequence of actions. Its plans are often plausible but not always correct, and it states them with the same confidence whether they are right or wrong 2307.01928 §1. Long plans fail in particular: the LLM+P paper shows GPT-4 writing an incorrect plan for a five-block Blocksworld task 2304.11477 §1.

The wiki has ten method papers on this concept, plus a 2024 survey of foundation models in robotics that organises the field. The survey states that long-horizon tasks need several skills in the right order, and that training one policy for a whole task at once is hard 2402.02385 §2 intro. The variants below come from the ten method papers. LLM+P and KnowNo take opposite routes. LLM+P keeps the model out of the search. It turns the task into a formal problem file, and a classical planner that is guaranteed to find optimal plans solves it 2304.11477 §3.3. KnowNo keeps the model as the step chooser and lets it decline to act. Its option likelihoods are calibrated against labelled examples, and the robot executes an action only when the calibrated set contains a single option 2307.01928 §3.2. Text2Motion sits between them. The LLM proposes skill sequences, and learned feasibility checks decide which candidate runs 2303.12153 §4.4. Chameleon takes another route: the planner writes an ordered list of tools from a fixed inventory, and that list runs without revision 2304.09842 §3 2304.09842 app. B. LLM-Planner is a fourth route, for a simulated household. It has no formal model and no calibration step. The model writes the next subgoals from a few retrieved examples, and it plans again when the robot fails 2212.04088 §4.4. Hindsight Planner keeps LLM-Planner's retrieval and adds a closed-loop beam search over sub-goals, with rewritten failed attempts as extra examples 2412.19562 §4.5 2412.19562 §4.3. DCIP lets the model edit a grid map and a goal, and the route search runs on the edited map 2412.02655 §4. DexDiff fine-tunes a vision-language model to output a plan, and a diffusion policy conditioned on that plan produces the motions 2409.05493 §3.1 2409.05493 §3.2. Two Heads Are Better Than One has no formal model either. Agents write robot code, and the question is whether a reviewer agent helps 2411.16723 §3.1. SMART-LLM covers the multi-robot case. One instruction is split into sub-tasks, the model picks a robot or a team for each, and it writes code that assigns them, with independent sub-tasks run in parallel 2309.10062 §4. The variants below follow these routes. The choice depends on whether a formal domain model, labelled calibration data, a tool inventory, a simulator with labelled examples, a grid map, demonstrations or a team of robots with known skills is available.

Forms of planning in the survey

The survey, which covers robotic manipulation, sorts LLM planning by the form of the plan. In order from strict to loose, the forms are PDDL, which an external planner solves; policy code, a program that calls control primitives; and natural language 2402.02385 §2.1. It adds three kinds of assistant that support planning: vision that grounds the plan in the scene, extra knowledge fed back for re-planning or kept in memory, and feedback from the environment or from a human after each step 2402.02385 §2.2 2402.02385 §2.2.3. Its named examples are other systems, so this page does not place the method papers in these forms 2402.02385 §2.1.1 2402.02385 §2.1.2. The survey reports no measured results to compare them. Its taxonomy of low-level control is qualitative, with no numbers 2402.02385 tab. 1. Card: A Survey on Robotics with Foundation Models.

Variants

PDDL problem generation with a classical planner (LLM+P)

The pipeline has three stages. The LLM writes a PDDL problem file from the new task description, conditioned on one example pair of a short description and its problem PDDL 2304.11477 §3.2. A classical planner, given a fixed domain file and the generated problem file, returns a plan. The authors use Fast Downward with the optimal alias seq-opt-fdss-1 and a 200 s search limit 2304.11477 §5.2. The paper also describes a final step in which the LLM rewrites the plan in English 2304.11477 §3.3; the released code omits it (see Contradictions).

The domain file, which holds the action preconditions and effects, is written once by a person per domain and is fixed across tasks 2304.11477 §3.3. The LLM setting is GPT-4 with temperature 0 and top_p 1 2304.11477 §5.2. The paper's main reported failure is a mis-specified problem file, for example a missing initial condition that makes the problem unsolvable 2304.11477 §5.3.

On seven IPC-derived domains with 20 tasks each, the optimal-alias success rates are 90% on Blocksworld, 95% on Grippers and 85% on Storage. Floortile scores 0%, and Barman, Termes and Tyreworld score 10% to 20% 2304.11477 tab. 1. Without the example, the same pipeline scores 0 in every domain 2304.11477 tab. 1. Card: LLM+P.

Skill sequences checked by learned feasibility models (Text2Motion)

The planner works over a fixed library of four learned skills: Pick, Place, Pull and Push. Each skill has a policy, an ensemble of Q-functions and a dynamics model 2303.12153 §3.1 2303.12153 app. A.1. A candidate skill sequence is optimised with geometric feasibility planning (STAP) before it is accepted 2303.12153 §3.3. A skill is treated as out of distribution when the variance of its Q-ensemble at the optimised parameters reaches a per-skill threshold. Plans that contain such a skill are rejected 2303.12153 §4.5 2303.12153 app. A.2.

Two search styles use the LLM in different ways. Shooting asks for K complete skill sequences in one query and keeps those that reach a predicted goal 2303.12153 §4.2 2303.12153 alg. 1. Greedy search adds one skill per step. It scores each of the K candidates by an LLM usefulness term multiplied by a Q-value feasibility term 2303.12153 §4.3 2303.12153 eq. 8. The hybrid tries shooting from the latest predicted state and takes one greedy step when shooting fails 2303.12153 §4.4 2303.12153 alg. 3. The LLM also names the goal propositions, and a predicted goal decides when planning stops, in place of a stop skill 2303.12153 §4.1 2303.12153 §6.4. During execution the remaining skills are re-optimised after each executed skill, and the LLM is not asked to replan the task 2303.12153 §5.5.

The authors report 82% success on six simulated tabletop tasks, against 13% for earlier language-based planners 2303.12153 §1 2303.12153 §5.5. The hybrid matters most on the tasks with partial affordance perception. On task 4, shooting alone gave 14% of the successful plans and the combination gave 86%. On tasks 5 and 6, every successful plan needed both planners 2303.12153 tab. 1. Shooting had planning failures in 80% of the runs on these tasks 2303.12153 §6.2 2303.12153 fig. 6. Card: Text2Motion.

The paper names its LLMs as text-davinci-003 and code-davinci-002 without snapshot dates, at temperature 0 2303.12153 §5.2. It reports 2 to 10 seconds per LLM query 2303.12153 §7. The maximum search depth and the SAC entropy coefficient are not stated 2303.12153 alg. 2 2303.12153 app. A.1. The OOD thresholds are given only for this skill library 2303.12153 app. A.2.

Multiple-choice options with conformal calibration (KnowNo)

The planner is prompted with a few-shot example set and asked to produce up to four distinct candidate next steps, labelled A to D. A fifth option, E ("an option not listed here"), is always appended, so the correct action can fall outside the list 2307.01928 §2. Options are read as single-token choices, so one forward pass yields a likelihood for each label 2307.01928 §2 2307.01928 fig. 2.

The nonconformity score is one minus the normalised confidence of the correct option. A quantile of these scores on the calibration set gives a threshold for a user-set error level ε 2307.01928 §3.1 2307.01928 eq. 1 2307.01928 eq. 2. The paper's dataset-conditional form 2307.01928 eq. 2 picks an adjusted level so that the bound holds with probability 0.99 over the calibration sample (δ = 0.01) 2307.01928 §3.1. The card for the paper is KnowNo.

At test time, the prediction set keeps every option whose confidence is at least 1 minus the threshold. A singleton set means the robot acts; a larger set means it asks a person to pick 2307.01928 §3.2. The paper states the guarantee only under i.i.d. calibration and test scenarios, a correct human answer and correct low-level execution 2307.01928 §3.2 2307.01928 app. A.8.

Multi-step episodes

For plans with several steps, each calibration episode gets one sequence score: the minimum, over its steps, of the confidence given to the correct option 2307.01928 eq. 4. A single threshold is fitted on these scores. At test time the set is rebuilt one step at a time from the context reached so far, and the robot acts or asks at each step 2307.01928 §3.3 2307.01928 alg. 1. The paper's appendix argues that the product of per-step sets keeps the sequence guarantee 2307.01928 app. A.2.

Several acceptable options

When more than one next step is correct, the calibration label is the correct option that the planner ranks highest. The set must then contain that option, and a human is asked to choose the highest-ranked correct option 2307.01928 app. A.3 2307.01928 app. A.4.

Module-sequence planning from a tool inventory (Chameleon)

The planner receives the task instruction, a description of each module in an inventory with its constraints, and a few demonstrations. It returns a list of module names for each query 2304.09842 §3 2304.09842 eq. 1. Nothing is trained; the planner and the LLM-based modules run on few-shot prompts 2304.09842 §3. The inventory has thirteen module types, from LLM prompts to an image captioner, OCR, web search and Python execution. Each task exposes only a subset of them to the planner 2304.09842 §4.1 2304.09842 tab. 8 2304.09842 tab. 9.

Modules run in plan order. Each reads the current input and a shared cache and writes an output. Two task-specific update functions then change the input and the cache for the next module 2304.09842 eq. 2 2304.09842 eq. 3 2304.09842 eq. 4. The last module's output is the answer 2304.09842 eq. 5. The plan is written before any module runs and is not revised during the run 2304.09842 app. B.

If a plan fails the task's validity check, the run falls back to a fixed default. For ScienceQA that default is the chain-of-thought sequence; for TabMWP it is the program-of-thought sequence with verification 2304.09842 §4.2 2304.09842 §4.3.

With GPT-4 the paper reports 86.54% on ScienceQA and 98.78% on TabMWP. These are author-reported task accuracies, not plan success 2304.09842 tab. 3 2304.09842 tab. 4. In an ablation with ChatGPT on 500 ScienceQA examples, removing Knowledge Retrieval costs 7.8 points 2304.09842 tab. 5. Card: Chameleon.

Grounded subgoal planning with retrieved examples (LLM-Planner)

This route suits a simulated household agent with a fixed set of high-level actions and a low-level controller. The planner is GPT-3 (text-davinci-003) at temperature 0 2212.04088 §5.3. For each instruction it retrieves the nine nearest examples from a pool of 100 training pairs, using Euclidean distance on frozen BERT-base instruction embeddings 2212.04088 §4.3 2212.04088 §5.3. The prompt lists the allowed actions, the examples, the task, the visible objects and the subgoals already completed. It ends with a marker where the model writes the next plan 2212.04088 §4.2. The output is a list of (action, object) subgoals, drawn from one Navigation action and seven ALFRED interaction actions 2212.04088 §3. A low-level controller turns each subgoal into primitive actions.

When a subgoal fails, the planner is called again with the objects seen so far and the completed subgoals 2212.04088 §4.4 2212.04088 alg. 1. The paper also allows re-planning after n time steps, but gives no value for n 2212.04088 §4.4. Logit biases of +0.1 are added to the allowed action and object tokens and to the observed objects 2212.04088 §4.1 2212.04088 §5.3.

With 100 pairs and HLSM as the controller, the paper reports unseen-test success of 11.58% for static plans and 13.41% with re-planning, using goal instructions. With step-by-step instructions the figures are 15.83% and 16.42% 2212.04088 tab. 1. HLSM trained on all 21,023 pairs reaches 20.27% on the same split with goal instructions 2212.04088 tab. 1. Leave-one-out cross-validation on the 100 pairs gives 40.59% high-level plan accuracy for the full method, 17.48% without the retriever and 38.10% without the logit bias 2212.04088 tab. 2. Card: LLM-Planner.

Closed-loop sub-goal search with hindsight relabelling (Hindsight Planner)

The planner is GPT-4 Turbo at temperature 0. It uses few-shot examples from a pool of 140 solved ALFRED trajectories, 20 for each of seven task types, and it retrieves the nearest examples by Euclidean distance as LLM-Planner does 2412.19562 §5.1 2412.19562 §4.2. At every step an adapter predicts seven hidden PDDL-style arguments from the object list, such as the target object, the receptacle and the object state. The paper argues for predicting them at each step rather than once per task 2412.19562 §4.4 2412.19562 §2. Two actors and a critic then run a beam search over candidate sub-goals, and the first action of the best rollout is executed 2412.19562 §4.5 2412.19562 alg. 2. The low-level controller is the pretrained HLSM stack, reused without retraining 2412.19562 §5.1.

The hindsight step turns the agent's failed rollouts into training examples. An LLM writes a reflection on each rollout, using the ground-truth rollout as reference, then completes the rollout, guided by that reflection. The rewritten attempts go into the prompt of the second actor and of the critic 2412.19562 §4.3 2412.19562 alg. 3. The paper's argument that this keeps the information statistic unchanged is an inequality sketch, not a theorem 2412.19562 §4.3 2412.19562 eq. 4 2412.19562 eq. 5 2412.19562 eq. 6.

The text gives the beam depth and width as 2, but the algorithm uses B, W and U, and the paper does not map the 2 onto them 2412.19562 §5.1 2412.19562 alg. 2. Each depth calls the adapter, both actors, the generator and the critic. The paper does not report the number of model calls per sub-goal, the token cost or the latency 2412.19562 alg. 2.

On ALFRED the paper reports test-seen SR 25.51 and test-unseen SR 18.77, against 15.33 and 13.41 for LLM-Planner in the same few-shot setting 2412.19562 tab. 1. On the validation-seen split, removing the hindsight step lowers SR from 25.61 to 23.53, and removing the adapter lowers it to 23.17 2412.19562 tab. 4. The results come from one benchmark and one LLM 2412.19562 §5.1. Card: Hindsight Planner.

Grid-map edits with a classical route search (DCIP)

DCIP runs a loop of observe, update goals and constraints, generate actions, and execute them while updating the grid 2412.02655 alg. 1 2412.02655 §4. The LLM turns the instruction into a task definition, an action format and semantic constraints 2412.02655 §4.1. Three action types can then change the map or the goal: reset the map to its initial state, set a cost on a region, or set a goal 2412.02655 §4.2. The paper gives no completion test for the loop 2412.02655 alg. 1.

The grid is binary in the problem statement, with 0 for free and 1 for occupied cells 2412.02655 §3. The method writes cost values into it, so the grid's data type is not clear from the text 2412.02655 §4.2 2412.02655 §4.3. The route-search algorithm is never named. A* appears only as the baseline 2412.02655 §5.1.1. The prompts, output format and checkpoints of the LLMs are not given, so the instruction-to-action mapping cannot be reproduced 2412.02655 §5.1.

In simulation on an industrial workspace, the Llama3 row with the balance strategy expanded 370 nodes against 2,254 for A*, and both paths had length 177 2412.02655 tab. 2. The DCIP path cost is 102.0 against 177.0 for A*, but DCIP costs include zone weights that the paper does not define, so the two costs are not comparable 2412.02655 §5.1.1 2412.02655 tab. 2. Instruction adherence across the three LLMs and three strategies ranges from 28.57% to 92.85% 2412.02655 tab. 1. The paper gives 90% as the average instruction-to-task accuracy 2412.02655 §5.5. Card: LLM-Enhanced Path Planning.

Fine-tuned VLM plans with a diffusion controller (DexDiff)

The route targets large flat objects that cannot be grasped from above. The robot first pushes such an object against a wall or table edge, so that one side hangs over and can be grasped 2409.05493 §1. A VisualGLM-6B model is fine-tuned with LoRA to read a scene image and a prompt, and to output a plan that names a skill, a structure, a direction and a target object 2409.05493 §3.1. The plan is turned into return-to-go values. The paper says these come from goal-conditioned grasp poses, but gives no formula for them 2409.05493 §3.1 2409.05493 §3.2.

A goal-conditioned diffusion policy (GCAD) conditioned on those values predicts a chunk of actions, and the controller executes it without replanning 2409.05493 §3.2 2409.05493 app. A.5. The training data are large: about 1.6k simulated and 1.4k real labelled images for the VLM, and 400 simulated and 200 real trajectories per scenario for the policy 2409.05493 §3.1 2409.05493 §4.1. The real demonstrations come from human teaching 2409.05493 app. A.3.

Across four simulated layouts the average success rate is 70% for the full method, against 23% for a transformer diffusion policy with the same VLM planner 2409.05493 tab. 1. On a UR5e in four daily-life setups, with ten trials each, the paper reports 65% over 40 trials 2409.05493 tab. 2. The real-robot setup relies on human knowledge of which surfaces act as walls or table edges 2409.05493 §4.5. The paper calls the transformer model demanding in demonstrations and computation 2409.05493 §5. Card: DexDiff.

Robot code from a team of agents with a reviewer (Two Heads)

The planner here writes program code for a legged robot, not a plan in a formal language 2411.16723 §3.1. Three setups are compared: one coder (A); a coder with a reviewer that checks the code and sends it back with feedback (B); and a planner, a coder and a reviewer (C) 2411.16723 §3.1 2411.16723 fig. 5. The chat ends when the reviewer's message contains "terminate", or after 12 rounds in configurations B and C 2411.16723 §3.1 coder_reviewer.py coder_reviewer.py.

The error rate counts first attempts only. Runs that error are regenerated, and the regenerated run is not counted 2411.16723 §3.3 2411.16723 fig. 6. On seven prompts with three runs each, configuration B had 9 points fewer first-attempt errors than A and 14 fewer than C, and C had about 5 points more than A 2411.16723 §4 2411.16723 fig. 6. Observer scores for success, safety, sociability and the expectation gap showed no significant difference across configurations in the text 2411.16723 §4 2411.16723 fig. 8. The sample is small: 63 scored runs, and no statistical test is reported 2411.16723 §4. Card: Two Heads Are Better Than One.

Team allocation and assignment code for several robots (SMART-LLM)

The planner is prompted in four stages, and each stage uses few-shot examples written as Python code. Robot skills and object properties are given as Python dictionaries, and each example is a commented code block 2309.10062 §4 2309.10062 fig. 2. The first stage splits the instruction into ordered sub-tasks, each with the skills it needs 2309.10062 §4.1. The second stage decides, for each sub-task, whether one robot has every required skill or a team is needed, and which robots form it 2309.10062 §4.2. The third stage writes code that assigns robots to sub-tasks and runs independent sub-tasks in parallel 2309.10062 §4.3. The fourth stage runs that code, with each call mapped to a low-level skill of a robot 2309.10062 §4.4.

The feasibility test in the formal setting is a set condition. A single robot takes a sub-task if its skills include every required skill. A team works if the union of its members' skills covers the sub-task 2309.10062 §3. The paper gives no algorithm for choosing teams. The model makes that choice from the examples 2309.10062 §3 2309.10062 §4.2.

On a 36-task AI2-THOR benchmark in four categories, GPT-4 reaches a success rate of 1.00 on elemental tasks and 0.62, 0.69 and 0.71 on the simple, compound and complex categories 2309.10062 tab. 1. Success is stricter than task completion. On simple tasks GPT-4's completion rate is 1.00 but its success rate is 0.62. The paper attributes the gap to sequential execution where parallel execution was possible, which lowers robot utilisation 2309.10062 §6.1 2309.10062 tab. 1. GPT-3.5 scores 0.42 and 0.14 on the compound and complex categories 2309.10062 tab. 1. The paper's ablation without the coalition stage scores 0.60 against 0.75 for the full method, but the task scope of that ablation is not stated 2309.10062 tab. 3. The real-robot runs are described qualitatively, with no metrics. They use skills that the paper assumes exist 2309.10062 §5.3 2309.10062 §6.2. Card: SMART-LLM.

Baselines reported by the papers

These are alternatives the papers measured against their own methods, not separate method cards.

• SayCan. It ranks candidate skills with the LLM, so it calls the model once per candidate. The paper reports 22 GPT-3 calls per task, against 7 for LLM-Planner, even with SayCan given oracle object knowledge 2212.04088 §5.4. Its valid-unseen success with step-by-step instructions is 9.88% 2212.04088 tab. 1.
• Direct LLM plans. The LLM writes the whole plan itself, with or without the in-context example. In LLM+P's table this reaches at most 35% in any domain 2304.11477 tab. 1.
• LLM tree search. A tree-of-thoughts style search over LLM-proposed steps, adapted to breadth-first search. The paper reports it mostly times out because of the number of LLM calls 2304.11477 §5.2 2304.11477 §5.3.
• Simple Set. Rank options by LLM confidence and keep adding them until their cumulative confidence reaches 1 − ε. Its ε is tuned to match plan success, not the target 2307.01928 §4.1 2307.01928 §4.2.
• Ensemble Set. Sample the planner with 20 prompts built from random few-shot subsets and pool the answers. The paper reports it needs about 20 times the inference cost and can still give the correct option zero probability 2307.01928 §4.1.
• Prompt Set and Binary. Ask the model to report its own uncertainty, either as a predicted set or as a certain/uncertain label. The paper notes that neither exposes a success-rate control 2307.01928 §4.1.
• Raw probabilities. Use the sequence probability or perplexity of a free-form sentence plan as the score. The paper reports larger sets and more help than the multiple-choice score, because the score depends on answer length 2307.01928 app. A.9.
• Random and rule-based allocation (SMART-LLM). Both keep GPT-4 for decomposition and replace the model's team and allocation step with a random or a rule-based allocator. On compound and complex tasks, random allocation scores 0.08 and 0.00, and rule-based allocation scores 0.57 and 0.14 2309.10062 §5.2 2309.10062 tab. 1.
• Behaviour cloning and Decision Transformer (DexDiff). The low-level baselines use the same VLM planner. The paper reports no successful trajectories for either 2409.05493 §4.2 2409.05493 tab. 1.
• Heuristic high-level planner (DexDiff). A YOLOX detector with a threshold rule for the table edge. Paired with GCAD, it reaches 35% on the real robot, against 65% for the VLM planner 2409.05493 §4.1 2409.05493 tab. 2.

When to use which

Note that the papers' own results show a gap to the target. On the mobile manipulator the KnowNo target was 0.85 and task success was 0.76 2307.01928 tab. 2. The authors attribute the gap to about 86% low-level success on the base 2307.01928 app. A.8. Budget for the controller, not only for the planner. The method papers' numbers measure different things, and they cannot be compared directly. LLM+P reports optimal-plan success on benchmark problem files. KnowNo reports robot task success and help. Chameleon reports answer accuracy on QA benchmarks. LLM-Planner and Hindsight Planner report unseen-test success on the ALFRED simulator. DCIP reports instruction adherence and search cost on one grid. DexDiff reports simulated and real grasp success. Two Heads reports first-attempt code errors and observer scores on a Spot robot. SMART-LLM reports task success on AI2-THOR tasks, with its real-robot runs unmeasured 2309.10062 §6.2. Text2Motion reports task success on six simulated tabletop tasks, and its myopic baselines received task-level retries that it did not 2303.12153 §5.5.

Implementation sketch

The routes share a common shape: the LLM produces something checkable, and a second component decides whether to trust it. The LLM+P core follows 2304.11477 §3.3:

# LLM+P: generate the problem, then plan  [arxiv:2304.11477 §3.3]
# domain_pddl is fixed per domain; example_* is one (description, PDDL) pair
def llm_plus_p(task_nl, example_nl, example_pddl, domain_pddl):
    # Ask the LLM for the problem file, conditioned on the example  [§3.2]
    prompt = build_pddl_prompt(example_nl, example_pddl, task_nl)
    # Temperature 0 and top_p 1 were used in the paper  [§5.2]
    problem_pddl = query_llm(prompt)
    # Optimal planner, 200 s search limit  [§5.2]
    plans = run_planner(domain_pddl, problem_pddl, time_limit=200)
    # No plan means no answer; a bad problem file looks the same  [§5.3]
    return min(plans, key=plan_cost) if plans else None

The KnowNo core follows 2307.01928 §3.1 and 2307.01928 §3.2:

# Calibrate once on labelled scenarios  [arxiv:2307.01928 §3.1]
def calibrate(calib_probs, calib_labels, epsilon):
    # One score per scenario: 1 minus confidence of the correct option
    scores = 1 - calib_probs[range(N), calib_labels]    # (N,)
    # Quantile level ceil((N + 1)(1-eps))/N, capped at 1  [arxiv:2307.01928 eq. 2]
    q_level = min(1.0, ceil((N + 1) * (1 - epsilon)) / N)
    return quantile(scores, q_level, method="higher")

# Decide per context  [arxiv:2307.01928 §3.2]
def decide(option_probs, qhat):
    # Keep every option at or above 1 - qhat  [arxiv:2307.01928 §3.1]
    kept = [y for y in options if option_probs[y] >= 1 - qhat]
    if len(kept) == 1:
        return execute(kept[0])
    return ask_human(kept)    # human picks; E means a new action is given

For multi-step plans, replace the per-scenario score with the minimum over the episode's steps 2307.01928 eq. 4, fit one threshold on episodes, and call decide at each step on the context reached so far 2307.01928 alg. 1.

The Chameleon loop, with the planner's output fixed up front 2304.09842 §3:

# Chameleon: plan once, then run the modules in order  [arxiv:2304.09842 §3]
def chameleon_answer(query, inventory, planner_prompt, default_plan):
    # The planner returns module names; the plan is fixed here  [eq. 1]
    plan = parse_plan(query_llm(planner_prompt, query, inventory))
    # Use the task default when the plan fails its validity check  [§4.2]
    if not is_valid_plan(plan):
        plan = default_plan
    cache, module_input = {}, query
    for name in plan:
        # Module reads the input and cache, and writes an output  [eq. 2]
        output = modules[name](module_input, cache)
        # Task-specific updates for the next module  [eq. 3] [eq. 4]
        module_input = update_input(name, module_input, output)
        cache = update_cache(name, cache, output)
    # The last module's output is the answer  [eq. 5]
    return output

The LLM-Planner loop, with the replanning rule from the text 2212.04088 §4.4. Algorithm 1 differs on what happens to the index after a replan (see Contradictions):

# LLM-Planner: retrieve, plan, and replan on failure  [§4.2] [§4.3] [§4.4]
def llm_planner_episode(instruction, pool, env, k=9):
    # Retrieve the k nearest pool pairs by instruction embedding  [§4.3]
    examples = nearest_k(embed(instruction), pool, k)
    observed, done = set(), []
    plan = plan_subgoals(instruction, examples, observed, done)
    index = 0
    while index < len(plan):
        # Run the controller on the current subgoal  [alg. 1]
        ok = env.run_subgoal(plan[index])
        # Add the objects the perception model reports  [alg. 1]
        observed |= env.visible_objects()
        if not ok:
            # Replan from the new object list and the completed subgoals  [§4.4]
            plan = plan_subgoals(instruction, examples, observed, done)
            index = 0
        else:
            done.append(plan[index])
            index += 1
    return done

The Hindsight Planner beam search, with the critic scoring each extended rollout 2412.19562 alg. 2. The paper's B, W and U map onto keep, width and depth, and the text's "2" is not mapped (see the variant above):

# Beam search over rollouts, scored by a critic  [arxiv:2412.19562 alg. 2]
def beam_search(state, actors, critic, generator, depth, width, keep):
    # Start from the current state only  [alg. 2 l.12]
    frontier = [[state]]
    # Expand each rollout by one action per depth  [alg. 2 l.15]
    for _ in range(depth):
        candidates = []
        for rollout in frontier:
            # Actors split the proposals between them  [alg. 2 l.16-18]
            proposals = [a.propose(rollout, width / len(actors))
                         for a in actors]
            for action in flatten(proposals):
                # The generator gives the next state  [alg. 2 l.19]
                extended = rollout + [action, generator(rollout[-1], action)]
                # The critic scores the extended rollout  [alg. 2 l.20]
                candidates.append((critic(extended), extended))
        # Keep the best rollouts for the next depth  [alg. 2 l.21]
        frontier = [r for _, r in top_k(candidates, keep)]
    # The first action of the best final rollout is the sub-goal  [alg. 2]
    best = max(frontier, key=critic)
    return best[1]

Contradictions

Between the method papers there is no shared benchmark, so no numbers conflict across papers. The survey reports no experiments of its own 2402.02385 §1. LLM+P reports optimal-plan success on IPC-derived PDDL problems 2304.11477 §5.1, KnowNo reports help rates and task success on robots 2307.01928 §4, Chameleon reports accuracy on ScienceQA and TabMWP 2304.09842 tab. 3, and LLM-Planner reports success on the ALFRED unseen and seen splits 2212.04088 tab. 1. Hindsight Planner reports ALFRED success on the same splits 2412.19562 tab. 1, DCIP reports adherence and search cost on one simulated grid 2412.02655 tab. 1 2412.02655 tab. 2, DexDiff reports simulated and real grasp success 2409.05493 tab. 1 2409.05493 tab. 2, Two Heads reports first-attempt code errors and observer scores 2411.16723 §4, and SMART-LLM reports success on a 36-task AI2-THOR benchmark 2309.10062 tab. 1. Within each paper, the released code differs from the text in several places, which matters for an implementer.

LLM+P:

• Back-translation. The text has the LLM rewrite the plan in English 2304.11477 §3.3. The released planner writes only the raw planner output to a file, and the translation call is commented out with the note "exceeding token limit of gpt-4" 2304.11477 code: main.py#L481-L488.
• Planner alias. The text reports the optimal alias, with lama results in parentheses 2304.11477 §5.2. The released planner calls use only the optimal alias; the lama line is commented out 2304.11477 code: main.py#L12-L13 2304.11477 code: main.py#L464-L467.
• Plan selection. The live text gives no selection rule. The code picks the lowest reported cost across all plan files 2304.11477 code: main.py#L469-L478.
• Context. The text's context is the example description plus its problem PDDL 2304.11477 §3.2. The code loads an example solution as well, but the PDDL prompt does not use it 2304.11477 code: main.py#L324-L333.

KnowNo:

• LLM. The text uses PaLM-2L in all examples. The released notebooks call text-davinci-003 (GPT-3.5), as the README says 2307.01928 §4 2307.01928 code: language_model_uncertainty/README.md.
• Calibration size. The text uses N = 400. The tabletop notebook uses 100 and the mobile notebook uses 200 2307.01928 app. A.7 2307.01928 code: language_model_uncertainty/KnowNo_TabletopSim.ipynb#L1625 2307.01928 code: language_model_uncertainty/KnowNo_MobileManipulation.ipynb#L1735.
• Temperature. The text does not mention temperature scaling. The notebooks divide the five option log-probabilities by 5 before the softmax, which changes the scores and the threshold 2307.01928 code: language_model_uncertainty/KnowNo_TabletopSim.ipynb#L2193-L2210.
• Dataset-conditional bound. The text uses eq. 2 with δ = 0.01. The notebooks use the requested ε directly, so they give only the marginal guarantee 2307.01928 §3.1 2307.01928 code: language_model_uncertainty/KnowNo_TabletopSim.ipynb#L2232-L2233.
• Multi-step and bimanual. The released code covers single-step tabletop and mobile manipulation only. The sequence procedure 2307.01928 alg. 1 cannot be checked against code 2307.01928 §3.3.

Chameleon:

• Bing results. The text says the search module returns the top three responses 2304.09842 app. A.2. The code default is one result run.py.
• Plan constraints on TabMWP. The text says the ordering constraints force the default program 2304.09842 §4.3. The code's ordering checks compare module names with a nested list, so they never fire model.py.
• Row and column thresholds. The appendix uses 18 cells 2304.09842 app. A.2. The code defaults to 12 cells run.py.
• Captioner. The text describes a live captioner 2304.09842 app. A.2. The ScienceQA code loads precomputed captions, and its live captioner is a TODO that returns an empty string model.py.

LLM-Planner:

• Object list. The paper takes objects from HLSM's perception model with confidence above 80% 2212.04088 §5.3. The released ALFRED evaluator reads the simulator's visibility flag, so its object list is ground truth thor_connector.py run_eval.py.
• Replanning. The text says the new plan continues the completed part of the plan, but Algorithm 1 restarts at the first subgoal 2212.04088 §4.4 2212.04088 alg. 1. The released evaluator replans only when an action raises an error, with a retry cap, and its config ships with dynamic re-planning off run_eval.py config_alfred.yaml.
• Retriever. The paper uses frozen BERT-base with Euclidean distance 2212.04088 §4.3. The hlp/ code uses MiniLM sentence embeddings with cosine similarity hlp_planner.py hlp_planner.py.
• Logit bias. The paper biases allowed action and object tokens and observed objects 2212.04088 §4.1 2212.04088 §5.3. The hlp/ generator biases only the object string hlp_planner.py, and the e2e/ call has no logit-bias argument run_eval.py.
• Model. The paper uses text-davinci-003 2212.04088 §5.3. The hlp/ default is text-davinci-002 hlp_planner.py, and the e2e/ config uses gpt-4o-mini config_alfred.yaml.

Hindsight Planner:

• Headline percentages. The introduction gives 60% and 39% improvements over the previous baseline 2412.19562 §1. Against LLM-Planner, Table 1 gives about 66% and 40% for the two test splits 2412.19562 tab. 1. The paper does not say which baseline the introduction uses.
• Dataset size. The text gives ALFRED as 25k annotations, then says the few-shot examples come from a 21k training set 2412.19562 §5.1.

DCIP:

• Text against Table 1. The text says Mistral is strong on Navigate Quickly, that Llama3 is best in every strategy, and that Llama3.1 lagged behind 2412.02655 §5.1. In Table 1, Mistral's Navigate Quickly row is its lowest on all three metrics, Llama3.1 has higher completeness than Llama3 on that strategy, and Llama3.1's balance row is above Mistral's on all three metrics 2412.02655 tab. 1.
• Headline accuracy. Section 5.5 gives an average instruction-to-task accuracy of 90%, and the conclusion says "over 90%" 2412.02655 §5.5 2412.02655 §6.

Two Heads:

• Significance. The Results say the configurations do not differ significantly on observer scores 2411.16723 §4. The Discussion claims a significant impact and concludes that a two-agent setup beats a three-agent one 2411.16723 §5. No statistical test is reported 2411.16723 §4.
• Model. The text says all agents are GPT-4 2411.16723 §3.1. The released code calls gpt-4 for configurations A and C, and gpt-4o for B 2411.16723 code: src/coder.py#L18 2411.16723 code: src/coder_reviewer.py#L25 2411.16723 code: src/planner_coder_reviewer.py#L31. The B results therefore may mix model and architecture.

DexDiff:

• Simulated demonstrations. Section 4.1 says the simulated trajectories combine reinforcement learning with human demonstration, while the appendix says they come only from SAC policies 2409.05493 §4.1 2409.05493 app. A.1.

SMART-LLM:

• Stage count. The text describes four stages, with execution as the fourth 2309.10062 §4. The released code makes three model calls and runs execution as a separate script 2309.10062 code: scripts/run_llm.py#L134-L221 2309.10062 code: scripts/execute_plan.py#L17-L67. The match between the middle call and the coalition stage is inferred from the example contents, not stated.
• Mass constraint. The formal problem uses skills only 2309.10062 §3. The code's allocation rules also require the team's combined mass capacity to meet each object's mass 2309.10062 code: data/pythonic_plans/train_task_allocation_solution.py#L129-L131.
• Backbones. The text reports GPT-4, GPT-3.5, Llama-2-70B and Claude-3-Opus 2309.10062 §5.2 2309.10062 tab. 1. The released script accepts only three GPT model names 2309.10062 code: scripts/run_llm.py#L68-L69.
• Metric definitions. The text defines goal condition recall as a set difference over the goals, and executability over all actions 2309.10062 §5.4. The code counts a goal once for each scene object whose name matches it, and counts only some action calls 2309.10062 code: data/aithor_connect/end_thread.py#L29-L77 2309.10062 code: data/aithor_connect/aithor_connect.py#L88-L159.
• Variability. Table 2 gives a complex-category success rate of 0.48 ± 0.40 over five GPT-4 runs. Table 1 gives 0.71 for the same backbone and category. The paper does not explain the gap 2309.10062 tab. 1 2309.10062 tab. 2 2309.10062 §6.1.

Text2Motion:

• Task labels. The Figure 4 caption, the appendix table and the results text assign the LH, LG and PAP properties to tasks differently 2303.12153 fig. 4 2303.12153 app. B 2303.12153 §6.2. Treat any per-task comparison as unresolved.

Papers

• Text2Motion (arXiv:2303.12153): an LLM proposes skill sequences over learned Pick, Place, Pull and Push skills, and feasibility planning and Q-ensemble checks decide which plan runs. Lin et al.
• LLM+P (arXiv:2304.11477): has GPT-4 write a PDDL problem with one in-context example and solves it with a classical optimal planner. Liu et al.
• KnowNo (arXiv:2307.01928): multiple-choice LLM planning calibrated with split conformal prediction, with the robot asking for help on uncertain steps. Ren et al.
• Chameleon (arXiv:2304.09842): an LLM planner writes an ordered list of tools from a fixed inventory for each query, with no training. Lu et al.
• LLM-Planner (arXiv:2212.04088): GPT-3 writes ALFRED household subgoals from nine retrieved examples and replans when a subgoal fails. Song et al.
• Hindsight Planner (arXiv:2412.19562): a few-shot ALFRED planner with actor-critic beam search over sub-goals, and hindsight-rewritten failed rollouts as extra examples. Yang et al.
• LLM-Enhanced Path Planning (arXiv:2412.02655): an LLM edits a grid map and goal, and the route search runs on the edited map, with replanning. Only simulated. Doma et al.
• DexDiff (arXiv:2409.05493): a fine-tuned VLM writes a plan for pushing large flat objects to a wall or table edge, and a diffusion policy executes it. Ma et al.
• Two Heads Are Better Than One (arXiv:2411.16723): one, two or three GPT-4 agents write robot code for a Spot robot; a reviewer agent lowers first-attempt errors. Rosser and Carmichael.
• SMART-LLM (arXiv:2309.10062): four prompted stages (decompose, form teams, allocate, write code) turn one instruction into code for a multi-robot team in AI2-THOR. Author-reported GPT-4 success is 1.00 on elemental tasks and 0.62 to 0.71 on the others. Kannan et al.
• A Survey on Robotics with Foundation Models (arXiv:2402.02385): a taxonomy of foundation models for robotic manipulation, with planning forms, assistants and low-level control. No experiments. Xu et al.