--- kind: concept slug: task-and-motion-planning title: Task and motion planning summary: >- Checking a language model's robot plan before the robot moves. Text2Motion has an LLM propose skill sequences over learned Pick, Place, Pull and Push skills, then scores each with geometric feasibility planning and out-of-distribution rejection. Shooting, greedy and hybrid search are compared. Author-reported: 82% vs 13% for prior language planners on six simulated tasks. tags: [planning, robot-manipulation, llm-planning, motion-primitives, out-of-distribution-detection] papers: ["2303.12153", "2304.11477"] related: [llm-planning, motion-primitives, out-of-distribution-detection, task-decomposition, neural-symbolic-integration, robot-simulation] status: draft updated_at: 2026-10-31 --- ## Overview A language model can propose a sequence of robot actions, but it cannot tell whether the arm can reach an object or whether an earlier action blocks a later one [arxiv:2303.12153 §2.1]. The shared idea is to keep the model in charge of the sequence and to check each proposal at the level of the low-level parameters before anything moves. Text2Motion does this with a library of learned skills, a geometric feasibility planner and out-of-distribution rejection [arxiv:2303.12153 §3.1] [arxiv:2303.12153 §3.3] [arxiv:2303.12153 §4.5]. Only one paper is compiled for this page so far, so the variants below are planners inside one method, not a comparison across methods. ## Variants ### Shooting: whole plan per query One LLM query returns K complete skill sequences. Each is optimised with STAP, and the best sequence that reaches a predicted goal is kept [arxiv:2303.12153 §4.2] [arxiv:2303.12153 alg. 1]. Card: [Text2Motion](/papers/2303.12153). On the partial-affordance tasks, shooting alone reaches 0% on tasks 5 and 6 [arxiv:2303.12153 §6.2]. ### Greedy search: one skill per step Each step asks the LLM for K next skills. Each candidate is appended, optimised and scored by its usefulness multiplied by its feasibility, and the best one is added to the plan [arxiv:2303.12153 §4.3] [arxiv:2303.12153 alg. 2]. On the partial-affordance tasks 4 to 6 the authors report 40% to 60% success [arxiv:2303.12153 §6.2]. Each step runs STAP once per candidate, which the authors say increases planning time [arxiv:2303.12153 §7]. ### Hybrid: shooting first, one greedy step as fallback At each iteration the planner tries shooting from the latest predicted state. If shooting fails, it takes one greedy step and checks the goal again [arxiv:2303.12153 §4.4] [arxiv:2303.12153 alg. 3]. This is the method's headline planner, reported at 82% success across six tasks [arxiv:2303.12153 §1]. On tasks 5 and 6 every successful plan used both planners [arxiv:2303.12153 tab. 1]. ### Myopic baselines (not checked) SayCan-GS and Inner Monologue choose one skill at a time without checking the sequence [arxiv:2303.12153 §2.1] [arxiv:2303.12153 §5.1]. The authors report at most 20% success on the long-horizon tasks for them [arxiv:2303.12153 §6.1]. ## When to use which The choice depends on constraints. The rows below follow what the card reports. | Situation | Use | Why | |---|---|---| | Short scenes where whole-plan search rarely fails (tasks 1 to 3) | Shooting | Matches the hybrid there, because shooting had no planning failures [arxiv:2303.12153 §6.3] [fig. 6]. One LLM query per attempt [arxiv:2303.12153 §4.2] | | Scenes with a hidden precondition, such as a hook needed first | Hybrid | Shooting alone reaches 0% on tasks 5 and 6 [arxiv:2303.12153 §6.2]. On those tasks all successful plans used both planners [arxiv:2303.12153 tab. 1] | | Tight latency or few LLM calls | Shooting, not greedy | Each LLM query takes 2 to 10 s, and greedy steps add STAP runs per candidate [arxiv:2303.12153 §7] [arxiv:2303.12153 alg. 2] | | No trained skills and no Q-functions | Not this design | Each skill needs 1M training tuples, and four skills took about 12 h on one GPU [arxiv:2303.12153 app. A.1] | | Open-world or high-dimensional observations | Not covered | Predicates and object poses are hand-built, and open-world perception is future work [arxiv:2303.12153 §3] [arxiv:2303.12153 §7] | | Domain PDDL available, classical planner acceptable | See LLM+P | LLM+P writes a PDDL problem and calls a classical planner, with a domain file supplied per domain [arxiv:2304.11477 §3.3]. It does not check learned feasibility | The authors use a closed set of predicates and few-shot prompting only [arxiv:2303.12153 §3] [arxiv:2303.12153 §5.2]. The LLMs are named without snapshot dates [arxiv:2303.12153 §5.2]. If those models are unavailable, a substitute may not reproduce the results [arxiv:2303.12153 §5.2]. ## Implementation sketch The shared core of the planners: propose, optimise, reject, keep the best. The planners differ in how many skills the LLM proposes per query and when the plan is accepted. ```pseudocode # Propose-then-check core shared by the three planners [§4.2] [§4.3] # Input: instruction (str), start_state, goals (set, from the LLM [§4.1]) # Output: list of skills to execute, or None if nothing passes def propose_and_check(instruction, start_state, goals, K): # One LLM query proposes K candidate skill sequences [§4.2] proposals = llm_propose(instruction, start_state, goals, K) best, best_p = None, -1.0 for seq in proposals: # Optimise primitive parameters with STAP [§3.3] [alg. 1] states, params, p = stap(start_state, seq) # p is eq. 5 # Reject the whole plan if any skill is out of distribution [§4.5] if any(is_ood(s, a, k) for s, a, k in zip(states, params, seq)): continue # Keep the most likely plan that reaches a predicted goal [§4.2] if reaches_goal(states, goals) and p > best_p: best, best_p = seq, p return best ``` Notes for implementers, from the paper's own pseudocode: - The hybrid in [alg. 3] returns the shooting result alone. It does not join it to the greedy prefix, and it passes the state trajectory where a start state is expected. Return the prefix plus the shooting result, and pass the latest predicted state [arxiv:2303.12153 alg. 3]. - The shooting planner returns a cut index that is set inside its loop [arxiv:2303.12153 alg. 1]. Store the cut index with each candidate. - The usefulness term has three definitions in the paper. See Contradictions. ## Contradictions - **Task labels.** The Figure 4 caption lists LG and PAP for tasks 4 to 6 [arxiv:2303.12153 fig. 4]. The appendix labels task 3 as LH plus PAP and task 4 as LG plus PAP [arxiv:2303.12153 app. B]. The results text calls task 3 "LH + LG" [arxiv:2303.12153 §6.2] and treats tasks 1 to 3 as non-PAP [arxiv:2303.12153 §6.3]. The property each task has is unresolved. - **Combination share.** The text says over 90% of PAP solutions combine both planners [arxiv:2303.12153 §6.3]. The table gives 86% for task 4 [arxiv:2303.12153 tab. 1]. - **Usefulness score.** The paper gives it as a probability [arxiv:2303.12153 eq. 9] [arxiv:2303.12153 eq. 10], as a sum of token log-probabilities [arxiv:2303.12153 §4.3], and as a softmax log-probability damped by β = 0.3, with no damping formula [arxiv:2303.12153 app. A.3]. - **Greedy success and greedy share.** The text gives greedy success of 40% to 60% on PAP tasks 4 to 6 [arxiv:2303.12153 §6.2]. The table lists 0% for "greedy only" on the same tasks [arxiv:2303.12153 tab. 1]. The table does not define its columns, so the two figures are not directly comparable. ## Papers - [Text2Motion: From Natural Language Instructions to Feasible Plans](/papers/2303.12153): an LLM proposes skill sequences; STAP and Q-ensemble checks gate them; shooting, greedy and hybrid planners. - [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](/papers/2304.11477): an LLM writes PDDL for a classical planner, with no learned feasibility check.