Task and motion planning

Overview

A language model can propose a sequence of robot actions, but it cannot tell whether the arm can reach an object or whether an earlier action blocks a later one 2303.12153 §2.1. The shared idea is to keep the model in charge of the sequence and to check each proposal at the level of the low-level parameters before anything moves. Text2Motion does this with a library of learned skills, a geometric feasibility planner and out-of-distribution rejection 2303.12153 §3.1 2303.12153 §3.3 2303.12153 §4.5. Only one paper is compiled for this page so far, so the variants below are planners inside one method, not a comparison across methods.

Variants

Shooting: whole plan per query

One LLM query returns K complete skill sequences. Each is optimised with STAP, and the best sequence that reaches a predicted goal is kept 2303.12153 §4.2 2303.12153 alg. 1. Card: Text2Motion. On the partial-affordance tasks, shooting alone reaches 0% on tasks 5 and 6 2303.12153 §6.2.

Greedy search: one skill per step

Each step asks the LLM for K next skills. Each candidate is appended, optimised and scored by its usefulness multiplied by its feasibility, and the best one is added to the plan 2303.12153 §4.3 2303.12153 alg. 2. On the partial-affordance tasks 4 to 6 the authors report 40% to 60% success 2303.12153 §6.2. Each step runs STAP once per candidate, which the authors say increases planning time 2303.12153 §7.

Hybrid: shooting first, one greedy step as fallback

At each iteration the planner tries shooting from the latest predicted state. If shooting fails, it takes one greedy step and checks the goal again 2303.12153 §4.4 2303.12153 alg. 3. This is the method's headline planner, reported at 82% success across six tasks [arxiv:2303.12153 §1]. On tasks 5 and 6 every successful plan used both planners 2303.12153 tab. 1.

Myopic baselines (not checked)

SayCan-GS and Inner Monologue choose one skill at a time without checking the sequence 2303.12153 §2.1 2303.12153 §5.1. The authors report at most 20% success on the long-horizon tasks for them 2303.12153 §6.1.

When to use which

The choice depends on constraints. The rows below follow what the card reports.

The authors use a closed set of predicates and few-shot prompting only 2303.12153 §3 2303.12153 §5.2. The LLMs are named without snapshot dates 2303.12153 §5.2. If those models are unavailable, a substitute may not reproduce the results 2303.12153 §5.2.

Implementation sketch

The shared core of the planners: propose, optimise, reject, keep the best. The planners differ in how many skills the LLM proposes per query and when the plan is accepted.

# Propose-then-check core shared by the three planners  [§4.2] [§4.3]
# Input:  instruction (str), start_state, goals (set, from the LLM [§4.1])
# Output: list of skills to execute, or None if nothing passes

def propose_and_check(instruction, start_state, goals, K):
    # One LLM query proposes K candidate skill sequences [§4.2]
    proposals = llm_propose(instruction, start_state, goals, K)
    best, best_p = None, -1.0
    for seq in proposals:
        # Optimise primitive parameters with STAP [§3.3] [alg. 1]
        states, params, p = stap(start_state, seq)  # p is eq. 5
        # Reject the whole plan if any skill is out of distribution [§4.5]
        if any(is_ood(s, a, k) for s, a, k in zip(states, params, seq)):
            continue
        # Keep the most likely plan that reaches a predicted goal [§4.2]
        if reaches_goal(states, goals) and p > best_p:
            best, best_p = seq, p
    return best

Notes for implementers, from the paper's own pseudocode:

• The hybrid in alg. 3 returns the shooting result alone. It does not join it to the greedy prefix, and it passes the state trajectory where a start state is expected. Return the prefix plus the shooting result, and pass the latest predicted state 2303.12153 alg. 3.
• The shooting planner returns a cut index that is set inside its loop 2303.12153 alg. 1. Store the cut index with each candidate.
• The usefulness term has three definitions in the paper. See Contradictions.

Contradictions

• Task labels. The Figure 4 caption lists LG and PAP for tasks 4 to 6 2303.12153 fig. 4. The appendix labels task 3 as LH plus PAP and task 4 as LG plus PAP 2303.12153 app. B. The results text calls task 3 "LH + LG" 2303.12153 §6.2 and treats tasks 1 to 3 as non-PAP 2303.12153 §6.3. The property each task has is unresolved.
• Combination share. The text says over 90% of PAP solutions combine both planners 2303.12153 §6.3. The table gives 86% for task 4 2303.12153 tab. 1.
• Usefulness score. The paper gives it as a probability [arxiv:2303.12153 eq. 9] 2303.12153 eq. 10, as a sum of token log-probabilities 2303.12153 §4.3, and as a softmax log-probability damped by β = 0.3, with no damping formula 2303.12153 app. A.3.
• Greedy success and greedy share. The text gives greedy success of 40% to 60% on PAP tasks 4 to 6 2303.12153 §6.2. The table lists 0% for "greedy only" on the same tasks 2303.12153 tab. 1. The table does not define its columns, so the two figures are not directly comparable.

Papers

• Text2Motion: From Natural Language Instructions to Feasible Plans: an LLM proposes skill sequences; STAP and Q-ensemble checks gate them; shooting, greedy and hybrid planners.
• LLM+P: Empowering Large Language Models with Optimal Planning Proficiency: an LLM writes PDDL for a classical planner, with no learned feasibility check.