AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers

In plain words

Robots need to follow instructions such as "visit the kitchen, then wait five seconds in the bedroom, and never step into the bathroom", and they must also respect how fast they can move and how long the whole job may take. Asking a language model to output the robot's path directly often fails, because the model is poor at counting distances and times. Asking it to list sub-goals also fails, because the best order of visits depends on the room layout and on the path the robot would actually drive.

AutoTAMP splits the job. The language model only translates the instruction into a formal rule written in a timed logic language. A separate planner then searches for a path that obeys that rule and the robot's speed limit. Because translation can go wrong, the system checks its own output. A rule checker catches malformed rules. The language model also reads the path the planner produced and asks whether it matches what the person meant. If not, the rule is revised and the loop runs again. The paper reports that this gives higher success than having the language model plan the path itself, on the tasks it tests.

TL;DR

AutoTAMP has a few-shot LLM translate each instruction into Signal Temporal Logic (STL), then runs an STL planner that outputs timed waypoints. Two re-prompting loops repair the translation: a rule-based syntax checker, and an LLM semantic check that compares the planned state sequence with the instruction. On six 2D scenarios with geometric and timing constraints, author-reported success with GPT-4 is 79-100% for the full method, which beats direct LLM trajectory planning and LLM task planning on five of six scenarios (with GPT-4; task planning is higher on HouseWorld1). Code is MIT-licensed; the paper's prompts are partly reconstructable from it, but several baselines are not in the release.

Problem

Given a natural-language instruction, the environment as named polygons, a start and a goal, produce a trajectory of timed waypoints that satisfies the instruction, does not exceed the robot's maximum speed and fits the task time limit. The trajectory is the linear interpolation between waypoints §II.

The authors argue that task sub-goals cannot be planned separately from motion when constraints couple them, for example when the best visiting order depends on geometry §I. They also argue that LLMs produce unreliable trajectories directly, which they attribute to weak spatial and numerical reasoning §III-A. A translation step can produce rules that are malformed (syntax errors) or that mean something other than the instruction (semantic errors) §I.

Method

The pipeline has four stages. (1) A few-shot LLM prompt translates the instruction, together with a list of named boxes and their coordinates, into a prefix-form STL formula built from proposition placeholders §III-C. (2) A rule-based syntax checker gives error messages to the LLM, which re-prompts up to five times §III-C. (3) The multi-agent STL planner computes timed waypoints that maximise the formula's validity under speed limits §III-C. (4) The LLM receives the planned state sequence (rooms and times) with the instruction and the formula, and either keeps the formula or revises it; a revision goes back through stages 2 and 3, for up to three rounds §III-C.

The logic is STL with the operators not, and, or, imply, equal, globally, finally and until, each with optional time bounds. Atomic predicates are "enter(room)" and "not_enter(room)" §III-C. The paper gives the grammar as an unnumbered display §III-C.

Baselines are described alongside it. End-to-end planning has the LLM emit the whole trajectory, re-prompted up to five times on constraint violations §III-A. LLM task planning has the LLM emit a sub-task sequence, handled by the same STL planner; it is evaluated as naive generation, as SayCan with K=5 candidates §III-B, and with feedback on infeasible sub-tasks §III-B.

# AutoTAMP translation and checking loop  (no algorithm box in the paper;
# reconstructed from §III-C and the released code, see Paper vs code)
# Input:  instruction (text), boxes (named squares), start, goal
# Output: STL formula and timed waypoints, or failure

def autotamp(instruction, boxes, start, goal):
    # 1. Few-shot translation to a prefix STL  [§III-C] [§IV]
    formula = llm_translate(instruction, boxes)
    # 2. Syntax loop: rule checker feedback, up to 5 rounds  [§III-C]
    for round in range(5):
        errors = syntax_check(formula)
        if not errors:
            break
        formula = llm_fix_syntax(formula, errors)
    # Deterministic repair after the loop: pad with 'and' or drop
    # leading tokens  [code: env_and_optimize_single_agent.py#L385-L391]
    formula = repair_arity(formula)
    # Formulas that still contain negation or imply are not planned
    # [code: env_and_optimize_single_agent.py#L405-L409]
    if has_negation_or_imply(formula):
        return failure
    # 3. Plan timed waypoints with the STL planner  [§III-C]
    trajectory = stl_plan(formula, boxes, start, goal)
    if trajectory is None:
        return failure
    # 4. Semantic loop: the LLM reads the planned states  [§III-C]
    for round in range(3):
        states = state_sequence(trajectory, boxes)
        revised = llm_review(instruction, formula, states)
        revised = repair_arity(revised)
        if revised == formula:
            # no change means the check accepts the formula
            break
        formula = revised
        new_trajectory = stl_plan(formula, boxes, start, goal)
        if new_trajectory is not None:
            # a failed plan leaves the old trajectory in place
            trajectory = new_trajectory
    return formula, trajectory

Hyperparameters

Results

Author-reported. Task success rates per scenario. Single-agent scenarios: HouseWorld1 (soft time limit), HouseWorld2 (hard time limit), Chip's Challenge (hard geometric constraints) tab. I. Multi-agent scenarios: Overcooked, Rover, Wall, each with hard time and geometric constraints tab. II. Setup: GPT-3 and GPT-4 as the LLM, planner from the cited MILP work §III-C; planner and checker are hard-coded §IV.

Single-agent tab. I:

Multi-agent tab. II. SayCan and task planning with feedback are not run for multi-agent scenarios §IV:

Reading the numbers §V: the full method beats task-planning baselines on HouseWorld2 and on all geometric scenarios. The authors note that task planners do better on HouseWorld1, which has no geometric challenge. They also report that most task-planning failures come from time violations and action order, and that translation failures remain the main cause for the translation methods. Replacing the LLM translator with a fine-tuned NL2TL pipeline gives small gains on HouseWorld2 and Rover §V. Physical differential-drive demonstrations on the Robotarium and 3D simulations are shown in supplementary videos, not in the tables §V.

Paper vs code

Code: https://github.com/yongchao98/AutoTAMP at commit 71e5000964dd234a4b08fbaf5e7703b85732bb8b, MIT licence. The repository contains the translation and checking loops, the STL planner wrapper and the experiment scripts. It does not contain the SayCan or feedback baselines.

• Baselines. Paper: LLM task planning, including SayCan and feedback variants §III-B. Code: llm_task_plan.py sends one prompt asking for timed (x, y, time) waypoints, closer to end-to-end planning llm_task_plan.py llm_task_plan.py. SayCan scoring and the feedback loop are not in the released files.
• In-context examples. Paper: five per method §IV. Code: the translation prompt holds dozens of instruction/formula pairs openai_func.py and the environment-translation prompt holds worked examples of its own env_and_optimize_single_agent.py. The "six example sets" selection §IV is not in the release.
• Semantic loop stop rule. Paper: stop when no error is found or the STL is unchanged §III-C. Code: stop only when the revised formula equals the previous one; there is no separate error flag env_and_optimize_single_agent.py.
• Plan failure inside the semantic loop. Code keeps the old trajectory for the next review when the revised formula has no plan, but still returns the revised formula env_and_optimize_single_agent.py env_and_optimize_single_agent.py.
• Negation and imply. Code marks a formula as a syntax failure and skips planning if negation or imply survive the rewriting step env_and_optimize_single_agent.py. The paper does not describe this filter, and it also skips the semantic check for such formulas env_and_optimize_single_agent.py.
• Syntax repair. Paper: re-prompt until the checker is satisfied, up to five times §III-C. Code: after the loop, a deterministic fix pads the formula with leading "and" tokens or drops leading tokens to match arity env_and_optimize_single_agent.py env_and_optimize_single_agent.py.
• Test pairs. Paper: five random start-goal pairs per instance §IV. The Chip script draws one jittered start and goal per instruction autotamp_single_agent.py autotamp_single_agent.py.
• Silent failures. Exceptions in the main scripts and in the translation function are swallowed, so a failed case writes nothing autotamp_single_agent.py env_and_optimize_single_agent.py. Result counts in experiment_result can therefore be lower than the number of test cases.
• Timeout. The 90-minute cut-off §IV is not found in the released code (searched for "90", "minute", "timeout").
• Model mapping. The code uses text-davinci-003 for "gpt-3" openai_func.py, which the paper does not state.

No discrepancy was found for the STL operators, the planner call or the 2D environment layouts, which were compared only by name, not by numbers.

Pitfalls

• Ambiguous translation target. The paper's STL uses real-time bounds on until, finally and globally. The released prompt asks for integer bounds or "infinite" openai_func.py. Use the same convention when re-implementing the prompt, or the bounds will not match the planner.
• Re-check cost. Each semantic round needs a fresh LLM call and a planner run. The paper's cost estimate covers the whole evaluation, not one instruction §IV. Planning time is named as a main limitation §VII.
• Prompt sensitivity. The reported numbers use the best of six prompt sets §IV. The paper says variation across sets was small, but it does not say whether the selection used held-out data, so the scores may be optimistic.
• Semantic check depends on a plan. The semantic check runs only after a plan exists and only for formulas without residual negation or imply env_and_optimize_single_agent.py. Results for such formulas depend on the planner succeeding first.
• Physical results are not measured. The physical demonstrations use tracking with a control barrier function and padded obstacles §V; they are not part of the success tables.
• Scope of planner. The authors say the STL planner used does not suit manipulation tasks and that their method is not tied to it §VII.
• Dataset. The introduction mentions a released dataset of 1400 test cases §I. This card did not check whether the repository contains it.

Lineage

• Builds on: the syntax re-prompting technique with an external verifier, which the paper says it adopts §I §VI; the authors name it by its paper title, no arXiv id is given in the source.
• Compared against: naive LLM task planning, SayCan and a feedback variant of task planning, and direct end-to-end LLM trajectory planning §III §V; NL2TL, a modular LLM-to-temporal-logic pipeline, in the translator ablation §V (the released README names it as arXiv:2305.07766).
• Superseded by: not known from content on this wiki.