SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models

In plain words

SMART-LLM lets a team of different robots take one job written in plain language, such as "turn off the desk light and the floor light, then watch TV", and get back a plan they can run. A language model does the planning in steps. It first breaks the job into smaller pieces. Then it works out which robot, or which small group of robots, can do each piece, based on what each robot is able to do. Last, it writes the code that tells each robot what to do, with pieces that do not depend on each other running side by side.

The language model is not trained for this. It is shown a few worked examples written as Python code, a list of the robots' abilities and the objects in the room, and then continues the same pattern for the new job. The plans were run in a simulated house, and on real mobile robots.

TL;DR

SMART-LLM prompts a language model in four stages (decompose, form teams, allocate, emit code) to turn one natural-language instruction into an executable plan for heterogeneous robots. It is evaluated on a 36-task AI2-THOR benchmark in four difficulty categories and on real mobile robots; all results are author-reported. It is relevant if you need a prompt-only multi-robot planner and can accept LLM-written code running on robots without a check in between. The repository runs only three OpenAI GPT names and has no code for the baselines or the real robots.

Problem

Multi-robot task planning usually relies on fixed algorithms, which are hard to change when the task changes §1. Natural-language instructions are often incomplete: the paper's example, "closing the laptop and watching TV in a dimly lit room", does not say to turn the lights off, and a planner has to supply that step from common knowledge §1 fig. 1. Earlier methods cover only some of the four planning phases (decomposition, coalition formation, allocation, execution) §2. Some learned methods need task-specific demonstration data, which does not generalise to unseen tasks §2.

The formal setting has N heterogeneous robots, each with a set of skills drawn from a common skill set Δ. An instruction I is decomposed into K ordered sub-tasks, some of which may run in parallel §3. A sub-task goes to one robot if that robot has all the skills it needs. If no single robot has them, a team is formed whose combined skills cover the sub-task §3. The instruction is assumed to be executable in the environment §3. Constraints such as a robot's maximum payload are named as an example of a single-robot limit §3 §4.2. The paper does not give an algorithm for choosing teams; the language model does that step.

Method

The approach uses the language model for three planning steps, and a robot interpreter for the fourth §4. Prompts are written in Python style: robot skills and object properties are Python dictionaries, and each example is a commented code block §4. Each prompt contains the skills, the environment's objects, a few worked examples and the current input §4.1 §4.2 §4.3 fig. 2.

• Stage 1, task decomposition. The instruction is split into sub-tasks, each with the skills it needs and a sequence of actions. The prompt holds the skills, the environment and decomposed examples §4.1 fig. 2.
• Stage 2, coalition formation. For each sub-task, the model writes a coalition policy: whether one robot has all the skills, or whether a team is needed and which robots form it. Samples cover one robot doing a sub-task, several robots when no one robot has every skill, and a robot that has the skill but is limited (for example by a weight limit) so that helpers are added §4.2.
• Stage 3, task allocation. Using the sub-tasks and the coalition policy, the model writes code that assigns robots to sub-tasks and runs independent sub-tasks in parallel §4.3.
• Stage 4, task execution. An interpreter runs the generated code. Each call maps to a robot's low-level skill, such as going to an object or picking it up §4.4.

The pipeline in pseudocode (the paper gives no algorithm; this follows the stages above):

# Planning pipeline  [§4] [fig. 2]
# Input:  instruction (str); robots (skill sets); objects (name, mass)
# Output: multi-robot plan as Python code (str)

def plan_multi_robot_task(instruction, robots, objects):
    # Stage 1: split the instruction into sub-tasks with their skills.
    # Prompt = skills + objects + examples + instruction  [§4.1]
    subtasks = call_llm(decomposition_prompt(instruction, robots, objects))
    # Stage 2: choose the robot or team for each sub-task  [§4.2]
    coalitions = call_llm(coalition_prompt(subtasks, robots, objects))
    # Stage 3: write code that assigns sub-tasks, in parallel where
    # the order allows  [§4.3]
    plan_code = call_llm(allocation_prompt(subtasks, coalitions, robots))
    # Stage 4: an interpreter runs plan_code  [§4.4]
    return plan_code

The feasibility rule from the problem formulation, for one sub-task:

# Single-robot and team tests for one sub-task  [§3]
# A single robot takes the sub-task if it has every required skill.
def single_robot_can_do(required_skills, robot_skills):
    return required_skills <= robot_skills
# A team works if the union of its members' skills covers the sub-task.
def team_can_do(required_skills, team_skill_sets):
    combined = set().union(*team_skill_sets)
    return required_skills <= combined

Key equations: none are numbered in the paper; the set conditions above are stated in the text §3.

Hyperparameters

Values stated in the paper or set in the official code are listed. Where the paper is silent, the code's value is given and marked.

Results

Author-reported. Setup: 36 tasks in AI2-THOR, unseen by the prompt examples, which come from other floor plans §5.1 §5.2. Each row is one backbone or baseline on one category tab. 1. Baselines: random allocation, and rule-based allocation after Gunn et al. (cited as [gunn2015dynamic] in the paper), both using GPT-4 for decomposition §5.2. The number of runs behind tab. 1 is not stated.

Table 1: benchmark results tab. 1

Source for all rows: tab. 1. SR = success rate, TCR = task completion rate, GCR = goal condition recall, RU = robot utilisation, Exe = executability (definitions in §5.4).

Variability, SMART-LLM with GPT-4, mean ± standard deviation over 5 runs tab. 2 §6.1

Ablation, GPT-4 tab. 3 §6.1 (the task scope and number of runs are not stated)

Real robots §6.2 fig. 3. No metrics are reported. The paper describes qualitative runs: robots assigned to patrol regions by visibility, and image capture of named objects, using skills (GoToLocation, ClickPicture, Patrol) that the paper says are assumed to exist §5.3.

Paper vs code

The official repository is SMARTlab-Purdue/SMART-LLM at be42930, linked from the abstract's project page; the project page links it as "Code & Dataset". It has no LICENSE file, so cite it by path and line only. Discrepancies, each with citations:

• Stage count. The paper describes four stages §4. The code makes three model calls: a decomposition call run_llm.py, a call whose examples are coalition-style team policies run_llm.py, and a code-writing call run_llm.py. Execution is a separate script execute_plan.py. The mapping of the middle call to the paper's coalition stage is inferred from the example contents.
• Backbones. The paper evaluates GPT-4, GPT-3.5, Llama-2-70B and Claude-3-Opus §5.2. The script accepts only three GPT names run_llm.py. Nothing in the repo runs Llama-2, Claude-3 or the baselines.
• Baselines and real robots. The repo has no code for random allocation, the rule-based allocator, or the real-robot skills (GoToLocation, ClickPicture, Patrol). A search of the repo for these names and for other model names found nothing.
• Prompt wording. The paper does not print its prompts. The code is the only source of the wording run_llm.py. Every allocation prompt asks for the minimum number of robots run_llm.py, and the GPT-4 allocation call adds a long system message run_llm.py. Neither appears in the paper.
• Mass rule. The formal problem uses only skills §3. The code's allocation examples and rules also require the team's combined mass capacity to meet each object's mass train_task_allocation_solution.py train_task_allocation_solution.py.
• Robot records. Robots 1 to 4 and 16 to 27 use the key mass. Robots 5 to 15 and 28 use mass_capacity robots.py. Both keys go into the prompt as written, so the model sees different field names for the same property.
• Robot utilisation. The paper gives the rule in words §5.4. The code adds 1 to both the maximum and the ground-truth transition counts before applying it end_thread.py.
• Transition count. The code counts transitions by splitting the generated plan at blank lines and counting blocks that end with a closing parenthesis and contain no def, threading.Thread or join execute_plan.py. This is a string heuristic, not a count from execution.
• Goal condition recall. The paper defines GCR as a set difference over the goals divided by the number of goals §5.4. The code increments a counter for each scene object whose name contains the goal's name and whose state matches end_thread.py. A goal matched by more than one object adds more than one.
• Executability. The paper defines Exe over all actions in the plan §5.4. The code counts only pick-up, put, toggle, open, close, slice, throw and break calls; navigation is not counted aithor_connect.py.
• Success and completion. TCR and SR match the paper: TCR is 1 only when GCR equals 1, and SR is 1 only when TCR and RU are both 1 end_thread.py end_thread.py.
• Benchmark. The 36 tasks are in 7 floor-plan files . The records have no category field, so the per-category split in tab. 1 cannot be rebuilt from the repo.

Pitfalls

• Success rate is strict. For GPT-4 on simple tasks, TCR is 1.00 but SR is 0.62. The paper attributes the gap to sequential execution where parallel was possible, which lowers RU §6.1 tab. 1. Read SR together with TCR and RU.
• Small variability sample. tab. 2 uses 5 runs of one random task per category. The Complex SR there is 0.48, but tab. 1 reports 0.71 for the same backbone. The two are not comparable, and the paper does not explain the difference §6.1.
• Ablation scope. tab. 3 has no stated task scope and no stated number of runs, so its rows cannot be compared with the category rows of tab. 1.
• Infeasible tasks are qualitative. The paper says GPT-4 and Claude-3 refused to allocate impossible tasks, while GPT-3.5 and Llama-2 produced plans with robots ill-suited to the tasks. No counts are given §6.1.
• Generated code runs unchecked. The allocated code is appended to a script with helper functions and run as a subprocess execute_plan.py. Nothing checks it between stage 3 and execution. Run it only in simulation or inside a safety wrapper.
• Output caps may truncate. The GPT-4 allocation call is capped at 400 tokens run_llm.py. A longer coalition explanation would be cut off without an error. This is an inference from the cap; the paper does not discuss it.
• Object lookup is by prefix. Object names are matched as regular-expression prefixes against scene object IDs, and the first match is used aithor_connect.py aithor_connect.py. A short name can match a different object.
• Hard-coded thresholds. The goal threshold (0.25) and the stall rule (movement under 0.2, 8 stalls) are not in the paper and have no stated units aithor_connect.py aithor_connect.py.
• Results are sampled, not fixed. The paper reports run-to-run variation, mostly in Complex tasks §6.1 tab. 2. Temperature is 0 in the code run_llm.py, but the paper cites LLM non-determinism as a source of variation §6.1. Pin the model snapshot before comparing runs; none is stated.
• Cost per task. Each call includes the full object list with masses run_llm.py [L168], plus the examples. The paper notes that token limits stop more prompt samples from being added §6.1.
• Generalisation claim. The paper says new tasks need no code changes §6.1. The skill vocabulary and examples are hand-written for this simulator, and the real-robot tests use skills that the paper says are assumed §5.3 §6.2. The claim rests on the prompt examples the authors wrote.
• Pipeline has no retry. A failed stage feeds its output into the next prompt unchanged run_llm.py. There is no check between stages.

Lineage

• Builds on: Pythonic prompts. The paper cites Wu et al. (cited as [wu2023reasoning] in the paper) for the benefit of structured prompt syntax §4.
• Metrics: follow ProgPrompt (cited as [singh2023progprompt] in the paper) §5.4.
• Compared against: the same pipeline with other backbones, and two baselines (random allocation; rule-based allocation) tab. 1.
• Superseded by: not stated.