RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

In plain words

Two or three robot arms that share a table have to agree on who does what, and they must not bump into each other while moving. This paper gives every robot its own language model. The models talk to each other in a short chat, the way colleagues would, and end up proposing one job for each robot.

The proposed plan is then checked before anything moves. The checks cover the task rules, whether each arm can reach its target, whether a joint setting exists for that target, whether the arms would collide, and whether the intermediate points of a path are spaced evenly. When a check fails, the reason is written out and given back to the models, which talk again and try a new plan. Once a plan passes, a motion planner works out the arms' paths.

The authors test this on six simulated tables with two or three arms. The method needs no task-specific training, and on some tasks it does well, but not on all of them. A single planner that sees the whole scene and plans for every arm at once does better on average. The authors also show a human talking with one robot in a real block-sorting setup.

TL;DR

• Each robot is an LLM agent. The agents take turns in a dialog and end with one sub-task per robot §3.1 §3.2.
• A proposed plan passes ordered checks: task rules, reach, inverse kinematics, collisions and waypoint spacing. Failures return as text feedback for re-planning §3.2 feedback.py.
• A bidirectional RRT planner for all arms together turns validated goals and waypoints into joint trajectories §3.3 rrt_multi_arm.py.
• Zero-shot with GPT-4 on RoCoBench (six MuJoCo tasks). Author-reported success is 0.44 to 0.95 per task for the full dialog method, and the oracle central planner has the higher mean tab. 2.
• The official MIT-licensed code implements the pipeline, but several checks and defaults differ from the text. See Paper vs code.

Problem

Multi-robot systems can split work and speed it up, but three things make them hard §1:

• Choosing who does what needs a high-level grasp of the task and of each robot's reach and payload.
• Collision-free motion planning gets much harder as the number of arms grows, because the joint configuration space grows with them §2.
• Traditional systems hard-code the task structure and cannot adapt to new scenarios without re-engineering §1.

The paper argues that LLM agents can supply the high-level reasoning without task-specific training, while a classical planner handles the motion §1.

Method

Setting §2. A cooperative task with N robots and a finite horizon T. Each agent n sees only part of the observation space, Ω^n ⊂ O, and may have different capabilities, so the robots must communicate. Hand-written description functions turn each agent's observations into prompt text. Parsing functions map the LLM's text onto sub-tasks, and each sub-task becomes one or more gripper goal configurations §2. The planner solves for a collision-free path σ* from the start configuration to the goal region in the joint space of all arms §2.

Step 1: multi-agent dialog §3.1 fig. 2. Before each environment step, every robot gets its own LLM agent. Each agent's prompt has six parts: task context, round history, agent capability, communication instruction, current observation and optional plan feedback §3.1. The agents answer in turn. Each reply either passes the discussion on or proposes a final plan, and a final plan is allowed only after every agent has spoken once in the round §3.1. The code applies that rule only in the first attempt of a round, see Paper vs code dialog_prompter.py.

Step 2: sub-task plan and ordered checks §3.2. The last speaker writes the sub-task plan: one sub-task per agent, and optionally a list of 3D waypoints per agent. The checks run in this order, and each one assumes the previous one passed §3.2:

1. Text parsing: the format and required keywords are present.
2. Task constraints: each action fits the task and the agent's constraints.
3. IK: each arm's target pose has an inverse-kinematics solution.
4. Collision: the resulting arm joint configurations are collision-free.
5. Valid waypoints (optional): every intermediate waypoint is IK-solvable and collision-free, and the steps are evenly spaced.

If a check fails, its feedback is appended to every agent's prompt and a new dialog round starts. After a maximum number of attempts the round ends without execution, and the episode fails if the task is not finished within the finite number of rounds §3.2.

Step 3: motion planning §3.3. A validated plan gives one goal configuration for all arms together, from IK. Each waypoint gives one more intermediate configuration. A centralized RRT-based multi-arm planner plans for all arms jointly and outputs one trajectory per robot. The robots execute the trajectories open-loop §3.3 §7.

The pseudocode below combines the dialog and the checks. The first block follows the outline in alg. 1 and §3.1-§3.2. The second and third blocks follow the code, which adds a reach check that the paper's list does not name.

# One planning round: dialog, parsing, checks, feedback  [alg. 1] [§3.2]
# Input:  agents (one LLM per robot), env, K (re-plans), M (calls per dialog)
# Output: a plan that passed every check, or None

def plan_round(env, agents, K, M, max_failed_waypoints):
    # Feedback texts from failed attempts, given to agents in the next dialog
    #   [§3.2] [code: prompting/dialog_prompter.py#L119-L144]
    feedbacks = []
    # Messages from earlier attempts in this round  [§3.1]
    chat_history = []
    for attempt in range(K):
        # Agents speak in turn, at most M calls, until one writes a plan
        #   [§3.1] [code: prompting/dialog_prompter.py#L184-L237]
        plan_text, messages = run_dialog(
            env, agents, chat_history, feedbacks, M, attempt
        )
        chat_history += messages
        # Text parsing comes first  [§3.2] [code: prompting/parser.py#L48-L57]
        plan, error = parse_plan(env, plan_text)
        if error is None:
            # Ordered checks; each runs only if the previous one passed
            error = check_plan(env, plan, max_failed_waypoints)
        if error is None:
            return plan
        feedbacks.append(error)
    return None

# Ordered checks on a parsed plan  [§3.2]
#   [code: prompting/feedback.py#L163-L201]
# Output: None when every check passes, otherwise a feedback string

def check_plan(env, plan, max_failed_waypoints):
    # Task constraints from the environment  [§3.2]
    #   [code: prompting/feedback.py#L168-L170]
    task_error = env.get_task_feedback(plan, plan.ee_target_poses)
    if task_error:
        return task_error
    # Goal poses: reach, IK, collision  [§3.2]
    #   [code: prompting/feedback.py#L172-L178]
    goal_error = check_pose_set(env, plan, plan.ee_target_poses)
    if goal_error:
        return goal_error
    # Waypoints: a fixed number of failures may be tolerated
    #   [§3.2] [code: prompting/feedback.py#L180-L189]
    failed = 0
    for pose_dict in plan.ee_waypoints_list:
        if check_pose_set(env, plan, pose_dict):
            failed += 1
    if failed > max_failed_waypoints:
        return "too many failed waypoints"
    # Spacing: per arm, the spread of step lengths must stay under the
    # threshold  [§3.2] [code: prompting/feedback.py#L107-L128]
    if spacing_std(plan) > env.waypoint_std_threshold:
        return spacing_feedback(plan)
    return None

# Checks on one set of arm targets  [code: prompting/feedback.py#L136-L161]
# Output: None when all arms pass, otherwise a feedback string

def check_pose_set(env, plan, pose_dict):
    # Each target must lie inside its arm's reach; not named in [§3.2]
    if not all_in_reach(env, pose_dict):
        return "out of reach"
    # Inverse kinematics must solve for every arm  [§3.2]
    ik_solutions = solve_ik(env, pose_dict)
    if any(solution is None for solution in ik_solutions):
        return "IK failed"
    # The joint configuration must be collision-free  [§3.2]
    if collides(plan, ik_solutions):
        return "collision"
    return None

# Joint-space motion for one validated plan  [§3.3]
#   [code: rocobench/rrt_multi_arm.py#L620-L637]
# Input:  start_q, goal_q (all arms stacked), waypoint_qs (from IK)
# Output: a collision-free joint path, or None

def plan_motion(start_q, goal_q, waypoint_qs, timeout):
    # LLM waypoints are used first, reversed, as seed samples  [§3.3]
    seeds = list(reversed(waypoint_qs))
    # Bidirectional RRT, greedy extension, then path smoothing
    path = birrt(
        start_q,
        goal_q,
        init_samples=seeds,
        iterations=800,
        smooth_iterations=200,
        timeout=timeout,
    )
    return path

Benchmark: RoCoBench §4 tab. 1. Six tabletop tasks in MuJoCo, built from common objects, and each task is rated on three properties: whether the sub-tasks can run in parallel or must be ordered, whether the robots share their observations or each sees only part of the scene, and how much the workspaces overlap §4 tab. 1. The robots and their skills are listed in the appendix §10:

Toy example: LLM path planning in 3D §3.2 §13. GPT-4 is asked to plan collision-free paths for three agents in a 3D grid, with feedback on failed plans and up to 5 attempts. It succeeds in 86.7% of 30 random runs, using 2.73 attempts on average. The authors note that performance falls as the grid and the number of obstacles grow §3.2.

Round history §3.1 dialog_prompter.py. Past dialog and executed actions are kept per episode. A round is added to the history only after its plan has executed successfully.

Hyperparameters

Values in the paper are given where it states them; values in the code are marked as code. "Not stated" means the paper does not give a value.

Results

Author-reported. Setup: GPT-4 for all main results, six RoCoBench tasks, 20 runs per task. The oracle "Central Plan" gets the full observation, all robots' capabilities and the same plan feedback, and plans for all robots in one prompt. The ablations are "Dialog w/o History", which drops past rounds, and "Dialog w/o Feedback", which drops environment feedback and gives the episode twice the budget §5.1. Metrics: success rate, steps in successful runs, and average re-plan attempts per round §5.1.

Success rate tab. 2 (mean ± std over runs as reported):

Steps in successful runs / average re-plans per round tab. 2. The "N/A" entry is as printed in the table: no run succeeded.

The authors say that the dialog agents sometimes match the oracle, that the oracle is better at waypoints on Pack Grocery, and that the full prompt gives "the best overall results" §5.1. Averages across the six tasks, computed from tab. 2 for this card and not reported by the authors: Central Plan 0.81, Dialog (ours) 0.75, Dialog w/o Feedback 0.56, Dialog w/o History 0.53.

Waypoints §5.2 fig. 4. On Pack Grocery and Move Rope, LLM-proposed waypoints gave no clear benefit for picking, but sped up placing, where arms and objects are more likely to collide. The numbers are only in the figure. The comparison uses a 300 s planning budget §5.2.

Zero-shot variations §5.3 fig. 5. On Make Sandwich, the authors show qualitative adaptation to randomized food positions, to a different recipe order and to a reduced reach for one robot. No numbers are reported.

Real world §5.4 app. §12.2 tab. 3. A UR5e with a suction gripper (robot "Bob") talks with a human ("Alice") over a block-sorting task. Perception uses OWL-ViT on top-down RGB-D images. Ten runs per setting.

The authors attribute most failures to object-detection errors from OWL-ViT §5.4.

Benchmark properties tab. 1:

Text-only reasoning dataset §6 tab. 4. Accuracy by category, one correct answer per question. The question counts are 57 (capability), 44 (memory), 41 (inquiry), 96 (responsiveness) and 31 (adaptation) §14.

Paper vs code

Official code: https://github.com/MandiZhao/robot-collab at commit 9943870298e650de7ff1f9231ed279b35f196ddc, MIT license. The owner is the paper's first author, Zhao Mandi (the README spells the name "Mandi Zhao"). The repository contains the dialog loop, the checks, the RRT planner and the six RoCoBench environments. The RoCoBench-Text data is described in the repository README README.md but its data files are not in the repository, so the reasoning results cannot be reproduced from it.

• Dialog ending rule. The paper allows a final plan only after every agent has spoken §3.1. The code applies that rule in the first attempt of a round only. From the second attempt on, any reply containing EXECUTE ends the dialog dialog_prompter.py.
• Ending token. The paper describes two options, pass on or propose §3.1. The code only checks for the string EXECUTE dialog_prompter.py.
• Checks. The paper lists five checks §3.2. The code adds a reach check before IK for the goal and for each waypoint feedback.py.
• Waypoint tolerance. The paper says every intermediate waypoint must be IK-solvable and collision-free §3.2. The code tolerates failures up to max_failed_waypoints, which is 1 by default in the CLI feedback.py run_dialog.py.
• Spacing threshold. The paper says "evenly spaced" with no number §3.2. The code uses an environment-specific threshold, 1.0 by default, 0.19 for Pack Grocery and 0.3 for Move Rope base_env.py.
• Planning time. The paper's waypoint study uses 300 s §5.2. The code's default is 200 policy.py. Its comment calls the value a count of planning steps policy.py, but the RRT compares it with elapsed wall-clock seconds rrt.py.
• Temperature. The main results do not state one §5.1. The code defaults to 0.0 run_dialog.py.
• Episode budget. The paper says the no-feedback variant gets twice the budget §5.1. The code does not double it. The budget is the -t flag, default 10 run_dialog.py, and no-feedback runs require one re-plan run_dialog.py.
• Model provider. The code asserts that claude is a valid source dialog_prompter.py, but every call goes to the OpenAI ChatCompletion API dialog_prompter.py.
• Code-only settings. Seeds are set per run run_dialog.py. The RRT extension resolution (0.006) and the iteration counts appear only in code rrt_multi_arm.py rrt_multi_arm.py.

Not compared in detail: the MuJoCo environment files and the real-world code.

Pitfalls

• Abstract overstates. The abstract says the method achieves "high success rates across all tasks". Per tab. 2, the full dialog method reaches 0.44 on Pack Grocery and 0.65 on Move Rope, and 0.93 on Sort Cubes.
• Oracle beats the method on average. Central Plan's mean success is 0.81 and the full dialog's is 0.75, computed from tab. 2. The authors describe the dialog as "sometimes comparable" §5.1.
• Ablations are not uniformly worse. Dialog w/o History beats the full method on Arrange Cabinet (1.00 vs 0.75) tab. 2, so "full prompt best" holds only on average.
• Small runs. Each cell is 20 runs tab. 2. Differences of 0.1 are within a few runs. Report intervals before comparing variants.
• Perception is assumed exact in simulation. The authors note that the simulator supplies accurate object detection, collision checks and poses, and that this fails with real perception §7. The real-world run shows this: object-detection errors drive most failures §5.4.
• Open-loop execution. Trajectories run without feedback, so execution errors such as a knocked object are not corrected §7 app. §12.1.
• LLM cost scales with the loop. Code defaults allow up to K × M calls per round, which is 5 × 10 = 50 calls at the CLI defaults, plus up to three retries per call on API errors run_dialog.py run_dialog.py dialog_prompter.py. The authors list query cost and latency as a limitation §7.
• Feedback is only as good as the checks. A plan that passes the checks can still fail when executed. Each task has its own spacing threshold, so reusing one for a new task changes which plans are rejected base_env.py.
• Snapshot drift. The reasoning results differ by GPT-4 snapshot. Inquiry accuracy is 0.79 for 0314 and 0.57 for 0613 tab. 4. The code uses the unversioned name gpt-4 run_dialog.py.
• Legacy SDK. The code uses the pre-1.0 openai.ChatCompletion API and an API key read from openai_key.json at import dialog_prompter.py dialog_prompter.py. Pin openai==0.27.2 as in requirements, or port the calls.
• Parser is strict. The parser needs exactly one NAME … ACTION line per robot after EXECUTE, else it returns feedback parser.py.
• History is written only on success. A failed plan is shown to the next attempt as text, but not added to the round history dialog_prompter.py dialog_prompter.py.
• Appendix numbering is inferred. The source does not use \appendix, so appendix sections are numbered by source order here (§10 to §14). Check them against the PDF before citing.
• Adaptation is qualitative. The claims about variations in object positions, recipes and reach are not measured §5.3.

Lineage

• Builds on: GPT-4 as the LLM §5.1. The planner is an RRT method; the code uses a bidirectional variant rrt.py.
• Related work named by the paper: SayCan, Inner Monologue (in-context feedback), CaP, TidyBot, Text2Motion and AutoTAMP §8. None has a card here.
• Compared against: the Central Plan oracle and two ablations §5.1 tab. 2.
• Superseded by: not stated.