Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction

In plain words

Could a team of chatbots tell a robot what to do better than one chatbot working alone? This paper tests that idea. A large language model (an AI that writes text and code) receives a spoken request, such as "take me to the nearest chair", and writes a short program that makes a legged robot do it. The robot is a Boston Dynamics Spot. The authors compare one model working alone with two team setups: a writer plus a checker, and a planner plus a writer plus a checker. Seven requests were tried, each in three runs, and a person watched each run and scored it.

The answer in the paper is that more agents did not clearly help. The writer and checker pair made the fewest errors in the first attempt, and it did best on vague requests. The team with a planner in front made the most errors. The authors say the data are limited, so the findings are tentative.

TL;DR

• Three agent setups turn a spoken request into Python that calls a small robot library on a Boston Dynamics Spot: one coder (A), a coder with a reviewer (B), and a planner, coder and reviewer (C) §3.1 fig. 5.
• All agents are GPT-4 with system prompts and no fine-tuning §3.1. The code repository says otherwise for config B: it uses gpt-4o coder_reviewer.py.
• Author-reported: config B had 9 and 14 percentage points fewer first-attempt code errors than A and C, and C had about 5 points more than A §4 fig. 6. Observer scores did not differ significantly across configurations §4 fig. 8.
• The paper's Discussion concludes that a two-agent setup beats a three-agent one §5. The Results text says the differences are not significant §4. Treat the conclusion as tentative.
• Small study: 7 prompts, 3 runs, 63 scored samples §4. No statistical test is reported in the text.

Problem

Large language models can hallucinate, and their code can fail or drift away from the task. The authors ask whether the gains from collaborating LLM agents, reported in other domains, carry over to human-robot interaction §1 §2.3. Single-agent planners that emit code have been shown to work, but with limits: their output cannot be checked before it runs §2.1. The paper treats these as motivation and does not measure the claims against other methods §2.1 §2.2.

Method

Setup

1. The robot is a Boston Dynamics Spot. The agents control it through a custom library, which talks to the robot through the Spot ROS driver §3.
2. Perception is simplified with fiducial markers on objects from seven categories: chair, person, donut, apple, refrigerator, oven and microwave §3 fig. 1 fig. 3.
3. The system prompt tells the agents that they are part of a team of AIs controlling a guide-dog robot that helps a visually impaired user move safely §3.1. The coder prompt asks for executable Python only, safety first, and use of the say() function to keep the user informed [code: knowledge/coder_prompt.xml#L1-L14].
4. The robot library exposes detect, move_to, say, ask and pose coder_prompt.xml ai.py.

Configurations

5. Config A: one coder agent that interprets the request and writes the code §3.1. The user's message goes to the coder, and the code is taken from the coder's reply coder.py.
6. Config B: a coder, a reviewer and a group-chat manager §3.1. The reviewer checks the code for correctness, task fit and safety. It sends the code back to the coder with feedback, or forwards it to a code executor §3.1 review_prompt.xml. The allowed transitions are user to coder, coder to reviewer, reviewer to coder or executor, and executor back to reviewer coder_reviewer.py.
7. Config C: a planner runs first and writes a natural-language plan. The coder turns the plan into code, and the reviewer cycle from B follows §3.1 planner_coder_reviewer.py [code: src/planner_coder_reviewer.py#L100-L104]. Config C uses a coder prompt that takes a plan as input coder_w_planner_prompt.xml.

Chat loop (configs B and C)

8. The manager runs a group chat with at most 12 rounds [code: src/coder_reviewer.py#L74] planner_coder_reviewer.py.
9. The chat ends when the reviewer writes a message containing the word "terminate", in any letter case coder_reviewer.py. Config A ends the same way on the coder's messages coder.py.
10. The next speaker is chosen among the allowed transitions. The code sets no selection method for config B, so the library default applies [code: src/coder_reviewer.py#L69-L76]. Config C fixes the first speaker to the planner planner_coder_reviewer.py.

# Pick the code for the robot from the final chat message  [code: src/coder.py#L66]
# Input:  final_message (str), the last message of the chat
# Output: code (str), the text inside the first triple-backtick fence

def extract_code(final_message):
    # Non-greedy match across newlines; a missing fence raises an error
    match = re.search(r"```(.*?)```", final_message, re.S)
    return match.group(1)

Execution

11. Code runs in a local command-line executor. The executor asks a human for approval before every run, because human_input_mode is set to ALWAYS coder.py coder_reviewer.py planner_coder_reviewer.py. The paper does not describe this approval step.
12. The executor timeout is 600 seconds in configs A and C [code: src/coder.py#L22-L25] planner_coder_reviewer.py. Config B sets no timeout in code coder_reviewer.py.

Protocol

13. Seven prompts are used, each with a room context and a challenge type §3 tab. 1. Each configuration is run on each prompt, and the whole set is repeated three times, each time with an independent observer §3.
14. Code is generated before the observer arrives. If a run errors, it is not recorded and the code is regenerated §3.3. The error rate counts first attempts only fig. 6.
15. The observer rates each run: task success, safety and sociability on 1 to 5, and the gap between expected and actual behaviour on −5 to 5. A tester records code execution failure, inference time, execution time, and input and output tokens §3.2 tab. 2.
16. A case study reviews 12 code samples: prompts 3 and 5, two attempts per configuration, anonymised. One technically proficient reviewer does a qualitative review §3.4.

Hyperparameters

Results

Author-reported. Setup: seven prompts, three configurations, three runs with three observers, GPT-4-based agents, Boston Dynamics Spot §3 §4. The paper gives no absolute values in the text. Figures carry the numbers, and I did not read them off the images.

Conclusion, as the paper states it: a two-agent configuration performs better than a three-agent one §5. The text also proposes RAG and a pseudo-code planner as future fixes §5.

Paper vs code

The official repository is sheepskins/spottyai at commit 3161092e169e7d02b1d75b58252a1ef4541bde27, which belongs to the first author (its LICENSE names Mitchell Rosser). It is linked from the paper's bibliography §3. I searched no further. Discrepancies:

• Model. Paper: all agents are GPT-4 §3.1. Code: gpt-4 for A and C, but gpt-4o for B coder.py [code: src/coder_reviewer.py#L25] planner_coder_reviewer.py.
• Usage key. The B usage code reads a usage entry keyed gpt-4-0613 coder_reviewer.py, which does not match gpt-4o. The config B log files also record gpt-4-0613 config_b_11.py.
• Config B logs. No script in the repo writes the config B trial files. coder_reviewer.py only writes a usage CSV, and its path is absolute /data because of a bad os.path.join coder_reviewer.py. The config A and C scripts write files with the same format as the B logs [code: src/coder.py#L43-L66] planner_coder_reviewer.py.
• Round limit and timeout. Paper gives neither. Code sets 12 rounds and a 600-second timeout for A and C, but no timeout for B coder_reviewer.py coder.py.
• Human approval. Paper does not mention it. Code requires a human yes before each run in all three configs coder.py [code: src/coder_reviewer.py#L39] planner_coder_reviewer.py.
• Trial wording. Trial 4 reads "wait until I'm ready and then" in code, and "wait until I'm ready, and then" in Table I. Trial 7 has a trailing period in code only trial_prompts.csv [code: src/trial_prompts.csv#L8] tab. 1.
• Sample count. The repo holds 87 log files: 63 without a suffix and 24 with an a or b suffix . The suffixed files look like regenerated attempts. The paper describes regeneration §3.3 but gives no count of regenerated runs.
• Extra material. knowledge_dance/, src/coder_reviewer_dance.py and src/tmp/ hold a dance-motion variant that the paper does not describe coder_reviewer_dance.py.

Pitfalls

• Robot API mismatch. The coder prompt documents move_to(x, y) [code: knowledge/coder_prompt.xml#L46-L59], but the library defines move_to(x, y, yaw) ai.py. Code that follows the prompt raises a TypeError. Some errors in the error-rate counts may come from this mismatch and not from the model.
• Backtick dependency in config C. The extractor needs triple backticks planner_coder_reviewer.py. The config C coder prompt drops the backtick instruction that the other prompt keeps [code: knowledge/coder_w_planner_prompt.xml] coder_prompt.xml. The planner and reviewer prompts do not ask for backticks either.
• Contradictory termination rule. The reviewer prompt asks for the word "terminate" when the code is not to be terminated [code: knowledge/review_prompt.xml#L6-L7]. The chat stops on any reviewer message containing "terminate", so "do not terminate" ends the chat [code: src/coder_reviewer.py#L54].
• Routing is model-chosen. Config B's next speaker is picked by the group-chat manager from the allowed transitions. The code sets no selection method, so it relies on the library default (not pinned; see requirements.txt). Repeated runs can take different paths.
• Confounded comparison. If config B ran on gpt-4o and A and C on gpt-4, the B vs A and B vs C differences mix model and architecture. The repo cannot settle which model produced the logs (see Paper vs code).
• Statistics. The Results text says the configurations do not differ significantly §4, but the Discussion says the study "clearly demonstrates a significant impact" §5. The text reports no test, three runs, and one observer per run. Variance is unknown.
• Error-rate definition. Only first attempts count, and errored runs are regenerated §3.3 fig. 6. A configuration that errors often but recovers on regeneration looks worse here than it would in use.
• Hidden token cost. The paper says input tokens rise almost exponentially with the number of agents §5. The text gives no numbers beyond fig. 7. Config C's long system prompt is named as the likely cause §5.
• Human gate. The paper does not describe the approval gate in the code. If a human approved runs, the approval itself may have shaped the observer scores. The paper does not discuss this.
• Sampling. No temperature or seed is set in code, so the defaults of the OpenAI API and the AutoGen library apply coder.py.
• Stated limitations. Few configurations and few observers. Perception and locomotion failures may have affected scores, and the authors say they were spread across all systems §5.

Lineage

• Builds on: AutoGen, the multi-agent framework whose group chat the code uses §2.2 fig. 2 coder_reviewer.py.
• Methodology inspired by MechAgents, a multi-agent LLM study on mechanics problems §3 (no arXiv id in the bibliography).
• Related background: Code as Policies (LLM-written robot code; arXiv:2209.07753) §2.1; SayCan (affordance grounding; arXiv:2204.01691) §2.1.
• Compared against: the single-agent configuration A, which is the paper's own baseline §3.1.
• Superseded by: not stated.