CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

In plain words

Usually a person has to keep steering a chatbot with careful prompts until it finishes a job. This paper asks whether two chatbots can steer each other. One chatbot is given the role of a helper, such as a Python programmer, and the other the role of a client, such as a stock trader. A third step first turns a vague idea into a concrete task. After that the two bots take turns: the client asks for one step at a time, the helper answers, and the client finally says a fixed stop word when the job is done.

The careful part is the wording of the starting instructions. They tell each bot who it is, forbid swapping roles, require a fixed answer format, and make the client end the chat with the stop word. The chats the bots produce can then be used as training data for smaller models.

TL;DR

• Two LLM agents (assistant and user) hold a role-play. A task-specifier agent turns an idea into a concrete task §3.1.
• Fixed "inception" system prompts set roles, the one-instruction-at-a-time protocol, the answer format and the end token <CAMEL_TASK_DONE> §3.2.
• Author-reported: CAMEL solutions beat single-shot gpt-3.5-turbo in human and GPT-4 votes, and a LLaMA-7B fine-tuned on the generated data scores 14.0 pass@1 on HumanEval §5.
• The repo is Apache-2.0 and its HEAD is a later library version. Prompt wording and termination rules differ from the paper (see Paper vs code).

Problem

Chat models need a human to write each prompt and react to each answer §1. Autonomous cooperation between two chat agents runs into four failure modes the authors saw in pilot runs: role flipping, the assistant repeating instructions, "flake" replies that only promise action ("I will ..."), and endless polite loops §1 §4. Writing prompts that avoid these is hard and needs domain knowledge; the paper argues a non-expert cannot do it reliably §3.1.

Method

The framework is task-oriented role-play between one AI assistant and one AI user. A human supplies an idea and two roles, or the roles and ideas are generated by an LLM for the datasets §3.1. A task-specifier agent expands the idea into a task of at most a word limit fig. 2. The assistant (A) and the user (U) are each a language model that receives its own system message P_A or P_U §3.1. The user gives an instruction and the assistant gives a solution, round after round §3.1:

1. Specify: idea and roles go to the task-specifier, which returns a concrete task §3.1 fig. 2.
2. Assign: P_A and P_U are passed as system messages before the chat starts; in the experiments both agents are gpt-3.5-turbo §3.1 §4.
3. Converse: at each step the user writes an instruction from the message history, the assistant answers it, and the pair is appended to the history §3.1 eq. 1–4.
4. Stop: the chat ends on the end token, a turn-limit, or one of the termination rules in §4.
5. Critic (optional): a critic agent or a human picks among proposals or gives feedback, giving a tree-search-like loop; details are given in the appendix, not covered here §3.1.

Message history and update rules, with the paper's notation:

\mathcal{M}_t = \{(\mathcal{I}_i, \mathcal{S}_i)\}_{i=0}^{t} eq. 1

\mathcal{I}_{t+1} = \mathcal{U}(\mathcal{M}_t) eq. 2

\mathcal{S}_{t+1} = \mathcal{A}(\mathcal{M}_t, \mathcal{I}_{t+1}) eq. 3

\mathcal{M}_{t+1} = \mathcal{M}_t \cup (\mathcal{I}_{t+1}, \mathcal{S}_{t+1}) eq. 4

The paper notes the same formulation extends to more than two agents through message-passing graphs §3.1. The loop below follows these equations and the repository's two-agent example. It is written in our words and cites the repository where the code shows the exact order role_playing.py.

# CAMEL two-agent role-play loop  [eq. 1-4]
# Input: idea, assistant_role, user_role, turn_limit
# Output: list of (instruction, solution) pairs

def role_play(idea, assistant_role, user_role, turn_limit):
    # Task specifier makes the idea concrete  [§3.1] [fig. 2]
    task = specify_task(idea, assistant_role, user_role)
    # Fixed system prompts hold roles, rules, end token  [§3.2]
    assistant = Agent(assistant_prompt(assistant_role, user_role, task))
    user = Agent(user_prompt(user_role, assistant_role, task))
    # Seed message enters through the user agent
    # [code: examples/ai_society/role_playing.py#L49]
    last_message = seed_message(task)
    history = []
    for turn in range(turn_limit):
        # User writes the next instruction  [eq. 2]
        instruction = user.reply(last_message)
        # Assistant answers the instruction  [eq. 3]
        solution = assistant.reply(instruction)
        # Append the pair to the history  [eq. 4]
        history.append((instruction, solution))
        # Stop on the end token  [§3.2]
        # [code: examples/ai_society/role_playing.py#L81-L82]
        if "<CAMEL_TASK_DONE>" in instruction:
            break
        last_message = solution
    return history

Inception prompts §3.2 fig. 2. The inception prompt has three parts: one for the task specifier and one system prompt per agent, P_A for the assistant and P_U for the user. Key rules in P_A:

• It names the assistant's role and the user's role and forbids flipping them and forbids the assistant from instructing the user.
• It lets the assistant decline an instruction it cannot perform for physical, moral, legal or capability reasons, and requires it to explain why fig. 2.
• It fixes the answer format: start with "Solution:", then the answer, and end with "Next request." fig. 2 §3.2.

Key rules in P_U:

• It requires exactly two instruction forms: with an input, or with the word "None" as input fig. 2 §3.2.
• It requires one instruction per turn and no questions to the assistant fig. 2.
• It asks the user to reply only with <CAMEL_TASK_DONE> once the task is solved, and never before fig. 2 §3.2.

Data generation §4.1 fig. 3. Roles and tasks are generated by prompting the LLM: 50 assistant roles, 50 user roles and 10 tasks per role pair, giving 25,000 AI Society conversations §4.1 fig. 3. The Code scenario uses a similar pipeline with 20 languages and 50 domains app. "Meta Data". Math and Science problems are generated by GPT-4 from 25 topics, 25 subtopics per topic and 80 (math) or 32 (science) problems per pair app. "Math and Science Datasets Generation Details".

Hyperparameters

Results

Author-reported. Setup: gpt-3.5-turbo single-shot baseline; CAMEL solutions are summarised by GPT-4 before comparison §5.1. Human votes are for AI Society only §5.1.

Emergence §5.2 tab. 2: LLaMA-7B is fine-tuned in stages (AI Society, then +Code, +Math, +Science) and judged by GPT-4 against the previous stage. The paper reports that adding a dataset improves its own domain, and sometimes others: training on Code also helps Science, and training on AI Society helps Code. The table is a count table (draws, wins for each model), not a single score. One row is a counter-example to the trend: on Math, the model with AI Society alone wins 8 to 7 against the model with AI Society plus Code tab. 2.

The paper reports that the human and GPT-4 votes are highly aligned §5.1.

Paper vs code

The repository is camel-ai/camel at commit e5e55256e9446f744bf7309c478b084587dbf985, the HEAD of the clone. It is a later library version (its files carry 2023–2026 copyright and use the RolePlaying and ChatAgent classes); the 2023 experiment code is not in the clone, so the discrepancies below are between the paper and this later code. The abstract links the repository, so it is the official one. Code: Apache-2.0 LICENSE. Discrepancies:

• Message limit. Paper: 40 messages §4. The repo example caps at chat_turn_limit=50 turns role_playing.py role_playing.py. A turn is two messages, so the example allows about 100 messages.
• Termination rules. The paper lists five conditions: no instruction for 3 rounds, assistant instructs (role flip), end token, token limit and the 40-message cap §4. In the example script only the turn limit, the agents' own termination flags and the end token are checked role_playing.py. A search of camel/ and examples/ai_society/ found no "no instruction for 3 rounds" counter.
• Assistant prompt wording. Paper: the solution "should be specific, and provide preferable implementations and examples" fig. 2. Repo: the same requirement, plus more detailed explanations, more detailed implementations and examples, and lists ai_society.py. The repo's first rule also asks the assistant to "explain your solutions"; the paper's version does not.
• User prompt wording. Paper: "to complete the task ONLY" fig. 2. Repo: "to solve the task ONLY" ai_society.py.
• Word limit. Paper: placeholder only fig. 2. Repo default: 50 words task_agent.py.
• End token. The repo's RolePlaying.step calls the assistant once more after the user writes the end token, and only then does the example loop break role_playing.py role_playing.py. The paper says the chat ends when the token is received §4.
• Sampling. The paper gives no temperature. Repo examples set other values, e.g. temperature=1.4 role_playing_multiprocess.py and temperature=0.8, n=3 for the critic example role_playing_with_critic.py. These are examples, not the paper's settings.
• Fine-tuning. No LLaMA training script matching the paper's configuration was found in the repository, so the training table app. "Training Details" is not checked against code.

No discrepancy was checked for the data-generation prompts, because the repo's generation prompts were not compared line by line against fig. 3.

Pitfalls

• Prompt drift. The prompts in the paper and in the repo differ. An agent that copies the repo's prompts will not reproduce the paper's setup. Use the paper's text from fig. 2 if you want to replicate the experiments.
• Limit ambiguity. "40 messages" does not say whether a message is one side of a turn. The repo counts turns. Pick one and say which.
• Role-flip detection is not specified. The paper names "Assistant Instruct" as a termination reason and says it was the main reason for AI Society conversations app. "Dataset Analysis". It does not say how it is detected. The repo example does not detect it.
• Prompt ablation text is inconsistent. The text says Prompt V2 removes the "Solution" format and "Next request" app. "Dataset Analysis". The figure caption says it removes an output format that starts with "Output:". The ablation numbers are shown only in figures; no values are given in the text.
• Flake replies. The authors report that flake replies ("I will ...") rose under both ablations, and their caption says the original prompt has the fewest app. "Dataset Analysis". Flake replies are not eliminated by the original prompt.
• Evaluation confounds. The CAMEL conversation is summarised by GPT-4 before comparison, which the paper says hides its format §5.1. GPT-4 is both summariser and judge. Human votes are for AI Society only. No error bars or variance across runs are given; the checklist says experiments use seed 0 and does not report error bars app. "Check List Requirements".
• Cost grows quickly. The paper says cost grows quadratically with conversation length, which is why the message cap exists §4. Total API cost and token counts for the datasets are not stated.
• Licensing for reuse. The paper's arXiv licence (CC BY 4.0) covers the paper, not the datasets or model weights. The datasets were produced with gpt-3.5-turbo and GPT-4 through OpenAI's API; the paper does not state the terms that apply to the generated data. The fine-tuned models are based on LLaMA-7B, which the paper describes as under a non-commercial licence app. "Training Details". Do not assume the released artefacts can be used commercially.
• Dataset sizes not fully stated. The number of CAMEL Code conversations is not stated in the sections read for this card.

Lineage

• Builds on: gpt-3.5-turbo for the agents and GPT-4 for summarising, judging and generating Math and Science data §4 app. "Math and Science Datasets Generation Details". The fine-tuned model is LLaMA-7B §5.2.
• Related work named in the paper: automated instruction-data generation such as Self-Instruct and Unnatural-Instructions §2.
• Compared against: gpt-3.5-turbo single-shot solutions §5.1; LLaMA-7B and Vicuna-7B on HumanEval tab. 3.
• Superseded by: not known from content already in this wiki.