MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

In plain words

MetaGPT treats a team of language-model agents like a small software company. One agent plays the product manager and writes down what the product should do. Another plays the architect and designs the parts of the system, a third splits the work into tasks, a fourth writes the code, and a fifth tests it. Each role hands over a fixed-format document rather than chatting freely, which the authors argue limits errors as the messages pass along.

The engineer also runs the code it writes. If the tests fail, it reads the errors and tries again, using its own earlier attempts as a memory. The authors report that this produces more working programs than earlier chat-based agent teams on standard coding benchmarks.

TL;DR

• Five LLM agents follow a software-company workflow: product requirements, system design, task split, coding, testing §3.1.
• Agents publish structured documents to a shared message pool and act on messages they subscribed to §3.2.
• An engineer writes unit tests, runs them, and debugs on failure, with a retry limit of 3 in the text §3.3.
• Author-reported Pass@1: 85.9% HumanEval and 87.7% MBPP; SoftwareDev results on seven tasks §4.2 tab. 1.
• The official repository (MIT) has moved on since the paper, so its roles and retry limits differ; see Paper vs code.

Problem

Chat-style multi-agent systems let agents talk in free natural language. The authors argue that over several rounds the information gets distorted, and that agents also keep discussing when they should be producing outputs §1 §3.2. Complex tasks then suffer from inconsistent logic and cascading errors from chained LLM calls [abstract]. Free chat also invites "idle chatter" that feeds hallucinations §1. The paper also argues that generated code is often not executable, and that reviewing it without running it misses runtime errors §3.3.

Method

The paper defines the framework in three parts: role specialization inside a standard operating procedure (SOP) §3.1, a communication protocol with structured outputs and a publish-subscribe pool §3.2, and executable feedback for self-correction §3.3. The paper has no numbered algorithm or equation blocks for these parts; the pseudocode below is our rendering of the text.

Roles and workflow §3.1. Five roles: Product Manager, Architect, Project Manager, Engineer and QA Engineer fig. 1 §3.1. Each agent has a profile (name, profile, goal, constraints), its own context and skills; the product manager may use web search and the engineer may execute code §3.1. Agents follow a ReAct-style loop §3.1. The work runs in sequence: the product manager writes a PRD with user stories and a requirement pool; the architect produces a file list, data structures, interface definitions and a sequence flow; the project manager allocates tasks; engineers implement the files; the QA engineer writes test cases §3.1 fig. 3.

Structured communication §3.2. Agents exchange documents and diagrams rather than dialogue. The architect produces two outputs, a system interface design and a sequence diagram, which the engineers use as inputs §3.2 fig. 3.

Message pool and subscriptions §3.2. All messages go to one shared pool fig. 2, left. An agent reads only the messages matching its role profile. An agent starts an action only after all of its prerequisite inputs have arrived §3.2.

# SOP pipeline over a shared message pool  [§3.1] [§3.2] [fig. 3]
# Input:  requirement (text from the user)
# Output: PRD, system design, tasks, code files and test files

def run_software_company(requirement, roles):
    # Publish the user requirement to the shared pool  [§3.2]
    pool = MessagePool()
    pool.publish(requirement)
    # Act while there are unhandled messages  [§3.1]
    while pool.has_unhandled_messages():
        for role in roles:
            # Read only the messages this role subscribed to  [§3.2]
            news = pool.messages_for(role.subscriptions)
            # Start only when all prerequisite inputs are present  [§3.2]
            if not role.prerequisites_met(news):
                continue
            # Produce a structured document, not a chat reply  [§3.2]
            document = role.act(news)
            pool.publish(document)
    return pool.code_and_test_files()

Executable feedback §3.3 fig. 2. After the first version of a file is written, the engineer writes unit tests and executes them. If the tests pass, the next task starts. If not, the engineer debugs the code, using the execution results, its own memory of past runs, and the PRD, design and code files. The paper states the loop ends when the tests pass or after a maximum of 3 retries §3.3. The paper's Section 3.3 text describes the Engineer doing this, but the official code assigns the test and debug steps to a QA role (see Paper vs code).

# Iterative programming with executable feedback  [§3.3] [fig. 2]
# Input:  task (one code file), PRD, system design, memory of past runs
# Output: code file after the loop ends

def develop_file(task, design, memory, max_retries=3):
    # Write the first version from requirements and design  [§3.3]
    code = write_code(task, design)
    for retry in range(max_retries):
        # Write unit tests and run them  [§3.3]
        tests = write_unit_tests(task, code)
        # The code runs with a 10 s limit, not stated in the paper
        # [code: metagpt/actions/run_code.py#L113]
        run = execute(tests)
        if run.passed:
            # Tests pass: stop and move on  [§3.3]
            return code
        # Debug with the errors, past messages and design  [fig. 2]
        code = debug(code, run.errors, memory, design)
        memory.append(run)
    return code

Hyperparameters

Results

Author-reported. Setup: HumanEval (164 tasks) and MBPP (427 tasks) with Python prompts slightly modified to fit the response format; SoftwareDev (70 tasks, of which seven were used in comparisons); baselines as named in the right-hand column §4.1. Hardware and the main-result backbone are not stated.

Pass rates are reported in a bar chart fig. 4; the text gives only the headline values.

Other reported points: 100% task completion rate across the authors' experiments §1 (the paper does not define "completion"); token usage of about 24.6k to 31.3k per project tab. 1; average cost of $1.12 per project over 70 tasks without feedback app. C.

Pass@k is computed with the unbiased estimator the paper cites §4.1:

\operatorname{Pass}@k = \mathbb{E}_{\text{Problems}}\left[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right] §4.1

Ablations §4.4 tab. 3: going from one engineer to all four added roles lowered human revisions (10 to 2.5) and raised executability (1.0 to 4.0), at a higher cost (about $0.92 to $1.39 per run). The main results report no variance; only the GPT-variant table gives standard deviations app. C.

Paper vs code

Code: official repository https://github.com/geekan/MetaGPT, MIT license (Copyright 2024 Chenglin Wu), cloned at commit 11cdf466d042aece04fc6cfd13b28e1a70341b1f. Chenglin Wu is a listed author and the corresponding author, and the paper's abstract links the repository. The code is the repository's current head, not the version the paper describes, so the differences below may partly reflect later development.

• Roles. Paper: five roles that act in a ReAct-style loop §3.1. Code: the Product Manager, Architect and Project Manager subclass RoleZero product_manager.py architect.py project_manager.py; the QA Engineer and Engineer subclass Role qa_engineer.py engineer.py.
• Who tests. Paper: the Engineer writes and runs unit tests and debugs §3.3. Code: the QA Engineer writes the test file, runs it and sends the error to debug qa_engineer.py qa_engineer.py. The Engineer's action set is WriteCode only engineer.py.
• Retry limit. Paper: max 3 retries §3.3. Code: the QA role stops after more than 5 test rounds, test_round_allowed = 5 qa_engineer.py qa_engineer.py.
• Subscriptions. The paper's publish-subscribe pool §3.2 matches the code's design: each role sets a watch set of action types role.py, and news is filtered by cause_by or addressee role.py. Watch sets: Architect watches the PRD architect.py; Project Manager watches the design project_manager.py; Engineer watches tasks and code events engineer.py.
• Execution limits. A 10-second timeout that kills the process, and truncated output in the debug prompt run_code.py run_code.py. The paper states neither.
• Benchmarks. The repository has HumanEval and MBPP harnesses only under metagpt/ext/aflow/benchmark/. We did not find the script that produced 85.9% and 87.7% humaneval.py mbpp.py; that is unverified.
• Self-improvement (Outlook). The paper's handover-feedback and long-term memory idea app. A has related modules, such as longterm_memory.py. We did not check whether they implement the paper's design.

Pitfalls

• Time figure in the text. Section 4.2 says MetaGPT takes 503 s, but in tab. 1 503 s is the value for MetaGPT without feedback; MetaGPT is 541 s.
• Two executability numbers for the same setup. tab. 1 gives 3.75 (MetaGPT) and 2.25 (ChatDev). The per-task table on the same seven tasks app. C averages to 3.9 and 2.1. The paper does not explain the gap.
• Productivity column. The paper defines productivity as tokens divided by lines §4.1. In the instruction-level table app. C, the rows do not match that definition: 28,384.2 / 178.2 ≈ 159.3 (reported 163.8) and 29,657.0 / 257.0 ≈ 115.4 (reported 118.0). tab. 1 is consistent.
• Baseline for GPT-4. The headline "improves over GPT-4" uses GPT-4's published 67% HumanEval score. The authors' own GPT-4 runs gave 72.4%, 81.2% and 80.0% depending on prompt and parser app. C. The size of the gain depends on which baseline is used.
• Undefined metrics. "100% task completion rate" §1 is not defined. The "Revisions" metric counts manual edits of up to three lines each §4.1, so it measures human effort, not model quality.
• Model for main results. The backbone for Table 1 and Figure 4 is not stated in Section 4. The appendix names GPT-4 32K only for Table 9 app. C.
• Retry semantics. The text says "maximum of 3 retries" §3.3 without saying whether the last debug edit is run again, or whether tests are rerun after each fix. Our rendering reads it as: test, debug, up to three times.
• Sampling is not described. Temperature, sample count and seeds are not stated. Results from a single run, without variance (apart from app. C), do not show how stable the 4.2 and 5.4 point gains are.
• Prompt changes. HumanEval and MBPP prompts were "slightly modified" to fit the output format §4.1. The modifications are not listed.
• Token accounting. tab. 1 reports one "token usage" number, while app. C splits prompt and completion tokens. Check which one a cost estimate uses.
• Dataset release. Whether the 70-task SoftwareDev set is released is not stated in the text.
• Table 9 caption. It says 10 tasks; the table lists 11 rows and the text says 11.
• Human-in-the-loop. The revision metric counts human interventions, such as fixing import errors, class names and reference paths §4.1. Read the 0.83 vs 2.25 gap with this in mind.

Lineage

• Builds on: the ReAct-style reasoning and acting loop §3.1; self-reflection and reflection-based agents such as Reflexion §2. No cards in this wiki yet.
• Compared against: AutoGPT, LangChain (with a Python REPL tool), AgentVerse and ChatDev on SoftwareDev §4.1; AlphaCode, InCoder, CodeGeeX, CodeGen, CodeX, CodeT, PaLM and GPT-4 on HumanEval and MBPP §4.1.
• Superseded by: not stated in this wiki.