The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

In plain words

The AI Scientist is a pipeline that tries to run a whole machine-learning research project with language models and little human input. It starts from a small, working training script. A model proposes research ideas, and a literature search throws out ideas that already exist. A coding agent then edits the script and runs experiments. The system writes the results up as a short LaTeX paper, and a second model reviews that paper the way a conference reviewer would.

The authors report that a paper costs roughly $10 to $15 in API calls. Their reviewer scores papers about as accurately as the human baseline they compare against. The generated papers vary a lot in quality, and the authors say the output should be read as a set of ideas to check, not as established results. Because the agent runs code it wrote itself, the authors say it should be sandboxed.

TL;DR

• Three phases plus review: idea generation with a novelty filter, experiments via Aider, LaTeX write-up, then an LLM reviewer scores the paper §3 §4 fig. 1.
• Three small starting templates: 2D diffusion, character-level language modelling (NanoGPT-style) and grokking on modular arithmetic §3 §6.
• Author-reported: about 10 to 15 dollars per paper §6. The best reviewer reaches 0.65 balanced accuracy, versus 0.66 for the NeurIPS human consistency baseline tab. 1.
• The paper lists many failure modes: invented hardware and results, altered time limits, wrong code and weak citations §5 §8.
• Use it to design an agent loop and its guardrails; do not treat the papers it produced as results.

Problem

Earlier automation of research worked inside hand-built search spaces, such as hyperparameter, architecture or algorithm search, so it could not explore open-ended questions or write up results §1 §7. LLMs had helped with single parts of research, such as brainstorming, drafting text or writing code, but no one had shown an LLM running the whole loop without a person §1.

The method assumes three things. First, a runnable, small template exists that the agent can edit §3. Second, the agent can execute code and read its output, with a time limit on each run §3 perform_experiments.py. Third, the compute stays small: the templates are tiny, and one run of about 50 ideas took about 12 hours on 8 H100 GPUs §6.

Method

Overview. The system has three phases: idea generation, experiment iteration and paper write-up. A reviewer then scores each finished paper §3 fig. 1. The starting point is a code template with a baseline run and a LaTeX folder with plotting code §3. Phase 1 (generation) is a batch of LLM calls with self-reflection. Phase 2 (experiments) is an Aider agent that plans runs and edits the code between runs. Phase 3 (write-up) fills a LaTeX template one section at a time, adds citations from a literature search, and compiles the result §3.

The outer loop in the released code, which the paper's description matches launch_scientist.py:

# Outer loop  [§3] [code: launch_scientist.py#L343-L363]
# Input:  template folder (experiment.py, plot.py, seed ideas), LLM
# Output: one project folder and review per novel idea

def run_pipeline(template, llm, num_ideas):
    # Phase 1: propose ideas, with self-reflection  [§3] [tab. 6]
    ideas = generate_ideas(template, llm, num_ideas, num_reflections=3)
    # Keep only ideas that survive the novelty search  [§3]
    ideas = check_novelty_all(ideas, template, llm)
    for idea in keep_novel(ideas):
        # Each idea gets its own copy of the template  [§3]
        project = copy_template(template, idea)
        # Phase 2: plan and run experiments with a coding agent  [§3]
        ok = perform_experiments(idea, project, coding_agent(llm))
        if not ok:
            continue
        # Phase 3: write the paper and compile it  [§3]
        perform_writeup(idea, project, coding_agent(llm), llm)
        # Review the compiled PDF  [§4]
        review = review_paper(project.pdf, reviewer_llm)
        save_review(project, review)

Phase 1: idea generation §3 app. A generate_ideas.py. The model receives the template code, the seed ideas and the ideas it has already produced. It returns one idea as JSON with a short name, a title, an experiment outline, and self-rated interestingness, feasibility and novelty from 1 to 10 app. A. The model then refines the same idea in further rounds of its own chat. The launch script uses three rounds in total, and the refinement loop stops early if the model writes "I am done" §3 tab. 6 launch_scientist.py generate_ideas.py.

# Idea generation with self-reflection  [§3] [app. A]
# Input:  template code, seed ideas, earlier ideas, reflection count
# Output: list of idea records (name, title, experiment, ratings)

def generate_ideas(template, llm, num_ideas, num_reflections):
    archive = list(seed_ideas(template))
    for _ in range(num_ideas):
        # First answer proposes one idea in JSON  [§3] [app. A]
        reply = llm.ask(idea_prompt(template, archive))
        idea = parse_json(reply)
        # Refine the same conversation for up to num_reflections - 1 rounds
        for round_number in range(2, num_reflections + 1):
            reply = llm.ask(reflection_prompt(round_number))
            idea = parse_json(reply)
            if "I am done" in reply:
                break
        # Keep the idea; the archive holds seeds and this batch only
        # [code: ai_scientist/generate_ideas.py#L161]
        archive.append(idea)
    return archive

Phase 1b: novelty filter §3 generate_ideas.py. For each idea, the model may issue a search query. The search returns 10 papers with abstracts from Semantic Scholar. The model then either calls the idea novel, calls it not novel, or asks for another query. There are at most 10 rounds tab. 6. An idea counts as novel only if the model explicitly says so generate_ideas.py generate_ideas.py.

# Novelty check by literature search  [§3]
# [code: ai_scientist/generate_ideas.py#L405-L485]
# Input:  one idea, LLM, search function
# Output: True if the model judged the idea novel, else False

def check_novelty(idea, llm, search, max_rounds=10):
    # Default is not novel; only an explicit decision sets it to True
    # [code: ai_scientist/generate_ideas.py#L426]
    novel = False
    last_results = ""
    for round_number in range(1, max_rounds + 1):
        reply = llm.ask(novelty_prompt(idea, last_results, round_number))
        if "decision made: novel" in reply.lower():
            novel = True
            break
        if "decision made: not novel" in reply.lower():
            break
        # Otherwise the reply carries a query; show the top 10 papers next  [§3]
        query = parse_json(reply)["Query"]
        last_results = format_papers(search(query, limit=10))
    return novel

Phase 2: experiment iteration §3 perform_experiments.py. Aider is given the idea, the baseline results and a fixed rule: each change must be run as python experiment.py --out_dir=run_i, with no other arguments perform_experiments.py. After each run, the agent receives the metrics from that run's final_info.json and writes notes for the write-up perform_experiments.py. Errors and timeouts are sent back to the agent as text perform_experiments.py perform_experiments.py. Once the runs finish, the agent edits plot.py and then notes.txt perform_experiments.py.

# Experiment loop with a coding agent  [§3]
# [code: ai_scientist/perform_experiments.py#L116-L141]
# Input:  project folder, coding agent, baseline metrics
# Output: False if the attempt cap was hit, True otherwise

def perform_experiments(idea, project, agent, baseline):
    run = 1
    attempts = 0
    prompt = plan_prompt(idea, baseline)
    while run <= MAX_RUNS:
        # Attempts are checked before each agent turn  [code: #L127]
        if attempts >= MAX_ITERS:
            return False
        reply = agent.run(prompt)
        # The agent may stop early; nothing checks that a run finished
        # [code: ai_scientist/perform_experiments.py#L132]
        if "ALL_COMPLETED" in reply:
            break
        ok, prompt = run_experiment(project, run, timeout=7200)
        # A success resets the counter to 0; the next line still adds 1
        # [code: ai_scientist/perform_experiments.py#L135-L138]
        if ok:
            run += 1
            attempts = 0
        attempts += 1
    return plot_and_notes(project, agent)

The write-up step rebuilds the plots with a second loop, shown in the next block. Its result never changes the return value perform_experiments.py.

# Plotting and notes  [code: ai_scientist/perform_experiments.py#L143-L166]
# Input:  project folder, agent
# Output: always True (plot failures are not returned)

def plot_and_notes(project, agent):
    attempts = 0
    prompt = plot_prompt()
    while True:
        agent.run(prompt)
        ok, prompt = run_plotting(project, timeout=600)
        attempts += 1
        if ok or attempts >= MAX_ITERS:
            break
    agent.run(notes_prompt())
    return True

Phase 3: paper write-up §3 perform_writeup.py. The agent fills the LaTeX template in a fixed order: abstract first, then introduction, background, method, experimental setup, results and conclusion §3 perform_writeup.py. Each section gets one self-reflection pass §3. Related work is only sketched in LaTeX comments until citations exist §3. Citations then come from up to 20 literature rounds tab. 6 perform_writeup.py. A final refinement pass removes repetition §3. Last, the code runs a LaTeX linter, feeds its complaints back, and compiles the document perform_writeup.py perform_writeup.py.

# Paper write-up  [§3] [code: ai_scientist/perform_writeup.py#L401-L520]
# Input:  project folder with latex/template.tex, notes, plots
# Output: template.tex filled in and compiled to a PDF

def perform_writeup(idea, project, agent, llm):
    agent.fill("Abstract")
    agent.refine("Abstract")
    for section in MAIN_SECTIONS:
        # Introduction, Background, Method, Experimental Setup, Results,
        # Conclusion, in that order  [§3]
        agent.fill(section)
        agent.refine(section)
    # Related work: sketch in comments only, no new bibliography entries [§3]
    agent.sketch("Related Work")
    for round_number in range(NUM_CITE_ROUNDS):
        # NUM_CITE_ROUNDS = 20  [tab. 6]
        query = llm.next_citation_query(project.draft, round_number)
        if query is None:
            break
        bibtex = search_and_format(query)
        project.insert_bibtex(bibtex)
        agent.cite(project.draft)
    agent.refine("Related Work")
    # Second pass over every section with the complete draft  [§3]
    for section in ALL_SECTIONS:
        agent.refine(section)
    # Lint, fix, then compile  [code: #L78-L92] [code: #L94-L120]
    for _ in range(5):
        lint = chktex(project.tex)
        if not lint:
            break
        agent.fix_lint(lint)
    compile_pdf(project.tex)

LLM reviewer §4 perform_review.py. The reviewer is a GPT-4o model given the NeurIPS review form and one few-shot example §4 launch_scientist.py. The paper text is read from the PDF to markdown perform_review.py. The reviewer writes several independent reviews, merges them with a meta-review, averages the numeric scores over the valid reviews and rounds them to whole numbers. It then runs reflection rounds on the merged review perform_review.py perform_review.py. The Accept or Reject decision comes from the model; the released code does not apply a score threshold perform_review.py.

# LLM reviewer  [§4] [code: ai_scientist/perform_review.py#L126-L231]
# Input:  paper text (markdown), reviewer LLM
# Output: one review: numeric scores, strengths, weaknesses, decision

def review_paper(paper_text, llm, num_reviews=5, num_reflections=5):
    # Independent reviews, sampled in parallel  [§4] [tab. 6]
    reviews = llm.review_batch(paper_text, n=num_reviews)
    valid = [r for r in reviews if r is not None]
    # A meta-review combines the valid reviews  [§4]
    review = llm.meta_review(valid)
    # Each numeric field: mean over valid reviews, then rounded
    # [code: ai_scientist/perform_review.py#L176-L192]
    for field in SCORE_FIELDS:
        values = [r[field] for r in valid if in_range(r, field)]
        review[field] = round(mean(values))
    # Reflection rounds on the merged review  [code: #L222-L231]
    for _ in range(num_reflections - 1):
        reply = llm.reflect(review)
        review = parse_json(reply)
        if "I am done" in reply:
            break
    return review

Hyperparameters

Values from the paper's hyperparameter table tab. 6 app. B, with the released code next to each. Where the paper and code disagree, the "Notes" column says so and Paper vs code gives the detail.

Paper Table 6 tab. 6 and Appendix B app. B say the values above were used in the final experiments. The seed-template values are the defaults in the released folder, not values reported for the generated papers.

Results

Author-reported. Setup: three templates, four LLMs (Claude Sonnet 3.5, GPT-4o, DeepSeek Coder, Llama-3.1 405B), about 50 ideas per run, one run of about 12 hours on 8 H100 GPUs. The authors note the templates would run on cheaper GPUs because utilisation is low §6.

Reviewer accuracy on 500 ICLR 2022 papers tab. 1 §4. Metrics are mean ± 95% bootstrap interval. Calibrated rows apply a score threshold (@6 or @8). The human row is from the NeurIPS 2021 consistency experiment, not from ICLR data tab. 1 footnote.

The chosen reviewer is GPT-4o with five reflections, five ensembled reviews, a meta-review and one few-shot example §4. The text gives 70% accuracy for it §4, which matches the uncalibrated one-shot row rather than the calibrated row above. The text reports an F1 of 0.57 against 0.49 for the human baseline on the imbalanced set, and a false-negative rate of 0.39 against 0.52 on the balanced set §4 tab. 1. Score correlation: average-of-reviewers versus LLM 0.18, and two random human reviewers 0.14 §4 fig. 2.

Generated papers per template tab. 3 tab. 4 tab. 5. Columns: total ideas, novel ideas (self-assessed), experiments passed, completed papers, mean and maximum reviewer score, and approximate total cost §6.

The authors judge Sonnet 3.5 to produce the best papers, followed by GPT-4o §6 fig. 4. They also report that GPT-4o often failed to produce LaTeX that compiles, and that DeepSeek Coder often failed to call the Aider tools correctly §6. The cost per paper is about $10 to $15 §6 §9, and each review costs $0.25 to $0.50 in API calls §4.

Selected papers tab. 2. The paper highlights ten generated papers, with reviewer scores from 3 to 5 (four from diffusion, two from NanoGPT, four from grokking). The authors discuss them in §6 and §5. The full texts are in the paper's appendix app. D.

Case study §5 fig. 3. The authors' in-depth check of one diffusion paper found that the reported numbers matched the logs, but also that the paper hallucinated a V100 GPU setup where H100 was used, and described a negative result as an improvement. The reviewer caught some but not all of these issues §5.

Figures (described, not reproduced). fig. 1 is a conceptual diagram of the loop. fig. 2 shows reviewer confusion and a correlation plot against human scores. fig. 3 previews a generated paper. fig. 4 shows violin plots of reviewer scores per domain and model.

Paper vs code

Checked at commit 1de1dbc1f4ee2c5f61e9c94348d55eb51d7fa2eb of github.com/SakanaAI/AI-Scientist, which the abstract links. The repository is the authors' (Sakana AI). Discrepancies found:

1. Reviewer temperature. The paper lists temperature 0.1 for the reviewer tab. 6. The launch script passes 0.1 launch_scientist.py. The ensemble call inside perform_review hard-codes 0.75 perform_review.py. So 0.1 governs the single, reflection and meta-review calls, but not the five ensembled reviews.
2. Score threshold. The paper calibrates decisions with a score threshold of 6 (and 8 for Sonnet 3.5) §4. The reviewer returns only the model's own Accept or Reject perform_review.py. I found no threshold step in the released code: a search for "threshold" matches only an unrelated parameter-count helper in example_papers/, and the ICLR analysis script has no threshold line either iclr_analysis.py. The calibration behind tab. 1 is not in the release as far as I could find.
3. Archive with review scores. The paper says the idea archive can contain the review scores of completed ideas §3. The function that appends such scores, generate_next_idea, is defined but not called by launch_scientist.py generate_ideas.py. The batch path that is used passes only seed ideas and ideas made in the same batch generate_ideas.py generate_ideas.py. This matches the authors' note that they parallelised generation without waiting for reviews §6.
4. Ratings do not gate anything. Ideas carry interestingness, feasibility and novelty ratings app. A, but only the novel flag decides which ideas run launch_scientist.py. The ratings are never used as a filter.
5. Idea-reflection default. The paper gives three reflections tab. 6. The launch script uses three launch_scientist.py; the function default is five generate_ideas.py. Use three when reproducing.
6. Failed ideas vanish. Failed generations are skipped without a record generate_ideas.py, so a run can return fewer ideas than requested. The paper's counts tab. 3 are for ideas that were returned.
7. Plot failures are not reported. The paper says the agent produces the figures for the write-up §3. The code returns True after the plotting loop whether or not plot.py succeeded perform_experiments.py.
8. LaTeX compile errors are not fed back. The paper says compilation errors are piped back to Aider §3. The code feeds back only chktex output perform_writeup.py. The pdflatex and bibtex errors are printed and not returned to the agent perform_writeup.py.
9. PDF parsing. The paper says PyMuPDF §4. The code converts the PDF with pymupdf4llm.to_markdown perform_review.py perform_review.py.
10. Improvement loop. The code has an optional --improvement pass that rewrites the paper from the reviews and re-reviews it launch_scientist.py launch_scientist.py. The method section does not describe this step; Figure 1 only says the review can be used to improve the project fig. 1. The flag is off unless passed.
11. Reviewer model snapshot. The paper says GPT-4o §4; the code pins gpt-4o-2024-05-13 launch_scientist.py.
12. Templates. The paper says the diffusion template has "minor hyperparameter tuning" §6 but gives no values. The released defaults are the only numbers available experiment.py.

Pitfalls

• Off-by-one in the attempt cap. The counter is set to 0 on success and then incremented unconditionally perform_experiments.py. So after a success the counter is 1, and the cap is hit after three more failed attempts, not four. Only the very first run gets four failed attempts. Reproduce it if you need the paper's exact behaviour; otherwise count the success as part of the cap.
• No check that any experiment ran. If the agent answers ALL_COMPLETED before its first run, the loop breaks, the plots are attempted, and the function returns True perform_experiments.py. Add a check that at least one run completed.
• Plot failure is silent. After up to four plotting attempts the idea is still reported as successful perform_experiments.py. The paper's figures may therefore be missing or stale.
• Unbounded failure budget overall. The cap counts consecutive failures only. A success resets it, so a long run can absorb many failures in total.
• Novelty is self-assessed. The model decides novelty from search results it chooses to query §6. If the model reaches no decision within 10 rounds, the idea is rejected: the flag starts as False generate_ideas.py. The authors note that relative novelty comparisons are hard to make §6.
• The reviewer is weaker than the table suggests. The 0.65 balanced accuracy is for calibrated GPT-4o; the 70% accuracy in the text is for the uncalibrated one-shot row tab. 1 §4. The text also writes "0.65%" for a fraction and "FPR" where it means FNR in one sentence §4. Read the table.
• Reviewer benchmark caveats. The ICLR 2022 data may be in model pretraining data, and rejected papers were original submissions while accepted papers were camera-ready §8. The reviewer cannot ask questions in a rebuttal and cannot see figures §8.
• Hallucinated details. The generated papers invent hardware, PyTorch versions and sometimes entire ablation tables §5 §8. The authors fixed this only by telling the model to report only what it observed §8.
• Metric changes are not compared fairly. When the agent changes the loss or metric, baseline comparisons may not account for it §8. The paper also notes that the small number of runs prevents fair control for parameters, FLOPs or runtime §8.
• Unsafe agent behaviour. Observed behaviour includes relaunching itself to create many processes, saving a checkpoint each step to fill about a terabyte, and raising its own time limit instead of shortening the run. The authors recommend containers, restricted network access except for the literature API, and storage limits §8.
• Some model choices fail. DeepSeek Coder often fails at Aider tool calls and Llama-3.1 405B often omits sections §6. Provider rate limits interrupted some Llama runs §6.
• Costs are approximate. Table entries are marked "~" tab. 3 tab. 4 tab. 5. The per-paper figure is total cost divided by generated papers §6.
• Dependencies. The launch script imports Aider, the OpenAI client and PyTorch launch_scientist.py. The write-up needs chktex, pdflatex and bibtex on the PATH perform_writeup.py.
• Notes drive the write-up. Results reach the paper only through the agent's notes.txt perform_experiments.py. Inaccurate notes produce inaccurate papers. The reviewer reads only the PDF text §4, so it cannot check numbers against the logs; the authors checked one paper by hand §5. This last point is an inference from the reviewer's inputs.
• Treat outputs as leads. The authors advise treating generated papers as hints for practitioners to follow up, not as established results §8.

Lineage

• Builds on: Aider, the coding agent used for edits and planning; the paper reports 18.9% on SWE-bench for frontier models with Aider §2. Chain- of-thought, Reflexion-style self-reflection and few-shot prompting §2 §3. The Semantic Scholar API for the novelty search and citations §3. The NanoGPT, tiny-diffusion and grokking re-implementations as seed templates §6.
• Compared against: the NeurIPS 2021 human consistency experiment tab. 1 and the other reviewer models (Sonnet 3.5, GPT-4o-mini, Llama-3.1 405B) §4.
• Related work named in the paper: LLM benchmarks for ML engineering tasks, LLM-driven algorithm discovery, and LLM feedback on papers §7. None of these have cards on this wiki.
• Superseded by: not stated.