An AI system to help scientists write expert-level empirical software

In plain words

Many scientific programs are written by hand, one guess at a time: a researcher tries a method, checks the score, and tries again. This paper builds a system that does that loop automatically. A language model writes a program, a computer runs it and records how well it scored, and the system keeps the better programs as starting points for new attempts.

The system can also be handed written ideas from papers or from other AI tools, and it tries to turn them into working code. It can mix two earlier programs into a new one. The authors used it on several scoring problems in biology, disease forecasting, general time-series forecasting, satellite images, brain activity and numerical maths. The results are reported by the authors, and the code they released is a smaller version of the system.

TL;DR

• ERA is a search loop around an LLM: the LLM rewrites a Python program, the program is run and scored, and the next parent is chosen by a PUCT rule.
• Research ideas, from paper summaries or deep-research outputs, are injected into the prompt. Two programs can also be recombined by prompting.
• Author-reported headline results: 40 of 87 scRNA-seq batch-integration variants beat published OpenProblems entries; 14 COVID-19 models beat the CDC ensemble; GIFT-Eval per-dataset MASE 0.671 on the 18 May 2025 snapshot.
• The released repository implements only the generic search (futs.py) and a California Housing demo. The six task pipelines, prompts and sandbox are not in it.

Problem

Writing domain-specific empirical software is slow, and the design space is rarely searched systematically §Introduction. The paper treats software creation as a search for a program whose output maximises a fixed quality score, a "scorable task" §Introduction; §Discussion. The problem assumes the score can be computed automatically for each candidate, for example from a held-out dataset or a benchmark leaderboard [§Results, Overview of Scorable Tasks].

Method

Overview. Each node of the search tree holds one Python program and its score. The LLM is prompted with a task description, the evaluation metric and the relevant data, and it returns a program that is executed and scored in a sandbox §Methods, Code mutation system. The search then chooses the next parent to expand from the whole tree, not only from the latest generation §Methods, Code mutation system; alg. 1.

1. Scoring. Each candidate is executed and receives a scalar score, which the tree uses together with logs and other feedback §Methods, Code mutation system.
2. Rank score. Scores are converted to ranks so that the exploration constant works across tasks with different scales. The rank of a node is normalised to [0, 1] within the current tree; a single-node tree gets 1 §Methods, Code mutation system.
3. Flat prior and visit counts. Every node has the same prior 1/|T|. Visit counts are kept per node, and each new child adds one visit to every ancestor §Methods, Code mutation system; alg. 1.
4. Selection. The node with the largest PUCT value is chosen, over all nodes in the tree. PUCT is the rank score plus an exploration bonus that is larger for rarely visited nodes eq. 1; alg. 1. The authors contrast this with AlphaZero, which recurses from the root, and with beam search or a (1+λ) evolution strategy, which discard candidates §Methods, Code mutation system.
5. Expansion. The LLM writes one child conditioned on the parent's code and score. The child is executed, scored, added to the tree, and its visit is backpropagated §Methods, Code mutation system; alg. 1.
6. Output. After the fixed node budget, the program with the highest score anywhere in the tree is returned alg. 1; code: implementation/futs.py#L154-L155.

# Flat PUCT tree search over programs  [alg. 1]
# Input:  root program and score, generate(), execute(), n_iter, c_puct
# Output: highest-scoring program in the tree

def search(root, generate, execute, n_iter, c_puct):
    # Start the tree with the root; its visit count starts at 1  [alg. 1]
    nodes = [root]
    visits = {root: 1}
    for _ in range(n_iter):
        # Rank every node by score, normalised to [0, 1]  [§Methods]
        ranks = rank_scores(nodes)
        # Total visits over the tree and the flat prior 1/|T|  [alg. 1]
        total_visits = sum(visits[u] for u in nodes)
        prior = 1 / len(nodes)
        # PUCT: rank plus exploration bonus per node  [eq. 1] [alg. 1]
        puct = {}
        for u in nodes:
            explore = prior * sqrt(total_visits) / (1 + visits[u])
            puct[u] = ranks[u] + c_puct * explore
        # Expand the largest-PUCT node, anywhere in tree  [§Methods]
        parent = argmax(nodes, key=lambda u: puct[u])
        # Ask the LLM for one child and score it  [§Methods]
        child = generate(parent.program, parent.score)
        child.score = execute(child.program)
        nodes.append(child)
        visits[child] = 1
        # Add a visit to each ancestor, not to the child  [alg. 1]
        for a in ancestors(child):
            visits[a] += 1
    # Return the best program found, by score  [alg. 1]
    return argmax(nodes, key=lambda u: u.score)

Idea injection. Research ideas are added to the prompt as a method description. For the batch-integration task, each of nine published methods was summarised by Gemini 2.5 Pro from its paper, and the summary was put in the prompt §Methods, scRNA-seq batch integration. For COVID-19, the public method descriptions of eight CDC-hub models were added as instructions §Methods, COVID-19 prediction. Ideas from Gemini Deep Research and from an AI co-scientist tool were filtered by hand and then run through ERA §Methods, Adding research ideas.

Recombination. For pairs of already-found methods, the LLM first explains the technical similarities and differences of the two parent programs. That explanation is then added to a tree-search prompt that asks for a hybrid §Methods, Recombination experiments. The authors report that most successful hybrids combined models from different families, such as a climatology baseline with an autoregressive model §Results, COVID-19.

Scoring used by each task. Batch integration uses control-scaled OpenProblems metrics, averaged over 78 measurements, with failed metrics set to 0 §Methods, Evaluating scRNA-seq batch integration. COVID-19 uses the weighted interval score (WIS) §Results, COVID-19. GIFT-Eval uses the mean absolute scaled error (MASE) §Results, GIFT-Eval. The integral task uses a negative log of the relative error [supp. Notes, Numerical Analysis].

Equation (the only numbered equation in the main text):

\mathrm{PUCT}_i = r_i + c_{\mathrm{puct}} \cdot E(i) \qquad \text{[eq. 1]}

Here r_i is the rank score and E(i) is an exploration term based on node i's visit count relative to the tree's total visits eq. 1. Algorithm 1 writes the exploration term as P_T(u)\sqrt{N_{total}}/(1+V(u)) alg. 1. The reference code uses the algorithm's form [code: implementation/futs.py#L79-L86].

Hyperparameters

Results

Author-reported throughout. The search is stochastic, and the paper reports the best of several runs in some comparisons §Methods, ERA versus best-of-N.

scRNA-seq batch integration (OpenProblems v2.0.0, six holdout datasets, 13 metrics, 1,747,937 cells). The best ERA variant, BBKNN (TS), is reported at about 14% overall improvement over the best published method, ComBat, and it equals or beats published BBKNN on every dataset and on 11 of 13 metrics §Results, scRNA-seq; fig. 2. Eight of nine ERA re-implementations beat the corresponding published method in overall score §Results; fig. 2b. Across 87 variants, 40 outperform all published leaderboard entries §Results; fig. 2d. Exact overall scores are only in the figure: "not stated" in the text.

COVID-19 hospitalisation forecasts (2024–25 season, 52 US jurisdictions, four horizons, WIS). The retrospective "Google Retrospective" model has average WIS 26; the CovidHub ensemble has 29 §Results, COVID-19. Fourteen strategies beat the ensemble over the three-week period, 10 of them recombinations fig. 3e. Of the eight replicated models, six beat their originals §Results, COVID-19.

ERA versus best-of-N tab. 1. Batch integration and epidemiology scores, on validation data, for five models. Best of 1000 attempts compared with ERA using 128 nodes.

ERA wins on every row except GPT-5 batch integration. The authors attribute that exception to GPT-5's strong one-shot result on that task [§Methods, ERA versus best-of-N].

GIFT-Eval (28 datasets, MASE). The per-dataset solution outperformed the whole 18 May 2025 leaderboard snapshot. Its models were mostly gradient boosting and ensemble or decomposition methods [§Results, GIFT-Eval; supp. fig. S13]. The unified library, restricted to numpy, pandas and holidays, reached 0.734 [supp. tab. S12]. Its breakthroughs were date and trend features, and a holiday feature in one configuration [supp. tab. S13].

Other domains §Results, Other Problems. Geospatial segmentation: the three top solutions used UNet++, SegFormer and U-Net with large pre-trained encoders and heavy test-time augmentation [supp. Notes, Geospatial]. Neural activity: two tree-search models, one for multi-step and one for 1-step prediction, trained in under two hours on one T4 GPU, versus 36 hours on 16 A100 GPUs for the video U-Net [supp. Notes, Neuroscience]. Integrals: the best program always calls quad() first and uses an Euler-transform fallback on failure [supp. Notes, Numerical Analysis].

Paper vs code

Repository: https://github.com/google-research/era, commit 012c2d3957a3a6a249af2bbeeae53bbbd2182558, Apache-2.0. The paper links it in its Code Availability section §Code Availability. It is the authors' own organisation, and the README links the paper.

• Scope. The repository has a generic search (implementation/futs.py), a LLM wrapper (llm.py), an abstract sandbox (sandbox.py) and one demo (playground_s3e1.py, California Housing). It does not contain the six task pipelines, the prompts from the supplement, or the research-idea tooling. The generated programs are on the project page, not in the repository §Code Availability; code: implementation/README.md.
• Visit counts. The paper's Algorithm 1 starts the root at visit count 1, and increments ancestors but not the new child alg. 1. The code starts the root at 0 futs.py and increments the new node and all its ancestors futs.py. The total visit count is therefore one lower per iteration in the code. Effect: a small shift in the exploration term early in the search.
• Single-node rank. Paper: 1 §Methods, Code mutation system. Code: 0.5 futs.py. This has no effect on selection, because one node is the only candidate.
• Rank direction. Code sorts ascending by score and gives the best node rank 1 futs.py, matching the paper's intent. Ties get arbitrary ranks; the paper does not say how ties are handled.
• Exploration term notation. Paper eq. 1 uses E(i) and Algorithm 1 uses the explicit form. Code uses the algorithm form [code: implementation/futs.py#L84-L86].
• Model string. Paper: Gemini 2.5 Flash §Methods, Code mutation system. Code default: gemini-2.5-flash-image [code: implementation/llm.py#L12]. The name looks like an image model; set the model explicitly. No temperature is set llm.py.
• Failure score. The demo gives failed or malformed programs -inf, so they rank last playground_s3e1.py. The batch-integration pipeline instead sets failed metrics to 0 [§Methods, Evaluating scRNA-seq batch integration]. These are different scoring schemes.
• Sandbox. sandbox.py defines only run(), which raises NotImplementedError sandbox.py. The demo's Sandbox(timeout_seconds=60) call playground_s3e1.py therefore needs a user-supplied sandbox. Run generated code only in an isolated environment.
• API key. The README says GOOGLE_API_KEY, but the demo reads GEMINI_API_KEY playground_s3e1.py.

Pitfalls

• Unstated budgets. Node counts for the Kaggle benchmark, geospatial and ZAPBench are not stated, and neither are sampling temperature and seeds. Reproducing the headline numbers needs the paper's budget, which is only partly given.
• Recombination count. The COVID-19 hybrid count is 26 in the Results §Results, COVID-19 and 28 in the Methods [§Methods, Recombination experiments]. The paper does not explain the difference.
• Compute is not reconciled. Table S1 lists per-node sandbox durations of 8.0 min for batch integration and 53.3 min for GIFT-Eval [supp. tab. S1]. At 500 nodes that is about 67 hours of sandbox time if run in sequence, but the Methods say one experiment took about seven hours [§Methods, scRNA-seq batch integration]. The parallelism is not stated.
• Token cost. The integral task's request token count (224,415 per node) is inflated by image tokens [supp. tab. S1, footnote a]. Budget by tokens, not by node count, when estimating cost.
• Scale of noise. The authors report typical run-to-run variation of about 0.01 on batch integration and O(1) WIS on epidemiology, and state that GPT-5 varies more than other models §Methods, ERA versus best-of-N. Single runs can mislead.
• "40 novel methods". The abstract's 40 is the count of 87 variants that beat all published leaderboard entries. The 87 include recombinations and Deep-Research ideas, not only hand-written base methods §Results, scRNA-seq.
• Retrospective leakage. COVID-19 uses data as available on 1 May 2025 for the whole season, so later data revisions are ignored [§Methods, COVID-19 prediction]. GIFT-Eval is a frozen 18 May 2025 snapshot; the authors did not submit to the live leaderboard §Methods, GIFT-Eval benchmark.
• Selection on validation. For the scRNA-seq task, the best replicate is chosen on validation, which is a separate dataset from the OpenProblems holdout §Methods, scRNA-seq batch integration. The COVID-19 comparison uses only models with state-level forecasts covering 75% of the season §Methods, COVID-19 prediction. The comparison set is therefore a subset of CovidHub.
• Unified library. The search objective is the geometric mean of normalised MASE over all datasets, but the text does not name the split it uses §Methods, GIFT-Eval benchmark. The per-dataset choice among the eight presets is made on validation data §Methods, GIFT-Eval benchmark. The refinement history (MASE 0.82, then 0.77, then 0.734) is given in the text §Methods, GIFT-Eval benchmark.
• Score saturation. The authors say scores saturate after 300 to 1000 nodes §Methods, Code mutation system. The per-task node budgets above are the paper's choices, not a general rule.

Lineage

• Builds on: PUCT from AlphaZero §Methods, Code mutation system, with the tree-wide selection of flat UCB. No card.
• Compared against: AIDE and AlphaEvolve for code search §Introduction; the OpenProblems batch-integration leaderboard §Results; CovidHub ensemble; GIFT-Eval leaderboard snapshot.
• Superseded by: not known from this wiki.