MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses

In plain words

Scientists often come up with a new idea by joining two things that were not linked before: the problem they are working on, and a useful piece of knowledge from somewhere else. MOOSE-Chem asks whether a language model can do that for chemistry. You give it a research question and maybe a short survey of existing methods, plus a pool of paper titles and abstracts.

The system first searches the pool for papers that could supply that missing piece. It then writes several different ways of joining the question with each piece, has the model critique and rewrite each attempt a few times, and joins the best attempts again. Finally it scores every candidate and sorts them, so a scientist can read the best ones first.

The authors test it on a set of recent chemistry papers. They check whether the model's ideas match the core ideas that the real papers reported, and report that many do. These are the authors' own numbers, judged by a language model and partly by chemists.

TL;DR

• Chemistry hypotheses are cast as a background plus one to three "inspirations" from other papers, and the generation problem is split into retrieving inspirations, composing hypotheses and ranking them §4.1 eq. 1 eq. 2.
• The framework is a GPT-4o multi-agent pipeline with an evolutionary step: several mutation lines per inspiration, a recombination step, and beam search over inspirations §4.2.1 fig. 1.
• Author-reported: top Matched Score 4.020 vs 2.882 for MOOSE; average 2.564 vs 2.464 tab. 10. The Matched Score is judged by GPT-4o, not by experts.
• Use the card with care: the released run differs from the text on beam size, corpus size and screening rounds (see Paper vs code).

Problem

Modelling P(hypothesis | background) directly is treated as intractable. The paper argues that most chemistry hypotheses can be built from a background and a few inspirations, which makes the search decomposable into smaller steps §4.1. Earlier LLM hypothesis work covered social science, NLP and biomedicine §2 §6.1. Chemistry work either relied on rediscovering existing catalysts, which may be contaminated §1 §2. The paper's own evaluation tries to control contamination: the benchmark papers appeared online after January 2024, and the model's training data ends in October 2023 §3 §5.

Method

The method has three components, matching the decomposition eq. 2 and the ranking function eq. 3:

1. Inspiration retrieval, P(i_j | b, h_{j-1}, I): an LLM reads a window of candidate papers (title and abstract) together with the background and, after the first step, the current hypothesis. It names three titles and gives reasons. Only the surviving titles move on to the next screening round, so the pool shrinks: 20% of the corpus after one round, 4% after two §4.2.2 §5.1 tab. 3. The released code keeps only the titles, without reasons inspiration_screening.py. The paper avoids citation or semantic neighbours on purpose, since an inspiration already linked to the background would make the hypothesis less novel §4.2.2.
2. Hypothesis composition, P(h_j | b, h_{j-1}, i_j): an evolutionary unit (EU) per inspiration. It writes several distinct drafts ("mutations") that each join the background to the inspiration. Each draft is refined by critique on validness, novelty, clarity and significance, then the final drafts are recombined into one more draft §4.2.3. The paper does not state how many mutations or refinement passes to use; the released run uses three of each main.sh. Beam search keeps the best nodes and, in later steps, pairs them with another inspiration's best draft §4.2.1 §4.2.3.
3. Ranking, R(h): an LLM scores each final draft on validness, novelty, significance and potential. The rank is the mean of the four scores §4.2.4 eq. 3 evaluate.py.

Default settings in the text: three inspiration rounds per background, beam size 15, screening window 15 keeping 3 §4.2.1 §4.2.2. The released configuration changes several of these; see Paper vs code.

The pseudocode below follows the released code's control flow, which the paper describes in prose. It is in our own words; the names are ours.

# MOOSE-Chem: from background to ranked hypotheses  [§4.2.1] [eq. 2] [eq. 3]
# Input:  background (question, optional survey); corpus of (title, abstract)
# Output: hypothesis nodes, sorted by average self-score (best first)

def moose_chem(background, corpus):
    # Coarse search: screen the corpus in several rounds  [§4.2.2]
    # Text: 3 rounds; released run: 4 rounds, step 1 uses round index 2
    pool = corpus
    for _ in range(3):
        pool = screen(background, pool, window=15, keep=3)
    # Step 1: develop each surviving inspiration  [§4.2.3]
    nodes = []
    for inspiration in pool:
        nodes += develop(background, inspiration)
    # Steps 2..k: beam search over recombination  [§4.2.1]
    # Text: beam 15; released run: beam 5 with tie handling (see Pitfalls)
    previous = nodes
    for step in range(2, max_steps + 1):
        beam = top_nodes(previous, beam_size)
        previous = []
        for node in beam:
            previous += recombine_across(background, node, nodes)
        nodes += previous
    # Rank every node by the mean of its four self-scores  [§4.2.4] [eq. 3]
    return sorted(nodes, key=average_score, reverse=True)


def develop(background, inspiration):
    # Three mutation lines, each told to differ from earlier final drafts
    # [§4.2.3]; the avoid list is used only for each line's first draft
    # [code: hypothesis_generation.py#L476-L512]
    lines = []
    for index in range(3):
        avoid = [line.draft for line in lines] if index > 0 else None
        lines.append(refine_line(background, inspiration, avoid))
    # Recombine the final drafts of all lines into one more node  [§4.2.3]
    combined = combine_lines(background, inspiration, lines)
    return lines + [combined]


def refine_line(background, inspiration, avoid):
    # First draft; `avoid` is used only here  [§4.2.3]
    # [code: hypothesis_generation.py#L548-L549]
    draft = propose(background, inspiration, avoid)
    # Critique on validness, novelty, clarity and significance  [§4.2.3]
    feedback = critique(draft)
    # Refinement passes: three drafts in total in the released run
    # [code: main.sh#L77]
    for _ in range(2):
        draft = revise(background, inspiration, draft, feedback)
        feedback = critique(draft)
    # Score only the final draft; these scores drive ranking  [§4.2.4]
    # [code: hypothesis_generation.py#L566-L574]
    scores = score(draft)  # [validness, novelty, significance, potential]
    return draft, scores


def recombine_across(background, node, nodes):
    # Best node of every other inspiration, screened in windows of 12,
    # keeping 3  [code: main.sh#L77]
    others = best_node_per_other_inspiration(nodes, node)
    partners = screen(background, others, window=12, keep=3)
    children = []
    for partner in partners:
        # Refine a new draft that joins both inspirations  [§4.2.3]
        children.append(combine_pair(background, node, partner))
    return children


def screen(background, candidates, window, keep):
    # Split candidates into consecutive windows; the LLM names `keep` of each
    # (temperature 0)  [§4.2.2] [code: inspiration_screening.py#L131-L163]
    kept = []
    for start in range(0, len(candidates), window):
        chunk = candidates[start:start + window]
        if len(chunk) <= keep:
            kept += chunk
        else:
            kept += ask_llm_for_titles(background, chunk, keep)
    return kept

Key equations, as written in the paper:

h = f(b, i_1, \ldots, i_k), \quad k \in [1, 3] \text{ in chemistry} eq. 1

P(h \mid b) \approx \prod_{j=1}^{k} P(i_j \mid b, h_{j-1}, I) \cdot P(h_j \mid b, h_{j-1}, i_j), \quad h_0 = \emptyset eq. 2

P(H_{\text{ranked}}) = P(H, R), \quad R(h_i) \ge R(h_{i+1}) eq. 3

Hyperparameters

Values in the paper text are cited to the paper; values only in the released run are cited to the code. Where the paper is silent, the table says so.

Results

Author-reported. Setup: 51-paper TOMATO-Chem benchmark, GPT-4o as generator and judge, corpus |I| = 300, only the top 4% of the corpus used for drafting, strict background and survey §6 tab. 10.

The Matched Score is judged by GPT-4o unless marked "experts". Paper counts above are read from the table cells.

Paper vs code

Official code: https://github.com/ZonglinY/MOOSE-Chem (first author's repository, linked from the paper's source as "All code and data"). Cloned at f8cda98a3bed30b0f0d85fba625e343d688461ab, MIT licence.

• Beam size: paper says 15 §4.2.1; the released run passes 5 main.sh and the argument default is 10 hypothesis_generation.py.
• Corpus size: the wild and copilot-style setting uses |I| = 300 §6; the released run passes 150 main.sh.
• Screening rounds: paper says three per background §4.2.1; the released run performs four main.sh and feeds round index 2 into the composition step main.sh.
• Mutations and refinements: not stated in the paper. Released run: three mutation lines and three refinement iterations main.sh; the argument defaults are three and two hypothesis_generation.py.
• Recombination screening window: not stated in the paper; released run uses 12 main.sh.
• Self-explore: the paper does not describe it; the code has a self-explore module, switched off in the released run main.sh.
• Temperature: not stated in the paper. Screening and inspiration extraction use 0.0 inspiration_screening.py hypothesis_generation.py; drafts and self-scores use the default 1.0 utils.py; the Matched Score judge uses 0.0 evaluate.py.
• Ranking: paper says the rank is the average over the four aspects §4.2.4; code averages the four scores of the final draft of each node, excluding self-explore nodes evaluate.py. These agree.
• Screening output: the paper says the LLM gives reasons for each title §4.2.2; the released code extracts titles only inspiration_screening.py.

Other items were compared only at the level of the sections above; the released code's prompt wording was not compared with the paper's text.

Pitfalls

• Beam ties. When recom_num_beam_size is set and no recommended inspirations are listed, the code keeps every node whose score equals the cut-off, so the beam can be larger than its nominal size hypothesis_generation.py.
• Which screening round feeds step 1. The text says three rounds, the released run four, and the composition step reads round index 2 main.sh main.sh. Decide on purpose; the Hit Ratio table is per round tab. 3.
• Empty drafts survive. If extraction of a draft fails after retries, the code sets the hypothesis to an empty string and continues, so the empty node is still refined and scored hypothesis_generation.py hypothesis_generation.py. Extraction retries up to 15 times, raising the temperature by 0.1 each time up to 1.5 utils.py utils.py.
• Title matching is fuzzy. The LLM's titles are mapped back to corpus titles by a similarity match inspiration_screening.py; the recombination step also looks up abstracts by a similar-but-not-exact key hypothesis_generation.py. A near-miss title can attach the wrong abstract to a node.
• Self-scores drive both beam and rank. The same model scores drafts used to choose the beam and the final order, and only the last draft of each line is scored hypothesis_generation.py. The paper's own check shows a weak link: drafts with two matched inspirations have average rank ratio 0.411 against 0.521 for none tab. 8; the GPT-4o Matched Score trend is not monotone and rests on small groups (for example 29 drafts at score 1 and 210 at score 5) tab. 9.
• The judge is a language model. Matched Score is produced by GPT-4o, and the authors report experts usually rate one to two points lower app. A.14. Expert-only numbers are in tab. 7 and tab. 11.
• Ground-truth leakage in the retrieval test. The Q1 corpora include the roughly 120 ground-truth inspirations by construction §5.1. Retrieval numbers are therefore a recall test on a corpus that contains the answers, not a search over an unseen pool.
• Corpus for the copilot setting. §1 says "up to 3,000 papers" and §6 says |I| = 300. Whether the §6 corpus contains the ground-truth inspirations is not stated.
• No significance tests. The ablation claims ("significantly improve") are not backed by any statistical test or run-to-run variance §6.2 tab. 10.
• Cost is not reported. Each mutation line costs three generation calls, three critiques and one scoring call hypothesis_generation.py. Recombination repeats this per beam node and per screened partner, and the paper gives no call count.
• Model version. GPT-4o is named without a snapshot date, so later GPT-4o snapshots may not reproduce the numbers.
• Novelty is judged indirectly. The benchmark's novelty test is that the inspirations were not known to be related to the background §3. Nothing in the pipeline checks that claim; it relies on the LLM's choice.

Lineage

• Builds on: MOOSE, a social-science hypothesis framework using LLM inspiration retrieval and self-refinement §2 §6.1; the paper's bibliography gives its title as "Large Language Models for Automated Open-domain Scientific Hypotheses Discovery" (ACL 2024 Findings). No arXiv id is given in the paper.
• Builds on: Self-Refine, used for the critique loop §6.1.
• Compared against: MOOSE; SciMON, which uses semantic and citation neighbours for inspiration §6.1; arXiv:2311.05965, a biomedical hypothesis method that retrieves background-related material rather than inspirations §6.1 tab. 10.
• Superseded by: not known from wiki content.