ToolUniverse: An open platform for democratizing AI scientists

In plain words

Scientists increasingly use language models that can call software: look up a gene in a database, run a prediction model, or fit a statistic. Every such tool has its own way of being called, so each one needs custom setup before a model can use it. ToolUniverse is an open platform that puts many scientific tools behind one description format. Each tool states what it is for in plain language, lists its inputs and outputs with their types, and can be invoked the same way whether it runs as local code, as a remote service or as a model on someone else's server.

On top of that format, the platform helps a model find the right tool for a request, chain several tools into a workflow, and wrap a new tool so others can use it. Two helper systems can write a new tool from a description, or rewrite a tool's description so that a model uses it correctly. The authors also bundle common research tasks as ready-made skills that a biologist can request in ordinary language.

The authors report that agents given the platform answer biology database questions more accurately than the same agents without it, and more accurately than a purpose-built biomedical agent on the same test items. These are the authors' own measurements. The released code differs from the paper in several places, listed below.

TL;DR

ToolUniverse is an open platform that exposes more than 2,700 scientific tools through one typed call format. Six components cover registering, searching, calling, composing, generating and refining tools [main text] [SN 3]. The released code contains all six, but the keyword scoring formula in the paper is not what the code runs, several defaults differ, and the tool-generation loop has no human approval step. The headline result is author-reported: on LAB-Bench DbQA and SeqQA, the same Claude Code and Codex agents improve with the platform, and both exceed Biomni on the released items [SN 8] fig. S2.

Problem

AI scientists are usually bespoke systems tied to one workflow. The tools they need differ in purpose, inputs, runtime and access: a local library, a web API, a model on a private server, or a laboratory instrument [main text]. A model that only reasons in text cannot run experiments or query data, so it needs a reliable way to discover and call tools without per-tool configuration [SN 1]. The design assumes that a model's context window can hold the tool descriptions it needs [SN 2].

Method

Overview

ToolUniverse sits between a language model or agent and a library of tools fig. 1a [SN 1]. Clients reach it through a Python API, HTTP, the Model Context Protocol (MCP), or direct function calls [main text]. The library spans machine learning models, agents, databases, APIs, embedding stores, expert-feedback systems and robotics or lab tools [SN 5]. The authors also ship more than 130 research skills, which are natural-language workflows that run several tools for one task [SN 6].

Interaction standard

1. Every tool declares a natural-language purpose, a typed schema for each input and output field (name, type, required status, constraints), and a return schema [SN 2, Tool Description Schema] fig. S3 fig. S4.
2. A call is a dictionary with a tool name and an arguments object, passed to a single run entry point [SN 2, Accessing Tools] fig. S5.
3. The same call format serves local functions, remote MCP tools and model endpoints, so the client does not see the backend [SN 2] [SN 3, Tool Manager].

The six modules

1. Tool Manager. Local tools need a configuration dictionary and a class, registered with a decorator. Remote tools are loaded by giving an MCP server address; the loader registers all of the server's tools [SN 3, Tool Manager] fig. S7 fig. S8.

2. Tool Finder. Three strategies: keyword search, LLM in-context search, and embedding search with a fine-tuned GTE-Qwen2-1.5B encoder. The paper states the TF-IDF formula given below [SN 3, Tool Finder].

   \text{Relevance} = \text{TF} \times \text{IDF} \times \log(1 + \text{QueryFrequency}) [SN 3, Tool Finder] (unnumbered in source)

   The keyword scorer in code uses a different formula (see Pitfalls).

3. Tool Caller. Validates arguments against the declared types, loads a tool the first time it is requested, and returns a structured result or a readable error [SN 3, Tool Caller].

4. Tool Composer. Chains tools in sequence, broadcasts one query to several tools in parallel, or runs an agentic loop in which a reasoning model issues calls and reads the results [SN 3, Tool Composer] fig. S9.

5. Tool Discoverer. A multi-stage agent pipeline that searches for reference implementations (GitHub, PyPI, existing tools), writes a tool definition, generates code and tests, scores the result, and refines it until a target score or an iteration limit is reached [SN 3, Tool Discoverer].

6. Tool Optimizer. Generates test cases, executes the tool, rewrites the tool and parameter descriptions from the results, scores them on six criteria, and repeats [SN 3, Tool Optimizer].

Pseudocode for the three paths an implementer most often needs follows. Steps are written from the paper and checked against the code; the code citation is given where they differ.

# Tool Caller: one request to one checked call  [SN 3, Tool Caller]
# Input:  request = {"name": ..., "arguments": {...}}
# Output: structured result, or a readable error message

def call_tool(request, engine):
    # 1. Read the tool name and its arguments
    name = request["name"]
    arguments = request["arguments"]

    # 2. Load the tool on first use and keep the loaded instance
    #    [code: src/tooluniverse/execute_function.py#L195]
    tool = engine.get_tool_instance(name, cache=True)

    # 3. Check the arguments against the declared types and required fields
    #    [code: src/tooluniverse/execute_function.py#L153]
    check_arguments(arguments, tool.schema)

    # 4. Run the tool; any failure becomes a message the client can act on
    try:
        return tool.run(arguments)
    except Exception as error:
        return describe_error(error)

# Keyword Tool Finder score, as the released code computes it
# Input:  query_phrases (unigrams, bigrams, trigrams after stemming)
# Output: score per tool; tools scoring 0 are dropped
# [code: src/tooluniverse/tool_finder_keyword.py#L419-L470]

def keyword_score(query_phrases, tool, index):
    # 1. Sum a BM25-style term score over the query phrases
    total = 0.0
    for phrase, query_freq in count(query_phrases).items():
        if phrase not in tool.terms:
            continue
        tf = tool.terms[phrase]
        idf = log((index.num_docs + 1) / (index.doc_freq[phrase] + 0.5))
        # 2. Length-normalise the term frequency (k1 = 1.5, b = 0.75)
        length_ratio = tool.length / index.avg
        bm25_tf = tf * 2.5 / (tf + 1.5 * (0.25 + 0.75 * length_ratio))
        # 3. Longer phrases count for less: 0.3 per extra word
        weight = 0.3 ** (len(phrase.split()) - 1)
        total += idf * bm25_tf * log(1 + query_freq) * weight
    # 4. Add exact-match bonuses on name, description and type
    #    [code: src/tooluniverse/tool_finder_keyword.py#L659-L724]
    return total + exact_match_bonus(query_phrases, tool)

# Embedding Tool Finder  [SN 3, Tool Finder]
# Offline: one unit vector per tool description, built once
# Online:  one unit vector per query; rank tools by cosine similarity

def embedding_search(query, tool_vectors, tool_names, k):
    # 1. Encode the query; the default encoder adds no instruction prefix
    #    [code: src/tooluniverse/tool_finder_embedding.py#L107-L116]
    query_vector = encode(query, normalize=True)
    # 2. Cosine similarity against every tool vector
    #    [code: src/tooluniverse/tool_finder_embedding.py#L483-L559]
    scores = tool_vectors @ query_vector
    # 3. Return the k best tool names
    return top_k(tool_names, scores, k)

# Tool Optimizer loop  [SN 3, Tool Optimizer]
# Defaults in code: at most 3 rounds; stop when the score reaches 8 of 10
# [code: src/tooluniverse/compose_scripts/tool_description_optimizer.py#L11-L14]

def optimize_description(tool_config, max_rounds=3, threshold=8):
    # 1. Generate test cases and run the tool on them
    tests = generate_test_cases(tool_config)
    results = execute(tool_config, tests)
    description = tool_config["description"]
    for round_index in range(max_rounds):
        # 2. Rewrite tool and parameter descriptions from the results
        description = rewrite(description, results)
        # 3. Score on six criteria, each 1 to 10; stop if good enough
        #    [code: src/tooluniverse/data/optimizer_tools.json#L270]
        score, satisfied, feedback = evaluate(description, results)
        if satisfied or score >= threshold:
            break
        # 4. Generate new tests that target the feedback, then run them
        tests = generate_test_cases(tool_config, feedback)
        results = execute(tool_config, tests)
    return description

Tool Discoverer loop

# Tool Discoverer loop  [SN 3, Tool Discoverer]
# Defaults in code: 2 iterations, target score 8.5 of 10
# [code: src/tooluniverse/compose_scripts/tool_discover.py#L2080]
# [code: src/tooluniverse/compose_scripts/tool_discover.py#L2127]

def discover_tool(description, max_iterations=2, target=8.5):
    # 1. Collect reference implementations from GitHub, PyPI and existing tools
    references = collect_references(description)
    # 2. Write the tool definition: name, parameters, return schema
    definition = write_definition(description, references)
    # 3. Generate code and tests, then score the code
    code = write_implementation(definition, references)
    for iteration in range(max_iterations):
        # Pass ratio times 10, or the CodeQualityAnalyzer score when given
        #    [code: src/tooluniverse/compose_scripts/tool_discover.py#L1169]
        score = evaluate(code, definition)
        if score >= target:
            break
        code = refine(code, score)
    # 4. Return the output for review; no approval step exists in this code
    return definition, code

Hyperparameters

Results

Author-reported. Setup: LAB-Bench [laurent2024labbench] DbQA (60 items) and SeqQA (70 items), the same items Biomni released. Each platform-enabled agent is compared with the identical agent and base model without the platform. Claude arm: Claude Code with Claude Opus 4.8. Codex arm: GPT-5.5, three independent runs per condition [SN 8]. The paper does not state the Claude run count; the reference file says the ablation averages three repetitions [code: examples/benchmarks/labbench/reference_results.json]. Grading is exact letter match; a refusal or an unparseable answer counts as wrong [SN 8]. Panels in fig. S2.

The paper says the Claude Code arm's refusals on some questions lower its score whether or not the platform is present [SN 8]. Biomni uses its own base model, so the gap to Biomni is not a same-model comparison [SN 8].

The released reference file reports the same four pairs and the same two Biomni values reference_results.json.

Other results in the paper: a case study on a BLM question, a case study on an OXTR question, and a clinical-trial statistics case on BCG adverse events. Each is a worked example built from live tool executions, described step by step with its tool names and outputs [SN 9]. The main text says Tool Finder was benchmarked against its three strategies, but the LaTeX source contains no Tool Finder accuracy numbers.

Paper vs code

Searched: the paper links github.com/mims-harvard/ToolUniverse directly in its code availability statement, so no further search was needed. Cloned at depth 1 at commit 4bf80d1f25ee679febcf6fc25eb14f9bb4a2ad72. The repository declares version 1.5.9 pyproject.toml; the paper names v1.3.1 [main text, code availability]. Everything below describes the cloned commit, not v1.3.1.

• Keyword scorer. Paper: TF × IDF × log(1 + query frequency), with a 2.0 multiplier for name matches and a 1.5 multiplier for phrase matches [SN 3, Tool Finder]. Code: BM25 term scoring with k1 = 1.5 and b = 0.75, a 0.3 discount per extra word, and additive bonuses (3.0 or 2.0 for names, 1.5 for description phrases, 1.0 for type or category) [code: src/tooluniverse/tool_finder_keyword.py#L419-L470] [code: src/tooluniverse/tool_finder_keyword.py#L659-L724].
• Stemming. Paper: 20 morphological rules [SN 3, Tool Finder]. Code: 30 suffix rules tool_finder_keyword.py.
• Stop words. Paper: over 45 [SN 3, Tool Finder]. Code: 63 [code: src/tooluniverse/tool_finder_keyword.py#L40-L104]. Consistent with the paper's wording.
• Embedding finder. Paper: fine-tuned GTE-Qwen2-1.5B with cosine similarity [SN 3, Tool Finder]. Code: the encoder is mims-harvard/ToolRAG-T1-GTE-Qwen2-1.5B finder_tools.json; embeddings are normalised tool_finder_embedding.py. Consistent. The alternative 7B encoders are used only if already cached, unless the environment variable TOOLUNIVERSE_ALLOW_ENCODER_DOWNLOAD is set [code: src/tooluniverse/tool_finder_embedding.py#L22-L27].
• Tool Optimizer. Paper: at most 3 rounds, threshold 8, six criteria [SN 3, Tool Optimizer]. Code: 3 rounds, threshold 8 [code: src/tooluniverse/compose_scripts/tool_description_optimizer.py#L11-L14]; the six criteria (clarity, accuracy, completeness, conciseness, user-friendliness, redundancy avoidance) appear in the evaluator prompt [code: src/tooluniverse/data/optimizer_tools.json#L270]. Consistent.
• Tool Discoverer. Paper: target minimum 9 of 10, and five weighted quality dimensions (functionality, reliability, maintainability, performance, test coverage) [SN 3, Tool Discoverer]. Code: target 8.5 [code: src/tooluniverse/compose_scripts/tool_discover.py#L2127], two iterations by default tool_discover.py. The score is the share of passed generated tests times 10, or the overall score from the CodeQualityAnalyzer tool when it returns one [code: src/tooluniverse/compose_scripts/tool_discover.py#L1169-L1178]. The five named dimensions do not appear in that file.
• Human review of generated tools. Paper: every generated tool gets human review before it enters the platform [SN 3, Tool Discoverer] [SN 7]. Code: no approval step in the generation loop [code: src/tooluniverse/compose_scripts/tool_discover.py#L1285-L1289] auto-accepts package installation in a debug path. The review is a process claim outside the code.
• Tool registration. Paper: register_tool(Class_name, tool_config) and a remote decorator register_remote_tool [SN 3, Tool Manager]. Code: the decorator is register_tool(tool_type_name=None, config=None) [code: src/tooluniverse/tool_registry.py#L58], and register_remote_tool is an alias of the MCP registration function mcp_tool_registry.py. The argument names differ; the behaviour is consistent.
• Tool Composer. Paper: compose(arguments, tooluniverse, call_tool) [SN 3, Tool Composer]. Code: the same signature in the composer script [code: src/tooluniverse/compose_scripts/tool_graph_composer.py#L12]. Consistent.
• LAB-Bench. Paper: 56.7 → 78.3, 81.1 → 92.8, 94.3 → 99.5, 91.4 → 96.2, and Biomni 74.4 and 81.9 [SN 8]. Repository reference file: the same values reference_results.json. Consistent. LAB-Bench question text is not in the repository (the README excludes it because of a canary string) and must be built from Biomni's release.

Pitfalls

• Version. The paper describes v1.3.1 and the cloned code is 1.5.9. Keyword scoring, defaults and tool lists may have changed. Pin the commit you test.
• Keyword scoring. Do not implement the paper's TF-IDF formula and expect the released behaviour. The shipped scorer is BM25-style with additive bonuses. Use the code path, and cite it.
• No Tool Finder benchmark numbers. The main text says the three finder strategies were compared. The LaTeX source contains no precision, recall or rank numbers for them, so there is no reference to check the finders against.
• Default encoder has no query prefix. The alternative encoders need an instruction prefix and are 7B models; the default does not use one [code: src/tooluniverse/tool_finder_embedding.py#L107-L116].
• Generated tools run as code. The Tool Discoverer produces Python and runs test code. Review generated code yourself before it reaches production, because the release has no approval step and a debug path auto-accepts package installs tool_discover.py.
• Discoverer score is weak. With no failing tests the pass ratio is 1.0, so a generated tool can pass its own generated tests and still be wrong. The paper's five-dimension score is not what the code computes.
• Optimizer parse failure. If the evaluator's output cannot be parsed, the code sets the score to 5 [code: src/tooluniverse/compose_scripts/tool_description_optimizer.py#L487]. That is below the threshold of 8, so the loop continues. A broken evaluator therefore costs rounds rather than failing visibly.
• Caching. The paper says tools are cached "for a duration" after loading. The call path keeps loaded instances with cache=True [code: src/tooluniverse/execute_function.py#L195]; the only expiry setting found is for the credential-isolated instance cache. Check this before relying on cache-expiry behaviour.
• Silent loss of tools. The LAB-Bench README warns that an MCP server that has not finished loading produces a complete "with platform" run that has no platform tools. Run the repository's preflight check before any ablation README.md.
• Benchmark replication. The base models (Opus 4.8, GPT-5.5) are needed to reproduce the numbers. A different model measures a different system [code: examples/benchmarks/labbench/README.md]. Refusals count as wrong, so the Claude numbers are lower bounds [SN 8].
• Case studies are not reproducible from the repository. They are single live runs against external databases and ML services [SN 9]. The ML predictions (for example ADMET-AI risk scores) are estimates, as the paper itself says. The tool names used do exist in the repository, for example the clinical-trial severity test [code: src/tooluniverse/tools/clinical_trial_ae_severity_test.py].
• Counts not verified. The "over 2,700 tools" and "130 skills" figures are dated July 2026 in the paper. The cloned skills/ folder holds 230 entries. The two counts are not reconciled here. I did not count the tool registry.
• Licensing of wrapped data. The platform code is Apache-2.0. Each tool calls an external resource (for example OpenTargets, PubChem, ChEMBL, ClinVar, or a DrugBank vocabulary) under that resource's own terms. The paper says the sources are listed per tool in the repository. Check each source before commercial redistribution of tool outputs or bundled data.

Lineage

• Builds on: general tool-use approaches for language models, as the paper's own introduction frames them [main text]. No prior-work arXiv id is given in the source for the platform itself.
• Compared against: Biomni, a purpose-built biomedical agent, on the same released LAB-Bench items [SN 8].
• Used by: the paper names TxAgent, GeneAgent, SpatialAgent and ATHENA as systems that use or train with the platform [SN 4] [main text].