wikillm

Paper · 2026-11-08 · cs.CL, q-bio.QM

AOP-Smart-Finder: A High-Throughput Human-AI Hybrid Framework for AOP Discovery and Evidence Mapping from Large-Scale Literature

LLM pipeline over Web of Science abstracts: a relevance screen, LLM choice of 10 AOP-Wiki key events, causal-pair extraction, then one LLM call that merges unmatched event names into candidates. Author-reported: DEHP case 12,983 records to 1,093 included, 2,519 relations in 30 min; ALS case 16 events and 23 relations after manual curation.

Relation extractionRetrieval-augmented generationtoxicologyadverse-outcome-pathwayllm-extractionliterature-miningweight-of-evidence

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Toxicologists link chemicals to harm by tracing chains of biological steps from an initial trigger to an adverse outcome. Those chains are scattered across thousands of papers, and building them by hand takes a lot of time. AOP-Smart-Finder uses a language model to read the titles and abstracts of a large search result, keep the papers that describe such a chain, and write down the cause-and-effect pairs it finds. Each step is matched to a standard event from an official knowledge base where possible. Where no standard event fits, the name is kept and marked as a candidate. A person then checks the resulting network against the original papers and rates how strong the evidence is for each link. The language model does the reading; people make the final judgment.

TL;DR

  • Web of Science search results go through a screen, a KE-selection step, and a causal-pair extraction step, each an LLM call that also sees the AOP-Wiki knowledge base §2.2 fig. 1.
  • Two modes: a single-compound chain search (Mode 1) and a search around a user-given event set (Mode 2) §2.1.
  • Author-reported: 12,983 DEHP records processed in 30 min, 1,093 included, 1,589 events, 2,519 relations §3.1.
  • Author-reported: an ALS network of 16 events and 23 relations, each with a manual evidence rating §3.2 tab. 1.
  • Final curation and weight-of-evidence ratings are manual, and the code does not perform them.

Problem

AOPs organise biological processes into molecular initiating events, key events and key event relationships. Building and updating them depends on manual screening, evidence integration and causal judgment across many papers §1. Language models can speed this up but may invent relations or drop evidence, so the authors pair extraction with a standard knowledge base and with traceable literature links §1 §4.3.

Method

The workflow has three stages: manual retrieval and task definition, automated analysis and output, and manual curation with weight-of-evidence (WoE) evaluation. Eight steps, with the task split between human, LLM, a knowledge-base layer and Python §2 fig. 1.

Inputs. A Web of Science export. The paper asks for title, source and abstract; the screen and extraction use title and abstract only §2.1 §2.2. The code reads a different set of columns WOS_manage.py.

Mode 1 (single compound). Find chains of at least three nodes: target compound, then event A, then event B. The link from the compound to the first event must have experimental or strong evidential support. Later links may be hypotheses, if the original text proposes them and they are biologically meaningful §2.1 p. 3.

Mode 2 (predefined event set). Prioritise relations among the user's events, and keep other upstream and downstream events that connect to them §2.1 p. 3.

Both modes accept a free-text task prompt that focuses the search §2.1 p. 4.

Knowledge base (AOP-Smart). A local index is built from the official AOP-Wiki XML: key events, key event relationships and AOPs §2.2 p. 4. The authors call this retrieval-augmented generation §2.2 p. 4. In the code, the retrieval is a language-model choice from the full KE index, not a similarity search AOP-Smart-Finder.py.

Per-article pipeline.

Pseudocode
# Per-article pipeline  [§2.2] [fig. 1]
# Input:  article (title, abstract), key_event_index (id -> title), mode
# Output: list of (upstream, downstream, species) lines, or nothing

def process_article(article, key_event_index, mode):
    # 1. Screen title and abstract with the LLM; keep only a "yes"  [§2.2]
    #    [code: Script/AOP-Smart-Finder.py#L192-L216]
    if not screen_says_yes(article, mode):
        return None

    # 2. Ask the LLM for the 10 most relevant official KE ids  [§2.2]
    #    The whole index goes into the prompt; no embedding search
    #    [code: Script/AOP-Smart-Finder.py#L222-L249]
    selected_ids = llm_select_key_events(article, key_event_index, top_n=10)

    # 3. Add KEs one KER step up and down; add AOPs with >= 2 member KEs
    #    [§2.2] [code: Script/AOP-Smart-Finder.py#L352-L385]
    context = build_context(selected_ids, min_aop_hits=2)

    # 4. Extract pairs; use a KE id and title if one fits, else a candidate
    #    name  [§2.2] [code: Script/AOP-Smart-Finder.py#L391-L416]
    text = llm_extract_pairs(article, context, mode)

    # 5. Keep only lines of the form [upstream|downstream|species]  [§2.2]
    #    [code: Script/AOP-Smart-Finder.py#L165-L186]
    return parse_pair_lines(text)

Candidate events. Names with no official match stay as candidates. Their original description is kept, and a candidate is not a new official KE §2.2 p. 4. After all articles are processed, every unique candidate name goes to one LLM call, which returns groups of names for the same event. A Python step then rewrites those names. Official KE names are never changed §2.2 p. 5.

Pseudocode
# Merge candidate names  [§2.2] [code: Script/AOP-Smart-Finder.py#L589-L650]
# Input:  all unique names that contain "candidate"
# Output: mapping from each name to the name it should become

def unify_candidates(candidate_names):
    # 1. One LLM call over the full list, temperature 0, no chunking
    #    [code: Script/AOP-Smart-Finder.py#L619-L629]
    groups = llm_group_candidates(candidate_names)
    # 2. Start from identity, so unmatched names stay unchanged  [§2.2]
    #    [code: Script/AOP-Smart-Finder.py#L552-L566]
    mapping = {name: name for name in candidate_names}
    claimed = set()
    for canonical, synonyms in groups.items():
        # 3. Accept a group only if its canonical is an input name that
        #    contains "candidate"  [code: Script/AOP-Smart-Finder.py#L562-L563]
        if canonical not in candidate_names:
            continue
        if "candidate" not in canonical.lower():
            continue
        mapping[canonical] = canonical
        for synonym in synonyms:
            # 4. A name listed under two groups keeps the first group
            #    [code: Script/AOP-Smart-Finder.py#L577-L584]
            if synonym not in candidate_names or synonym in claimed:
                continue
            claimed.add(synonym)
            mapping[synonym] = canonical
    return mapping

Edges. Each pair is grouped by its exact upstream and downstream names. Count is the number of distinct WoS records supporting the edge, and species are the union across records visual.py. Official KEs get a URL to their AOP-Wiki page and each record gets a DOI link §2.2 p. 5.

Curation and WoE (manual). Import into Cytoscape. Check KE names and IDs against the original DOIs. Remove unreasonable or redundant relations. Merge, delete or add nodes, and classify endpoints by toxicity. The paper says the tool does not classify endpoint toxicity or lay out the network itself §2.3 p. 5. Each retained relation is then rated on three dimensions, each High, Moderate or Low §2.3 p. 5-6:

  • Biological plausibility: High means a clear, well-established mechanism; Moderate means plausible but not fully explained; Low means only an association §2.3.
  • Empirical support: High means direct experimental support across several conditions, with timing and dose agreement; Moderate means some support, few studies or limited agreement; Low means no direct support or clear contradictions §2.3.
  • Essentiality: High means blocking the upstream event prevents or reduces the downstream one; Moderate means indirect evidence that intervening upstream affects downstream; Low means no intervention evidence or results that do not support it §2.3.

The paper gives no rule for combining the three into the overall rating. Each overall rating is a researcher's judgment from the cited literature §2.3. In the paper's final table the same three ratings give different overall ratings tab. 1.

Hyperparameters

The paper states few settings. Code values are given where the paper is silent.

NameValueWhereNotes
ModelDeepSeek-Flash (DEHP run)§3.1 p. 6ALS model not stated; code takes any model from the provider list
Threads (concurrent articles)20 default; 200 in the DEHP run§2.2 p. 4; §3.1 p. 6; AOP-Smart-Finder.pyuser input at run time
Temperature0, every LLM callAOP-Smart-Finder.py [#L412] [#L628]not stated in paper
KEs selected per article (top N)10AOP-Smart-Finder.pynot stated in paper
KE expansionone step up and one step down along KERsAOP-Smart-Finder.pynot stated in paper
AOP inclusionat least 2 member KEs, counted per roleAOP-Smart-Finder.pynot stated in paper
Max output tokens, screening10,000AOP-Smart-Finder.pynot stated in paper
Max output tokens, KE selection30,000AOP-Smart-Finder.pynot stated in paper
Max output tokens, extraction20,000AOP-Smart-Finder.pynot stated in paper
Max output tokens, unification50,000AOP-Smart-Finder.pynot stated in paper
Reasoning effort"high" for model names containing "deepseek", screening call onlyAOP-Smart-Finder.pynot stated in paper
Minimum publications per edge1 in the Excel outputvisual.pythe paper keeps only edges with 2 or more publications, manually §3.2 p. 7
AOP-Wiki XML release2026-10-01§3.1 p. 6the local index is built from the XML
Web of Science fieldstitle, abstract, UT, DOIWOS_manage.pypaper lists title, source, abstract §2.1
Web of Science query, DEHPDEHP, its names and CAS 117-81-7§3.1 p. 6the query text is in the paper
Web of Science query, ALSALS and amyotrophic lateral sclerosis, with oxidative stress or ROS§3.2 p. 7top 500 by relevance kept

Results

Author-reported. Setup: a DEHP run in Mode 1 without an event restriction, and an ALS run in Mode 2 built around the draft AOP 636 on AOP-Wiki §3.1 §3.2.

CaseRecordsAnalysedOutputEventsMapped to official KEsRelationsNotes
DEHP, Mode 112,9831,093 included; 11,890 skipped1,589 events; 2,519 relations1,589458 (28.82%) §4.12,519546 endpoints with no outgoing relation §3.1
ALS, Mode 24,790 retrieved; top 500 kept437 analysed; 63 skipped55 stressors; 278 events; 548 relations27885548after curation: 16 events (11 official, 5 candidate) and 23 relations, from 108 publications §3.2
  • DEHP: the run took under 30 min, with 53,730,793 input and 19,799,749 output tokens §3.1 p. 6.
  • DEHP: 100 randomly chosen mapped events were checked by hand and all were correct §3.1 p. 6. This is a manual check, not a code check.
  • DEHP: Figure 2 shows the resulting network; endpoints by category are reproductive 132, metabolic 123, endocrine 71, neurotoxicity 49, developmental 35, hepatotoxicity 30, immunotoxicity 30, cardiovascular 27, nephrotoxicity 14, respiratory 7, other 28 fig. 2 §3.1.
  • ALS: Figure 3 shows the curated network, with edges coloured by overall WoE fig. 3 §3.2.

WoE ratings for the 23 ALS relations tab. 1 (author-reported, manual):

Upstream → downstreamBiological plausibilityEmpirical supportEssentialityOverall
Activation, NADPH Oxidase(1174) → Increase, Reactive oxygen species(1115)HighHighHighHigh
Increased intracellular Iron accumulation(2149) → Increase, Reactive oxygen species(1115)HighModerateLowModerate
Activation, Rac1 → Increase, Reactive oxygen species(1115)HighHighHighHigh
Activation, EP2 receptor → Increase, Reactive oxygen species(1115)HighHighHighHigh
Activation, Microglia(1998) → Increase, Reactive oxygen species(1115)HighHighModerateModerate
SOD1 mutations → Increase, Reactive oxygen species(1115)HighHighModerateModerate
SOD1 mutations → Increase, Mitochondrial dysfunction(177)HighModerateLowModerate
Increase, Reactive oxygen species(1115) → Increase, Mitochondrial dysfunction(177)HighHighHighHigh
Increase, Mitochondrial dysfunction(177) → Increase, Reactive oxygen species(1115)HighModerateModerateModerate
Activation, Drp1 → Increase, Mitochondrial dysfunction(177)HighHighHighHigh
Increase, Reactive oxygen species(1115) → Increase, Oxidative Stress(1392)HighHighModerateHigh
Increase, Mitochondrial dysfunction(177) → Increase, Oxidative Stress(1392)HighModerateModerateModerate
Depletion, GSH(130) → Increase, Oxidative Stress(1392)HighHighModerateHigh
Depletion, GSH(130) → Increase, Mitochondrial dysfunction(177)HighHighModerateModerate
Increase, Mitochondrial dysfunction(177) → Neuroinflammation(188)HighModerateLowModerate
Increase, Mitochondrial dysfunction(177) → N/A, Neurodegeneration(352)HighHighModerateHigh
Increase, Oxidative Stress(1392) → Increase, TDP-43 aggregation(2451)HighHighModerateHigh
Increase, TDP-43 aggregation(2451) → Increase, Oxidative Stress(1392)HighModerateModerateModerate
Increase, TDP-43 aggregation(2451) → Increase, Mitochondrial dysfunction(177)HighHighModerateHigh
Increase, Reactive oxygen species(1115) → ExcitotoxicityHighModerateLowModerate
Excitotoxicity → human amyotrophic lateral sclerosis (ALS)(2416)HighHighModerateHigh
N/A, Neurodegeneration(352) → human amyotrophic lateral sclerosis (ALS)(2416)HighHighModerateHigh
Neuroinflammation(188) → N/A, Neurodegeneration(352)HighHighModerateHigh

Numbers in parentheses are official AOP-Wiki KE IDs. The paper notes that the spelling "intracellular" in KE 2149 matches AOP-Wiki as published tab. 1 note. Overall: 13 High, 10 Moderate, 0 Low §3.2 p. 10.

Paper vs code

Code: AOP-Smart-Finder.py at commit e6921c6 (MIT).

  • Mode 1 screen is looser than the paper. The paper requires the compound-to-first-event link to have experimental support §2.1 p. 3. The screen prompt accepts an article if either of the two links has direct or strong evidence screen_article1.txt. Only one link must be strong, so the first link can be a hypothesis.
  • Mode 1 adds a rule the paper does not state. Only evidence from exposure to the compound alone may be used, and combined-exposure effects are ignored screen_article1.txt get_ker1.txt. The paper does not describe this restriction §2.1.
  • Mode 2 screen and extraction disagree. The Mode 2 screen accepts any stated relation, including associative ones, and does not require direct experimental evidence screen_article2.txt. The Mode 2 extraction prompt asks for an explicitly stated causal link get_ker2.txt.
  • Mode 2 compound name. The paper says Mode 2 is not restricted to a single compound §2.1 p. 3. The run still needs an input name, which becomes a folder and file name AOP-Smart-Finder.py [#L761]. The Mode 2 prompts do not name the compound get_ker2.txt.
  • Mode choice is a run-time prompt. Mode 1 and Mode 2 are prompt versions 1 and 2, chosen interactively, default 1 AOP-Smart-Finder.py. A wrong choice silently applies the wrong screen and extraction rules.
  • Input fields. The paper asks for a Web of Science export with title, source and abstract §2.1 p. 3. The code reads article title, abstract, UT and DOI, and does not read the source WOS_manage.py.
  • Knowledge-base IDs and titles are not checked. The paper says the LLM "strictly adheres" to official IDs and terms §4.3 p. 12. The code has no step that checks a KE id or title against the knowledge base after extraction. Only the number of fields is checked AOP-Smart-Finder.py. The 100-event check was manual §3.1 p. 6.
  • Prompt and output ID formats differ. The context lists KE ids as bare numbers AOP-Smart-Finder.py, while the extraction prompt demands the KE prefix get_ker1.txt. The Excel step accepts both forms visual.py.
  • Candidate groups. The prompt asks for a canonical name taken from its own group AOP-Smart-Finder.py. The code accepts any canonical that is an input name containing "candidate" AOP-Smart-Finder.py.
  • Edge filter. The paper keeps edges with at least 2 publications, applied by hand in Cytoscape §3.2 p. 7. The Excel output keeps every edge with at least 1 publication visual.py [#L430].
  • WoE. The paper rates WoE by hand §2.3. The code has no WoE step, and the Excel output marks each verification field "Not_checked" visual.py.
  • Model settings. The paper states the DEHP model and thread count only. Temperature, top N, and token limits are code-only §3.1 p. 6 AOP-Smart-Finder.py.

Pitfalls

  • Silent empty context. If the KE-selection reply has no bracketed list, or returns an empty list, the context is empty. Extraction still runs, with no knowledge base and no error AOP-Smart-Finder.py.
  • Malformed lines are dropped silently. A pair line must split into exactly three pipe-separated fields. A title containing "|" is lost without any warning AOP-Smart-Finder.py.
  • The screen is a prefix test. An article is kept when the reply starts with "yes", ignoring case. Any other text, including an unclear one, drops it AOP-Smart-Finder.py.
  • Failures are not retried. An exception writes the article with empty events and an error field. An empty model reply is also written as an article with no events. Both count as done on re-run, so they are not retried unless the output file is edited AOP-Smart-Finder.py [#L474-L484] [#L767-L790].
  • Unification failure is quiet. If the single unification call fails or returns unparseable text, the mapping is empty. The map file is still written, and candidate names are left as they are AOP-Smart-Finder.py [#L690-L695].
  • Unification has no size control. All unique candidate names go into one prompt with a 50,000-token output cap. The DEHP run produced 1,131 candidate events §4.1 p. 12, but the paper does not say how many unique names reached the call. The code has no chunking or length check AOP-Smart-Finder.py.
  • Node identity is by exact string. "Increase, X(1115)" and "Increase, X(KE1115)" become two different nodes. The official-KE count matches on the number alone, so the two counts can disagree visual.py [#L480] [#L664].
  • Overall WoE is not a function of the three ratings. In tab. 1, "Depletion, GSH(130) → Increase, Oxidative Stress(1392)" and "Depletion, GSH(130) → Increase, Mitochondrial dysfunction(177)" both have High, High, Moderate ratings but different overall ratings (High and Moderate). Plan for a judgment step, not a formula.
  • Cost is high. The DEHP run used about 53.7 million input and 19.8 million output tokens for 12,983 records §3.1 p. 6. The repository's README estimates about $15 to $20 for 10,000 screens and 1,000 extractions on the deepseek-flash model README.md. This is the authors' estimate, not a figure from the paper.
  • Output is rewritten on every article. Each completed article rewrites the whole results file under a lock AOP-Smart-Finder.py. Run time grows with the number of articles.
  • Retrieval is an LLM choice. The full index goes into every KE-selection call AOP-Smart-Finder.py. Results depend on the model's choice of 10 ids. Cite this as an LLM-judged step, not as embedding retrieval.

Lineage

  • Builds on AOP-Smart, the authors' earlier retrieval-augmented framework over the AOP-Wiki knowledge base §2 p. 2 (not on this wiki).
  • Related text-mining tool: AOP-helpFinder, which uses term matching and co-occurrence statistics §1 p. 2 (not on this wiki).
  • Compared against: no quantitative comparison in the paper.
  • Superseded by: not stated.

Source and license

Adapted from “AOP-Smart-Finder: A High-Throughput Human-AI Hybrid Framework for AOP Discovery and Evidence Mapping from Large-Scale Literature” by Qinjiang Niu, Lu Yan (arXiv:2611.11349), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/qinjiang-lab/AOP-Smart-Finder at commit e6921c6a8def (MIT). Any quoted code is shown under that license.