BioRED: A Rich Biomedical Relation Extraction Dataset

In plain words

Biomedical papers state facts such as "this gene variant is linked to this disease" or "this chemical treats this condition". Software that reads papers at scale needs labelled examples of those facts. Earlier corpora mostly labelled one kind of fact, usually inside a single sentence, so a system trained on them could not see a relation spread across several sentences of an abstract.

BioRED labels 600 PubMed abstracts with six kinds of biomedical terms and the links between them. Each link also gets a type, such as positive correlation or association, and a flag saying whether the abstract presents it as a new finding or as background knowledge. That flag is the part earlier corpora did not have.

The authors trained standard models on it. Finding the terms works well. Finding the links is much harder, and finding the new ones is the hardest part, so the dataset is mainly a benchmark for the open problem.

TL;DR

• Dataset of 600 PubMed abstracts: six entity types (gene, disease, chemical, variant, species, cell line) linked to database identifiers, and 6,503 relations between them §3.3 tab. 4.
• Relations are document-level, cover eight concept pairs and eight relation types, and each carries a novel-or-background label §3.1 fig. 2.
• Author-reported baselines: 89.3 strict F for NER (PubMedBERT-CRF), 58.9 F for relation plus type, and 47.7 F for relation plus type plus novelty (PubMedBERT) tab. 5 tab. 6.
• Use it when you need multi-type, cross-sentence biomedical relations with novelty labels. Do not use it as a source of gold standard for full-text or end-to-end extraction: the RE benchmark gives the gold entities and the corpus is abstracts only §4.2 §5.

Problem

Most biomedical relation extraction (RE) datasets label one entity pair, often one relation type, and only within a sentence §2.2. Relations that span sentences need annotation at document level, which is costly and was rarely done §2.2. Relations also carry a meaning the existing corpora ignored: whether the claim is new or already known background §1.

Named entity recognition (NER) and linking (NEL) corpora exist for most single concepts, but multi-concept corpora usually have no relation annotations §2.1 tab. 1.

The paper argues that a single corpus with several entity types and several relation types should make one model simpler and better, especially for rare types §1 §4.3.

Method

The contribution is a dataset plus benchmarks. There is no new model.

Annotation scope §3.1.

1. Public PubMed search queries were tagged to find the entity and relation types that matter most. The authors call this a data-driven choice of key entities and relations §3.1.
2. Six concept types are kept. Gene covers genes, proteins, mRNA and other gene products. Disease covers diseases, symptoms and some disease-related phenotypes. Variant covers substitutions, deletions and insertions. Species covers taxonomic species. Cell line is kept as its own type §3.1.
3. Entity spans are linked to identifiers in a database or controlled vocabulary for each type. For genes this is the NCBI Gene ID §3.1.
4. Eight concept pairs are annotated: Disease-Gene, Disease-Chemical, Gene-Chemical, Gene-Gene, Disease-Variant, Chemical-Variant, Chemical-Chemical and Variant-Variant §3.1 fig. 2a.
5. Eight non-directional relation types are used: Positive_Correlation, Negative_Correlation, Association, Bind, Drug_Interaction, Cotreatment, Comparison and Conversion §3.1 fig. 2b.
6. A relation with more than two entities is split into several pairwise relations §3.1.

Annotation process §3.2.

1. Abstracts were sampled from existing corpora (NCBI Disease, NLM-Gene, GNormPlus, BC5CDR, tmVar). A small set was used to write the guideline and train annotators on the TeamTat tool §3.2 fig. 1.
2. Entities were pre-annotated with PubTator and then corrected by hand §3.2.
3. Each article was annotated by three annotators with a biomedical-informatics background. A disagreement goes to a senior annotator with a molecular-biology background §3.2.
4. Relations were annotated by hand from scratch, with one exception: chemical-induced-disease relations were taken from BC5CDR §3.2.
5. For each relation, two more biologists judged whether it is a novel finding or background knowledge §3.2.
6. All 600 abstracts were annotated in 30 batches of 20 articles. Per batch the authors report about 2 hours per annotator for entities, 8 hours for relations and 6 hours for novelty labels §3.2.

Benchmark models §3.4.

• NER: BiLSTM-CRF, BioBERT-CRF and PubMedBERT-CRF with BIO tags. Documents are split into sentences before tagging §3.4.
• RE and novelty: BERT-GT and PubMedBERT, used as text classifiers over the entity pair §3.4 §4.2.
• Training and development sets are used for model development and parameter choice. The test set is used once for evaluation §3.4.

Evaluation §3.4 §4.1 §4.2.

• NER strict: type and boundary must match exactly §4.1.
• NER relaxed: the type must match and the predicted boundary must overlap the gold span §3.4 §4.1.
• RE: the gold concepts are given. The model must find the concept pair (schema 1), the pair plus relation type (schema 2), or the pair, type and novelty (schema 3) §4.2 tab. 6.

# NER span scoring  [§3.4] [§4.1] [tab. 5]
# Input:  predicted span, list of gold spans; span = (start, end, type)
# Output: True if the prediction counts as correct

def relaxed_match(predicted, gold_spans):
    # Relaxed: same type, and the boundaries overlap at all  [§3.4]
    for gold in gold_spans:
        if gold.type == predicted.type and overlaps(predicted, gold):
            return True
    return False

def strict_match(predicted, gold_spans):
    # Strict: same type, and identical start and end  [§4.1]
    return predicted in gold_spans

# RE scoring with gold concepts given  [§4.2] [tab. 6]
# Input:  relation = (concept_a, concept_b, type, novel)
# Output: key used to match a predicted relation to a gold one

def schema_key(relation, schema):
    # Relation types are non-directional, so sort the pair  [§3.1]
    pair = tuple(sorted([relation.concept_a, relation.concept_b]))
    # Schema 1: entity pair only  [§4.2]
    if schema == 1:
        return pair
    # Schema 2: entity pair plus relation type  [§4.2]
    if schema == 2:
        return (pair, relation.type)
    # Schema 3: entity pair, relation type and novelty label  [§4.2]
    return (pair, relation.type, relation.novel)

Hyperparameters

The paper body gives no training hyperparameters. It says benchmark implementation details are in Supplementary Materials A.1 §3.4. That supplement is not in the PDF we read, so the values below are "not stated".

Results

Author-reported. Setup: BioRED test split of 100 abstracts tab. 4. NER models are BiLSTM-CRF, BioBERT-CRF and PubMedBERT-CRF §3.4. RE models are BERT-GT and PubMedBERT §3.4. Hardware is not stated.

Corpus statistics tab. 4. Numbers in brackets are unique identifiers.

The novel share of all relations is 69% §3.3. Per document there are 34 entity spans and 10.8 relations. Of the relations, 52% are Association, 27% Positive_Correlation, 17% Negative_Correlation, and 2% are multi-entity relations §3.3.

NER, test set, F-score tab. 5. Strict is exact match; relaxed allows boundary overlap.

G = gene, D = disease, C = chemical, S = species, CL = cell line, V = variant tab. 5.

RE, test set, F-score tab. 6. Gold concepts are given. Column headers are concept pairs.

The table has no <V,V> column. The text says the <V,V> F-score is 0% for two models §4.2, so those numbers are not in tab. 6.

Single-type versus multi-type training, strict F-score tab. 7. Columns: all entities or relations trained together versus one type trained alone, both with PubMedBERT. Precision and recall are in the paper and are omitted here.

tab. 7

Headline numbers to quote: NER strict 89.3 (PubMedBERT-CRF, all types) tab. 5; NER relaxed 93.5 tab. 5; RE schema 2 best 58.9 and schema 3 best 47.7, both PubMedBERT tab. 6. The authors call the novelty result the open problem §1 §4.4.

Paper vs code

Official code: the paper links no code repository; it names only the NCBI FTP site for the dataset [abstract]. A GitHub search for "BioRED" returned ncbi/BioRED, owned by the NCBI organisation of the authors, whose README describes this paper. Its commit is 39e36f9 (cloned with --depth 1).

• The repository holds a README only. It has no LICENSE file, so code_license is recorded as none.
• The README points to two further items. biored_re_source_code.tar on the NCBI FTP holds the PubMedBERT relation and novelty code, and biored_re_model.tar holds trained models. A BERT-GT implementation is at github.com/ncbi/bert_gt.
• A WebFetch of the source tarball returned a binary archive. Its file names (scripts and src/ Python files) came back in a summary, but the content is compressed and no hyperparameters were visible. A local listing was denied. The training code, configs and hyperparameters were therefore not compared with the paper.
• The BERT-GT repository was not read.

Discrepancies found: not checked. Hyperparameters, the novelty classifier and the RE pipeline may differ from the paper, and no statement can be made until the tarball is read.

Pitfalls

• Gold concepts in RE. The RE benchmark gives the gold entities and their identifiers §4.2. Scores are not end-to-end. Running a real pipeline needs an NER step in front, and the paper's NER results do not transfer to it directly.
• Treatment relation is ambiguous. §3.1 says treatment falls under Negative_Correlation, but Figure 2b has no Treatment type and lists Cotreatment as its own type §3.1 fig. 2. Check the guideline before mapping labels.
• Concept merging. Gene includes proteins and mRNA; Disease includes symptoms and syndromes §3.1. Models trained on this label set will not match labels from corpora that keep those types apart.
• Chemical-induced disease relations were not annotated from scratch. They were taken from BC5CDR §3.2. The paper does not say how their quality compares with the rest.
• Relations are not all equal in size. Variant-Variant and some pairs are rare. The paper says some pairs and types barely occur §5 §3.1. Results for those rows rest on small counts.
• Single-type claim is partial. §4.3 says training on all types generally helps, and Table 7 shows it helps most for cell line and most RE pairs. But single-type models beat the joint model for disease (84.4 vs 83.5), chemical (89.8 vs 88.6), and the G,D and C,D pairs tab. 7.
• Novelty labels are hard. The paper names three error sources: abstracts that do not say which concepts are new, unknown mechanisms, and background repeated in conclusions §4.4. Structured section labels (TITLE, PURPOSE, METHODS, RESULTS) helped novelty classification on a subset of 191 abstracts §4.4. The 47.7 to 48.9 gain comes from adding a knowledge pair (<Rg1, NMDA receptor>) on top of the model, so it is not a plain model score §4.4.
• Abstracts only. The corpus has PubMed abstracts, not full text. The authors say the relation mechanism is often only in the full text §4.2 §5.
• Novelty IAA. Inter-annotator agreement is 77.91% for relations and 85.01% for novelty tab. 4. Expect noise in the labels. The relation figure is the lowest of the three.
• Dataset licence is not the paper licence. The card is CC BY 4.0. The paper says the dataset is freely available on the NCBI FTP and the annotation guideline is there too [abstract] §5. The paper does not state a dataset licence, and the repository has no LICENSE. Check the terms before redistributing data.
• Supplement missing. Benchmark details, the dictionary method for NER and the novelty ablations are in the Supplementary Materials §3.4 §4.4. None of them was available for this card.

Lineage

• Builds on: BioBERT §3.4 (ref. 51), PubMedBERT §3.4 (ref. 67), BERT-GT §3.4 (ref. 66), BiLSTM-CRF with a CRF output layer §3.4 (refs. 64, 65). Chemical-induced disease relations come from BC5CDR §3.2 (ref. 9).
• Compared against: the corpora in tab. 2 and tab. 3. The paper states that, to its knowledge, none of them carries a novelty attribute §1, and that few cover several concepts at document level §2.2 tab. 3.
• Superseded by: not stated in content already in this wiki.
• Citation: the repository README gives Briefings in Bioinformatics, 2022, doi 10.1093/bib/bbac282. This card reads the arXiv v2 PDF.