MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction

In plain words

General language models learn word meaning from large amounts of ordinary text. Materials science uses its own names, notations and abbreviations, and models trained on general text handle them poorly.

MatSciBERT starts from a model already trained on scientific text and keeps training it on a large collection of materials papers, so it learns that vocabulary. The result is then fine-tuned on small labelled tasks: finding material and property names in sentences, linking two entities in a sentence by the relation between them, and deciding whether an abstract belongs to the glass topic. The authors report it does better than the science model it started from on each task they tested.

TL;DR

• Continued pretraining of SciBERT on a materials corpus of about 285M words, followed by standard task-specific fine-tuning §2.2 tab. 1.
• Three NER heads (linear, CRF, BiLSTM-CRF), an entity-marker relation classifier and a CLS-based abstract classifier §2.4.
• Author-reported: MatSciBERT scores higher than SciBERT on every test cell of tab. 2, tab. 3, tab. 6 and tab. 7, and on all but one test cell of tab. 5 (a tie) §3.3.
• Code is GPL-3.0; checkpoints are on the Hugging Face hub under m3rg-iitd/matscibert.
• Reproducing the pretraining corpus is not possible from the released materials: the corpus is not in the repository.

Problem

General-domain and biomedical BERT-style models do not contain materials-science notation and jargon, so direct application to materials text may give suboptimal results §1. Materials papers are also mostly unstructured text, and annotated datasets for the field are scarce §3.4. Pretraining a model from scratch needs large compute and a large corpus, so the authors start from SciBERT instead §2.2.

Method

1. Corpus §2.1 tab. 1. The corpus covers four families: inorganic glasses and ceramics, bulk metallic glasses, alloys, and cement and concrete §2.1. Candidate papers come from a Crossref query (more than 1M hits) and full text from Elsevier ScienceDirect via its API, parsed from XML. Where only an abstract is available, the abstract is used §2.1. To filter relevant papers, the authors manually labelled 500 abstracts and fine-tuned one SciBERT classifier per family on them §2.1. The final corpus has 153,978 papers and about 284.5M words tab. 1 §3.1.

2. Text normalisation §2.1. Unicode characters that are noise are mapped to a space; similar characters are mapped to one standard form. The authors use a BERT normaliser from the Hugging Face tokenizers library, with their own mapping list normalize_text.py. Unmapped characters become spaces normalize_text.py.

3. Initialisation and vocabulary §2.2 §3.2. Weights are initialised from SciBERT (uncased) and tokenisation uses the SciBERT uncased WordPiece vocabulary. Section 2.2 also describes building a materials-specific vocabulary, which overlaps 53.64% with SciBERT's uncased vocabulary §3.2; the text then says the final model keeps SciBERT's vocabulary §3.2.

4. Pretraining recipe §2.2. The RoBERTa-style changes are: dynamic whole-word masking of 15% of words, no next-sentence-prediction loss, full-length sequences up to 512 tokens that may span several documents separated by [SEP], and larger batches. Training used AdamW, linear decay with warmup, and a peak learning rate of 1e-4 §2.2. Validation perplexity falls from about 3.8 to 3.112 over roughly 10 days fig. 2 §3.2.

5. NER heads §2.3.1 §2.4.1. Entities use BIO tags over words §2.3.1. Each word is represented by the encoder output of its first WordPiece §2.4.1. Three heads are compared: LM-Linear (softmax), LM-CRF (CRF in place of softmax), and LM-BiLSTM-CRF (a 2-layer BiLSTM with hidden size 300 between the encoder and the CRF) §2.4.1 §2.4.4.

6. Relation classification §2.3.1 §2.4.2. The input is a sentence with two entity spans. The spans are wrapped by the special markers [E1], [\E1], [E2], [\E2]. The output embeddings of the first and second entity markers are concatenated and passed through a linear layer with softmax, trained with standard cross-entropy §2.4.2.

7. Abstract classification §2.4.3. The CLS output of the whole abstract goes through a simple classifier head (the transformers BERT sequence-classification head) §2.4.3.

# Pretraining input packing  [§2.2] [code: pretraining/matscibert_pre_train.py#L77-L106]
# Input:  corpus lines (one text unit per line), max_len = 512
# Output: blocks of token ids, each starting with [CLS]
# Simplified: the code also pads the final partial block.

def pack_blocks(lines):
    blocks = []
    current = [CLS]
    for line in lines:
        ids = tokenize(line)
        # documents are separated by [SEP]; long lines are cut across blocks
        for piece in split_to_fit(ids, max_len - len(current)):
            current.extend(piece + [SEP])
            if len(current) == max_len:
                blocks.append(current)
                current = [CLS]
    return blocks

# NER labels and BiLSTM-CRF head  [§2.4.1] [§2.4.4]
# Input:  word_pieces (T), tags (one per word)
# Output: CRF negative log-likelihood (training)

def word_labels(word_pieces, tags):
    # only the first WordPiece of each word carries the label  [§2.4.1]
    labels = [IGNORE] * len(word_pieces)
    for word in words:
        labels[first_piece(word)] = tags[word]
    return labels

def bilstm_crf_loss(word_pieces, labels):
    # hidden states of the encoder's last layer  [code: ner/models.py#L91]
    states = encoder(word_pieces)
    # two dropout layers at 0.2 are in the code  [code: ner/models.py#L83-L85]
    states = bilstm(dropout(states), hidden=300, layers=2)
    scores = linear(dropout(states))
    # CRF scores only the labelled positions  [code: ner/models.py#L29-L39]
    return crf_nll(scores[labelled], labels[labelled])

Hyperparameters

Pretraining §2.2 unless stated.

Fine-tuning §2.4.4 unless stated.

Results

Author-reported. Setup: test set averaged over 3 seeds and, for SOFC and SOFC-Slot, 5 cross-validation splits. Validation scores in parentheses tab. 2 tab. 3 tab. 5. Baseline is SciBERT.

SOFC-Slot, Macro-F1 and Micro-F1 tab. 2:

SOFC, Macro-F1 and Micro-F1 tab. 3:

Matscholar, Micro-F1 and Macro-F1 tab. 5:

Abstract classification, glass vs non-glass, F1 tab. 6: MatSciBERT 93.57 (93.04 validation), SciBERT 91.04 (90.51).

Relation classification, Materials Synthesis Procedures tab. 7: MatSciBERT Macro-F1 87.87 (87.99), SciBERT 87.22 (87.21); Micro-F1 MatSciBERT 91.26 (91.40), SciBERT 91.04 (91.03). The paper says these numbers are not comparable with prior work, because that work assumes gold entity types and its label set includes an empty relation class §3.3.3.

Pretraining: final validation perplexity 3.112 §3.2 fig. 2. Macro-F1 for SOFC-Slot on the test set, tracked over pretraining days, trends upward but is not monotonic fig. 3.

Headline improvements in the text: on SOFC-Slot test, LM-BiLSTM-CRF is about 4.2 points higher in Macro-F1 and 2.1 in Micro-F1 than SciBERT §3.3.1. Against the best prior numbers the paper cites, it is about 3.3 points higher on SOFC-Slot and about 0.6 on SOFC, in Macro-F1 §3.3.1.

Per entity on the SOFC-Slot validation set, MatSciBERT scores higher on 15 of 17 types; SciBERT is higher on Current Density and Fuel Used tab. 4.

Paper vs code

Code: M3RG-IITD/MatSciBERT at commit 564750d9a8be0c058ee20f41def9c3de3718e079, GPL-3.0 (LICENSE). The owner is the M3RG research group at IIT Delhi, whose page links to co-author N. M. Anoop Krishnan's site. The repository README points to a 2022 npj Computational Materials article; this card follows the arXiv v1 text and did not read the journal version.

• Pretraining batch: paper says 256 per GPU §2.2; code uses per-device batch 16 with 8 accumulation steps, i.e. 128 per device per update matscibert_pre_train.py.
• Pretraining vocabulary: paper describes a materials WordPiece vocabulary and then says SciBERT's vocabulary is used §2.2 §3.2; code loads only the SciBERT tokenizer matscibert_pre_train.py.
• Pretraining corpus: paper describes the corpus construction §2.1; the repo does not contain the corpus or the annotation and classifier code. run.sh expects prepared files run.sh.
• Pretraining gradient clipping and epochs are set in code (max_grad_norm 0.0 and 30 epochs) and are not stated in the paper matscibert_pre_train.py.
• NER learning rates, epoch counts, seeds and warmup match the paper ner.py ner.py ner.py ner.py.
• NER BiLSTM: the code also applies dropout 0.2 on the encoder output before the LSTM and before the output layer. The paper mentions dropout only between LSTM layers models.py §2.4.4.
• NER CRF: the code sets invalid BIO transitions to a large negative value (−10000) at initialisation, which the paper does not mention models.py.
• NER model selection: code keeps the epoch with the best validation Macro-F1 for SOFC and SOFC-Slot and Micro-F1 for Matscholar. The paper does not say which metric picks the epoch ner.py ner.py.
• SOFC-Slot labels: the code maps experiment_evoking_word to O at prediction time and excludes it from the averaged metric ner.py ner.py. The paper says this type is used for training §3.3.1. The code also maps interconnect_material to O at data loading, which the paper does not mention ner_datasets.py.
• Relation context: the paper defines the input as a sentence §2.3.1. The code uses the whole line, and drops any pair whose two entities are on different lines relation_classification.py.
• Relation batch: the paper states batch size 16 for all architectures §2.4.4. The code adds gradient accumulation of 4, so 64 per update relation_classification.py. The code also uses dropout 0.1 before the head and 10 epochs relation_classification.py relation_classification.py.
• Abstract classification metric: the code keeps the epoch with the best validation accuracy and reports accuracy lists, though it also computes F1. The paper reports F1 cls.py cls.py §2.4.5.
• Baselines: the code runs SciBERT and bert-base-uncased with no text normalisation; only the MatSciBERT path normalises ner.py cls.py. The paper does not say that baselines differ in preprocessing. The paper's tables show SciBERT only.
• Learning-rate grid and non-LM learning rate match the paper ner.py ner.py.

Pitfalls

• The pretraining corpus is not released. Reproducing it needs Elsevier API access, the manual abstract labels and the SciBERT family classifiers, none of which are in the repository §2.1 run.sh.
• Two vocabularies are described. Implement with the SciBERT uncased tokenizer, which the code uses matscibert_pre_train.py.
• The baseline comparison is not like-for-like. MatSciBERT inputs are normalised; SciBERT and BERT inputs are not ner.py. Gains may partly come from preprocessing.
• Gradient clipping differs between stages. Pretraining disables clipping matscibert_pre_train.py. Fine-tuning does not set max_grad_norm, so the Hugging Face Trainer default applies.
• Hyperparameters beyond those listed are in code only. Several fine-tuning details are not in the paper: input and output dropout, invalid-transition masking, 512-token truncation, and per-task epoch counts for relation and classification models.py ner.py cls.py.
• Variance is not reported. Tables give means over 3 seeds with no spread. Gaps below about one point cannot be judged from the paper, for example the Matscholar Macro-F1 tie for LM-BiLSTM-CRF in tab. 5.
• Figures are described in the text as trends. Figure 3 shows dips, including a sharp drop late in training, so "increasing trend" is a summary, not a monotone result fig. 3 §3.2.
• Section 3.4 applications are illustrative. Entity counts from figure captions and the example relation/QA use cases have no evaluation §3.4.
• Relation results are not comparable to prior work. The authors say their label set and entity-type assumptions differ from earlier work §3.3.3.
• Model size is not stated in the paper. The code initialises from allenai/scibert_scivocab_uncased ner.py matscibert_pre_train.py; parameter count is not given.
• Weights and licences. Fine-tuned models are on Zenodo (DOI 10.5281/zenodo.6413296) and the pretrained checkpoint is on the Hugging Face hub under m3rg-iitd/matscibert README.md. The licence of the checkpoints was not checked here; check before reuse.
• Copyleft code. The repository is GPL-3.0. Only paths and line numbers are cited here; no code is copied.

Lineage

• Builds on: SciBERT, arXiv:1903.10676, initialisation and vocabulary §2.2; BERT, arXiv:1810.04805, the base architecture §1; the RoBERTa training recipe, arXiv:1907.11692 §2.2; BioBERT-style domain continued pretraining §3.2 (Lee et al., 2019, Bioinformatics, no arXiv id in the paper).
• Compared against: SciBERT, the only baseline in the paper's tables tab. 2 tab. 3 tab. 5 tab. 6 tab. 7. The code also runs bert-base-uncased run.sh.
• Superseded by: not known from content already in this wiki.