COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter

In plain words

BERT is a language model first trained on general text such as Wikipedia and books. You can then adapt it to a specific job with a small labelled dataset. The authors took BERT-Large and kept training it on a large set of tweets about the coronavirus, collected with COVID-19 keywords. Before training, they replaced usernames and web links with generic placeholders and turned emojis into plain-text codes, so the model sees a steady style of text.

The result, CT-BERT, is meant as a drop-in replacement for BERT-Large when a task involves social-media text about COVID-19. On five classification tasks it scored higher than BERT-Large. The gain was largest on COVID-19 tweets and smaller on the other sets. These numbers come from the authors' own experiments.

TL;DR

• CT-BERT is BERT-Large (uncased, whole-word masking) trained further on 22.5M cleaned tweets about COVID-19 §2.
• Pretraining uses masked-token and next-sentence objectives, sequences of 96 tokens and batches of 1024, for 500,000 steps §2 §3.1.
• After fine-tuning on five classification datasets, mean F1 is 0.833 against 0.802 for BERT-Large (author-reported) tab. 2.
• The gain is largest on the COVID-19 category set (ΔMP 25.88%) and smallest on SemEval (8.97%) tab. 2.
• The released code differs from the paper on the learning-rate schedule, the base checkpoint in one script, and the step count. Read "Paper vs code" before reimplementing.

Problem

General models such as BERT are trained on broad text, which holds little detail about any one subdomain §1. Domain-specific models such as BioBERT and SciBERT usually start from general weights and continue pretraining on in-domain text §1. Training from scratch needs a very large corpus, so continued pretraining is the common route §1. The authors assume the target domain has a large unlabelled corpus and that downstream tasks are classification tasks; they did not test other kinds of language understanding §4.

Method

CT-BERT is BERT-Large (English, uncased, whole-word masking) with further pretraining on in-domain tweets §1 §2.

1. Corpus. Tweets were collected from 12 January to 16 April 2020 with a filter on English COVID-19 keywords (160M tweets). Retweets, duplicates and near-duplicates were removed, leaving 22.5M tweets (0.6B words) §2.
2. Cleaning. Each tweet is one document. Usernames and URLs are replaced by one shared token each. Emojis become text aliases §2.
3. Sentences. Tweets are split into sentences with spaCy §2.
4. Instances. Each sequence has at most 96 tokens, drawn from a 30,000-word vocabulary (see Pitfalls). The duplication factor is 10, giving 285M training examples and 2.5M validation examples §2. Each pass over the corpus draws new masks and sentence pairings create_pretrain_data.py.
5. Pretraining. Masked language modelling and next-sentence prediction, with batch size 1024 and a constant learning rate of 2e-5 §2 §3.1. A checkpoint is saved every 100,000 steps §2. The final model is at 500,000 steps, about 512M examples or 1.8 passes over the data §3.1.
6. Fine-tuning. Each dataset is trained with learning rate 2e-5 for a dataset-specific number of epochs §2.1. The metric is F1 on the dev set §3.2.

Hardware: TensorFlow 2.2 on a TPU v3-8 for 120 hours §2.

# Step 1: clean each raw tweet  [§2]
# [code: utils/preprocess.py#L39-L62]
def clean_tweet(text):
    # Usernames and URLs become one shared token each  [§2]
    text = replace_usernames(text, filler="twitteruser")
    text = replace_urls(text, filler="twitterurl")
    # Emojis become text aliases such as :smile:  [§2]
    text = asciify_emojis(text)
    return text

# Step 2: build masked sentence pairs from the corpus  [§2]
# Input:  documents, each a list of sentences; dupe_factor = 10  [§2]
# Output: shuffled list of masked pairs, at most 96 tokens  [§2]
def make_instances(documents, dupe_factor):
    instances = []
    # Each pass draws fresh pairings and masks
    # [code: preprocess/create_pretrain_data.py#L128-L139]
    for _ in range(dupe_factor):
        for document in documents:
            # Pack sentences into segment A, then segment B
            # Half of the pairs take B from another document
            # [code: preprocess/pretrain_helpers.py#L215]
            pair = pack_pair(document, max_tokens=96)
            # Mask about 15% of tokens, at most 14 (paper is silent)
            # [code: preprocess/create_pretrain_data.py#L174-L176]
            instances.append(mask_tokens(pair, rate=0.15, cap=14))
    return shuffle(instances)

# Step 3: choose masked positions and corrupt them
# [code: preprocess/pretrain_helpers.py#L303-L345]
def mask_tokens(pair, rate, cap):
    # Predict rate x length tokens, at least 1 and at most cap
    # [code: preprocess/pretrain_helpers.py#L313-L314]
    count = min(cap, max(1, round(len(pair) * rate)))
    masked = []
    # Whole-word masking: wordpieces of one word are chosen together
    # [code: preprocess/pretrain_helpers.py#L303-L307]
    for position in choose_positions(pair, count):
        # 80% become [MASK], 10% stay as they are, 10% become a random token
        # [code: preprocess/pretrain_helpers.py#L337-L345]
        masked.append(corrupt(pair, position))
    return masked

# Step 4: continue pretraining BERT-Large  [§2] [§3.1]
# Start from BERT-Large uncased, whole-word masking  [§2]
def pretrain(model, instances):
    for step in range(500000):
        # Batches of 1024 sequences  [§2]
        batch = next_batch(instances, size=1024)
        # Masked-token and next-sentence losses  [§3.1]
        losses = mlm_and_nsp_losses(model, batch)
        # Learning rate 2e-5, held constant  [§2]
        # The released code uses warmup and linear decay instead
        update(model, losses, learning_rate=2e-5)
        # Save a checkpoint every 100,000 steps  [§2]
        if (step + 1) % 100000 == 0:
            save_checkpoint(model, step + 1)

# Step 5: fine-tune and score one labelled dataset  [§2.1] [§3.2]
def finetune_and_score(model, train, dev, epochs):
    # Learning rate 2e-5 on every dataset  [§2.1]
    # Epochs: 3 (SST-2, CC, SE), 5 (VC), 10 (MVS)  [§2.1]
    for _ in range(epochs):
        train_one_epoch(model, train, learning_rate=2e-5)
    # F1 on the dev set; the averaging is not stated  [§3.2]
    return f1_score(model, dev)

# Relative gain over the baseline  [§3.2] [tab. 2]
def delta_mp(f1_baseline, f1_model):
    # Gain divided by the room the baseline leaves; see Pitfalls  [§3.2]
    return (f1_model - f1_baseline) / (1 - f1_baseline)

Key equation, unnumbered in the paper §3.2. The paper prints the numerator as baseline minus model. The values in tab. 2 only fit model minus baseline, so the form above is used here (see Pitfalls):

\Delta\text{MP} = \frac{F_{1,\,\text{CT-BERT}} - F_{1,\,\text{BERT-Large}}}{1 - F_{1,\,\text{BERT-Large}}} §3.2

Figures: fig. 1 plots the masked-token and next-sentence loss and accuracy against pretraining steps, sampled every 25k steps with 1k-step validation evaluations §3.1. fig. 2 plots ΔMP for each dataset at checkpoints from step 0 (BERT-Large) to 500k, with the standard error over 10 repeated fine-tuning runs §3.3.

Hyperparameters

Values stated in the paper are cited to the paper. "Not stated" means the paper gives no value; code values are listed only as code-only and are cited to the repository.

Results

Author-reported. Setup: five classification datasets §2.1 tab. 1, each fine-tuned from BERT-Large or CT-BERT with the epochs above, on the dev split §3.2. The baseline is BERT-Large with the same recipe, and the number of fine-tuning runs behind tab. 2 is not stated.

Paper vs code

Code: linked from the paper's data-availability section and footnote §1 §5. The repository is digitalepidemiologylab/covid-twitter-bert at commit 86909698f104525b13d8ab0e22ae666e264ac5c6, MIT licence. The GitHub organisation name matches the Digital Epidemiology Lab at EPFL, which two of the authors list as their affiliation (author block).

1. Learning-rate schedule. Paper: constant 2e-5 §2. Code: warmup of 10,000 steps by default, then linear decay to end_lr = 0 run_pretrain.py run_pretrain.py optimizer.py. The pretraining README passes --learning_rate 2e-5 --end_lr 0 and no warmup flag README_pretrain.md.
2. Base checkpoint. Paper: whole-word-masking BERT-Large §2. The pretraining README and the default model class use that model README_pretrain.md config.py. The shell script scripts/run_pretrain.sh loads the non-WWM uncased_L-24_H-1024_A-16 checkpoint run_pretrain.sh. The released weights' origin is not stated.
3. Pretraining length. Paper: stopped at 500,000 steps §3.1. The shell script sets 10 epochs of 10,000 steps, or 100,000 steps run_pretrain.sh. The README command sets 1 epoch and no step count, so the default of 1,000 steps per epoch applies README_pretrain.md run_pretrain.py.
4. Masking and pairing details. Not in the paper: masking rate 0.15, cap 14 predictions, short-sequence probability 0.1, 80/10/10 replacement, and half of the pairs using a random next sentence create_pretrain_data.py pretrain_helpers.py pretrain_helpers.py. The README uses the same rate and short-sequence probability README_pretrain.md.
5. Vocabulary. Paper: "30000-word" vocabulary §2. Config vocab_size is 30,522 bert_config_covid_twitter_bert.json. The vocabulary file is bert-large-uncased-whole-word-masking-vocab.txt config.py.
6. MVS epochs. Paper: 10 epochs §2.1. The shell script for the MVS run (SHEET is maternal_vaccine_stance) sets --num_train_epochs=3 run_finetune.sh run_finetune.sh.
7. Fine-tuning optimizer. Paper: not stated §2.1. Code: AdamW with weight decay 0.01, 10% warmup, linear decay, gradient clipping at global norm 1.0 run_finetune.py optimizer.py optimizer.py.
8. Preprocessing beyond the paper. Paper describes usernames, URLs and emojis §2. Code also lowercases for the uncased model, removes Unicode "So" symbols, and standardises punctuation with transliteration; all three default to on preprocess.py preprocess.py preprocess.py prepare_pretrain_data.py.
9. Retweet and duplicate filtering. Paper: removed §2. I found no retweet or duplicate filter in the released preprocessing, which processes the files it is given prepare_pretrain_data.py.
10. F1 averaging. Paper: "mean F1" §3.2. Code computes micro, macro, weighted and per-label F1 finetune_helpers.py finetune_helpers.py. The README sample reports f1_macro README.md. The paper does not say which one.
11. Mixed precision. set_mixed_precision_policy refers to loss_scale in its fp16 branch, and that name is not defined in the function run_finetune.py run_pretrain.py. I did not run the code, so this is from reading only.

Pitfalls

• ΔMP formula. As printed, the numerator is baseline minus model §3.2, which gives negative values for a gain. The values in tab. 2 match model minus baseline over (1 − baseline) within about half a point, which is consistent with the three-decimal inputs. Use that form.
• Vocabulary size. The paper says 30,000 words §2 and the config gives 30,522 bert_config_covid_twitter_bert.json. The vocabulary file is the one that matters for tokenisation; take its size, not the paper's.
• Twitter-only claim. The paper says all datasets are Twitter-related §2.1 and then says SST-2 is not from Twitter §2.1. The SST-2 result is a non-Twitter control.
• Unpublished data. The MVS dataset is unpublished §2.1, so its result cannot be reproduced from the paper's own materials. The Data Availability statement covers the model, code and public datasets only §5. The 22.5M-tweet pretraining corpus is not in the repository.
• Tuning bias. Fine-tuning epochs and learning rate were tuned on BERT-Large and reused for CT-BERT. The authors say the fine-tuning recipe is not optimised for CT-BERT and that there is room for improvement §2.1 §4.
• Statistics. tab. 2 has no variance or significance test, and the number of fine-tuning runs behind it is not stated. The checkpoint study runs 10 repeats per checkpoint §3.3. The discussion calls the gains significant on all datasets §4, but no test is reported.
• Costs. Pretraining took 120 hours on a TPU v3-8 §2. The evaluation study used 300 fine-tuning runs, which the authors say costs as much as pretraining §4.
• Stopping point. Masked-token and next-sentence metrics did not show when to stop pretraining, so the stop at 500,000 steps is a judgement §3.1. Gains on downstream tasks flatten after about 200k steps §3.3.
• Out-of-domain text may need longer pretraining to show a similar gain §4.
• External code. The code imports TensorFlow's official BERT modules from a sibling tensorflow_models checkout run_pretrain.py. No version is pinned in this card.
• Test split. The authors kept a 20% test split unused §2.1, so reported results come from the dev split only.

Lineage

• Builds on: BERT, arXiv:1810.04805 §1. No card for it in this wiki yet.
• Compared against: BioBERT and SciBERT, which are continued-pretraining models for biomedical and scientific text §1. SciBERT is arXiv:1903.10676 [refs.bib]. Related general models named in the introduction are RoBERTa and ALBERT §1.
• Superseded by: not stated.