KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media

In plain words

Some tweets contain insults or hateful language, and the authors wanted a program that flags them automatically. Their starting point is BERT, a language model that has already read a large amount of text and so has a general sense of how words fit together. They fine-tune BERT for each language and add a small pattern detector (a convolutional neural network) on top. The detector looks at short runs of words in BERT's internal representations and outputs a yes or no for offensive.

The work was an entry in a shared competition that covered three languages: Arabic, Greek and Turkish. Because no BERT model existed for Arabic at the time, the team trained four new Arabic BERT models and released them publicly.

TL;DR

• Binary classifier for offensive versus non-offensive tweets, subtask A of OffensEval 2020, in Arabic, Greek and Turkish §1 §3.
• Model: a language-specific BERT-base whose last four layers feed a CNN with five filter widths and a sigmoid output §4.4 fig. 1.
• Author-reported macro-F1 on the test sets: 0.897 Arabic, 0.843 Greek, 0.814 Turkish; ranked 4th, 4th and 3rd §1 tab. 2.
• Against BERT alone, the average gain is 0.010, but Turkish is slightly lower tab. 2.
• Several training settings in the released code differ from the paper; see Paper vs code.

Problem

Offensive and hateful speech on social media is increasing. The authors say manual control and rule-based filters are slow or error-prone when the full context of a text is not considered §1.

The task is to label each tweet as Offensive or Non-offensive §3. The data comes from the organisers of OffensEval 2020 for three languages. Table 1 gives the counts per language and split tab. 1. The provided training data had no dev split, so the authors split it 90% train and 10% dev §3.

Each tweet is one input of at most 64 tokens §4.4. Evaluation uses macro-averaged F1 on the test set §5.

Table 1 (data, copied from the paper):

Source: tab. 1. Positive means Offensive §3.

Method

Preprocessing §3.1. Hashtags are turned into words by splitting them at case changes. Greek text is lowercased and its diacritics are removed. Each text is then tokenised with the WordPiece tokenizer that belongs to its BERT model §3.1. The code applies the case-based split to all text, not only hashtags (see Pitfalls) bert-cnn.py bert-cnn.py.

Encoders §4.2 §4.3. Each language has its own BERT-base model. Arabic uses ArabicBERT (base), which the authors trained themselves. Greek uses GreekBERT, and Turkish uses BERTurk. Multilingual BERT is used only as a baseline. The code names the checkpoints: Turkish dbmdz/bert-base-turkish-cased, Greek nlpaueb/bert-base-greek-uncased-v1, Arabic asafaya/bert-base-arabic bert-cnn.py. The paper does not give checkpoint names.

BERT-CNN §4.4 fig. 1. The model has two parts: a BERT-base encoder with 12 self-attention layers, and a CNN used as the classifier §4.4.

1. The tokens go through BERT. The outputs of the last four layers are taken. Fig. 1 labels them L9 to L12 fig. 1.
2. The four outputs are stacked as four channels, giving a 768 × 4 × 64 input per tweet §4.4.
3. Each of 160 filters is a convolution over consecutive tokens. Filters come in five heights, 1 to 5 tokens, with 32 filters per height. A filter spans all 768 dimensions and all four channels §4.4 fig. 1.
4. A ReLU follows each convolution, then a global max-pool keeps one value per filter §4.4.
5. The 160 values are concatenated, passed to a dense layer and a sigmoid, and give the probability of Offensive §4.4.

The paper does not describe dropout, the loss, or the threshold. The code adds dropout 0.1 before the dense layer and a 0.5 threshold on the output models.py bert-cnn.py.

ArabicBERT §4.3. Four models are trained (Large, Base, Medium, Mini) with masked language modelling and whole-word masking, for 4M steps on the same data §4.3. The corpus is an unshuffled version of OSCAR plus a Wikipedia dump, about 8.2 billion words. The WordPiece vocabulary has 32,000 entries. Non-Arabic characters are lowercased. There is no cased model. The corpus includes some dialectal Arabic §4.3.

# BERT-CNN forward pass  [§4.4] [fig. 1]
# Input:  token_ids, mask, type_ids: (B, 64) integer tensors
# Output: p_offensive: (B, 1), a probability

def forward(token_ids, mask, type_ids):
    # 1. Run BERT-base and keep the hidden states of every layer  [§4.4]
    hidden_states = bert(token_ids, mask, type_ids)
    # 2. Keep the outputs of the last four layers, L9 to L12  [§4.4] [fig. 1]
    last_four = hidden_states[-4:]
    # 3. Stack them as four input channels  [code: models.py#L71]
    maps = stack(last_four, axis=channel)        # (B, 4, 64, 768)
    features = []
    # 4. For each height K in 1..5, apply 32 filters  [§4.4]
    # [code: models.py#L61-L62]
    for K in (1, 2, 3, 4, 5):
        # 5. ReLU, then max over token positions, one value per filter  [§4.4]
        # [code: models.py#L72-L73]
        conv_out = relu(conv[K](maps))           # (B, 32, 64 - K + 1)
        features.append(max_over_positions(conv_out))    # (B, 32)
    # 6. Concatenate the five groups into 160 values  [§4.4]
    pooled = concat(features, axis=1)            # (B, 160)
    # 7. Dropout 0.1 (code only), dense layer, sigmoid
    # [code: models.py#L64-L77]
    pooled = dropout(pooled, p=0.1)
    return sigmoid(dense(pooled))                # (B, 1)

# Fine-tuning and checkpoint selection  [§4.4] [code: bert-cnn.py#L100-L163]
# Only the epoch count and the selection metric are stated in the paper.

def train(train_set, dev_set, epochs):
    # 1. Train all weights, BERT included, with AdamW at lr 2e-5  [§4.4]
    # [code: bert-cnn.py#L100]
    optimizer = adamw(params, lr=2e-5, weight_decay=0.9)
    # 2. Binary cross-entropy on the sigmoid output  [code: bert-cnn.py#L101]
    loss_fn = binary_cross_entropy
    # 3. Linear decay to zero, no warmup  [code: bert-cnn.py#L109-L111]
    scheduler = linear_decay(optimizer, warmup_steps=0, total_steps=steps)
    best_f1 = 0
    # 4. Train for a fixed number of epochs  [§4.4]
    # [code: bert-cnn.py#L170] runs 6 epochs; the paper says 10
    for epoch in range(epochs):
        for batch in shuffle(train_set):
            loss = loss_fn(forward(batch.inputs), batch.labels)
            loss.backward()
            optimizer.step()
            scheduler.step()
            model.zero_grad()
        # 5. Keep the epoch with the best dev score  [§4.4]
        # [code: bert-cnn.py#L149] computes binary F1, not macro F1
        dev_f1 = f1(dev_labels, predict(dev_set) >= 0.5)
        if dev_f1 > best_f1:
            best_f1 = dev_f1
            save(model)
    # 6. Predict the test set with the saved checkpoint
    # [code: bert-cnn.py#L175]
    return load_saved(model)

Hyperparameters

Results

Author-reported. Setup: OffensEval 2020 subtask A, test sets for Arabic, Greek and Turkish, macro-averaged F1 §5 tab. 2. Each BERT model is language-specific except mBERT. Hardware is not stated in the paper. The paper reports single numbers with no repeated runs or variance §5.

Ranking: 4th in Arabic, 4th in Greek and 3rd in Turkish, among more than 40 participants §1.

Table 2 (copied from the paper; values are macro-F1 on the test data):

Source: tab. 2. The authors note that BERT used language-specific models tab. 2 footnote.

Differences from BERT alone, computed from tab. 2: Arabic +0.013, Greek +0.021, Turkish −0.002, average +0.010.

Paper vs code

The official repository is alisafaya/OffensEval2020, linked from the paper's footnote §1. Its owner matches the first author's name and the ArabicBERT repository the paper links §4.3. Commit 2e166bc was checked. It has no LICENSE file.

• Epochs. Paper: 10 §4.4. Code: evaluate() calls training with n_epochs=6 bert-cnn.py. The function default is 10 bert-cnn.py.
• Checkpoint selection. Paper: best macro-averaged F1 on dev §4.4. Code: sklearn f1_score with default arguments, which is binary F1 on the positive class bert-cnn.py.
• Unstated settings in the paper, set in code. AdamW with weight decay 0.9 bert-cnn.py; linear decay with no warmup bert-cnn.py; batch size 24 × 6 = 144 bert-cnn.py; dropout 0.1 models.py; BCE loss bert-cnn.py; threshold 0.5 bert-cnn.py; seed 1234 bert-cnn.py.
• Dev split. Paper: 90/10 §3. Code takes the first 10% of the file in file order, with no shuffle bert-cnn.py.
• Preprocessing. Paper: hashtags are split into words §3.1. Code applies a regex that inserts a space at each lower-to-upper case change, and replaces # with a space, to all text bert-cnn.py.
• Seed timing. The model head is built before the seeds are set bert-cnn.py bert-cnn.py. The head's random initialisation is therefore not controlled by the seed. This is inferred from the order of calls.
• CNN-Text baseline. Paper: "same structure as the main model", with random embeddings of size 300 §5. Code: 36 filters per height, heights 1, 2, 3 and 5 models.py. This differs from the main model's 32 per height, heights 1 to 5.
• BiLSTM baseline. Paper: two BiLSTM layers, hidden size 128, embedding 300 §5. Code: hidden size 128 and embedding 300 models.py, but one bidirectional layer, since num_layers is not set models.py.
• BERT baseline. Code trains 5 epochs, batch 256, gradient clipping at 1.0 bert.py bert.py bert.py. It selects by macro-F1 bert.py. The paper gives none of these values.
• SVM baseline. Paper: CountVectorizer with 3000 features, TF-IDF, SVM §5. Code: the same, with an SVC using a linear kernel and C=10 tfidfsvm.py. The paper does not state C.
• Checkpoint names. Paper: names GreekBERT (with a URL) and BERTurk (cited, no checkpoint id) §4.2. Code: dbmdz/bert-base-turkish-cased for Turkish bert-cnn.py.

Pitfalls

• Selection metric. Dev selection uses binary F1 in the code, but macro-F1 is stated in the paper. The baseline script uses macro-F1, so the two models were selected differently bert.py bert-cnn.py.
• Turkish. BERT-CNN is slightly below BERT alone on Turkish (0.814 against 0.816) tab. 2. The average gain of 0.010 comes mostly from Arabic and Greek.
• No variance. The paper reports single numbers and does not repeat runs §5. Seed variance is not measured, so the size of the gains over BERT alone is uncertain.
• Unstated training setup. Optimizer, weight decay, schedule, batch size, dropout, loss and threshold are not in the paper. An implementer who follows the paper alone will not reproduce the code's settings. The values above come from code only.
• Epoch count. The paper says 10 epochs; the code uses 6 bert-cnn.py. Reported numbers may come from either setting; the paper does not say which was used for the submitted runs.
• Truncation. The paper states a maximum of 64 tokens but not whether longer tweets are truncated §4.4. The code passes max_length=64 with pad_to_max_length and does not set a truncation flag explicitly bert-cnn.py. The behaviour depends on the installed transformers version. The repository has no requirements file.
• Layer indexing. The paper says "last four hidden layers", and Fig. 1 labels them L9 to L12 §4.4 fig. 1. The code takes the last four entries of the hidden-states tuple models.py. The code does not name the layers, so L9 to L12 is inferred from the figure and from the library's output layout, not from the code itself.
• Camel-case split. The regex also splits ordinary capitalised words, for example McDonald into two pieces bert-cnn.py. The paper describes hashtags only §3.1.
• Dev split order. The dev set is the first 10% of the training file, with no shuffle or stratification bert-cnn.py. The paper does not say how the split was made. Results depend on file order.
• Fixed GPUs. The code uses GPUs 2 to 7 by default bert-cnn.py. Batch size is computed from the GPU count bert-cnn.py. On a machine with other GPU counts, the effective batch size changes.
• Missing data. The scripts read OffensEval/<lang>.<split>.tsv data.py. Data files are not in the repository. The organisers' data is needed.
• Baseline script. bert.py uses pretrained_model at line 183, but the only definitions are commented out bert.py bert.py bert.py. As shipped, it fails with an undefined name.
• ArabicBERT claim. The paper says dialect text in the corpus "boosted" performance on social media data §4.3. No ablation is shown, so this is an unsupported claim.
• Licence. The repository has no LICENSE file, so its code can only be cited by path and line, never quoted or copied.

Lineage

• Builds on: BERT, cited as Devlin et al. 2019, for the encoder §4.2. The paper's bibliography has no arXiv id, and there is no card in this wiki for it.
• Builds on: Kim 2014, for the CNN text classifier §4.1. No card in this wiki.
• The paper cites Devlin et al. for the choice of the last four layers §4.4 and Li et al. 2019 for the claim that BERT plus CNN beats either alone §1.
• Compared against: SVM with TF-IDF, multilingual BERT, Bi-LSTM, CNN-Text and language-specific BERT §5 tab. 2.