Paper · 2020-10-11 · cs.CL, cs.LG
Multilingual Offensive Language Identification with Cross-lingual Embeddings
Fine-tunes XLM-R large to classify offensive posts, then starts training for Hindi, Bengali and Spanish from weights learned on English OLID. Author-reported macro F1: 0.8415 Bengali, 0.8568 Hindi, 0.7513 Spanish, above the best shared-task systems on those sets.
Transfer learningText classificationToxicity detectionmultilingualcross-lingualtransfer-learningoffensive-language
arXivPDFCodeMarkdown10 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Abusive posts on social media need to be found, but most labelled examples of them are in English. This paper asks whether a model that learns to spot offensive posts in English can give a head start in other languages.
The authors use a large language model that was trained on text in 104 languages. They first train it on an English collection of offensive and non-offensive tweets. Then, for a new language, they start training from those learned weights instead of from scratch. For Bengali the label set is different, so only the core of the model is carried over.
On three languages the authors report results above the best systems from the shared tasks that published those datasets. Those numbers are the authors' own measurements. The paper's text also contains some numbers that disagree with its table, and the released code does not reproduce every setting described. The Pitfalls section lists both.
TL;DR
Ranasinghe and Zampieri fine-tune XLM-R large, a cross-lingual transformer, on the English OLID data §3 §4.1. They reuse those weights to start training for Hindi, Spanish and Bengali, with a variant for Bengali whose labels differ §4.2. Author-reported macro F1 is 0.8415 for Bengali, 0.8568 for Hindi and 0.7513 for Spanish, above the shared-task best systems §5 tab. 2. Relevant if you need offensive-language classification in a language with little labelled data. The released code does not contain a working Spanish or Bengali transfer path (see Paper vs code).
Problem
Offensive posts cause harm to users, and platforms need to set them aside or remove them §1. Most annotated offensive-language datasets and most prior systems are English §1 §2. Deep models usually need large annotated sets, which low-resource languages often lack §2. The goal is to carry knowledge from a resource-rich English task into other languages and into a different domain and label set §1 §3.
The English source is OLID, annotated with a three-level taxonomy. The authors use only its first level, offensive vs. non-offensive, and argue that it maps onto the labels of the target datasets §3.
Method
The method has two parts: a classifier and a transfer procedure. XLM-R is described as trained on 104 languages and as strong on cross-lingual benchmarks §4. Multilingual BERT (called BERT-m) is the comparison model §4 §5.
Classifier §4.1
- Feed the text to XLM-R large. The input is at most 512 tokens §4.1.
- Take the final hidden state of the first token as the representation of the whole text §4.1.
- Apply a softmax layer with a task-specific weight matrix to get label probabilities §4.1 eq. 1.
- Fine-tune all XLM-R parameters and the matrix together, maximising the log-probability of the correct label §4.1. The repository uses cross-entropy loss for this roberta_model.py.
p(c \mid \mathbf{h}) = \mathrm{softmax}(W \mathbf{h}) eq. 1
The architecture is shown in fig. 2.
Transfer strategies §4.2
The source model is trained on OLID level A. Two strategies then start the target-language model from its weights §4.2 fig. 1:
- Inter-language transfer (Hindi, Spanish). Save all weights from the English model, including the softmax layer, and use them to initialise the target-language model §4.2. Both target sets have two labels §3 tab. 1.
- Inter-task and inter-language transfer (Bengali). Save only the encoder weights and do not reuse the softmax layer, because the Bengali task has three classes instead of two §4.2 §3 tab. 1.
Datasets §3
| Language | Instances | Source platform | Label structure (paraphrased) |
|---|---|---|---|
| Bengali | 4,000 | three classes: overt aggression, covert aggression, none | |
| English (OLID) | 14,100 | two classes: offensive or not | |
| Hindi | 8,000 | two classes: hateful offensive or not | |
| Spanish | 6,600 | two classes: hateful or not |
Source: tab. 1 §3. Bengali comes from TRAC-2, Hindi from HASOC 2019 and Spanish from HatEval 2019 §3.
Procedure
# Transfer from an English source to a target language [§4.2]
# Input: english_train, target_train, target_test (text, label)
# Output: predicted label for each target_test text
def transfer_classify(english_train, target_train, target_test):
# Start XLM-R large with a softmax head [§4.1]
source_model = new_xlmr_large(num_labels=2)
# Fine-tune all weights on English OLID level A [§3] [§4.2]
source_model = fine_tune(source_model, english_train)
# Always keep the encoder weights [§4.2]
saved = source_model.encoder_weights()
# Keep the softmax weights only when the labels match [§4.2]
if same_label_set:
saved = source_model.all_weights()
# Start the target model from the saved weights [§4.2]
target_model = new_xlmr_large(num_labels=len(target_labels))
target_model.load(saved)
# Fine-tune all weights on the target language [§4.2]
target_model = fine_tune(target_model, target_train)
# Predict on the target test texts [§4.1]
return target_model.predict(target_test)Key equation: eq. 1 above. No algorithm box is given in the paper.
Hyperparameters
Values are either stated in the paper or taken from the repository. Repository values are for the XLM-R configuration of each language. Paper values apply to all languages unless noted.
| Name | Value | Where | Notes |
|---|---|---|---|
| model | XLM-R large | §4.1 bengali_deepoffense_config.py | Hindi starts from an English checkpoint, see Paper vs code |
| learning rate | 1e-5 | §4.2 bengali_deepoffense_config.py | Paper: tuned by hand on the validation set |
| epochs | 3 | §4.2 bengali_deepoffense_config.py | Paper: all languages. Code English config uses 6 |
| max sequence length | not stated (paper allows 512) | §4.1 bengali_deepoffense_config.py | Code: 128 for Bengali, Hindi, Spanish; 256 for English |
| train batch size | not stated | bengali_deepoffense_config.py | Code value: 8 |
| eval batch size | not stated | bengali_deepoffense_config.py | Code value: 8 |
| optimizer | not stated | classification_model.py | Code: AdamW, epsilon 1e-8 (config line 32) |
| learning-rate schedule | not stated | classification_model.py | Code: linear decay with 6% warmup (config line 33) |
| weight decay | not stated | bengali_deepoffense_config.py | Code value: 0 |
| gradient clipping | not stated | bengali_deepoffense_config.py | Code value: max norm 1.0 |
| train/validation split | 0.8:0.2 | §4.2 | Code: 0.9:0.1 per fold, see Paper vs code |
| folds | not stated | bengali_deepoffense_config.py | Code: 3, majority vote over folds |
| evaluation interval | not stated | bengali_deepoffense_config.py | Code: every 60 steps (Bengali, Hindi), 300 (English) |
| early stopping | not stated | bengali_deepoffense_config.py | Code: patience 10 on eval loss |
| random seed | not stated | bengali_deepoffense_config.py | Code: 777; fold split seed 777 × fold index |
| hardware | one Nvidia Tesla K80 GPU | §4.2 | About 1 h for English, about 30 min for the other languages |
The paper states learning rate 1e-5 and 3 epochs as the best values found, "for all the languages" §4.2.
Code path (not in the paper)
# Code path: three folds, majority vote over fold predictions
# Input: train (text, label), test_texts, init_checkpoint
# Output: one label per test text
def fold_vote(train, test_texts, init_checkpoint, seed=777):
# One column of predictions per fold [code: bengali_deepoffense.py#L65]
votes = zeros(len(test_texts), 3)
for fold in range(3):
# New model each fold [code: bengali_deepoffense.py#L72-L73]
model = new_model(init_checkpoint)
# 90/10 split, seeded by seed * fold [code: bengali_deepoffense.py#L74]
train_part, eval_part = split(train, 0.1, seed * fold)
# Keep lowest eval loss [code: classification_model.py#L762-L769]
model = train(model, train_part, eval_part)
# Reload best, predict [code: bengali_deepoffense.py#L76-L79]
votes[:, fold] = predict(load_best(model), test_texts)
# Majority vote over the three folds [code: bengali_deepoffense.py#L82-L87]
return majority(votes)Results
Author-reported. Setup: models are fine-tuned on each language's training data and scored on the organisers' test sets §5. Metrics are macro F1 and weighted F1. The TRAC-2 and HatEval tables report only one of them, as the paper notes §5. Baselines are a majority-class predictor, XLM-R and BERT-m with and without transfer, and the best published shared-task systems §5.
| Language | Setting | Macro F1 | Weighted F1 | Source |
|---|---|---|---|---|
| Bengali | XLM-R, transfer (TL) | 0.8415 | 0.8423 | tab. 2 |
| Bengali | best TRAC-2 system (Risch et al., 2020) | 0.8219 | not reported | tab. 2 §5 |
| Bengali | BERT-m, TL | 0.8197 | 0.8231 | tab. 2 |
| Bengali | XLM-R, no TL | 0.8142 | 0.8188 | tab. 2 |
| Bengali | BERT-m, no TL | 0.8132 | 0.8157 | tab. 2 |
| Bengali | majority-class baseline | 0.2498 | 0.4491 | tab. 2 |
| Hindi | XLM-R, TL | 0.8568 | 0.8580 | tab. 2 |
| Hindi | best HASOC system (Bashar et al., 2019) | 0.8149 | 0.8202 | tab. 2 §5 |
| Hindi | BERT-m, TL | 0.8211 | 0.8220 | tab. 2 |
| Hindi | XLM-R, no TL | 0.8061 | 0.8072 | tab. 2 |
| Hindi | BERT-m, no TL | 0.8025 | 0.8030 | tab. 2 |
| Hindi | majority-class baseline | 0.3510 | 0.3798 | tab. 2 |
| Spanish | XLM-R, TL | 0.7513 | 0.7591 | tab. 2 |
| Spanish | BERT-m, TL | 0.7319 | 0.7385 | tab. 2 |
| Spanish | best HatEval systems (Vega et al.; Pérez et al.) | not reported | 0.7300 | tab. 2 §5 |
| Spanish | XLM-R, no TL | 0.7224 | 0.7265 | tab. 2 |
| Spanish | BERT-m, no TL | 0.7215 | 0.7234 | tab. 2 |
| Spanish | majority-class baseline | 0.3700 | 0.4348 | tab. 2 |
- XLM-R with transfer is the top row for all three languages §5 tab. 2.
- The authors say XLM-R beats multilingual BERT in every language tested §5.
- For Bengali, the non-aggressive class scores higher than the two aggressive classes, which matches what TRAC-2 participants observed §5 fig. 3.
Paper vs code
Code paths below are relative to the repository root, at the commit in the
frontmatter. In citations the directory prefix is dropped when the file name is
unique (for example bengali_deepoffense.py is examples/bengali/…).
- Bengali transfer is not implemented. The paper uses English-initialised
weights for Bengali §4.2. The script creates each fold's model from the
xlm-roberta-largename bengali_deepoffense_config.py [code: bengali_deepoffense.py#L72-L73]. No English weights are loaded. - Spanish transfer is not implemented.
spanish_deepoffense.pyimportsglobal_argsfrom the English config spanish_deepoffense.py, which defines no such name english_deepoffense_config.py. It trains multilingual BERT on the English OLID file spanish_deepoffense.py spanish_deepoffense.py, then evaluates on an English split [code: spanish_deepoffense.py#L30]. No Spanish data is read. Its config, which sets learning rate 4e-5 global_args.py, is not imported. - The English source model differs. The paper says XLM-R §4.1 §4.2. The
English config uses
bert-large-casedwith 6 epochs and sequence length 256 english_deepoffense_config.py [code: english_deepoffense_config.py#L25] english_deepoffense_config.py. The README says the released English checkpoint is XLM-R large [code: README.md#L24]. - The Hindi source checkpoint is downloaded at run time.
MODEL_NAMEisdeepoffense/enhindi_deepoffense_config.py. The script fetches it from a Google Drive ID hindi_deepoffense_config.py [code: hindi_deepoffense.py#L24-L25]. That ID differs from the README link [code: README.md#L24]. The paper does not name the checkpoint, so the Hindi source model used cannot be confirmed from the code. - Split. The paper says 0.8:0.2 train/validation §4.2. The code uses 0.9:0.1 per fold bengali_deepoffense.py hindi_deepoffense.py. The Bengali script also merges the provided dev file into training first bengali_deepoffense.py. The English script holds out a random 20% of the OLID training file, unseeded english_deepoffense.py. That is not the official OLID test set.
- Folds and voting are not in the paper. The code trains three fold models per language and takes a majority vote bengali_deepoffense.py.
- Model selection. The best checkpoint is the lowest eval loss, not the best F1 classification_model.py [code: bengali_deepoffense_config.py#L70]. Early stopping has patience 10 [code: bengali_deepoffense_config.py#L68].
- Epochs and sequence length. The paper says 3 epochs for all languages §4.2. The English config uses 6 epochs english_deepoffense_config.py. The code caps sequence length at 128 for the non-English runs [code: bengali_deepoffense_config.py#L25]. The paper allows 512 §4.1.
- Spanish learning rate. The paper says 1e-5 for all languages §4.2. The unused Spanish config sets 4e-5 global_args.py.
- Head. The paper's eq. 1 is a single linear layer and softmax eq. 1. The
code imports
RobertaClassificationHeadfrom the Hugging Facetransformerslibrary roberta_model.py roberta_model.py. That head is not defined in this repository. Per the library, it adds a hidden dense layer with tanh before the output layer. This is inferred from the import, not read from the repository; check it against the pinnedtransformersversion (>=4.2.0inrequirements.txt).
Pitfalls
- The paper's text disagrees with its table in four places. Use tab. 2.
- §5 gives the Hindi XLM-R (TL) result as "0.8568 and 0.8580 weighted and macro F1". The abstract and tab. 2 give macro 0.8568 and weighted 0.8580.
- §5 gives the Bengali XLM-R (TL) result as "0.7513 and 0.7591 macro and weighted F1". Those are the Spanish numbers. The abstract and tab. 2 give Bengali 0.8415 macro and 0.8423 weighted.
- §5 says the TRAC-2 Bengali winner scored "0.821 weighted F1". The same section says TRAC-2 reported only macro F1, and tab. 2 lists 0.8219 under macro.
- §5 says the two best HatEval Spanish systems scored "0.7300 macro F1". The same section says HatEval reported only weighted F1, and tab. 2 lists them under weighted.
- Transfer is defined inconsistently. §5 defines TL as inter-language transfer only. The Bengali rows also use inter-task transfer §4.2 §5.
- Architecture figures contradict the model size. §4.1 says XLM-R large and
then gives 12 layers, 768 hidden states and 8 heads. This card cites the model
name and not those numbers. The code loads
xlm-roberta-large[code: bengali_deepoffense_config.py#L12]. - Learning rate and epochs were tuned by hand on the validation set. The paper gives no search grid §4.2. The reported numbers may depend on those choices and on the checkpoint rule above.
- Majority vote can tie.
max(set(row), key=row.count)breaks ties by set order. With three Bengali classes, three different fold predictions produce an arbitrary winner bengali_deepoffense.py. Use a fixed tie-break if you reimplement. - Costs are higher than the paper's timings. The paper reports about 1 h for English and 30 min for each other language on one K80 §4.2. The code trains three fold models per language bengali_deepoffense.py, so plan for roughly three times that, with evaluation every 60 steps [code: bengali_deepoffense_config.py#L47]. This is an inference from the loop, not a measured time.
- Single runs. The paper reports one number per cell, with no seeds or spread §5. Treat differences of a few points with care.
- Shared-task data. The repository ships copies of the shared-task datasets hindi_dataset.tsv. Their licence is not stated in the repository. Do not copy them into this wiki.
- Short inputs. Truncation at 128 tokens may cut long posts. The paper does not discuss this.
Lineage
- Builds on: XLM-R (cited in the paper as conneau2019unsupervised) §2 §4.1; multilingual BERT as the BERT-m comparison §4; the OLID dataset §3. None of these has a card in this wiki yet.
- Compared against: the best TRAC-2 (Bengali), HASOC 2019 (Hindi) and HatEval 2019 (Spanish) systems, as the organisers report them §5 tab. 2.
- Superseded by: not stated. The authors list SOLID and the OffensEval 2020 languages as future work §6.
Source and license
Adapted from “Multilingual Offensive Language Identification with Cross-lingual Embeddings” by Tharindu Ranasinghe, Marcos Zampieri (arXiv:2010.05324), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/tharindudr/DeepOffense at commit 838fd09a8898 (Apache-2.0). Any quoted code is shown under that license.