Paper · 2020-04-05 · cs.CL
Arabic Offensive Language on Twitter: Analysis and Experiments
Builds a 10,000-tweet Arabic dataset by sampling tweets with the vocative pattern "yA ... yA", so it is not tied to a seed word list, and labels offensive, vulgar, hate-speech and clean tweets. Compares lexicon, static-embedding, fastText and BERT/AraBERT classifiers with 5-fold cross-validation. Best result: AraBERT, F1 83.2 (author-reported).
Toxicity detectionText classificationInter-annotator agreementSupport vector machinearabicoffensive-languagetoxicitydatasetsocial-mediatext-classification
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Insults on social media are rare, so a random sample of tweets would mostly be harmless. The authors wanted a large set of labelled Arabic tweets that contained plenty of insults but did not favour one topic, dialect or target. Instead of filtering with a list of rude words, which would bias the set, they searched for a common grammatical habit: Arabic speakers often repeat a word meaning "O" before a name or insult, as in "O X, O Y". Tweets that matched this pattern were far more likely to be insulting, and the pattern works across dialects and topics.
The team then labelled the tweets as offensive, vulgar (sexual or private-part language), hate speech (attacks on a group) or clean, with a written guide and checks on how well people agreed. They used the labelled set to describe how Arabic speakers insult each other, and to test several classifiers. The best classifier was a pretrained Arabic BERT model, AraBERT.
TL;DR
- The paper builds a 10,000-tweet Arabic dataset for offensive language, labelled for vulgarity and hate speech, from tweets that match the vocative pattern "yA ... yA" §3.1 tab. 1.
- Offensive tweets are about 19% of this set, but only 1-2% of random tweets §3.1.
- The best reported result is AraBERT with F1 83.2 for offensive versus not offensive, under 5-fold cross-validation (author-reported) tab. 2.
- A Mazajak-embedding RBF SVM reaches F1 79.7, a smaller and simpler option tab. 2 tab. 3.
- The dataset is the main contribution. The paper's own licence does not cover it, and the dataset licence is not stated.
Problem
Offensive-language detectors need labelled data, but offensive tweets are rare. Among random tweets, the authors found 1-2% offensive, so annotating random tweets is inefficient §3.1. A seed list of offensive words solves the rarity problem but skews the data towards particular words, dialects or targets §3.1. Earlier Arabic work used small labelled sets, for example 1,100 tweets §2.
Method
The method has four parts: collecting candidate tweets, labelling them, checking label quality, and describing the data. Experiments then compare classifiers.
1. Candidate collection §3.1. The authors used the Twitter API to collect 660k Arabic tweets that contain the pattern "yA ... yA" (vocative particle, a word sequence, the particle again) between 15 April and 6 May 2019 §3.1. They sorted the word sequences between the particles by frequency, kept the 10,000 most frequent unique sequences, and took one random tweet for each sequence §3.1. The pattern appears in all dialects and is not tied to a topic, which is why the authors chose it §3.1. They also note that not every offensive tweet has the particle, so the method under-covers some styles §3.1.
# Collect candidate tweets [§3.1]
# Input: tweet stream, 15 Apr to 6 May 2019
# Output: one candidate tweet per frequent word sequence
def collect_candidates(stream):
# 1. Keep tweets that contain the pattern "yA ... yA" [§3.1]
matches = [t for t in stream if has_vocative_pattern(t)]
# 2. Take the word sequence between the vocative particles [§3.1]
sequences = [between_particles(t) for t in matches]
# 3. Keep the 10,000 most frequent unique sequences [§3.1]
top_sequences = most_frequent_unique(sequences, 10000)
# 4. For each sequence, pick one random tweet that contains it [§3.1]
return [random_tweet_with(s, matches) for s in top_sequences]2. Labels and guidelines §3.2. Each tweet gets one or more of four labels: offensive, vulgar, hate speech or clean §3.2. The offensive label covers both vulgar and hate speech, and a tweet can be offensive and also vulgar and/or hate speech §3.2. The guidelines follow the OffensEval 2019 guidelines §3.2. Hate speech means insults or threats against a group on the basis of nationality, ethnicity, gender, political affiliation, religion or other common traits. The authors also count insults against sports-club fans as hate speech §3.2. Clean tweets can contain offensive words used for humour, advice, conditions, condemnation, self-insult, a non-human target or a quotation, and the annotator searched Twitter for ambiguous cases §3.2.
3. Label quality §3.2. Three further annotators checked two samples. The first was 100 tweets (50 offensive, 50 not). Fleiss's kappa was 0.92. The second was three random samples of 100 tweets each, with agreement against the dataset of 0.97, 0.96 and 0.97 §3.2.
4. Description §3.3. The annotator tagged each offensive tweet by topic, dialect, gender and hate-speech type §3.3 fig. 1 fig. 2 fig. 3 fig. 4. Sports and politics are the most common topics fig. 1. Egyptian and Gulf dialects dominate; for vulgar tweets, 71% are Egyptian fig. 2 §3.3. Most offensive tweets, including vulgar and hate speech, come from male authors fig. 3 §3.3. The most common animal insults are "dog" and "donkey" §3.3.
5. Classifiers §4. The task is binary: offensive or not §4. Pre-processing uses Farasa tokenisation and a set of normalisation rules §4.1. Representations are a polarity-lexicon count, static embeddings (fastText, AraVec, Mazajak, and a fastText model trained on this data), and fine-tuned contextual encoders (BERT multilingual and AraBERT) §4.2. The static-embedding and lexicon systems use an SVM with an RBF kernel §4.3. The contextual systems add a dense softmax layer over the first-token state and fine-tune all weights end to end §4.2. Evaluation is 5-fold cross-validation with identical folds §4.4.
# Mazajak embeddings + RBF SVM, the best static-embedding system [§4.2] [§4.3]
# Input: tweets (strings); labels (1 = offensive, 0 = not offensive)
# Output: fitted classifier
def fit_mazajak_svm(tweets, labels, mazajak):
# 1. Tokenise with Farasa [§4.1]
tokens = [farasa_tokenise(tweet) for tweet in tweets]
# 2. Drop URLs, numbers, mentions, retweets and hashtags [§4.1]
tokens = [remove_noise(toks) for toks in tokens]
# 3. Normalise alef, ta marbouta, alef maqsoura; drop diacritics [§4.1]
tokens = [normalise_letters(toks) for toks in tokens]
# 4. Cap repeated letters at two [§4.1]
tokens = [cap_repeats(toks, 2) for toks in tokens]
# 5. Average the token vectors into one vector per tweet [§4.2]
features = [mean_vector(mazajak, toks) for toks in tokens]
# 6. Train an RBF-kernel SVM; C and gamma are not stated [§4.3]
return svm_rbf_fit(features, labels)# Fine-tune BERT or AraBERT with a softmax head [§4.2] [§4.3]
# Input: sub-word ids per tweet; labels; pretrained encoder
# Output: fine-tuned encoder and head W
def fine_tune(encoder, head_w, batches):
# 1. Read the hidden state of the first ([CLS]) token [§4.2]
for token_ids, labels in batches:
h = encoder(token_ids)[:, 0] # (B, 768)
# 2. Apply the softmax head p(l | h) = softmax(W h) [§4.2]
probs = softmax(h @ head_w.T) # (B, num_labels)
# 3. Compute the loss; the paper says binary cross-entropy [§4.3]
loss = cross_entropy(probs, labels)
# 4. Update encoder and head together [§4.2]
update(encoder, head_w, loss)Key formula §4.2, in the paper's own inline notation:
p(l \mid h) = \mathrm{softmax}(W h)
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Candidate pattern | "yA ... yA" | §3.1 | Vocative particle يا, repeated |
| Collection window | 15 Apr to 6 May 2019 | §3.1 | 660k tweets matched |
| Unique sequences kept | 10,000 | §3.1 | one random tweet per sequence |
| Random base-rate sample | 3 samples of 1,000 tweets | §3.1 | offensive share 1-2% |
| Guideline check sample | 100 tweets (50/50) | §3.2 | Fleiss kappa 0.92 |
| Cross-validation | 5 folds, identical across experiments | §4.4 | stratification not stated |
| Letter normalisation | alef variants to bare alef; ta marbouta to ha; alef maqsoura to ya | §4.1 | diacritics and kashida removed |
| Repeated letters | at most 2 | §4.1 | |
| Lexicon | NileULex: 3,279 MSA + 2,674 Egyptian terms; 4,256 negative, 1,697 positive | §4.2 | features are counts of positive and negative terms |
| fastText (pretrained Egyptian) | 300 dimensions | §4.2 | |
| AraVec skip-gram | 100 dimensions; 66.9M tweets | §4.2 | |
| Mazajak skip-gram | 300 dimensions; 250M tweets | §4.2 | |
| fastText trained on this data | 40 dimensions; learning rate 0.5; character n-grams 2 to 10; 30 epochs | §4.2 | tuned by 5-fold grid search; grid not stated |
| Encoder size (BERT, AraBERT) | 12 blocks, hidden size 768, 12 heads | §4.2 | |
| BERT pretraining data | Wikipedia, 104 languages | §4.2 | |
| AraBERT pretraining data | 8.5M news articles, about 2.5B tokens | §4.2 | |
| SVM kernel | RBF (libsvm via scikit-learn) | §4.3 | C, gamma, class weights not stated |
| Learning rate, batch size, epochs, max length, optimiser, warm-up | not stated | §4.2 §4.3 |
Results
Author-reported. Setup: the 10,000-tweet dataset, labels offensive versus not, 5-fold cross-validation. Baselines are the lexical-feature SVM and the other embedding systems in the table. The paper gives no held-out test set §4.4.
Headline: AraBERT, F1 83.2 (abstract; tab. 2).
Table 2: classification performance with different features and models tab. 2
| Model/classifier | Prec. | Recall | F1 |
|---|---|---|---|
| Lexical features, SVM | 68.5 | 35.3 | 46.6 |
| fastText/SVM (pretrained Egyptian) | 76.7 | 43.5 | 55.5 |
| AraVec/SVM | 85.5 | 69.2 | 76.4 |
| Mazajak/SVM | 88.6 | 72.4 | 79.7 |
| fastText/fastText (trained on this data) | 82.1 | 68.1 | 74.4 |
| BERT multilingual | 78.3 | 74.0 | 76.0 |
| AraBERT | 84.6 | 82.4 | 83.2 |
Table 3: other classifiers on Mazajak embeddings tab. 3
| Classifier | Prec. | Recall | F1 |
|---|---|---|---|
| Decision Tree | 51.2 | 53.8 | 52.4 |
| Random Forest | 82.4 | 42.4 | 56.0 |
| Gaussian NB | 44.9 | 86.0 | 59.0 |
| Perceptron | 75.6 | 67.7 | 66.8 |
| AdaBoost | 74.3 | 67.0 | 70.4 |
| Gradient Boosting | 84.2 | 63.0 | 72.1 |
| Logistic Regression | 84.7 | 69.5 | 76.3 |
| SVM | 88.6 | 72.4 | 79.7 |
Table 1: distribution of annotated tweets tab. 1
| Label | Tweets | Words |
|---|---|---|
| Offensive | 1,915 | 38k |
| -- Vulgar | 225 | 4k |
| -- Hate speech | 506 | 13k |
| Clean | 8,085 | 151k |
| Total | 10,000 | 193k |
Error analysis (one fold, Mazajak/SVM: 36 false positives, 121 false negatives) §4.5. False positives: fans of rival teams, quoted insults, idioms, and implicit sarcasm. False negatives: insults that sound affectionate, and implicit offence §4.5.
Paper vs code
No official code.
The paper gives only a dataset download link, a shared-task zip on alt.qcri.org §3.2. That is data, not code. I searched GitHub for "ASAD arabic offensive" (no hits), "arabic offensive tweets" (only third-party repositories), "offensive" under the authors' user name and under the qcri organisation (no hits), and "OSACT" (third-party repositories only). I also read the first 30 of 168 qcri organisation repositories, with no match. No author repository was found.
Pitfalls
- Biased sample. The 19% offensive rate is conditional on the "yA ... yA" pattern §3.1. Random tweets were 1-2% offensive §3.1. Do not use 19% as a base rate for Twitter. Each word sequence also contributes only one tweet §3.1.
- Table 1 does not add up. Offensive 38k words plus clean 151k words is 189k, but the total is 193k tab. 1. The paper gives no explanation, and word counts may be rounded.
- Label overlap is not defined. Vulgar and hate speech can overlap with offensive and with each other §3.2. Table 1 gives no overlap counts and does not say how the offensive class is built from them, so a single class per tweet cannot be derived from it.
- Agreement metric is unclear. The second check reports agreement of 0.97, 0.96 and 0.97 under a metric the paper calls "IIA", which is a typo for IAA. The coefficient is not named §3.2. The 100-tweet check uses Fleiss's kappa §3.2.
- Annotator roles. The guidelines were written with one experienced annotator §3.2, and that annotator "manually examined and tagged all the offensive tweets" §3.3. Whether that person also labelled all 10,000 tweets is not explicit. How gender and dialect were assigned is not stated §3.3.
- Softmax or binary cross-entropy. The head is a softmax over the first-token state §4.2, but the training loss is described as binary cross-entropy §4.3. The paper does not say whether W has one row or two for the offensive and not-offensive labels. Two labels with softmax and cross-entropy are a valid pair, but the implementation must choose one.
- Missing fine-tuning details. Learning rate, batch size, epochs, maximum sequence length, optimiser, warm-up and random seeds are not stated §4.2 §4.3. The results cannot be reproduced exactly from the paper.
- Missing SVM details. C, gamma, class weights and feature scaling are not stated. How lexicon counts are combined with embedding features is not stated. Handling of out-of-vocabulary tokens for Mazajak is not stated. The fastText grid search has no grid §4.2.
- Metric averaging. Table 2 does not say whether precision, recall and F1 are for the offensive class or macro-averaged tab. 2.
- Folds and tuning. The paper says 5-fold cross-validation with identical folds §4.4, but not whether folds are stratified or whether the fastText grid search used the same folds. If it did, the reported F1 is optimistic.
- Small error sample. The error analysis uses one fold, of which the counts are 36 false positives and 121 false negatives §4.5. Treat its breakdown as illustrative.
- Source tree. The LaTeX source has a file
classifiers.texwith an extra "Classification Models" section and an ASAD component table, but the main file does not include it [eacl2021.tex]. Commented-out drafts also hold alternative settings. None of that is part of the paper, so none of it is used here. - Dataset rights. The paper's CC BY 4.0 licence covers the paper. The dataset's licence is not stated, and this card does not redistribute the data.
- Figures are not reproduced. Four numbered figures are described in words and cited fig. 1 to fig. 4. A fifth figure, a word tag cloud, is described in the paper's captions only fig. 5.
Lineage
- Builds on: AraBERT, BERT multilingual, Mazajak embeddings, AraVec, fastText, NileULex lexicon, Farasa tokeniser and the OffensEval 2019 guidelines §4.1 §4.2 §3.2. Only the paper's citations are used. No arXiv ids are given in the source.
- Compared against: a lexical baseline and the static, contextual and classical-classifier systems in tab. 2 and tab. 3.
- Superseded by: not stated in the paper.
Source and license
Adapted from “Arabic Offensive Language on Twitter: Analysis and Experiments” by Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, Ahmed Abdelali (arXiv:2004.02192), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.