wikillm

Paper · 2020-04-05 · cs.CL

Arabic Offensive Language on Twitter: Analysis and Experiments

Builds a 10,000-tweet Arabic dataset by sampling tweets with the vocative pattern "yA ... yA", so it is not tied to a seed word list, and labels offensive, vulgar, hate-speech and clean tweets. Compares lexicon, static-embedding, fastText and BERT/AraBERT classifiers with 5-fold cross-validation. Best result: AraBERT, F1 83.2 (author-reported).

Toxicity detectionText classificationInter-annotator agreementSupport vector machinearabicoffensive-languagetoxicitydatasetsocial-mediatext-classification

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Insults on social media are rare, so a random sample of tweets would mostly be harmless. The authors wanted a large set of labelled Arabic tweets that contained plenty of insults but did not favour one topic, dialect or target. Instead of filtering with a list of rude words, which would bias the set, they searched for a common grammatical habit: Arabic speakers often repeat a word meaning "O" before a name or insult, as in "O X, O Y". Tweets that matched this pattern were far more likely to be insulting, and the pattern works across dialects and topics.

The team then labelled the tweets as offensive, vulgar (sexual or private-part language), hate speech (attacks on a group) or clean, with a written guide and checks on how well people agreed. They used the labelled set to describe how Arabic speakers insult each other, and to test several classifiers. The best classifier was a pretrained Arabic BERT model, AraBERT.

TL;DR

  • The paper builds a 10,000-tweet Arabic dataset for offensive language, labelled for vulgarity and hate speech, from tweets that match the vocative pattern "yA ... yA" §3.1 tab. 1.
  • Offensive tweets are about 19% of this set, but only 1-2% of random tweets §3.1.
  • The best reported result is AraBERT with F1 83.2 for offensive versus not offensive, under 5-fold cross-validation (author-reported) tab. 2.
  • A Mazajak-embedding RBF SVM reaches F1 79.7, a smaller and simpler option tab. 2 tab. 3.
  • The dataset is the main contribution. The paper's own licence does not cover it, and the dataset licence is not stated.

Problem

Offensive-language detectors need labelled data, but offensive tweets are rare. Among random tweets, the authors found 1-2% offensive, so annotating random tweets is inefficient §3.1. A seed list of offensive words solves the rarity problem but skews the data towards particular words, dialects or targets §3.1. Earlier Arabic work used small labelled sets, for example 1,100 tweets §2.

Method

The method has four parts: collecting candidate tweets, labelling them, checking label quality, and describing the data. Experiments then compare classifiers.

1. Candidate collection §3.1. The authors used the Twitter API to collect 660k Arabic tweets that contain the pattern "yA ... yA" (vocative particle, a word sequence, the particle again) between 15 April and 6 May 2019 §3.1. They sorted the word sequences between the particles by frequency, kept the 10,000 most frequent unique sequences, and took one random tweet for each sequence §3.1. The pattern appears in all dialects and is not tied to a topic, which is why the authors chose it §3.1. They also note that not every offensive tweet has the particle, so the method under-covers some styles §3.1.

Pseudocode
# Collect candidate tweets  [§3.1]
# Input:  tweet stream, 15 Apr to 6 May 2019
# Output: one candidate tweet per frequent word sequence

def collect_candidates(stream):
    # 1. Keep tweets that contain the pattern "yA ... yA"  [§3.1]
    matches = [t for t in stream if has_vocative_pattern(t)]
    # 2. Take the word sequence between the vocative particles  [§3.1]
    sequences = [between_particles(t) for t in matches]
    # 3. Keep the 10,000 most frequent unique sequences  [§3.1]
    top_sequences = most_frequent_unique(sequences, 10000)
    # 4. For each sequence, pick one random tweet that contains it  [§3.1]
    return [random_tweet_with(s, matches) for s in top_sequences]

2. Labels and guidelines §3.2. Each tweet gets one or more of four labels: offensive, vulgar, hate speech or clean §3.2. The offensive label covers both vulgar and hate speech, and a tweet can be offensive and also vulgar and/or hate speech §3.2. The guidelines follow the OffensEval 2019 guidelines §3.2. Hate speech means insults or threats against a group on the basis of nationality, ethnicity, gender, political affiliation, religion or other common traits. The authors also count insults against sports-club fans as hate speech §3.2. Clean tweets can contain offensive words used for humour, advice, conditions, condemnation, self-insult, a non-human target or a quotation, and the annotator searched Twitter for ambiguous cases §3.2.

3. Label quality §3.2. Three further annotators checked two samples. The first was 100 tweets (50 offensive, 50 not). Fleiss's kappa was 0.92. The second was three random samples of 100 tweets each, with agreement against the dataset of 0.97, 0.96 and 0.97 §3.2.

4. Description §3.3. The annotator tagged each offensive tweet by topic, dialect, gender and hate-speech type §3.3 fig. 1 fig. 2 fig. 3 fig. 4. Sports and politics are the most common topics fig. 1. Egyptian and Gulf dialects dominate; for vulgar tweets, 71% are Egyptian fig. 2 §3.3. Most offensive tweets, including vulgar and hate speech, come from male authors fig. 3 §3.3. The most common animal insults are "dog" and "donkey" §3.3.

5. Classifiers §4. The task is binary: offensive or not §4. Pre-processing uses Farasa tokenisation and a set of normalisation rules §4.1. Representations are a polarity-lexicon count, static embeddings (fastText, AraVec, Mazajak, and a fastText model trained on this data), and fine-tuned contextual encoders (BERT multilingual and AraBERT) §4.2. The static-embedding and lexicon systems use an SVM with an RBF kernel §4.3. The contextual systems add a dense softmax layer over the first-token state and fine-tune all weights end to end §4.2. Evaluation is 5-fold cross-validation with identical folds §4.4.

Pseudocode
# Mazajak embeddings + RBF SVM, the best static-embedding system  [§4.2] [§4.3]
# Input:  tweets (strings); labels (1 = offensive, 0 = not offensive)
# Output: fitted classifier

def fit_mazajak_svm(tweets, labels, mazajak):
    # 1. Tokenise with Farasa  [§4.1]
    tokens = [farasa_tokenise(tweet) for tweet in tweets]
    # 2. Drop URLs, numbers, mentions, retweets and hashtags  [§4.1]
    tokens = [remove_noise(toks) for toks in tokens]
    # 3. Normalise alef, ta marbouta, alef maqsoura; drop diacritics [§4.1]
    tokens = [normalise_letters(toks) for toks in tokens]
    # 4. Cap repeated letters at two  [§4.1]
    tokens = [cap_repeats(toks, 2) for toks in tokens]
    # 5. Average the token vectors into one vector per tweet  [§4.2]
    features = [mean_vector(mazajak, toks) for toks in tokens]
    # 6. Train an RBF-kernel SVM; C and gamma are not stated  [§4.3]
    return svm_rbf_fit(features, labels)
Pseudocode
# Fine-tune BERT or AraBERT with a softmax head  [§4.2] [§4.3]
# Input:  sub-word ids per tweet; labels; pretrained encoder
# Output: fine-tuned encoder and head W

def fine_tune(encoder, head_w, batches):
    # 1. Read the hidden state of the first ([CLS]) token  [§4.2]
    for token_ids, labels in batches:
        h = encoder(token_ids)[:, 0]        # (B, 768)
        # 2. Apply the softmax head p(l | h) = softmax(W h)  [§4.2]
        probs = softmax(h @ head_w.T)       # (B, num_labels)
        # 3. Compute the loss; the paper says binary cross-entropy  [§4.3]
        loss = cross_entropy(probs, labels)
        # 4. Update encoder and head together  [§4.2]
        update(encoder, head_w, loss)

Key formula §4.2, in the paper's own inline notation:

p(l \mid h) = \mathrm{softmax}(W h)

Hyperparameters

NameValueWhereNotes
Candidate pattern"yA ... yA"§3.1Vocative particle يا, repeated
Collection window15 Apr to 6 May 2019§3.1660k tweets matched
Unique sequences kept10,000§3.1one random tweet per sequence
Random base-rate sample3 samples of 1,000 tweets§3.1offensive share 1-2%
Guideline check sample100 tweets (50/50)§3.2Fleiss kappa 0.92
Cross-validation5 folds, identical across experiments§4.4stratification not stated
Letter normalisationalef variants to bare alef; ta marbouta to ha; alef maqsoura to ya§4.1diacritics and kashida removed
Repeated lettersat most 2§4.1
LexiconNileULex: 3,279 MSA + 2,674 Egyptian terms; 4,256 negative, 1,697 positive§4.2features are counts of positive and negative terms
fastText (pretrained Egyptian)300 dimensions§4.2
AraVec skip-gram100 dimensions; 66.9M tweets§4.2
Mazajak skip-gram300 dimensions; 250M tweets§4.2
fastText trained on this data40 dimensions; learning rate 0.5; character n-grams 2 to 10; 30 epochs§4.2tuned by 5-fold grid search; grid not stated
Encoder size (BERT, AraBERT)12 blocks, hidden size 768, 12 heads§4.2
BERT pretraining dataWikipedia, 104 languages§4.2
AraBERT pretraining data8.5M news articles, about 2.5B tokens§4.2
SVM kernelRBF (libsvm via scikit-learn)§4.3C, gamma, class weights not stated
Learning rate, batch size, epochs, max length, optimiser, warm-upnot stated§4.2 §4.3

Results

Author-reported. Setup: the 10,000-tweet dataset, labels offensive versus not, 5-fold cross-validation. Baselines are the lexical-feature SVM and the other embedding systems in the table. The paper gives no held-out test set §4.4.

Headline: AraBERT, F1 83.2 (abstract; tab. 2).

Table 2: classification performance with different features and models tab. 2

Model/classifierPrec.RecallF1
Lexical features, SVM68.535.346.6
fastText/SVM (pretrained Egyptian)76.743.555.5
AraVec/SVM85.569.276.4
Mazajak/SVM88.672.479.7
fastText/fastText (trained on this data)82.168.174.4
BERT multilingual78.374.076.0
AraBERT84.682.483.2

Table 3: other classifiers on Mazajak embeddings tab. 3

ClassifierPrec.RecallF1
Decision Tree51.253.852.4
Random Forest82.442.456.0
Gaussian NB44.986.059.0
Perceptron75.667.766.8
AdaBoost74.367.070.4
Gradient Boosting84.263.072.1
Logistic Regression84.769.576.3
SVM88.672.479.7

Table 1: distribution of annotated tweets tab. 1

LabelTweetsWords
Offensive1,91538k
-- Vulgar2254k
-- Hate speech50613k
Clean8,085151k
Total10,000193k

Error analysis (one fold, Mazajak/SVM: 36 false positives, 121 false negatives) §4.5. False positives: fans of rival teams, quoted insults, idioms, and implicit sarcasm. False negatives: insults that sound affectionate, and implicit offence §4.5.

Paper vs code

No official code.

The paper gives only a dataset download link, a shared-task zip on alt.qcri.org §3.2. That is data, not code. I searched GitHub for "ASAD arabic offensive" (no hits), "arabic offensive tweets" (only third-party repositories), "offensive" under the authors' user name and under the qcri organisation (no hits), and "OSACT" (third-party repositories only). I also read the first 30 of 168 qcri organisation repositories, with no match. No author repository was found.

Pitfalls

  • Biased sample. The 19% offensive rate is conditional on the "yA ... yA" pattern §3.1. Random tweets were 1-2% offensive §3.1. Do not use 19% as a base rate for Twitter. Each word sequence also contributes only one tweet §3.1.
  • Table 1 does not add up. Offensive 38k words plus clean 151k words is 189k, but the total is 193k tab. 1. The paper gives no explanation, and word counts may be rounded.
  • Label overlap is not defined. Vulgar and hate speech can overlap with offensive and with each other §3.2. Table 1 gives no overlap counts and does not say how the offensive class is built from them, so a single class per tweet cannot be derived from it.
  • Agreement metric is unclear. The second check reports agreement of 0.97, 0.96 and 0.97 under a metric the paper calls "IIA", which is a typo for IAA. The coefficient is not named §3.2. The 100-tweet check uses Fleiss's kappa §3.2.
  • Annotator roles. The guidelines were written with one experienced annotator §3.2, and that annotator "manually examined and tagged all the offensive tweets" §3.3. Whether that person also labelled all 10,000 tweets is not explicit. How gender and dialect were assigned is not stated §3.3.
  • Softmax or binary cross-entropy. The head is a softmax over the first-token state §4.2, but the training loss is described as binary cross-entropy §4.3. The paper does not say whether W has one row or two for the offensive and not-offensive labels. Two labels with softmax and cross-entropy are a valid pair, but the implementation must choose one.
  • Missing fine-tuning details. Learning rate, batch size, epochs, maximum sequence length, optimiser, warm-up and random seeds are not stated §4.2 §4.3. The results cannot be reproduced exactly from the paper.
  • Missing SVM details. C, gamma, class weights and feature scaling are not stated. How lexicon counts are combined with embedding features is not stated. Handling of out-of-vocabulary tokens for Mazajak is not stated. The fastText grid search has no grid §4.2.
  • Metric averaging. Table 2 does not say whether precision, recall and F1 are for the offensive class or macro-averaged tab. 2.
  • Folds and tuning. The paper says 5-fold cross-validation with identical folds §4.4, but not whether folds are stratified or whether the fastText grid search used the same folds. If it did, the reported F1 is optimistic.
  • Small error sample. The error analysis uses one fold, of which the counts are 36 false positives and 121 false negatives §4.5. Treat its breakdown as illustrative.
  • Source tree. The LaTeX source has a file classifiers.tex with an extra "Classification Models" section and an ASAD component table, but the main file does not include it [eacl2021.tex]. Commented-out drafts also hold alternative settings. None of that is part of the paper, so none of it is used here.
  • Dataset rights. The paper's CC BY 4.0 licence covers the paper. The dataset's licence is not stated, and this card does not redistribute the data.
  • Figures are not reproduced. Four numbered figures are described in words and cited fig. 1 to fig. 4. A fifth figure, a word tag cloud, is described in the paper's captions only fig. 5.

Lineage

  • Builds on: AraBERT, BERT multilingual, Mazajak embeddings, AraVec, fastText, NileULex lexicon, Farasa tokeniser and the OffensEval 2019 guidelines §4.1 §4.2 §3.2. Only the paper's citations are used. No arXiv ids are given in the source.
  • Compared against: a lexical baseline and the static, contextual and classical-classifier systems in tab. 2 and tab. 3.
  • Superseded by: not stated in the paper.

Source and license

Adapted from “Arabic Offensive Language on Twitter: Analysis and Experiments” by Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, Ahmed Abdelali (arXiv:2004.02192), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.