Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model

In plain words

Saudi Telecom Company (STC) receives far more customer tweets than people can read by hand. This paper asks whether a language model can sort those tweets by mood (positive, negative, neutral, sarcastic or unclear) and separately flag spam. Arabic has fewer ready-made language tools than English, and its dialects differ from each other, so the authors start from a BERT-style model that was pretrained on Arabic tweets (MARBERT) and fine-tune it on labelled STC tweets.

The hard part is the data more than the model. Most tweets are negative, while sarcastic and unclear tweets are rare, and the authors found the model learned those rare classes poorly. Rather than copy or drop existing tweets to balance the classes, they collected more sarcasm examples. They train one model for spam and a separate one for sentiment. Their scores are F1 values, which combine precision and recall, and they are reported on the authors' own data without a comparison on that data against other methods.

TL;DR

• Fine-tunes MARBERT, an Arabic BERT-style model pretrained on dialectal and standard Arabic tweets, once for sentiment (five classes) and once for spam (Yes/No), on private STC tweets §4.2 §5.1.
• Best sentiment F1 is 0.75, reached with train batch 96, learning rate 2e-5, max sequence length 256 and 5 epochs tab. 4 §6. Sarcasm and indeterminate classes stay weak (F1 0.54 and 0.56 at batch 96) tab. 5.
• No official code, and the dataset is private, so the paper cannot be reproduced from its text alone.
• Read it as a fine-tuning and imbalance report on one dataset, not as a method with a baseline comparison.

Problem

• Reading STC customer tweets by hand does not scale, and sentiment analysis is meant to help improve customer service §1.
• Arabic NLP is less developed than English NLP, and Arabic dialects differ across countries §1 [abstract].
• The sentiment data is unbalanced: sarcasm and indeterminate examples are rare, and the model did not learn them adequately at first §5.3.
• The paper also needs to separate spam tweets from others, using a second labelled dataset §5.1.

Method

The pipeline has two parallel branches, one for spam and one for sentiment. Each runs cleaning, then a MARBERT-based classifier fig. 2 §5.2. The branches are described as separate models, and the text does not state whether they share weights (see Pitfalls).

1. Data. The sentiment set holds more than 22K STC customer tweets labelled Negative, Positive, Neutral, Sarcasm or Indeterminate, labelled manually §5.1. The spam set holds more than 24K tweets labelled Yes or No; those labels were assigned automatically from a word-net of unwanted keywords and phrases, not by hand §5.1. The abstract gives a combined count of 24,513 tweets with per-class counts: 1,437 positive, 13,828 negative, 5,694 neutral, 1,221 sarcasm, 2,297 indeterminate [abstract].
2. Cleaning. Remove hashtags, usernames and URLs, extra spaces, extra symbols and repeated characters, and remove duplicate tweets §4.1. Some variant forms of Arabic letters are mapped to one form; §4.1 gives examples, not a full map §4.1.
3. Model. MARBERT is a BERT-based encoder trained on dialectal and modern standard Arabic tweets, about 15.6B tokens. It drops the next-sentence-prediction objective because it was trained on short tweets §4.2. The input tweet is tokenised after a leading [CLS] token, and the output label is read from the [CLS] position fig. 1.
4. Fine-tuning. Adam with epsilon 1e-8, learning rate 2e-5, training batch 96, max sequence length 256, 5 epochs. These are the best of six settings in Table 4 tab. 4 §5.6 §6.
5. Imbalance. Oversampling was rejected because it adds no new data and increases the weight of noise. Undersampling was rejected because it loses data on a small dataset. Balanced bagging is mentioned as an option but not described as used. The authors instead collected more sarcasm tweets, from about 200 to about 1,400 §5.3.
6. Evaluation. Precision, recall and F1 are computed for each class from true and false positives and negatives §5.5 eq. 1.

\text{Precision} = \frac{TP}{TP + FP}, \quad \text{Recall} = \frac{TP}{TP + FN}, \quad F_1 = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} eq. 1

# Clean and normalise each tweet  [§4.1]
# Input:  tweets (list of str); labels (list of str)
# Output: fine-tuned MARBERT classifier

def clean_tweet(text):
    # Remove hashtags, usernames and URLs  [§4.1]
    text = strip_hashtags_mentions_urls(text)
    # Collapse extra spaces, symbols and repeated characters  [§4.1]
    text = collapse_noise(text)
    # Map variant letter forms to one form; full map not stated  [§4.1]
    return normalise_arabic_letters(text)

# Remove duplicate tweets; order relative to cleaning not stated  [§4.1]
tweets = unique(tweets)
inputs = [clean_tweet(t) for t in tweets]

# Load MARBERT; no next-sentence-prediction objective  [§4.2]
model = load_marbert_pretrained()
# Adam, learning rate 2e-5, epsilon 1e-8  [tab. 4] [§6]
optimizer = adam(model.parameters(), lr=2e-5, eps=1e-8)

# Five epochs with training batch size 96  [§6] [tab. 4]
for epoch in range(5):
    for batch_inputs, batch_labels in batches(inputs, labels, size=96):
        # Max length 256; truncation and padding not stated  [tab. 4]
        token_ids = tokenize(batch_inputs, max_length=256)
        # Label read from the [CLS] position  [fig. 1]
        logits = model(token_ids)
        # Loss function not stated in the paper
        loss = loss_fn(logits, batch_labels)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

Hyperparameters

Results

Author-reported. Setup: STC sentiment tweets (five classes) and spam tweets (Yes/No), MARBERT fine-tuned per task. Baseline: the paper reports no run on the STC data; Table 3 compares spam results from other papers on other datasets. The split, the averaging of F1 across classes and the test set are not stated.

Sentiment, six configurations (Table 4). The F1 column is the only figure given per row:

Sentiment, per class at two batch sizes (Table 5). Column headers read "-ive" and "+ive"; they are taken to mean Negative and Positive:

Spam (Table 3). The table's cells are laid out by row, and the mapping from each F1 to each author is read from that layout:

The abstract says the scheme is "promising in terms of accuracy in contrast to existing techniques". The accuracies in Table 2 come from other papers, datasets and tasks, and are not run on the STC data tab. 2.

Paper vs code

No official code. The paper has no code link and no project page. The abstract and arXiv metadata give no repository. Searches: GitHub repositories for "MARBERT spam sentiment" (no results); "STC arabic tweets sentiment" (one repository by a third-party owner with no licence, not used); GitHub users matching the author names (no results). The sentiment data is held privately by the client §5.1, so no data release is known either.

Pitfalls

• Accuracy claim. The abstract claims accuracy gains over existing techniques. The paper reports F1, and Table 2's literature numbers come from other datasets. No baseline is run on the STC data.
• Dataset counts disagree. The abstract gives 24,513 tweets, but its per-class counts sum to 24,477. §5.1 gives "more than 22K" sentiment tweets. Which set is the training data, and whether counts are before or after cleaning, is not stated.
• Max sequence length. §5.6 says "265" and Table 4 gives 256. This card uses 256 from the table. §6 does not state a value.
• Table 4 row 3 lists epsilon 2e-5 and learning rate 1.215e-5. The epsilon value looks like a column slip, since every other row has 1e-8; treat it as unverified.
• Batch column. Table 4 writes batch sizes as pairs such as 96-32 and 40-14. The card reads them as train-eval, which matches §6's train 96 and eval 32. §5.6 also says batch 96 beats "16 or 14", and the "14" is not explained.
• Confounded search. The six Table 4 rows change learning rate, epochs, sequence length and batch together, so no single factor's effect is isolated. It is not stated whether the reported F1 is on held-out data or on the data used to pick the setting.
• F1 averaging. Table 4's F1 values do not say whether they are macro, weighted or micro averages. Table 5 gives per-class values only.
• Spam labels come from a keyword list §5.1. The spam model may be learning that list rather than spam itself. The 0.98 F1 should be read with that in mind.
• One model or two. The abstract calls it "the proposed model", and Figure 2 shows two pipelines. §5.2 says spam training came first and sentiment second. Whether the same weights were reused is not stated.
• Multi-label wording. §6 calls the task "multilabel". §5.1, Table 5 and Figure 2 read as one label per tweet. How labels are encoded is not stated.
• Imbalance handling. Over- and undersampling were rejected, and class-weighted loss is not mentioned. The sarcasm count was raised to about 1,400 by collecting data §5.3, but the other class counts after collection are not stated.
• Cleaning details. The Arabic letter mappings are examples only, and the cleaning order is not stated.
• Not stated. Loss function, optimizer betas and weight decay, warmup, scheduler, dropout, seeds, train/validation/test split, hardware, framework, MARBERT checkpoint, and how the five-class output is decoded.
• Dates. arXiv v1 is dated 2026-06-24. The PDF header gives Mathematical Modelling of Engineering Problems, Vol. 9 No. 6, December 2022, DOI 10.18280/mmep.090617, accepted 4 October 2022. The card uses the arXiv date.
• Table 1 lists Arabic BERT model sizes from the related-work section (ref. [14]). It is not part of this paper's method.

Lineage

• Builds on: MARBERT, Abdul-Mageed et al. 2020, cited as ref. [33] in the paper. The paper gives no arXiv id, and the paper is not on this wiki. BERT is cited to a blog post, ref. [1].
• Compared against: spam methods in Table 3 (Saab et al. [15], Yaseen [16]) on other datasets; sentiment methods in Table 2 on other datasets, reported as accuracy.
• Superseded by: not stated.