Text classification

Overview

Text classification gives a text one label, or a sentence pair one label. The twenty-four cards on this page cover a wide range of inputs. They include personal attacks in English Wikipedia talk comments 1610.08914 §3, medical sentences in 26 categories 1704.06841 §3, mental health conditions inferred from a user's Reddit posts 1806.05258 §3.1, rumour veracity on tweet reply branches 1806.03713 §3, sentiment and sarcasm in short sentences 1901.08014 §2, positive and negative tweets and reviews 2006.03541 §4.1, toxicity in chatbot prompts 2310.17389 §2, phishing and spam emails 2405.11619 §2, natural language inference hypotheses 1803.02324 §2, five GLUE sentence tasks 2111.02840 §3.1, few-shot tasks such as sentiment, emotion and spam 2209.11055 tab. 2, industrial labels from a private dataset 2102.00225 §5, and text-annotation tasks on political and content-moderation samples 2303.15056 §4.1, and dialog acts for each utterance in a sequence of dialogs 1603.03827 §3.1. Two newer cards add COVID-19 tweets in five classification sets 2005.07503 tab. 1 and Arabic tweets for five-class sentiment and spam 2606.25495 §5.1. Two more cover offensive tweets in Arabic, Greek and Turkish 2007.13184 tab. 2, and depression severity in English and Greek Reddit posts 2410.12985 tab. 2. A survey of text data augmentation 2110.01852 §2 and a byte-level multilingual toxicity encoder 2202.11176 tab. 1 complete the set.

Three questions come up again and again. The first is how many labelled examples a method needs. The second is whether a simple model is good enough: two cards report that a linear model on TF-IDF or a FastText model is among the strongest options on its own data. The third is whether a model that does well on its training domain still does well on another one. The chatbot-prompt card reports that detectors trained on other toxicity data score low on chatbot traffic 2310.17389 tab. 1 2310.17389 tab. 2. The adversarial benchmark asks a related question about altered input: a changed example is only a fair test if its label stays the same, so it keeps an example only when annotators agree on the original label 2111.02840 §3.3. A fourth question is where the labels come from. Two cards on this page study label quality directly: one shows that an input-only classifier can beat chance on a benchmark because of how the data was written 1803.02324 tab. 2, and the other reports a gain from relabelling disagreeing examples by hand 2102.00225 tab. 2.

Variants

Convolutional network over word vectors

A sentence is turned into a fixed-length matrix of 100-dimensional word2vec vectors, padded or cut to 50 tokens. Two groups of convolutions with 256 filters of width 5 are followed by max pooling, then a 128-unit dense layer and a 26-way softmax 1704.06841 §3 1704.06841 fig. 1. The authors report 0.68 accuracy, against 0.28 to 0.51 for a doc2vec classifier, mean embeddings and bag-of-words with k-means features 1704.06841 fig. 2. The training recipe (optimiser, learning rate, batch size, epochs) is not stated, and the dataset is not released. The training sentences come from encyclopedia articles, and the paper does not test patient notes 1704.06841 §5. Card: Medical Text Classification using Convolutional Neural Networks.

Sequential classification with earlier texts as context

For a sequence of short texts, such as dialog utterances, each text is first turned into a fixed-size vector. The encoder is an LSTM with max pooling or a CNN with filters of height 3 1603.03827 §2.1 1603.03827 tab. 2. A small two-layer network then labels the current text from its own vector and from the vectors of a few earlier texts 1603.03827 §2.2. The author-reported best accuracies are 66.2 on DSTC 4 (LSTM), 84.6 on MRDA (CNN) and 73.1 on SwDA (CNN), against majority-class scores of 25.8, 59.1 and 33.7 1603.03827 tab. 4. In the authors' grid, settings with history beat the setting without it, and raising the history at the vector level helped more than raising it at the label level 1603.03827 §4 [arxiv:1603.03827 tab. 3]. The paper says the CNN beats the LSTM on every dataset, but on DSTC 4 its own table gives the LSTM the higher headline score (66.2 against 65.5) 1603.03827 tab. 4. The grid averages are over five runs, while the headline numbers are single runs picked on validation data. The learning rate, batch size and epochs are not stated, and the paper does not say how a validation split was made for two datasets 1603.03827 §3.1 1603.03827 §3.2. Card: Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks.

Recurrent and convolutional models on word embeddings or TF-IDF

For binary positive and negative sentiment, a DNN, a CNN and an LSTM are each trained on eight tweet and review sets. Each model gets either learned word embeddings or TF-IDF vectors. The embedding input uses a 15,000-word vocabulary, width 300 and sequences of at most 40 tokens 2006.03541 §4.4. Scores come from 10-fold cross-validation, and the paper runs no classical baseline 2006.03541 §4.2. Word embeddings give higher accuracy than TF-IDF for the CNN and the LSTM on all eight sets. On Sentiment140 the CNN reaches 0.8006 with embeddings and 0.7669 with TF-IDF, and the LSTM reaches 0.8282 against 0.5696 2006.03541 tab. 2. The LSTM with embeddings is the most accurate on that largest tweet set, but it takes 1 h 4 min to train there, against 7 min 3 s for the CNN with embeddings 2006.03541 tab. 7 2006.03541 tab. 8. The authors judge the CNN to give the best balance of accuracy and training time 2006.03541 §6. The optimiser and learning rate are not stated, and the "echoes" setting is not defined 2006.03541 §5. Card: Sentiment Analysis Based on Deep Learning: A Comparative Study.

Classical pipeline as a map (survey)

This is a survey, not a method, and it runs no experiments of its own. It splits the task into four phases: text cleaning and features, optional dimensionality reduction, a classifier, and evaluation 1904.08067 §1 1904.08067 fig. 1. The feature options it lists are bag-of-words, TF-IDF, Word2Vec, GloVe and FastText 1904.08067 §2.3 1904.08067 §2.4. Its classifier chapter covers naive Bayes, logistic regression, support vector machines, decision trees and random forests, next to CNN, RNN and attention models 1904.08067 §4. Its comparisons are qualitative, with pros and cons per option 1904.08067 §6. It warns that accuracy misleads on imbalanced data 1904.08067 §1 1904.08067 §6. Card: Text Classification Algorithms: A Survey. Use it to pick options to test, and take the numbers from the experimental variants below.

Data augmentation for small labelled sets (survey)

This is a survey, not a method, and it reports no experiments. It sorts text augmentation into three families by how far the new text departs from the original: paraphrasing with small rewordings, noising with swaps, deletions, insertions or substitutions, and sampling of new text from rules, generators, self-training or mixup 2110.01852 §2. Paraphrasing and noising mostly ignore the label, while sampling methods use the labels and the task format 2110.01852 §2.4 2110.01852 §2.3. Random insertion and substitution can flip a label, so the survey suggests label-independent resources at word level, and same-label examples at sentence level 2110.01852 §2.2.3 2110.01852 §2.2.4. Mixup is the only online method, and it mixes the labels as well, so its targets can differ from the original labels 2110.01852 §2.3.5. The survey states no default values for its probabilities and counts, so those choices are yours to make 2110.01852 fig. 12. Card: Data Augmentation Approaches in Natural Language Processing: A Survey.

Linear model on TF-IDF features

Words are weighted by term frequency times inverse document frequency, and a linear classifier is trained on the result. On a merged set of 82,486 emails with an 80/20 split, a linear-kernel SVC on TF-IDF reaches accuracy 0.991 and F1 0.99 on the 16,498-email test split 2405.11619 tab. 2. The same classifiers on Word2Vec inputs reach F1 0.71 to 0.83 in that table 2405.11619 tab. 2. The paper uses one split and one random state, and the best row is picked from eleven rows scored on that test split, so the headline number is optimistic 2405.11619 tab. 2. The paper also adds LIME explanations for single predictions 2405.11619 §2. Card: Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection.

The same family does not win everywhere. On Reddit users, a linear SVM on TF-IDF features is best on one of nine binary conditions, and FastText is best on seven 1806.05258 tab. 4. See Contradictions.

Shallow neural models on user-level text

On the SMHD benchmark, each user is labelled with one or more of nine mental health conditions, and the classifier reads all of that user's posts. Models compared are logistic regression, linear SVM and XGBoost on bag-of-words features, a FastText model with hidden size 100 and character n-grams, and a CNN with one model per condition 1806.05258 §5. FastText has the best F1 on seven of nine binary tasks, the linear SVM on bipolar, and the CNN on eating 1806.05258 tab. 4. The authors note that bag-of-words models favour precision over recall, and they explain the CNN's lower scores by the fact that it reads only part of each user's text 1806.05258 §5. Scores are lowest for the three smallest groups 1806.05258 §5. The dataset is shared under a data-use agreement, not openly. Card: SMHD.

Multi-task learning with a shared encoder

One encoder is trained for several labels at once, and each label has its own output layers 1901.08014 §2. In the sentiment and sarcasm model, a shared GRU feeds two attention-pooled branches. A neural tensor network fuses them before the sarcasm output 1901.08014 §2. On 994 sentences with 10-fold cross-validation, the fused model with shared attention reaches an average F-score of 86.66. Standalone classifiers reach 83.75, and the cited prior system, which also uses gaze features, reaches 82.13 [arxiv:1901.08014 tab. 1]. The paper reports no variance, and a gap of about one point between the multi-task variants is within what small-data noise can produce. Card: Sentiment and Sarcasm Classification with Multitask Learning.

In rumour verification, one LSTM stack is shared by the stance, detection and veracity tasks 1806.03713 §4.2. On PHEME with five events, the three-task model reaches macro-F 0.396. The single-task branchLSTM reaches 0.336 and the strongest reported baseline, NileTMRG*, reaches 0.339 1806.03713 tab. 3. On RumourEval, the veracity-and-stance model reaches 0.558 against 0.539 for NileTMRG* 1806.03713 tab. 3. The paper describes the loss as a plain sum of task losses 1806.03713 §4.2. The released code uses fixed weights instead, and it differs from the paper in other respects, so matching the reported numbers requires choosing between the paper's text and the code. Card: All-in-one: Multi-task Learning for Rumour Verification.

Pretrained encoder fine-tuned on in-domain labels

A RoBERTa-base model trained on the training half of ToxicChat reaches F1 72.1 on the test half, with a standard deviation of 1.2 over five seeds 2310.17389 tab. 2. The best model trained on another dataset reaches 51.6, and a model trained on the combination of six other sets reaches 42.7 2310.17389 tab. 2. Off-the-shelf tools score lower still. OpenAI Moderation reaches F1 20.6, the Perspective API at threshold 0.7 reaches 5.3, and ToxDectRoberta reaches 34.6 2310.17389 tab. 1. The labelled data are real first-turn prompts to one chatbot demo, in English only. The dataset training hyperparameters are not stated 2310.17389 §4.1, and the paper does not state the dataset licence. Card: ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation.

Byte-level multilingual encoder with a multi-label head

The model reads UTF-8 bytes. A learned step scores blocks of one to four bytes at each position and sums them into latent subwords, so no per-language vocabulary is built 2202.11176 §3.1. The encoder is pretrained to fill in masked spans. For fine-tuning the decoder is dropped, the first position is pooled, and a sigmoid output per label lets several labels be on at once 2202.11176 §3.3. On the authors' production multilingual set the average AUC-ROC is 0.910 for the 102M model and 0.919 for the 268M model. The comparison rows are mT5-base at 0.906, mT5-small at 0.878 and an in-house mBERT at 0.840 2202.11176 tab. 1. On the two code-switching sets the 102M model is below mT5-base on JMTCC-CS, at 0.919 against 0.939, while the 268M model is highest on both sets, at 0.945 and 0.902 2202.11176 tab. 3. On obfuscated English Civil Comments, the authors report that the model beats every baseline at every obfuscation rate above 0%, and that all models lose accuracy as the rate rises 2202.11176 §6.2. The pretraining corpus is partly proprietary, with about 4.6 billion texts in one half of the mix, and the paper names no released weights or training code, so the design can be reproduced in outline but not the pretrained model 2202.11176 §3.4. The fine-tuning stopping rule is "until convergence" and the number of labels per task is not stated 2202.11176 §5 2202.11176 §3.3. Card: A New Generation of Perspective API: Efficient Multilingual Character-level Transformers.

Continued pretraining on in-domain text, then fine-tuning

BERT-Large is trained further, with its original masked-token and next-sentence objectives, on about 22.5M cleaned COVID-19 tweets. Usernames and links are replaced by shared placeholder tokens, and emojis become text aliases 2005.07503 §2. The continued model is then fine-tuned on each labelled set. On five classification sets the author-reported mean F1 rises from 0.802 for BERT-Large to 0.833 2005.07503 tab. 2. The authors report their largest relative gain on the COVID-19 category set (ΔMP 25.88%). That measure divides the gain by the room the baseline leaves, and the paper's printed formula has its numerator reversed 2005.07503 §3.2. On raw F1 the largest gain is on the vaccine-sentiment set, from 0.824 to 0.869 2005.07503 tab. 2. Downstream gains rise quickly up to about 200k pretraining steps and then flatten 2005.07503 §3.3. Pretraining took 120 hours on one TPU v3-8 2005.07503 §2. The fine-tuning recipe was tuned on BERT-Large and reused for the new model 2005.07503 §2.1, and the authors note that out-of-domain text may need longer pretraining 2005.07503 §4. Card: COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter.

Language-specific encoder on imbalanced sentiment and spam

MARBERT, a BERT-style encoder pretrained on Arabic tweets, is fine-tuned in two separate runs: one for five-class sentiment and one for spam. Both use private customer tweets from a Saudi telecom company 2606.25495 §4.2 2606.25495 §5.1. The model drops next-sentence prediction, because it was pretrained on short texts 2606.25495 §4.2. The best of six settings reaches sentiment F1 0.75, with learning rate 2e-5, training batch 96, maximum length 256 and 5 epochs 2606.25495 tab. 4. At batch size 96, per-class F1 is 0.85 for negative, 0.92 for positive, 0.89 for neutral, 0.54 for sarcasm and 0.56 for indeterminate 2606.25495 tab. 5. The sarcasm class was weak at batch size 16, at 0.33 2606.25495 tab. 5. The authors rejected over- and undersampling, and raised the sarcasm examples from about 200 to about 1,400 by collecting more tweets 2606.25495 §5.3. The spam F1 of 0.98 2606.25495 tab. 3 is on labels from a keyword list, not on hand-labelled data 2606.25495 §5.1. The paper does not run any baseline on its own data, and its comparison rows come from other papers and datasets 2606.25495 tab. 3. Card: Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model.

Language-specific BERT with a CNN head on the last layers

A language-specific BERT-base is fine-tuned with a small CNN on top. The CNN reads the outputs of the last four BERT layers as four channels. It has 160 filters in five heights, from one to five tokens, then global max pooling and a sigmoid output for offensive or not 2007.13184 §4.4 [arxiv:2007.13184 fig. 1]. On OffensEval 2020 subtask A the author-reported macro-F1 is 0.897 for Arabic, 0.843 for Greek and 0.814 for Turkish, which the authors report as 4th, 4th and 3rd place 2007.13184 tab. 2 2007.13184 §1. Against BERT alone the average gain is 0.010, and Turkish is slightly lower, at 0.814 against 0.816 2007.13184 tab. 2. The paper reports single numbers with no repeated runs 2007.13184 §5, so the size of the gain is uncertain. The training recipe is only partly stated, and the card's Paper vs code section lists where the released code differs on epochs and checkpoint selection 2007.13184 §4.4. Card: KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media.

Few-shot contrastive fine-tuning with a linear head

The method tunes a pretrained sentence encoder on pairs built from a few labelled examples. Pairs with the same label are pulled together and pairs with different labels are pushed apart. A logistic-regression head is then trained on the resulting embeddings 2209.11055 §3.1. No prompt or verbalizer is needed. Card: Efficient Few-Shot Learning Without Prompts.

With 8 examples per class, the 110M-parameter MPNet body averages 62.3 ± 4.9 over five English datasets, with AG News left out. T-Few 3B, which uses prompts, averages 63.4 ± 1.9 and ADAPET 58.3 ± 3.6. Fine-tuning RoBERTa-large averages 43.0 ± 5.2 2209.11055 tab. 2. With 64 examples per class the same body averages 75.3 ± 1.3, against 73.8 ± 2.2 for ADAPET and 70.3 ± 1.5 for T-Few 3B 2209.11055 tab. 2. These are the paper's own runs. The T-Few figure is a median over 5 seeds and the SetFit figure a mean over 10 splits 2209.11055 §4.3 2209.11055 §4.4.

Cost is reported as formula estimates, not measured values. The 110M body needs about 19 times fewer inference FLOPs than T-Few 3B, and a 15M distilled student needs about 123 times fewer 2209.11055 tab. 5. The paper also reports roughly 30 seconds of training per split on one AWS GPU instance 2209.11055 §7.2.

Two more settings are reported. With a multilingual sentence encoder, the method beats fine-tuning and ADAPET at 8 examples per class in every setting tested, on a lower-is-better regression metric 2209.11055 §6 2209.11055 tab. 4. A 15M student distilled from the 110M teacher reaches 60.3 ± 1.6 at 8 examples per class 2209.11055 tab. 5.

Curated adversarial evaluation (robustness)

This is an evaluation variant rather than a training method. The benchmark takes the GLUE dev sets for five tasks and runs 14 attack entries on them, grouped into 11 perturbation columns. The groups are synonym swaps, typos, knowledge-based substitutions, syntactic rewrites, distracting sentences and human-written sets 2111.02840 tab. 1 2111.02840 §3.2. Examples are kept only if they transfer across three surrogate models, change fewer than 15% of the words, and receive a 4-of-5 annotator vote for the original label 2111.02840 §3.3. Card: Adversarial GLUE.

The author-reported result is that ten models average 30.3 to 60.9 on this benchmark, against 85.7 to 93.2 on GLUE 2111.02840 tab. 4. BERT (Large) goes from 85.76 on GLUE to 33.68 on the adversarial set 2111.02840 tab. 4. The paper finds that the human-written sets (C8 to C11) hurt most, and that distracting sentences (C7) hurt more than syntactic rewrites (C6) 2111.02840 §4. Only the dev split is public. Test scores need a hidden test set 2111.02840 §3.5.

Word-substitution attack on a classifier

The attack is black-box. It ranks words by how much removing each one changes the model's output, then replaces the most important ones with embedding neighbours that pass a sentence-similarity check and a part-of-speech check, until the label flips 1907.11932 alg. 1 1907.11932 eq. 2. The paper attacks WordCNN, WordLSTM and BERT on five classification sets. For BERT, accuracy falls to 11.5 on MR, 13.6 on IMDB, 6.6 on Yelp and 12.5 on AG, with 6.1% to 22.0% of words perturbed 1907.11932 tab. 3. On entailment, BERT falls to 4.0 on SNLI and 9.6 on MultiNLI matched, with 15.2% to 18.5% of words perturbed 1907.11932 tab. 4. Human checks on 100 adversarial examples each for MR and SNLI give label agreement with the original of 92% and 85% 1907.11932 §Results, Human Evaluation. Adversarial training raises BERT's MR after-attack accuracy from 11.5 to 18.7, from one run 1907.11932 tab. 12. The official code differs from the paper's stated thresholds and similarity measure in several places 1907.11932 §Paper vs code. Card: Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment.

Masked-language-model substitution attack

BAE is also black-box. It scores each token by how much deleting it lowers the probability of the true label, then masks tokens in that order. A BERT masked language model proposes replacements or insertions for each mask, and the candidates are filtered by sentence similarity and, for replacements, by part of speech 2004.01970 §2 2004.01970 app. A. The paper defines four modes: replace only (BAE-R), insert only (BAE-I), either per token (BAE-R/I), and replace then insert (BAE-R+I) 2004.01970 §2. On BERT, the author-reported accuracy after attack is 19.2 on MR for BAE-R+I, against 30.7 for TextFooler, and 14.0 on Amazon, against 30.0 2004.01970 tab. 1. BAE-R alone is the exception: on BERT it leaves more accuracy than TextFooler on six of the seven datasets, and less only on TREC 2004.01970 tab. 1 2004.01970 tab. 2. In a small human check on BERT, rated naturalness is higher for BAE-R than for TextFooler on all three datasets tested 2004.01970 tab. 3. The paper's results use 100-example test sets for Amazon, Yelp and IMDB 2004.01970 tab. 5, so the variance of those numbers is large. Card: BAE: BERT-based Adversarial Examples for Text Classification.

Zero-shot LLM annotation

The model is given the same codebook that human coders use and labels each text with no training examples. The paper uses gpt-3.5-turbo and compares it with MTurk crowd-workers and trained research assistants on five tasks over four samples 2303.15056 §4.1 2303.15056 §4.2. On average across the four datasets, ChatGPT's accuracy against trained annotators is about 25 points above MTurk 2303.15056 §2. Intercoder agreement is 91% at the default temperature and 97% at temperature 0.2, against 56% for MTurk and 79% for trained annotators 2303.15056 §2. The stated cost is below $0.003 per annotation, which the authors put at about thirty times cheaper than MTurk 2303.15056 §1. The samples are US political and content-moderation texts. The language mix of three samples is not stated, and the card records no published code. Card: ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks.

Zero-shot LLM classification through machine translation

The same zero-shot idea, applied to a clinical label. GPT-3.5-turbo at temperature 0 assigns each Reddit post one of four depression-severity levels. It does this first on the English posts, then on Greek posts that the same model has translated from the English 2410.12985 §3 [arxiv:2410.12985 fig. 1]. The author-reported macro F1 is 0.17 in English and 0.16 in Greek 2410.12985 tab. 2. The Minimum class holds most posts, and the English recall for that class is 0.14 2410.12985 tab. 1 [arxiv:2410.12985 tab. 2]. Greek F1 is lower than English for three of the four levels 2410.12985 tab. 2. The Greek prompt is not stated, and the paper has no translation quality metric 2410.12985 tab. 1 2410.12985 §4. Each language has one run, and the English run is the only baseline 2410.12985 §4. The authors limit the approach to helping train professionals, not to diagnosis 2410.12985 §5. Card: Leveraging LLMs for Translating and Classifying Mental Health Data.

Crowd-labelled training data and a bag-of-n-grams labeller

Crowd workers label a sample of comments, a bag-of-n-grams classifier is trained on their labels, and the classifier then labels a corpus too large to read by hand. Each sampled comment gets at least 10 crowd labels, and annotators must pass a test before they contribute 1610.08914 §3 1610.08914 tab. 1. The classifier is checked against small annotator ensembles on 8,000 comments. On AUC it beats an ensemble of three raters (95.49) and reaches 97.19, against 97.13 for five raters. On Spearman correlation it sits between three raters (64.75) and five (68.27) 1610.08914 tab. 3. The paper reads this as matching about three raters 1610.08914 §4.3. The model is then run on 63 million English talk comments with a fixed score threshold of 0.425. At that threshold the reported precision and recall are both 0.63 1610.08914 §5.1. The paper does not name, in its text, the one classifier it uses downstream, and its own dev-split table gives several close candidates 1610.08914 §4.3 1610.08914 tab. 2. The labels are for personal attacks, and the measured quality holds for that property on that corpus, so validate the threshold on your own data. Card: Ex Machina: Personal Attacks Seen at Scale.

Relabel disagreeing examples, then train with the first model's output

A first model is trained on the original labels. Examples where its prediction disagrees with the label are relabelled by hand. A second model then gets the first model's one-hot prediction as an extra input and is trained on the corrected labels 2102.00225 §5 2102.00225 fig. 1. On a private industrial dataset with BERT-Base, the first model reaches 83.3% test accuracy and the second 92.5%. The second model without the extra input reaches 91.7% 2102.00225 tab. 2. On 5,000 human-checked items, the three models score 88.0%, 97.2% and 97.7% 2102.00225 tab. 2. The paper reports one run per model and no variance, and the gain from the extra input is 0.8 points 2102.00225 tab. 2. Card: Learning From How Humans Correct.

Input-only baseline as a dataset-artifact check

A classifier that sees only the hypothesis of an NLI pair, with no premise, is a fastText model over words and bigrams. On SNLI it reaches 67.0% against 34.3% for the majority class. On MultiNLI matched and mismatched it reaches 53.9% and 52.3%, against 35.4% and 35.2% 1803.02324 tab. 2. Words such as "nobody" and "no" are tied to the contradiction class, and words such as "outdoors" to entailment 1803.02324 tab. 4. The paper then splits the test sets into hard examples, which this classifier gets wrong, and easy ones. Three NLI models score 84.7, 85.8 and 86.5 on the full SNLI test set, and 69.4, 71.3 and 72.7 on its hard subset 1803.02324 tab. 5. The authors read this as most of the models' success coming from easy examples 1803.02324 §4. The method is a classifier plus a split rule, and the card records no official code. Card: Annotation Artifacts in Natural Language Inference Data.

Energy cost of running a classifier

This is a measurement study, not a method. It runs 88 models on ten tasks, with 1,000 inferences per dataset, one input at a time, and records energy with CodeCarbon on eight A100 GPUs in one AWS region 2311.16863 §3.2 2311.16863 tab. 1. Mean energy per 1,000 inferences is 0.002 kWh for text classification and 2.907 kWh for image generation 2311.16863 tab. 2 2311.16863 §4.1. Zero-shot multi-purpose models (Flan-T5 and BLOOMz) cost more than task-specific models on the same task, most clearly for classification and question answering 2311.16863 §4.2. The paper states a grid carbon intensity that its own data do not fully match, so treat the carbon figures with care and read the energy figures first 2311.16863 §3.2. Card: Power Hungry Processing: Watts Driving the Cost of AI Deployment?.

When to use which

Implementation sketch

Few-shot path, after 2209.11055 §3.1. The paper sets the number of pairs per class and the training epochs. The official repository's defaults for the pair count and the body learning rate differ from the paper, so set them explicitly when you reproduce a result (Efficient Few-Shot Learning Without Prompts).

# Build labelled pairs from the K examples  [arxiv:2209.11055 §3.1]
# Same label -> target 1.0; different label -> target 0.0
pairs = []
for i, j in sampled_index_pairs(K):      # R pairs per class in the paper
    target = 1.0 if label[i] == label[j] else 0.0
    pairs.append((text[i], text[j], target))

# Tune the encoder so the cosine similarity of each pair matches its target
# [arxiv:2209.11055 §3.1]; 1 epoch, batch 16 [arxiv:2209.11055 §4.4]
encoder = fine_tune_cosine(encoder, pairs)

# Fit a logistic-regression head on the embeddings of the K examples
# [arxiv:2209.11055 §3.1]
head = fit_logistic_regression([encoder.encode(t) for t in text], label)

# Classify an unseen text  [arxiv:2209.11055 §3.1]
predicted = head.predict(encoder.encode(new_text))

TF-IDF baseline, after 2405.11619 §2. The paper does not state whether the vocabulary and IDF weights were fitted on training rows only. Fit them on training rows, so that test information does not leak into the model.

# Weight words by TF-IDF; fit on training texts only  [arxiv:2405.11619 §2]
vectorizer = TfidfVectorizer()
train_x = vectorizer.fit_transform(train_texts)
test_x = vectorizer.transform(test_texts)

# Linear-kernel SVC with a fixed random state  [arxiv:2405.11619 §2]
# The regularisation strength is not stated in the paper; set and report it
model = SVC(kernel="linear", random_state=42)
model.fit(train_x, train_labels)
predicted = model.predict(test_x)

Multi-task path, after 1901.08014 §2. The paper gives equal priority to its two losses and sums them. The released code of 1806.03713 weights its losses differently, so state which choice you use.

# Shared encoder, one output head per task  [arxiv:1901.08014 §2]
hidden = shared_encoder(text)
loss_a = cross_entropy(head_a(hidden), labels_a)
loss_b = cross_entropy(head_b(hidden), labels_b)

# Equal priority: a plain sum of the task losses  [arxiv:1901.08014 §2]
loss = loss_a + loss_b

Robustness check, after 2111.02840 §3.3 and tab. 4:

# Score a model on adversarial dev examples for one task  [arxiv:2111.02840 §3.5]
correct = 0
for example in adversarial_dev_set:
    if model.predict(example.text) == example.label:
        correct += 1
accuracy = correct / len(adversarial_dev_set)
# Compare with clean GLUE accuracy on the same task; large gaps are expected
# [arxiv:2111.02840 tab. 4]

Relabel-then-stack path, after 2102.00225 §5. The paper's second model takes the first model's one-hot output as an extra input. Relabelling is by hand, so check the set you relabel on a held-out sample first.

# Train the first model on the original labels  [arxiv:2102.00225 §5]
model_a = fine_tune(encoder, texts, labels)

# Find the items where model A disagrees with the label, then relabel them
disagree = [i for i in range(len(texts))
            if model_a.predict(texts[i]) != labels[i]]
corrected = relabel_by_hand(texts, labels, disagree)

# Train model C on corrected labels, with model A's one-hot output as input
# [arxiv:2102.00225 tab. 2]
model_c = fine_tune(encoder, texts, corrected,
                    extra=one_hot(model_a.predict(texts)))

Contradictions

Two cards compare word-vector and TF-IDF features with different results, and the comparisons cannot be ranked against each other. On medical sentences, a CNN over word2vec reaches 0.68 against 0.51 for bag-of-words with logistic regression 1704.06841 fig. 2. On a merged email set, a linear SVC on TF-IDF reaches F1 0.99 while the Word2Vec-based classifiers reach F1 0.71 to 0.83 2405.11619 tab. 2. The two papers use different data, tasks, baselines and Word2Vec settings, and the email paper does not state how word vectors are turned into document vectors 2405.11619 §2.

On user-level text, a linear SVM on TF-IDF is best on one of nine conditions and FastText on seven 1806.05258 tab. 4. The phishing paper reports TF-IDF with a linear SVM as best among its three classifiers, and it tests no neural model 2405.11619 tab. 2. The two results do not conflict, because the tasks and inputs differ, but they show that a linear winner on one dataset does not carry over.

On binary sentiment, the pattern flips for the DNN. TF-IDF beats word embeddings for the DNN on five of eight sets, and the two tie on one [arxiv:2006.03541 tab. 2]. The embeddings win for the CNN and the LSTM on all eight. The TF-IDF rows feed neural models, not a linear classifier, so they do not test the linear-model result above. The paper gives one number per cell and no spread, so the small gaps in that table may be noise 2006.03541 tab. 2.

The few-shot results and the adversarial results measure different things. The few-shot paper does not test altered inputs 2209.11055 §4.4. The adversarial benchmark trains on full GLUE training data rather than few-shot samples 2111.02840 §3.1. The two sets of numbers cannot be compared directly, because the few-shot results are on five other English datasets 2209.11055 tab. 2.

The annotation numbers use different references. The LLM-annotation paper measures accuracy against trained annotators and intercoder agreement on four samples 2303.15056 §2. The relabelling paper measures accuracy against hand-checked labels on 5,000 items from one private dataset 2102.00225 tab. 2. Neither result can be read as a rate for the other, and the first paper does not give per-task accuracy in its text 2303.15056 fig. 1.

The two newest results are not comparable with each other either. CT-BERT reports F1 on five English sets against BERT-Large, under one recipe for both models 2005.07503 tab. 2. MARBERT reports F1 on one private Arabic dataset, and its comparison rows come from other papers and datasets 2606.25495 tab. 3. The MARBERT abstract's claim of accuracy gains over existing techniques is therefore not supported by a baseline on the same data 2606.25495 tab. 2.

The two zero-shot LLM results also differ. The annotation paper reports ChatGPT about 25 points above crowd-workers on accuracy against trained annotators, with intercoder agreement of 91% to 97% on four samples 2303.15056 §2. The severity paper reports GPT-3.5-turbo at macro F1 0.17 on four depression-severity levels, against gold labels, with the English run as its only baseline 2410.12985 tab. 2. The tasks, label scales and references differ, and macro F1 weights the rare classes equally, so the two numbers cannot be ranked against each other. Together they show that zero-shot LLM labels depend strongly on the task.

Papers

• Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks: an LSTM or CNN encoder with a history-aware two-layer classifier for dialog acts, with author-reported best accuracies on three dialog datasets.
• Medical Text Classification using Convolutional Neural Networks: a CNN over word2vec vectors for 26 medical sentence categories, with author-reported gains over simple baselines on an unreleased dataset.
• Annotation Artifacts in Natural Language Inference Data: a hypothesis-only fastText classifier that beats the majority class on SNLI and MultiNLI, and hard-subset scores well below full-test scores for three NLI models.
• All-in-one: Multi-task Learning for Rumour Verification: one shared LSTM for stance, detection and veracity on Twitter reply branches, with small author-reported gains over single-task models.
• SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions: a Reddit dataset of self-reported diagnoses with matched controls, and baseline comparisons of bag-of-words, FastText and CNN models.
• Sentiment and Sarcasm Classification with Multitask Learning: a shared GRU with attention and a tensor fusion for joint sentiment and sarcasm labels, reported on 994 sentences.
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment: a black-box word-substitution attack that cuts accuracy of several classifiers, with author-reported human checks.
• Learning From How Humans Correct: a case study of relabelling disagreeing examples in an industrial text classifier, then training with the first model's output as input.
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks: zero-shot ChatGPT annotation compared with crowd-workers and trained annotators on four samples, with author-reported agreement and cost figures.
• ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation: a benchmark of real chatbot prompts, showing that detectors trained on other domains transfer poorly.
• Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection: TF-IDF with linear and tree classifiers on merged spam emails, with LIME explanations and one reported split.
• Adversarial GLUE: an adversarial test set for five GLUE classification tasks, with large author-reported drops from clean GLUE scores.
• Efficient Few-Shot Learning Without Prompts: contrastive fine-tuning of a sentence encoder plus logistic regression, for few-shot text classification without prompts.
• Ex Machina: Personal Attacks Seen at Scale: crowd-labelled Wikipedia talk comments and a bag-of-n-grams classifier, checked against small annotator ensembles and run on 63 million comments.
• Text Classification Algorithms: A Survey: a survey of features, dimensionality reduction, classifiers and metrics, with qualitative comparisons and no experiments.
• BAE: BERT-based Adversarial Examples for Text Classification: a black-box attack that uses BERT's masked-LM guesses as word replacements and insertions, with author-reported gains over TextFooler.
• Sentiment Analysis Based on Deep Learning: A Comparative Study: a DNN, CNN and LSTM compared on word embeddings and TF-IDF for binary sentiment on eight tweet and review sets, with author-reported accuracy and training times.
• Power Hungry Processing: Watts Driving the Cost of AI Deployment?: energy and carbon per 1,000 inferences for 88 models on ten tasks, with text classification among the cheapest tasks measured.
• COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter: BERT-Large continued on 22.5M COVID-19 tweets, with author-reported mean F1 gains on five classification sets.
• Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model: MARBERT fine-tuned for five-class sentiment and for spam on private telecom tweets, with per-class F1 and no baseline run on its own data.
• KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media: a language-specific BERT-base with a CNN over its last four layers, with author-reported macro-F1 on three OffensEval 2020 sets.
• Data Augmentation Approaches in Natural Language Processing: A Survey: a survey that groups text augmentation into paraphrasing, noising and sampling, with the label risks of each, and no experiments.
• A New Generation of Perspective API: Efficient Multilingual Character-level Transformers: a byte-level multilingual toxicity encoder with a learned tokenizer and a multi-label head, with author-reported AUC-ROC on production, code-switched and public sets.
• Leveraging LLMs for Translating and Classifying Mental Health Data: GPT-3.5-turbo zero-shot depression-severity classification in English and in Greek translated by the same model, with low author-reported macro F1.