Text Classification via Large Language Models

In plain words

Large language models can label text without any training, just by being shown the task and a few examples in the prompt. On sentiment and topic labels they still trail models that were fine-tuned on the full labelled set. The authors see two causes: the model is weak at the kind of reading that sentiment and topic labels need (sarcasm, contrast, negation), and a prompt can only hold a handful of examples.

The method, CARP, changes both parts. Before the model answers, it is asked to point out the words and phrases that carry the label, then to explain how those clues add up, and only then to give the label. For the examples in the prompt, the system does not pick at random. A small classifier that was fine-tuned on the labelled data finds the training texts most like the new input, and the model writes the clues and explanation for each of those examples ahead of time, so the prompt shows the full reasoning pattern in each example.

The authors report that this gives better accuracy than plain prompting and plain chain-of-thought prompting on five standard benchmarks, and that it does well when labelled data is scarce. All numbers below are the authors' own.

TL;DR

• CARP prompts a GPT-3 model (text-davinci-003) in three steps: list clues, reason over them, then output the label §4.2.
• For few-shot prompts it uses 16 demonstrations per test input, chosen by nearest neighbour search with a RoBERTa-Large encoder fine-tuned on the training set §3.3.
• Each demonstration carries generated clues and reasoning, produced from its text and gold label §4.2.1.
• Author-reported: 97.39 on SST-2, 96.40 on AGNews, 98.78 on R8, 96.95 on R52 (weighted-probability vote); MR 92.39 against a cited 93.3 tab. 2.
• The released code does not contain the weighted-probability vote, and it regenerates demonstration explanations at every test call. Read the Paper vs code section before reproducing.

Problem

Plain in-context learning with an LLM trails fine-tuned encoders on text classification. The authors name two reasons: complex linguistic phenomena (for example intensification, contrast and irony) need more reasoning than a bare label request provides §1, and the context window holds only a few demonstrations, so the model sees little of the training set §1. They also note that reasoning methods of the time were mostly tuned for maths problems §4.2.1.

Assumptions: the task has a fixed label set that can be written as words §3.1; a labelled training set exists, which the method uses both to build demonstrations and to fit the retriever §3.3; the model is accessed through a completion API with a token limit §5 tab. 3.

Method

Prompt structure

A prompt has three parts: a task description, a list of demonstrations, and the test input §3.2. For zero-shot use there are no demonstrations §3.2. Each demonstration is a training text and its label word §3.2.

Zero-shot CARP

The task description asks the model to: (1) list clues such as keywords, phrases, context, semantic relations, tone and references; (2) derive a diagnostic reasoning process from those clues and the input; (3) give the label, Positive or Negative for sentiment, considering the clues, the reasoning and the input §4.2. No example is shown §4.2.

Few-shot CARP

Step 1, demonstration pool. For every training example the authors generate clues and reasoning with the LLM, using the text and its gold label as input §4.2.1.

• Clue call: the prompt asks for clues that support the label, with the limit stated as 15 words §4.2.1.
• Reasoning call: the prompt gives the text, label and generated clues, and asks for the reasoning that supports the label. The authors cap it at 50 words §4.2.1, footnote.

Step 2, retrieval. For a test input, a RoBERTa-Large model is fine-tuned on the training set. The sentence representation is the [CLS] embedding; the k nearest training examples by similarity are the demonstrations, with k = 16 §3.3 §5.

Step 3, ordering. The authors compare random, high-to-low and low-to-high ordering of retrieved demonstrations, and report low-to-high (most similar example closest to the test input) as best §6.5 tab. 10.

Step 4, assembly. Each demonstration becomes the text, then its clues, then its reasoning, then its label word. The test input follows in the same layout, so the model writes clues and reasoning before the label §4.2.1.

Voting

LLM outputs are sampled, so repeated runs can differ. The paper defines two votes: majority vote (most frequent label over runs) and weighted-probability vote (sum of weighted probabilities over runs) §4.3. The number of runs and the weighting are not stated in the paper.

Pseudocode

Few-shot CARP for one test input, following §3.3, §4.2.1 and §6.5:

# CARP few-shot prompt for one test input  [§3.3] [§4.2.1] [§6.5]
# Input:  test_text (str); train_pool (list of (text, label))
# Output: prompt (str)

def build_carp_prompt(test_text, train_pool, retriever, teacher, k=16):
    # 1. Find the k training examples nearest to the test input,
    #    by cosine similarity of fine-tuned [CLS] embeddings  [§3.3]
    neighbours = retriever.top_k(test_text, train_pool, k)
    # 2. Order them from lowest to highest similarity, so the most
    #    similar example sits next to the test input  [§6.5]
    neighbours = sorted(neighbours, key=similarity)
    demos = []
    for text, label in neighbours:
        # 3a. Clues for this example, from text and gold label.
        #     Prompt asks for at most 15 words  [§4.2.1]
        clues = teacher.generate(clue_prompt(text, label))
        # 3b. Reasoning from text, label and clues; cap 50 words  [§4.2.1]
        reasoning = teacher.generate(
            reasoning_prompt(text, label, clues))
        demos.append((text, clues, reasoning, label))
    # 4. Task description, then demonstrations, then the test input  [§4.2.1]
    return task_description + join(demos) + "INPUT: " + test_text

Voting over sampled answers. The weighting is not stated in the paper §4.3:

# Voting over repeated answers  [§4.3]
# Number of runs and weight() are not stated in the paper.
def vote(prompt, model, runs):
    # 1. Sample several answers; temperature 0.7  [tab. 3]
    answers = [model.sample(prompt) for run in range(runs)]
    labels = [parse_label(answer) for answer in answers]
    # 2a. Majority vote: most frequent label  [§4.3]
    majority = most_common(labels)
    # 2b. Weighted probability vote: weighted sum of probabilities  [§4.3]
    weighted = argmax_label(
        sum(weight(a) * prob(a) for a in answers))
    return majority, weighted

Hyperparameters

Model and decoding settings come from the paper's table of API parameters. The fine-tuning settings are from the appendix.

Temperature ablation (SST-2, appendix) app. B.2: tau 0.0 gives 96.39, 0.2 gives 96.48, 0.4 gives 96.40, 0.6 gives 96.59, 0.8 gives 96.68, 1.0 gives 96.70. The authors report CARP as robust to these settings app. B.2 §5.

Results

Author-reported. Setup: five benchmarks (SST-2, AGNews, R8, R52, MR) with the standard train, dev and test splits §5 app. A; text-davinci-003 for all LLM rows; supervised baselines fine-tuned on the full training set §5.1. Accuracy in percent. Table caption: mean and standard deviation over 5 runs, but the table shows means only tab. 2. The paper marks previously published best results with an asterisk; the asterisks are dropped here.

Full training set tab. 2

"Vanilla" is the plain prompt, "CoT" is the chain-of-thought baseline of §5, and "FT kNN" is the fine-tuned RoBERTa-Large retriever §3.3. Author-reported results from the paper's own runs.

Text claims for CARP over CoT: +1.48, +0.97, +2.76, +3.29 and +0.47 on SST-2, AGNews, R8, R52 and MR §5.2. These differences match the SimCSE kNN rows of the table, not the FT kNN rows, which differ by 1.32, 1.10, 2.70, 0.93 and 1.73.

Low-resource settings tab. 4

Training subsets of n examples per class. The authors compare fine-tuned RoBERTa-Large with 16-shot GPT-3 using SimCSE retrieval from the subset tab. 4 caption.

The abstract's claim that 16 examples per class match supervised models with 1,024 per class holds on SST-2, AGNews, R8 and R52, where CARP at n=16 is at or above FT RoBERTa at n=1024. On MR, CARP at n=16 (86.54) is 1.83 points below FT RoBERTa at n=1024 (88.37) tab. 4.

Domain adaptation tab. 5

Task: positive or negative polarity, with SST-2 and Yelp as the two domains §5.4. Accuracy in percent.

Test subsets tab. 6

Results on smaller test subsets (728, 760, 875, 1,027 and 888 instances for SST-2, AGNews, R8, R52 and MR) app. A. The subset-selection procedure is not stated.

Ablations

Components of a demonstration, SST-2 and R8 tab. 7:

Label words, SST-2 tab. 8. The authors' table lists six kinds of label words with the word pairs used for positive and negative. Accuracy for each kind, in the same order: index-style labels 95.66, the dataset's own annotation words 96.86 (best), synonyms 96.27, flipped meanings 64.63 (the only large drop), random vocabulary words 95.06, and special tokens 96.65.

Clue types, SST-2 and R8 tab. 9:

Demonstration ordering, SST-2 tab. 10:

Quality of generated reasoning, 500 sampled (text, reasoning) pairs each for SST-2 and R8 tab. 11. Reliability is judged by a zero-shot GPT-3 yes/no check; logic faithfulness by a 16-shot GPT-3 entailment check with SNLI demonstrations; fluency is perplexity §6.6:

Figures, described in words: fig. 1 shows zero-shot vanilla, CoT and CARP prompts; fig. 2 shows the 1-shot versions; fig. 3 and fig. 4 plot SST-2 accuracy against the number of demonstrations (0 to 24 in steps of 2) for three samplers, with vanilla and CARP prompts respectively. The authors report accuracy rising with more demonstrations for both §6.1.

Paper vs code

Repository: ShannonAI/GPT-CLS-CARP, Apache-2.0, commit 651eba00eaedc7f56a60a545d0c73ab557c5f34c. The abstract links it. Paths below are relative to the repository root.

• Demonstration explanations are generated per test call, not in advance. Paper: clues and reasoning are generated once for every training example §4.2.1. Code: for each test input, every retrieved demonstration is sent to the model with a single instruction that asks for clues and reasoning together, and those calls are repeated for each test input prompt.py gpt3_text_cls.py. No cache of demonstration explanations is visible.
• Two-step generation is one step in code. Paper: a clue call (at most 15 words) and then a reasoning call (at most 50 words) §4.2.1. Code: one combined call carp_16shot_davinci003.json. The 15- and 50-word limits do not appear in the few-shot prompt path.
• Explanation length cap differs. Code clips each generated explanation to 100 words by default and each demonstration text to 200 words by default; the configs do not override these values prompt.py prompt.py. The paper states neither cap, except the 50-word target.
• The demonstration-generation token limit is not applied. The forward call accepts an update_max_tokens argument, but the request function does not use it, so generation uses the global limit of 200 tokens gpt_model.py gpt_model.py.
• Retrieval matches. Fine-tuned RoBERTa, [CLS] embedding, unit-normalised vectors, inner-product index, which is cosine similarity data_retriever.py data_retriever.py. Top-k equals instance_num = 16 carp_16shot_davinci003.json prompt.py.
• Ordering matches. The code reverses the retrieval list for "score_l2h", which is low-to-high prompt.py carp_16shot_davinci003.json.
• Weighted-probability vote is not in the code. Paper: weighted sum of probabilities over runs §4.3, used for the headline numbers tab. 2. Code: no weighting logic exists. The only vote is "hard-vote" with 20 samples gpt3_text_cls.py. It runs only when a reply cannot be mapped to a label, and the CARP configs do not select it carp_16shot_davinci003.json. The 97.39 / 96.40 / 98.78 / 96.95 / 92.39 row cannot be reproduced from the released code alone.
• Out-of-label replies have four policies, and the paper does not say which it reports. Replies that do not map to a label word go to one of: "wrong" (a random label other than the gold label), "random", "retry" (up to 20 re-queries, then random), or "retry-various" gpt3_text_cls.py carp_16shot_davinci003.json. The "wrong" policy uses the gold label to pick its replacement gpt3_text_cls.py.
• "retry-various" never varies the request. The intended change to the request's user field is checked against the string with an underscore, "retry_various", while the policy name uses a hyphen, so the branch never runs gpt3_text_cls.py gpt3_text_cls.py.
• Zero-shot prompt has an extra length limit. The zero-shot CARP config asks for a reasoning of at most 130 words. The paper's zero-shot prompt does not contain this limit carp_davinci003.json §4.2.
• Prompt budget counts words, not tokens. Demonstrations are cut to a 3,800-word prompt budget, and max tokens is set to the smaller of 200 and 3,900 minus the prompt's word count carp_16shot_davinci003.json gpt_model.py. The paper states no prompt budget.
• Fine-tuning settings differ from the appendix. Paper: learning rate {2e-5, 3e-5, 4e-5}, batch {16, 32, 32}, dropout 0.3, weight decay 0.01, warmup 0.01 app. B.1. Script: learning rate {1e-5, 2e-5, 3e-5, 4e-5}, batch 16 with gradient accumulation 2, dropout 0.2, weight decay 0.05, warmup 0.06, max length 200, 5 epochs finetune_roberta_large.sh. The retriever checkpoint named in the CARP config matches the script, not the appendix: learning rate 1e-5, weight decay 0.1, dropout 0.2, warmup 0.06 carp_16shot_davinci003.json. The script requests 8 GPUs finetune_roberta_large.sh finetune_roberta_large.sh, so the effective batch depends on the launch setup.
• Seeds. The run script loops over five seeds: 2333, 8866, 9998, 6624 and 1314 carp_16shot_davinci003.sh. The paper does not list seeds.
• Sampling settings match. Temperature 0.7, max tokens 200, top p 1, penalties 0.0 carp_16shot_davinci003.json tab. 3. The config adds a batch size of 18, which the paper does not state.
• Label parsing. The paper does not describe how a reply becomes a label. The code's zero-shot parser takes the text up to the first newline and matches it against the label words prompt.py. I did not confirm that the few-shot class uses the same parser.

No discrepancy found for the zero-shot prompt wording beyond the length limit above.

Pitfalls

• Cost per test input. The released code makes one generation call per retrieved demonstration for each test input, so 16 explanation calls plus the answer call, with no cache prompt.py. The paper's pre-generation design costs one pass over the training set instead. Budget for the code's behaviour when reproducing.
• Weighted-probability vote is underspecified. The number of runs and the weight function are not stated, and the code has no implementation §4.3 gpt3_text_cls.py. Headline numbers cannot be reproduced without guessing.
• Unmappable replies are unscored in the paper. How invalid outputs count toward accuracy is not stated. The four code policies give different accuracies by design; "wrong" is a pessimistic bound and "retry" re-queries the LLM up to 20 times gpt3_text_cls.py.
• Abstract arithmetic. The abstract gives +1.24 for SST-2 over the previous best. The table's previous best on SST-2 is XLNet at 96.10, so 97.39 minus 96.10 is 1.29 tab. 2. Other rows match their stated gains (AGNews +0.72, R8 +0.25, R52 +0.6).
• "Mean and standard deviation" not shown. The table caption says 5 runs with mean and standard deviation; no standard deviations are printed anywhere in the table tab. 2.
• Which sampler the gains refer to. The text's gains over CoT match the SimCSE kNN rows, not the FT kNN rows, and the paper does not say so §5.2 tab. 2.
• Table inconsistencies. The SimCSE kNN CARP value for SST-2 is 95.69 in the main table tab. 2 but 96.39 for low-to-high ordering in the ordering ablation tab. 10. The 16-shot random-sampler CARP row in the subset table repeats the zero-shot CARP row exactly tab. 6. The label-word ablation reports 96.86 for annotation words, against 96.80 for the same setup in the main table tab. 8.
• Caption and typo errors. The low-resource table calls CARP "CRAP" and its caption describes "cls" and "reason-cls" variants that the rows do not contain tab. 4. The clue ablation caption refers to label words tab. 9. The reliability sentence in §6.6 says "R5" where R8 is meant §6.6.
• "Comparable" to full training is loose. At n=512 per class, CARP is 1.2 to 2.2 points below RoBERTa-Large on the full training set across all five datasets (for example SST-2 94.03 against 95.99) tab. 2 tab. 4. The introduction compares n=1,024 with "more than 1 thousand" examples per class §1, which is not the same as the full set.
• Low-resource retriever is not specified. The low-resource table uses SimCSE retrieval for CARP tab. 4 caption. The text does not say whether the fine-tuned retriever is retrained on each subset.
• Subset construction is not stated. The procedure for the test subsets is not described app. A.
• Yelp source is not specified. The domain-adaptation data is described only by a link and "Yelp-2" §5.4. Size and version are not stated.
• Robustness to temperature. The temperature ablation is on SST-2 only, with one value per setting and no variance app. B.2.
• Component and clue ablations cover two datasets. They run on SST-2 and R8 only, so the importance ranking is not shown for AGNews, R52 or MR tab. 7 tab. 9.
• Reasoning quality checks use the same model family. Reliability and logic-faithfulness are judged by GPT-3 prompts, so they measure agreement with GPT-3, not with humans §6.6.

Lineage

• Builds on: chain-of-thought prompting, from which the paper takes the step-by-step format; its zero-shot baseline is the cited zero-shot CoT method §2 §5.
• Compared against: vanilla in-context learning; the zero-shot CoT baseline; random, SimCSE (Sup-SimCSE-RoBERTa-Large) and fine-tuned nearest-neighbour demonstration samplers; RoBERTa-Large, DeBERTa, RoBERTa-GCN, XLNet, VLAWE and GCN-SB as supervised baselines §5.1 tab. 2.
• Superseded by: not stated in this card.