Noisy Channel Language Model Prompting for Few-Shot Text Classification

In plain words

A usual way to get a language model to classify text is to ask it which label word comes next after the text. This paper asks the question the other way round: given that the label is, say, "great", how likely is this review? The model then has to account for every word in the text, not just the one label word, so each example gives it more to learn from. The idea comes from machine translation, where a "noisy channel" model scores the source text given the translation.

The authors test this flip with a fixed, pretrained model on eleven text classification tasks. They try it both with a few worked examples placed in the prompt and with a small learned prompt, and compare each against the usual direction. Across most settings the flipped version is more stable: its results vary less from one wording or sample to the next, and its worst run is better. The paper also gives advice on when to pick which method, based on how many examples there are, whether the labels are balanced, and whether the model must handle labels it never saw in training.

TL;DR

• Score each label by the probability of the input text given that label (a channel model), and pick the best-scoring label. §3
• Works with frozen GPT-2 Large, using in-context demonstrations (concatenated or ensembled) or 20 tuned prompt vectors. §4 §5.3
• Author-reported: channel prompt tuning averages 61.7 (worst case 53.0) across 11 datasets with K=16, versus 57.7 (47.1) for direct head tuning. tab. 4
• Choose channel prompt tuning when examples are few, labels are imbalanced, or unseen labels must be predicted; direct head tuning is competitive with many examples or on TREC and Subj. §6.2 §7

Problem

Prompting a language model with a verbalized label and reading off the label word's probability ("direct" models, P(label | x)) is widely used §1. In few-shot settings it can be unstable: results vary a lot across verbalizers (label wordings) and random seeds, and the worst case is often close to chance §1. The paper focuses on classification where the model stays frozen or changes very few parameters, typically under 0.01% of the total §4.

Method

Formulation

Text classification maps input texts x to labels c in a set C. A verbalizer v maps each label to a natural-language string; for sentiment, v(great) and v(terrible) are short phrases §3. The paper compares three scores §3:

• Direct: P(c_i \mid x), estimated as P_{LM}(v(c_i) \mid x).
• Direct++: P(c_i \mid x) / P(c_i \mid \text{NULL}), a calibrated direct score that divides out the label's probability for a content-free input §3.
• Channel: P(x \mid c_i). Bayes' rule gives P(c_i \mid x) \propto P(x \mid c_i)P(c_i), and the paper assumes a uniform prior P(c_i) = 1/|C|, so only P(x \mid c_i) is computed §3.

tab. 1 lists the input and target for each score. The predicted label is the one with the highest score; the official code implements this as the lowest summed negative log-likelihood per label main.py.

Demonstration methods (no trainable parameters) §4.1

• Zero-shot. Direct uses P_{LM}(v(c_i) \mid x); channel uses P_{LM}(x \mid v(c_i)). §4.1.1
• Concat-based. Prepend K training examples as one sequence, as in Brown et al. §4.1.2. The channel version is P_{LM}(x \mid v(c^1), x^1, \ldots, v(c^K), x^K, v(c_i)) §4.1.2.
• Ensemble-based (the paper's proposal). Run the LM once per demonstration, each with one example, and multiply the probabilities: \prod_{j=1}^{K} P_{LM}(x \mid v(c^j), x^j, v(c_i)) §4.1.3 tab. 1. This uses O(K) memory instead of O(K²) and removes the dependence on example order §4.1.3.

Every method uses the same input format at training and test time §4.2.

Tuning methods (few trainable parameters) §4.2

• Head tuning (direct only): the LM output matrix O is copied and separated from the input embedding, and only O is trained §4.2.1.
• Transformation tuning (direct only): a new h×h matrix U, initialized to the identity, is inserted before the frozen output matrix O, so the output is O U h_x §4.2.2.
• Prompt tuning (direct and channel): n learned vectors are prepended to the input, and only those vectors are trained §4.2.3. Direct scores P_{LM}(v(c_i) \mid u_1 \ldots u_n, x); channel scores P_{LM}(x \mid u_1 \ldots u_n, v(c_i)) §4.2.3.

Channel scores are length-normalized: the summed token losses are divided by the number of scored tokens §4, footnote 1.

Pseudocode

# Predict a label by scoring every label  [§3] [tab. 1]
# Input:  lm (frozen causal LM), verbalizer v, labels C, input text x, demos
# Output: predicted label

def predict(lm, v, labels, x, demos):
    # Score each label; a lower loss means a higher probability
    # [code: main.py#L357-L364]
    losses = {}
    for label in labels:
        # Channel score: explain x given the verbalized label  [§3]
        losses[label] = channel_loss(lm, demos, v[label], x)
    # Return the label with the smallest loss  [code: main.py#L362]
    return min(losses, key=losses.get)

# Mean negative log-likelihood of x given demos and the label  [§4.1.1]
# Input:  lm, demos as (text, label) pairs, label_tokens, x
# Output: scalar loss per input

def channel_loss(lm, demos, label_tokens, x):
    # Start with BOS, then the demonstrations  [code: util.py#L27-L28]
    context = [BOS]
    for demo_text, demo_label in demos:
        # Each demonstration is its verbalized label, then its text
        # [§4.1.2] [code: data.py#L170-L171]
        context += verbalize(demo_label) + tokenize(demo_text)
    # Then the verbalized label of the candidate  [§4.1.2]
    context += label_tokens
    # Input text with a leading space, cut to max_length - 16 tokens
    # [code: data.py#L78-L79] [code: data.py#L86-L92]
    text_tokens = tokenize(" " + x)[: max_length - 16]
    # Score the text and then one EOS token; EOS is scored too
    # [code: util.py#L29-L30]
    targets = text_tokens + [EOS]
    # Summed token losses, divided by the number of scored tokens  [§4 fn. 1]
    total = total_nll(lm, context, targets)
    return total / len(targets)

# Ensemble-based demonstrations: one context per demonstration  [§4.1.3]
# Input:  lm, demos, label_tokens, x
# Output: scalar score (lower is better)

def ensemble_loss(lm, demos, label_tokens, x):
    # Each demonstration is used alone, so the cost grows as O(K)  [§4.1.3]
    total = 0.0
    for demo in demos:
        # One forward pass per demonstration; the losses are summed
        # [code: main.py#L361]
        total += channel_loss(lm, [demo], label_tokens, x)
    return total

# Prompt tuning: initialize n learned input vectors  [§4.2.3] [app. B]
# Input:  embedding_table (vocabulary embeddings), n
# Output: prompt matrix (n, h)

def init_prompt(embedding_table, n):
    # Pick n distinct token ids from the first 5000 ids  [app. B]
    # [code: model_util.py#L123-L126]
    rows = sample_without_replacement(range(5000), n)
    return embedding_table[rows]

# Training loop for prompt, head and transformation tuning  [app. B]
# Input:  lm (frozen), trainable params, train pairs (x, label), lr
# Output: trained params

def tune(lm, params, train_pairs, lr):
    # Adam with no weight decay and no warmup  [app. B]
    # [code: model_util.py#L102-L105] (the code also adds more; see below)
    optimizer = adam(params, lr=lr)
    # Run 100 global steps  [app. B]
    for step in range(100):
        # Batch of 32 examples, or 16 for long inputs  [app. B]
        batch = sample_batch(train_pairs)
        # Mean of the per-example losses over the batch  [code: run.py#L51-L52]
        loss = mean(channel_or_direct_loss(lm, params, batch))
        # Gradient step on the trainable parameters only  [§4.2]
        optimizer.step(loss)
    return params

Dataset and split

Eleven datasets from Gao et al. and Zhang et al.: sentiment (SST-2, SST-5, MR, CR, Amazon, Yelp), question type (TREC), topic (AG News, Yahoo), ontology (DBPedia) and subjectivity (Subj) §5.1 tab. 2. K training examples are sampled uniformly from the true distribution, without class balancing §5.2.

Hyperparameters

Learning rates per dataset and method are in tab. 7 (tuning, main results), tab. 8 (LM size ablation, in the appendix figure) and tab. 9 (K ablation).

Verbalizers tab. 6

The code's exact strings differ in punctuation spacing; see Paper vs code.

Learning rates tab. 7

Learning rates, LM size ablation tab. 8

Learning rates, K ablation tab. 9

Results

Author-reported, from the arXiv v3 source. All runs use GPT-2 Large. Cells show average / worst-case accuracy, in percent, across verbalizers and seeds §5.4 tab. 3 tab. 4. "Worst case" is the minimum over the runs in each cell.

Demonstration methods tab. 3

ZS = zero-shot (4 runs). Concat and ensemble: 20 runs. Per-run standard deviations and best cases are in tab. 11 and tab. 12.

Findings the paper states §6.1:

• Direct++ beats naive Direct in every setting.
• Ensemble beats concat for Direct++ by 7 points on average (55.8 vs 48.8) and about 7 in worst case (43.6 vs 36.9). For plain Direct the gap is smaller (41.4 vs 38.5) tab. 3.
• Among demonstration methods, the strongest channel model (concat, 58.9 average, 50.8 worst) beats the strongest direct model (ensemble Direct++, 55.8 and 43.6) by 3.1 and 7.2 points tab. 3.
• Few-shot channel models beat zero-shot ones on all datasets §6.1.

Tuning methods tab. 4

All methods use GPT-2 Large and 80 runs per cell tab. 4. Findings the paper states §6.2:

• Channel prompt tuning beats direct prompt tuning by 13.3 points on average and 23.5 in worst case (61.7 vs 48.4, 53.0 vs 29.5).
• Direct head tuning beats channel prompt tuning on TREC (54.1 vs 37.1) and Subj (86.3 vs 75.5), where the paper says the task is far from language modeling §6.2.
• Channel prompt tuning beats all finetuning of the LM on Yahoo and DBPedia, with K=16 §6.2.

Generalization to unseen labels tab. 5

Each cell averages over runs where at least one test label was unseen in training (20 runs each) tab. 5.

The paper states that direct models cannot predict unseen labels. Channel prompt tuning outperforms all finetuning on the 2-way datasets, and outperforms head tuning on five datasets, TREC excepted §6.4. Under zero-shot transfer across datasets, channel prompt tuning beats all direct models on all datasets except TREC §6.4 fig. 5.

Ablations (text only; figures not reproduced)

• Number of examples K §6.3 fig. 3: with K up to 16, channel prompt tuning is strong. At K=64, head tuning beats channel prompt tuning (the text says "channel head tuning", which the method sections do not define). With full training data, direct prompt tuning and head tuning both beat channel prompt tuning.
• Label imbalance §6.3 fig. 4: on SST-2 and MR, channel prompt tuning is insensitive to the share of negative labels in training, and beats direct models when that share is small. Near balance, direct head tuning matches or beats it.
• LM size app. C fig. 6: trends hold from GPT-2 Small to XL.

Variance and best case tab. 11 tab. 12 tab. 13

Average rows only; the paper's per-dataset std and best cases are in the appendix.

The paper says channel models have lower variance and higher worst-case accuracy, while the highest best-case results are often from direct models app. C.

Figures

No images are reproduced. The paper's figures are described here and cited.

• fig. 1: Direct and channel models for sentiment on one example.
• fig. 2: Finetuning variants. Trainable boxes are yellow, frozen boxes white.
• fig. 3, fig. 4, fig. 5, fig. 6: described under Ablations above.

Paper vs code

The official repository (https://github.com/shmsw25/Channel-LM-Prompting, commit fb9beeff) is the authors' own implementation README.md. It has no LICENSE file, so no code is quoted verbatim; everything below is pseudocode with citations. The following differences were found:

1. Training length and schedule. The appendix says 100 global steps app. B. The code trains for 400 steps with a linear learning-rate decay over those 400 steps main.py model_util.py run.py. It saves checkpoints at steps 100, 200, 300 and 400 main.py. But the evaluation loop returns after the first checkpoint, which is step 100 main.py. So the reported model is step 100, with the learning rate already at 0.75 of its starting value. The README says the same ("the checkpoint that is trained for 100 steps") README.md.
2. Optimizer extras. The paper says Adam, no weight decay, no warmup app. B. The code uses AdamW with weight decay 0 and epsilon 1e-8 model_util.py, clips the gradient norm to 1.0 run.py, and applies the linear decay described above. Clipping and decay are not in the paper.
3. Loss normalization. The paper divides the loss by the number of tokens in the batch app. B. The code takes the mean over label tokens within each example, then the mean over the batch run.py run.py.
4. Head tuning softmax. The paper's O is over the full vocabulary V §4.2.1. The code remaps the output to the set of tokens that occur in label positions in the training prompts, plus EOS, and the softmax runs over only that set util.py main.py model_util.py.
5. Concat demonstration length. The appendix says the sequence length is multiplied by the number of examples, capped at 1024 app. B. The code multiplies by K for 2-class tasks, by 1.5K for 4 or 5 classes, by 2K for 6 classes, and uses 1024 for anything else. The batch size is then reduced to keep the total tokens fixed main.py.
6. Template spacing. The appendix shows templates without spaces before punctuation (for example "A MASK one.") tab. 6. The code uses " ." and " :" for SST-2, SST-5, MR, CR, TREC and Subj, and no space for AG News, Yahoo, DBPedia, Amazon and Yelp. It also changes "." to " ." for Amazon and Yelp when training on SST-2, MR or CR util.py main.py. These are different token sequences.
7. Channel demonstration order. Table 1 writes the channel concat context as x^1, v(c^1), ..., while the §4.1.2 text writes v(c^1), x^1, ... tab. 1 §4.1.2. The code uses the text order, label first data.py.
8. Direct++ null input. The paper writes P(c | NULL) §3. The code uses the literal string "N/A", and for demonstration methods the null run includes the demonstrations data.py main.py README.md.
9. Input truncation. Not in the paper. The code cuts each text to max_length − 16 tokens, and it cuts demonstrations from the left when the concatenated context is too long data.py util.py.
10. Prompt tuning and BOS. Not in the paper. The code drops the GPT-2 BOS token and puts the 20 prompt vectors in its place, so the input is 19 positions longer util.py.
11. CR input length. The paper names short and long groups without CR app. B. The code treats CR as a long-input dataset main.py.
12. Dropout. Not stated in the paper. The code trains in train mode, so GPT-2 dropout is active, and evaluates in eval mode run.py main.py.
13. NaN check. The code sets a stop flag on a NaN loss, but the flag is never read, and the break only exits the batch loop, so training continues run.py.
14. "Worst case" definition. The paper's worst case is across all runs in a cell (20 or 80) §5.4. The code's worst case is across the 4 verbalizers of one seed main.py. The seed loop and aggregation are not in the repository; the README gives single-run commands only README.md.
15. Learning-rate selection. The paper picks the learning rate by lowest average training loss app. B. That selection loop is not in the repository; the code takes --lr as a single value, default 1e-5 main.py.

Pitfalls

• Concat versus ensemble ambiguity. The text says ensemble beats concat "by 7% absolute" for direct models §6.1. The 7-point gap is for Direct++ (55.8 vs 48.8); plain Direct differs by 2.9 tab. 3.
• Table 5 duplicates. The MR and CR rows of the unseen-label table show identical channel-prompt numbers (80.9/74.8) tab. 5. This may be a copy error; the card does not check it against run logs.
• "Channel head tuning". §6.3 uses this phrase, which the method section does not define. Read it as channel prompt tuning, which is the only channel tuning method in the paper §6.3 §4.2.
• Uniform prior. The channel score ignores P(c), so it does not use class frequencies at all §3. This is the mechanism the paper gives for its robustness to imbalance §7, but it also means the score cannot use a known prior.
• Cost. Channel scoring runs the LM over the full input once per label, and once more per demonstration for the ensemble, so inference cost grows with the number of classes. The compiled text reports no wall-clock or memory measurements beyond the O(K) versus O(K²) remark §4.1.3.
• Silent truncation. Long inputs are cut to max_length − 16 tokens, and concat contexts are cut from the left. Neither is in the paper; both change the scored text data.py util.py.
• Verbalizer sensitivity. Results depend on the exact verbalizer strings, including spacing and punctuation. Reproduce the code's strings, not the appendix's tab. 6 util.py.
• Labels are indices. The code maps label ids 0 to n−1 to the verbalizer list in order (for SST-5, index 0 is "terrible"). The data files must use the same ids util.py.
• Head tuning limits. Because the head's output is restricted to the training label-token set, it can predict only those tokens. Unseen labels are impossible for direct head tuning, which matches §6.4 but is a design choice to note util.py.
• Data preparation. The k-shot splits come from a separate script. The default is K examples in total, not per class, and the README's download is about 776 MB generate_k_shot_data.py README.md.
• Scope. Classification only, causal LMs only, and the paper names non-classification tasks as future work §7.

Lineage

• Builds on: noisy-channel models in machine translation, cited as §1 and §2 (not carded).
• Builds on: concatenated demonstrations from Brown et al., arXiv:2005.14165 (not carded).
• Builds on: prompt tuning from Lester et al., arXiv:2104.08691 (not carded).
• Builds on: Direct++ calibration from Zhao et al., arXiv:2102.09690, and Holtzman et al., arXiv:2104.08315 (not carded).
• Builds on: the verbalizers and data split from Gao et al., arXiv:2012.15723 (not carded).
• Compared against: direct models (zero-shot, concat, ensemble), direct head, transformation and prompt tuning, and standard finetuning §6.
• Superseded by: nothing in this wiki.