Noisy channel model

Overview

A classifier built on a language model can score a label in two directions. The direct score asks how likely the label is given the input, P(label | x). The channel score asks how likely the input is given the label, P(x | label), and picks the label that makes the input most likely. The idea comes from machine translation, where the source text is scored given the translation 2108.04106 §1. The shared rule is that the label with the best score wins. The channel score assumes a uniform label prior, so it never uses class frequencies 2108.04106 §3.

Variants

Each variant scores a label by the loss of the input under a verbalization of that label, a natural-language string such as "great" or "terrible" 2108.04106 §3. The variants differ in how the label is conditioned: on zero or more demonstrations, or on learned prompt vectors.

Zero-shot channel

Score the input given the verbalized label alone, with no demonstrations 2108.04106 §4.1.1. Averaged over 11 datasets, the zero-shot channel score reaches 52.0% (worst case 47.0%), against 50.5% (43.6%) for zero-shot Direct++ 2108.04106 tab. 3. Card: Noisy Channel Language Model Prompting for Few-Shot Text Classification.

Concatenated demonstrations

Put K labelled examples in one sequence, then score the input after the candidate label 2108.04106 §4.1.2. This is the strongest demonstration method in the paper: 58.9% average (worst case 50.8%), against 48.8% (36.9%) for concatenated Direct++ 2108.04106 tab. 3. The context grows with K, and the paper gives its cost as O(K²) 2108.04106 §4.1.3. Card: Noisy Channel Language Model Prompting for Few-Shot Text Classification.

Ensemble demonstrations

The paper's proposal. Run the LM once per demonstration, each with a single example, and combine the scores by multiplying probabilities [arxiv:2108.04106 §4.1.3] 2108.04106 tab. 1. Memory is O(K) rather than O(K²), and the result does not depend on example order 2108.04106 §4.1.3. The ensemble channel score averages 57.3% (worst case 49.8%), a little below concatenated channel 2108.04106 tab. 3. Card: Noisy Channel Language Model Prompting for Few-Shot Text Classification.

Channel prompt tuning

Freeze the LM, prepend 20 learned vectors to the input, and train only those vectors 2108.04106 §4.2.3 2108.04106 app. B. The trainable share is typically under 0.01% of the model 2108.04106 §4. This is the highest average among the tuning methods in the paper: 61.7% (worst case 53.0%), against 57.7% (47.1%) for direct head tuning and 48.4% (29.5%) for direct prompt tuning 2108.04106 tab. 4. Card: Noisy Channel Language Model Prompting for Few-Shot Text Classification.

Mutual-information score in dialogue

This is not a classifier. A dialogue model is trained toward a score that combines a forward model and a backward model 1606.01541 §4.2. The paper does not use the term "noisy channel", and its exact formula for the score is not written out in the card. Our reading is that the backward model plays the channel role, scoring the input given the reply, which is the same direction as P(x | label) above. The score is then used to start a policy-gradient stage 1606.01541 §4.3. Card: Deep Reinforcement Learning for Dialogue Generation.

When to use which

The paper's results come from GPT-2 Large with K=16 unless noted. Choose by your constraints, and treat the numbers as author-reported.

Implementation sketch

The core is the same for every demonstration-based variant. Prompt tuning replaces the demonstrations with trained vectors in front of the input.

# Predict a label by its channel score  [arxiv:2108.04106 §3]
# Input:  lm (frozen causal LM), verbalizer v, labels C, input x, demos
# Output: predicted label

def predict(lm, v, labels, x, demos):
    # Score each label; a lower loss means a higher P(x | label)
    losses = {c: channel_loss(lm, demos, v[c], x) for c in labels}
    # Return the label with the smallest loss
    return min(losses, key=losses.get)

# Mean negative log-likelihood of x given demos and the label  [arxiv:2108.04106 §4.1.2]
# Input:  lm, demos as (text, label) pairs, label_tokens, x
# Output: scalar loss per input

def channel_loss(lm, demos, label_tokens, x):
    # Demonstrations come first, each as its verbalized label then its text
    context = []
    for demo_text, demo_label in demos:
        context += verbalize(demo_label) + tokenize(demo_text)
    # The candidate label comes last, before the scored input  [§4.1.2]
    context += label_tokens
    # Score the input text and one end-of-sequence token  [§4 fn. 1]
    targets = tokenize(" " + x) + [EOS]
    # Summed token losses divided by the number of scored tokens  [§4 fn. 1]
    return total_nll(lm, context, targets) / len(targets)

# Ensemble demonstrations: one context per demonstration  [arxiv:2108.04106 §4.1.3]
# Input:  lm, demos, label_tokens, x
# Output: scalar score (lower is better)

def ensemble_loss(lm, demos, label_tokens, x):
    # Each demonstration is scored alone, so memory grows as O(K)
    return sum(channel_loss(lm, [d], label_tokens, x) for d in demos)

# Channel prompt tuning: train only the prompt vectors  [arxiv:2108.04106 §4.2.3]
# Input:  lm (frozen), prompt (n, h), train pairs (x, label), lr
# Output: trained prompt

def tune_channel_prompt(lm, prompt, train_pairs, lr):
    # Adam with no weight decay and no warmup  [arxiv:2108.04106 app. B]
    opt = adam([prompt], lr=lr)
    # The paper reports 100 global steps  [arxiv:2108.04106 app. B]
    for step in range(100):
        # Batch of 32 short inputs or 16 long ones  [arxiv:2108.04106 app. B]
        batch = sample_batch(train_pairs)
        loss = mean(channel_prompt_loss(lm, prompt, x, y) for x, y in batch)
        opt.step(loss)
    return prompt

Points to check before reproducing:

• The paper picks the learning rate from {0.1, 0.01, 0.001} by lowest training loss, and the learning rate varies by dataset 2108.04106 app. B 2108.04106 tab. 7.
• Verbalizer strings matter, and the released code's punctuation differs from the appendix. The card lists these differences under "Paper vs code" (step count, optimizer extras, head softmax, template spacing).
• Silent truncation is not in the paper. The code cuts each input to the maximum length minus 16 tokens and cuts demonstrations from the left 2108.04106 app. B. Check your own pipeline for the same effect.
• The paper scores only classification, with causal LMs. It names other tasks as future work 2108.04106 §7.

Contradictions

The two papers do not compare their scores with each other. The dialogue paper's score is not a label classifier, and its card does not write out the formula 1606.01541 §4.2. Within the classification paper:

• The text says ensemble beats concatenation by 7 points for Direct++, while the table gap for plain Direct is 2.9 points. Both are correct; they compare different scores 2108.04106 §6.1 2108.04106 tab. 3.
• §6.3 uses the phrase "channel head tuning", which the method section does not define. It reads most naturally as channel prompt tuning, the only channel tuning method the paper defines 2108.04106 §6.3 2108.04106 §4.2.
• The channel prompt rows for MR and CR in the unseen-label table are identical (80.9 / 74.8). This may be a copy error; the card does not check it against run logs 2108.04106 tab. 5.

Papers

• Noisy Channel Language Model Prompting for Few-Shot Text Classification: introduces the channel score for few-shot classification with a frozen GPT-2 Large, ensemble demonstrations and channel prompt tuning.
• Deep Reinforcement Learning for Dialogue Generation: starts policy-gradient dialogue training from a forward-and-backward mutual-information model.