---
kind: concept
slug: noisy-channel-model
title: Noisy channel model
summary: >-
  Scores each label by how well it explains the input, P(x | label), instead of P(label | x). With frozen GPT-2 Large on 11 text classification datasets and 16 examples, the author-reported channel prompt tuning averages 61.7% against 57.7% for direct head tuning.
tags: [prompting, few-shot-learning, text-classification, noisy-channel]
papers: ["1606.01541", "2108.04106"]
related: [few-shot-learning, in-context-learning, soft-prompt-tuning, prompt-engineering]
status: draft
updated_at: 2026-10-19
---

## Overview

A classifier built on a language model can score a label in two directions.
The direct score asks how likely the label is given the input, P(label | x).
The channel score asks how likely the input is given the label, P(x | label),
and picks the label that makes the input most likely. The idea comes from
machine translation, where the source text is scored given the translation
[arxiv:2108.04106 §1]. The shared rule is that the label with the best score
wins. The channel score assumes a uniform label prior, so it never uses class
frequencies [arxiv:2108.04106 §3].

## Variants

Each variant scores a label by the loss of the input under a verbalization of
that label, a natural-language string such as "great" or "terrible"
[arxiv:2108.04106 §3]. The variants differ in how the label is conditioned:
on zero or more demonstrations, or on learned prompt vectors.

### Zero-shot channel

Score the input given the verbalized label alone, with no demonstrations
[arxiv:2108.04106 §4.1.1]. Averaged over 11 datasets, the zero-shot channel
score reaches 52.0% (worst case 47.0%), against 50.5% (43.6%) for zero-shot
Direct++ [arxiv:2108.04106 tab. 3]. Card: [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106).

### Concatenated demonstrations

Put K labelled examples in one sequence, then score the input after the
candidate label [arxiv:2108.04106 §4.1.2]. This is the strongest demonstration
method in the paper: 58.9% average (worst case 50.8%), against 48.8% (36.9%) for
concatenated Direct++ [arxiv:2108.04106 tab. 3]. The context grows with K, and
the paper gives its cost as O(K²) [arxiv:2108.04106 §4.1.3]. Card: [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106).

### Ensemble demonstrations

The paper's proposal. Run the LM once per demonstration, each with a single
example, and combine the scores by multiplying probabilities [arxiv:2108.04106
§4.1.3] [arxiv:2108.04106 tab. 1]. Memory is O(K) rather than O(K²), and the
result does not depend on example order [arxiv:2108.04106 §4.1.3]. The
ensemble channel score averages 57.3% (worst case 49.8%), a little below
concatenated channel [arxiv:2108.04106 tab. 3]. Card: [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106).

### Channel prompt tuning

Freeze the LM, prepend 20 learned vectors to the input, and train only those
vectors [arxiv:2108.04106 §4.2.3] [arxiv:2108.04106 app. B]. The trainable
share is typically under 0.01% of the model [arxiv:2108.04106 §4]. This is the
highest average among the tuning methods in the paper: 61.7% (worst case 53.0%),
against 57.7% (47.1%) for direct head tuning and 48.4% (29.5%) for direct prompt
tuning [arxiv:2108.04106 tab. 4]. Card: [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106).

### Mutual-information score in dialogue

This is not a classifier. A dialogue model is trained toward a score that
combines a forward model and a backward model [arxiv:1606.01541 §4.2]. The
paper does not use the term "noisy channel", and its exact formula for the
score is not written out in the card. Our reading is that the backward model
plays the channel role, scoring the input given the reply, which is the same
direction as P(x | label) above. The score is then used to start a
policy-gradient stage [arxiv:1606.01541 §4.3]. Card: [Deep Reinforcement Learning for Dialogue Generation](/papers/1606.01541).

## When to use which

The paper's results come from GPT-2 Large with K=16 unless noted. Choose by
your constraints, and treat the numbers as author-reported.

| Situation | Use | Why |
|---|---|---|
| Frozen model, a few labelled examples, and you can afford a short tuning run | Channel prompt tuning | Highest tuning average and worst case at K=16 [arxiv:2108.04106 tab. 4]; beats direct head tuning on most datasets, though not on TREC, AG News average, or Subj [arxiv:2108.04106 tab. 4] [arxiv:2108.04106 §6.2] |
| Labels not seen in training must be predicted | Channel prompt tuning, or zero-shot channel | Direct models cannot predict unseen labels; channel prompt tuning beats all finetuning on the 2-way datasets [arxiv:2108.04106 §6.4] |
| Label shares in training are skewed, with few examples | Channel | On SST-2 and MR, channel prompt tuning is insensitive to the share of negative labels and beats direct models when that share is small [arxiv:2108.04106 §6.3] [arxiv:2108.04106 fig. 4] |
| No labelled data and no tuning, only prompting | Zero-shot channel | 52.0% average against 50.5% for zero-shot Direct++ [arxiv:2108.04106 tab. 3] |
| Many examples (K=64) or full training data | Direct head or direct prompt tuning | At K=64, head tuning beats channel prompt tuning; with full data, direct prompt and head tuning both beat it [arxiv:2108.04106 §6.3] [arxiv:2108.04106 fig. 3] |
| Near-balanced labels with plenty of data | Direct head tuning | Matches or beats channel prompt tuning near balance [arxiv:2108.04106 §6.3] [arxiv:2108.04106 fig. 4] |
| Task far from language modelling, such as question type or subjectivity | Direct head tuning | Beats channel prompt tuning on TREC (54.1% against 37.1%) and Subj (86.3% against 75.5%) [arxiv:2108.04106 §6.2] [arxiv:2108.04106 tab. 4] |
| Many classes and a tight latency budget | Direct scoring | Channel scoring needs a separate pass over the input for each label, and once more per demonstration in the ensemble form. The paper gives no wall-clock or memory measurements beyond the O(K) and O(K²) remarks [arxiv:2108.04106 §4.1.3] |

## Implementation sketch

The core is the same for every demonstration-based variant. Prompt tuning
replaces the demonstrations with trained vectors in front of the input.

```pseudocode
# Predict a label by its channel score  [arxiv:2108.04106 §3]
# Input:  lm (frozen causal LM), verbalizer v, labels C, input x, demos
# Output: predicted label

def predict(lm, v, labels, x, demos):
    # Score each label; a lower loss means a higher P(x | label)
    losses = {c: channel_loss(lm, demos, v[c], x) for c in labels}
    # Return the label with the smallest loss
    return min(losses, key=losses.get)
```

```pseudocode
# Mean negative log-likelihood of x given demos and the label  [arxiv:2108.04106 §4.1.2]
# Input:  lm, demos as (text, label) pairs, label_tokens, x
# Output: scalar loss per input

def channel_loss(lm, demos, label_tokens, x):
    # Demonstrations come first, each as its verbalized label then its text
    context = []
    for demo_text, demo_label in demos:
        context += verbalize(demo_label) + tokenize(demo_text)
    # The candidate label comes last, before the scored input  [§4.1.2]
    context += label_tokens
    # Score the input text and one end-of-sequence token  [§4 fn. 1]
    targets = tokenize(" " + x) + [EOS]
    # Summed token losses divided by the number of scored tokens  [§4 fn. 1]
    return total_nll(lm, context, targets) / len(targets)
```

```pseudocode
# Ensemble demonstrations: one context per demonstration  [arxiv:2108.04106 §4.1.3]
# Input:  lm, demos, label_tokens, x
# Output: scalar score (lower is better)

def ensemble_loss(lm, demos, label_tokens, x):
    # Each demonstration is scored alone, so memory grows as O(K)
    return sum(channel_loss(lm, [d], label_tokens, x) for d in demos)
```

```pseudocode
# Channel prompt tuning: train only the prompt vectors  [arxiv:2108.04106 §4.2.3]
# Input:  lm (frozen), prompt (n, h), train pairs (x, label), lr
# Output: trained prompt

def tune_channel_prompt(lm, prompt, train_pairs, lr):
    # Adam with no weight decay and no warmup  [arxiv:2108.04106 app. B]
    opt = adam([prompt], lr=lr)
    # The paper reports 100 global steps  [arxiv:2108.04106 app. B]
    for step in range(100):
        # Batch of 32 short inputs or 16 long ones  [arxiv:2108.04106 app. B]
        batch = sample_batch(train_pairs)
        loss = mean(channel_prompt_loss(lm, prompt, x, y) for x, y in batch)
        opt.step(loss)
    return prompt
```

Points to check before reproducing:

- The paper picks the learning rate from {0.1, 0.01, 0.001} by lowest training
  loss, and the learning rate varies by dataset [arxiv:2108.04106 app. B]
  [arxiv:2108.04106 tab. 7].
- Verbalizer strings matter, and the released code's punctuation differs from
  the appendix. The card lists these differences under "Paper vs code"
  (step count, optimizer extras, head softmax, template spacing).
- Silent truncation is not in the paper. The code cuts each input to the
  maximum length minus 16 tokens and cuts demonstrations from the left
  [arxiv:2108.04106 app. B]. Check your own pipeline for the same effect.
- The paper scores only classification, with causal LMs. It names other tasks
  as future work [arxiv:2108.04106 §7].

## Contradictions

The two papers do not compare their scores with each other. The dialogue
paper's score is not a label classifier, and its card does not write out the
formula [arxiv:1606.01541 §4.2]. Within the classification paper:

- The text says ensemble beats concatenation by 7 points for Direct++, while
  the table gap for plain Direct is 2.9 points. Both are correct; they compare
  different scores [arxiv:2108.04106 §6.1] [arxiv:2108.04106 tab. 3].
- §6.3 uses the phrase "channel head tuning", which the method section does
  not define. It reads most naturally as channel prompt tuning, the only
  channel tuning method the paper defines [arxiv:2108.04106 §6.3]
  [arxiv:2108.04106 §4.2].
- The channel prompt rows for MR and CR in the unseen-label table are
  identical (80.9 / 74.8). This may be a copy error; the card does not check
  it against run logs [arxiv:2108.04106 tab. 5].

## Papers

- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106): introduces the channel score for few-shot classification with a frozen GPT-2 Large, ensemble demonstrations and channel prompt tuning.
- [Deep Reinforcement Learning for Dialogue Generation](/papers/1606.01541): starts policy-gradient dialogue training from a forward-and-backward mutual-information model.