---
kind: concept
slug: pointwise-mutual-information
title: Pointwise mutual information
summary: >-
  PMI compares how often two things occur together with how often they would by chance. Two uses are covered: as a scorer that divides out an answer's prior for zero-shot multiple choice, and as a rule that keeps only positive-PMI pairs as graph edges over byte values. Both results are author-reported.
tags: [scoring-functions, zero-shot-inference, graphs, statistics]
papers: ["2104.08315", "2307.16713"]
related: [likelihood-based-text-evaluation, calibration, zero-shot-learning, graph-neural-network, mutual-information-maximization]
status: draft
updated_at: 2026-11-15
---

## Overview

Pointwise mutual information (PMI) is a log-ratio. It asks whether two
things occur together more often than their individual frequencies would
predict. Two papers in this wiki use it in different ways. In zero-shot
scoring, the log-probability of an answer given a question is corrected by
the log-probability of that answer alone [arxiv:2104.08315 §3.2]. In traffic
classification, byte values are linked into a graph when they co-occur inside
short sliding windows more often than chance [arxiv:2307.16713 §3.1]. The
invariant both share is that the score is a difference of logs, so a
frequent item is not favoured just for being frequent.

## Variants

### Domain-conditional PMI as a scorer

Each answer option y is scored by log P(y | x) minus log P(y | x_domain),
where x is the question and x_domain is a short, task-specific domain premise
such as the ending of a prompt [arxiv:2104.08315 §3.2] [eq. 2]. The domain
term estimates the option's prior, so answers with many valid wordings are not
penalised for those wordings. The method needs no training, and the paper
applies it to a causal language model without fine-tuning [arxiv:2104.08315
§4.1]. The full description, results and pitfalls are on the
[likelihood-based text evaluation](/concepts/likelihood-based-text-evaluation)
page; the paper card is [Surface Form Competition](/papers/2104.08315).

### Positive-PMI co-occurrence edges

Here PMI is an edge test. Within one packet's bytes, a window of w consecutive
bytes slides along the packet. Two distinct byte values become connected if
their PMI, computed from window counts, is positive [arxiv:2307.16713 §3.1]
[eq. 3]. The graph is undirected because PMI is symmetric [arxiv:2307.16713
§3.1]. The paper names PMI but gives no formula. The reference code uses a
window-count form; the details are recorded on the
[TFE-GNN card](/papers/2307.16713). The paper's default window is 5 bytes
[arxiv:2307.16713 §4.1.3], and it reports that smaller windows gave better F1
because larger windows made the graphs too dense [arxiv:2307.16713 §4.6].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Multiple choice with a causal LM, no labels, answer options vary in how many wordings they have | Domain-conditional PMI as a scorer [arxiv:2104.08315 §3.2] | Best or tied on 62.5% of splits for GPT-3 175B, against 18.75% for length-normalised log-likelihood [arxiv:2104.08315 tab. 2] (author-reported) |
| Datasets such as HellaSwag or ARC-Easy, where a baseline already wins | Compare the baselines first | The paper reports that HellaSwag is best with the length-normalised score and ARC-Easy with the raw probability [arxiv:2104.08315 §6] |
| Few-shot prompts on SST-2 with a large model | Measure raw probability too | At GPT-3 6.7B the raw probability scored 92.9 and the PMI score 79.8 [arxiv:2104.08315 tab. 4] |
| Discrete symbols (for example bytes) that should become a sparse graph before a GNN | Positive-PMI co-occurrence edges [arxiv:2307.16713 §3.1] | Only pairs that co-occur more than chance are linked, which keeps the graph small |
| Graph too dense or slow | Shrink the window | The paper reports better F1 with smaller windows [arxiv:2307.16713 §4.6] |
| Premise or domain template is unknown for a new task | Budget time for writing it | The domain premise is hand-written per dataset, and the paper reports no ablation over alternative premises [arxiv:2104.08315 §3.2] |

## Implementation sketch

Both variants share one core step: compute a log-ratio, then keep or compare
by its sign or value. The two sketches below are our own pseudocode.

```pseudocode
# Domain-conditional PMI selection  [arxiv:2104.08315 §3.2] [eq. 2]
# Input:  premise x, options [y_1..y_n], domain premise x_dom
# Output: index of the chosen option

def select_dcpmi(premise, options, domain_premise):
    scores = []
    for option in options:
        # Log-probability of the option given the question  [§3.2]
        conditional = log_prob(option, context=premise)
        # Log-probability of the same option given only the domain premise  [§3.2]
        prior = log_prob(option, context=domain_premise)
        # The difference is the PMI-style score  [eq. 2]
        scores.append(conditional - prior)
    return argmax(scores)
```

```pseudocode
# Positive-PMI co-occurrence edges  [arxiv:2307.16713 §3.1] [eq. 3]
# Input:  byte values of one packet, window size w
# Output: undirected edges between distinct byte values

def pmi_edges(values, w):
    # Slide a window of w bytes over the packet  [§3.1]
    windows = sliding_windows(values, w)
    n = len(windows)
    # Windows that contain each value  [§3.1]
    in_window = count_windows_containing(windows)
    # Windows in which each distinct pair co-occurs  [§3.1]
    pair_count = count_pairs_per_window(windows)
    edges = []
    for (i, j), count in pair_count.items():
        # PMI from window counts; the reference code's form, see card
        pmi = log(count * n / (in_window[i] * in_window[j]))
        # Keep only positive-PMI pairs  [eq. 3]
        if pmi > 0:
            edges.append((i, j))
    return edges
```

## Contradictions

None known. The two papers use PMI for different purposes, one as a scorer
over model outputs and one as a graph-edge rule over byte windows, so their
results do not compete. Within the first paper, the authors note that the
argmax under the two conditioning directions is not always identical in
practice [arxiv:2104.08315 §5.1]. That is a caveat about symmetry, not a
conflict between papers.

## Papers

- [Surface Form Competition](/papers/2104.08315): domain-conditional PMI as a training-free scorer for zero-shot multiple choice.
- [TFE-GNN](/papers/2307.16713): PMI between byte values in sliding windows as the positive-edge rule for per-packet graphs.
