Concept · 2 papers · updated 2026-11-15
Pointwise mutual information
PMI compares how often two things occur together with how often they would by chance. Two uses are covered: as a scorer that divides out an answer's prior for zero-shot multiple choice, and as a rule that keeps only positive-PMI pairs as graph edges over byte values. Both results are author-reported.
scoring-functionszero-shot-inferencegraphsstatistics
Markdown3 min read
AI-compiled overview, updated 2026-11-15, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
Pointwise mutual information (PMI) is a log-ratio. It asks whether two things occur together more often than their individual frequencies would predict. Two papers in this wiki use it in different ways. In zero-shot scoring, the log-probability of an answer given a question is corrected by the log-probability of that answer alone 2104.08315 §3.2. In traffic classification, byte values are linked into a graph when they co-occur inside short sliding windows more often than chance 2307.16713 §3.1. The invariant both share is that the score is a difference of logs, so a frequent item is not favoured just for being frequent.
Variants
Domain-conditional PMI as a scorer
Each answer option y is scored by log P(y | x) minus log P(y | x_domain), where x is the question and x_domain is a short, task-specific domain premise such as the ending of a prompt 2104.08315 §3.2 eq. 2. The domain term estimates the option's prior, so answers with many valid wordings are not penalised for those wordings. The method needs no training, and the paper applies it to a causal language model without fine-tuning [arxiv:2104.08315 §4.1]. The full description, results and pitfalls are on the likelihood-based text evaluation page; the paper card is Surface Form Competition.
Positive-PMI co-occurrence edges
Here PMI is an edge test. Within one packet's bytes, a window of w consecutive bytes slides along the packet. Two distinct byte values become connected if their PMI, computed from window counts, is positive 2307.16713 §3.1 eq. 3. The graph is undirected because PMI is symmetric [arxiv:2307.16713 §3.1]. The paper names PMI but gives no formula. The reference code uses a window-count form; the details are recorded on the TFE-GNN card. The paper's default window is 5 bytes 2307.16713 §4.1.3, and it reports that smaller windows gave better F1 because larger windows made the graphs too dense 2307.16713 §4.6.
When to use which
| Situation | Use | Why |
|---|---|---|
| Multiple choice with a causal LM, no labels, answer options vary in how many wordings they have | Domain-conditional PMI as a scorer 2104.08315 §3.2 | Best or tied on 62.5% of splits for GPT-3 175B, against 18.75% for length-normalised log-likelihood 2104.08315 tab. 2 (author-reported) |
| Datasets such as HellaSwag or ARC-Easy, where a baseline already wins | Compare the baselines first | The paper reports that HellaSwag is best with the length-normalised score and ARC-Easy with the raw probability 2104.08315 §6 |
| Few-shot prompts on SST-2 with a large model | Measure raw probability too | At GPT-3 6.7B the raw probability scored 92.9 and the PMI score 79.8 2104.08315 tab. 4 |
| Discrete symbols (for example bytes) that should become a sparse graph before a GNN | Positive-PMI co-occurrence edges 2307.16713 §3.1 | Only pairs that co-occur more than chance are linked, which keeps the graph small |
| Graph too dense or slow | Shrink the window | The paper reports better F1 with smaller windows 2307.16713 §4.6 |
| Premise or domain template is unknown for a new task | Budget time for writing it | The domain premise is hand-written per dataset, and the paper reports no ablation over alternative premises 2104.08315 §3.2 |
Implementation sketch
Both variants share one core step: compute a log-ratio, then keep or compare by its sign or value. The two sketches below are our own pseudocode.
# Domain-conditional PMI selection [arxiv:2104.08315 §3.2] [eq. 2]
# Input: premise x, options [y_1..y_n], domain premise x_dom
# Output: index of the chosen option
def select_dcpmi(premise, options, domain_premise):
scores = []
for option in options:
# Log-probability of the option given the question [§3.2]
conditional = log_prob(option, context=premise)
# Log-probability of the same option given only the domain premise [§3.2]
prior = log_prob(option, context=domain_premise)
# The difference is the PMI-style score [eq. 2]
scores.append(conditional - prior)
return argmax(scores)# Positive-PMI co-occurrence edges [arxiv:2307.16713 §3.1] [eq. 3]
# Input: byte values of one packet, window size w
# Output: undirected edges between distinct byte values
def pmi_edges(values, w):
# Slide a window of w bytes over the packet [§3.1]
windows = sliding_windows(values, w)
n = len(windows)
# Windows that contain each value [§3.1]
in_window = count_windows_containing(windows)
# Windows in which each distinct pair co-occurs [§3.1]
pair_count = count_pairs_per_window(windows)
edges = []
for (i, j), count in pair_count.items():
# PMI from window counts; the reference code's form, see card
pmi = log(count * n / (in_window[i] * in_window[j]))
# Keep only positive-PMI pairs [eq. 3]
if pmi > 0:
edges.append((i, j))
return edgesContradictions
None known. The two papers use PMI for different purposes, one as a scorer over model outputs and one as a graph-edge rule over byte windows, so their results do not compete. Within the first paper, the authors note that the argmax under the two conditioning directions is not always identical in practice 2104.08315 §5.1. That is a caveat about symmetry, not a conflict between papers.
Papers
- Surface Form Competition: domain-conditional PMI as a training-free scorer for zero-shot multiple choice.
- TFE-GNN: PMI between byte values in sliding windows as the positive-edge rule for per-packet graphs.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.