Surface Form Competition: Why the Highest Probability Answer Isn't Always Right

In plain words

A language model can answer a multiple choice question without any training: you show it the question, then compare how likely it finds each answer. The trouble is that the model spreads its belief over every way of writing an answer. If the right answer is "bathtub" and the model also likes "a bathtub" and "bath", the probability for the correct idea gets split across those forms. A rare but correct wording, such as "whirlpool bath", can then lose to a common wrong one, even though a human sees it is right.

The paper's fix is to ask a different question. Instead of "how likely is this answer after the question?", it asks "how much more likely does the question make this answer, compared with how likely the answer is anyway in this kind of task?" The second part is estimated with a short generic prompt that ends the same way the question does, so the common wordings of an answer are divided out. The method needs no training and no labelled examples, only a short prompt written by hand for each task.

The authors report that this is the best or tied-best scoring rule on most of their multiple choice datasets, for every GPT-2 and GPT-3 size they tested.

TL;DR

• Zero-shot multiple choice scores each option by how likely a causal language model finds it. Surface-form competition lets many wordings of one correct answer split probability, so rare correct wordings lose §1 §2.
• Domain Conditional PMI scores option y as log P(y | x) minus log P(y | x_domain), where x_domain is a short task-specific "domain premise" §3.2 eq. 2.
• No training. GPT-2 (125M to 1.6B) and GPT-3 (2.7B to 175B) on 16 splits of 13 datasets. Author-reported: best or tied on 62.5% of splits for GPT-3 175B, against 18.75% for length-normalised log-likelihood tab. 2.
• It loses on HellaSwag, ARC-Easy and BoolQ §6. Code is public, but the repository has no LICENSE file.

Problem

Generative language models assign a probability to every possible string. Several strings can express the same answer ("USA", "United States", "U.S. of A."), and they compete for the same probability mass. The probability of the correct option is therefore lower than its meaning deserves, and options with fewer lexical paraphrases are favoured §1 §2. The paper argues that this is the main reason raw probability underperforms in zero-shot multiple choice with GPT-2 and GPT-3 §5.1.

The paper does not assume fine-tuning or a labelled validation set. All methods use the models unchanged §4.1.

Method

The paper defines no new model or training procedure. It proposes a scoring function, applied to any causal language model and any prompt §3.

Baselines. Let x be the premise (question) and y_1..y_n the options.

• LM (raw probability). Pick argmax_i P(y_i | x) eq. 1 §3.1.
• Avg (length-normalised log-likelihood). For a causal model, log P(y | x) is a sum of token log-probabilities. Avg divides that sum by the number of tokens in y and takes the argmax §3.1.
• Unc (unconditional, in-domain). Pick argmax_i P(y_i | x_domain), ignoring the premise §3.3.
• CC (contextual calibration). Taken from Zhao et al. (2021) for five datasets only; the paper does not rerun it §3.3 tab. 1.

Domain Conditional PMI (the method). Pointwise mutual information is log P(y|x) − log P(y), which factors out the prior of the option. In a multiple choice question the premise is fixed across options, so ranking by PMI is ranking by the premise's probability given each option §3.2. The paper estimates the prior P(y) inside the domain instead of over the whole model, using a short domain premise x_domain such as "the answer is:". The score is

exp PMI_DC(x, y, domain) = P(y | x, domain) / P(y | domain)  [§3.2]

with P(y | x, domain) taken to equal P(y | x), and P(y | domain) estimated as P(y | x_domain). The unnumbered display equations in §3.2 give the same formula. Scores are compared on the log scale, so the rule is an argmax of log P(y | x) − log P(y | x_domain) §3.2.

Domain premises are written per dataset and are usually the ending of the premise, for example "the answer is:" for ARC and CommonsenseQA, "true or false? answer:" for RTE, and "The quote has a tone that is" for SST-2. Full templates are in tab. 7 and Appendix B §3.2.

# Domain Conditional PMI selection  [§3.2] [eq. 2]
# Input:  premise x (str), options [y_1..y_n] (str), domain premise x_dom (str)
# Output: index of the chosen option

def select_dcpmi(premise, options, domain_premise):
    # Score each option: log P(y|x) - log P(y|x_dom)  [§3.2] [eq. 2]
    scores = []
    for option in options:
        # Summed token log-probabilities of the option given the premise  [§3.2]
        conditional = sum_log_prob(option, context=premise)
        # Same option given only the domain premise (its in-domain prior)  [§3.2]
        domain = sum_log_prob(option, context=domain_premise)
        scores.append(conditional - domain)
    return argmax(scores)

# Length-normalised log-likelihood (Avg)  [§3.1]
# Input:  premise x (str), options [y_1..y_n] (str)
# Output: index of the chosen option

def select_avg(premise, options):
    scores = []
    for option in options:
        # Sum of log P(y^j | x, y^<j) over the option's tokens  [§3.1]
        total = sum_log_prob(option, context=premise)
        # Divide by the number of tokens in the option  [§3.1]
        scores.append(total / num_tokens(option))
    return argmax(scores)

Scoring by premise (diagnostic, COPA only). Reversing cause and effect turns each option into the premise and makes the premise the answer. The paper then scores argmax_i P(x | y_i), which removes competition between different option strings §5 fig. 3. Causal models cannot compute this directly, so the experiment uses the COPA Flipped rewrite §5 fig. 3.

Symmetry of PMI. The paper shows that the DC-PMI argmax is the same as argmax_i P(x | y_i, domain) / P(x | domain), so it can be computed in either direction §5.1.

Where the method stops: the model still scores each option separately, so it cannot reason about options that refer to each other, such as "all of the above" §7.

Hyperparameters

The method has no trained parameters. The table lists the settings that matter for reproduction.

Results

Author-reported. Setup: zero-shot, 16 splits of 13 datasets, GPT-2 and GPT-3 §4.1 §4.2. Datasets: COPA, StoryCloze, HellaSwag, RACE-M and RACE-H, ARC-Easy and ARC-Challenge, OpenBookQA, CommonsenseQA, BoolQ, RTE, CB, SST-2 and SST-5, AGN, TREC §4.2. Bold marks the best score in each row, as printed in the paper.

Headline (GPT-3 175B, tab. 1). On COPA, DC-PMI reaches 89.2, against 85.2 for LM, 82.8 for Avg and 56.0 for Unc. On OpenBookQA it reaches 58.0, against 33.2 for LM and 43.8 for Avg. On CommonsenseQA it reaches 66.7, against 61.0 for LM and 57.4 for Avg.

Summary across splits tab. 2 (percent of splits where a method is best or tied; rows are model sizes):

The paper states that the smallest margin between DC-PMI and the best competing method is for GPT-3 175B against Avg, and is over 40 percentage points §4.3.

GPT-3 zero-shot accuracy tab. 1 (accuracy in percent, as printed; CC from Zhao et al. where reported).

Robustness on SST-2 across 15 templates tab. 3 (mean ± std; Avg omitted because it equals LM for single-token answers):

Four-shot inference tab. 4 (mean ± std over 5 random sets of 4 examples; SST-2 closed set, CommonsenseQA open set; Avg omitted on SST-2 because it equals LM):

Removing surface form competition on COPA tab. 5 (accuracy as printed in the source; COPA Flipped reverses premise and option, so the target string is fixed):

The paper reads this as evidence that surface form competition explains most of the gap: on COPA Flipped, methods that do not correct for it give the same score as DC-PMI §5.1 tab. 5.

GPT-2 zero-shot accuracy tab. 6 (fractions as printed, not percentages; CC from Zhao et al. where reported):

Template used for each dataset tab. 7. Every dataset has a template with a premise (question), an optional domain premise, and a hypothesis (option). The rows are in tab. 7; the per-dataset strings are in the loaders (see Paper vs code).

Paper vs code

Official code: the repository is linked from the abstract footnote (the paper's own link, so no GitHub search was needed). Its owner peterwestuw matches the co-author Peter West in the author list. The README also names a project page at peterwestuw.github.io, which I did not open. No other repository was used. Commit ffc22c6352f41ae1519c370564c9f4cb993fed75. The repository has no LICENSE file, so no code is quoted here. Code is cited by path and line only.

Score rules match the paper. DC-PMI is the summed conditional log-probability minus the summed domain log-probability, with argmax utils.py utils.py. Avg divides by the number of hypothesis tokens for GPT-2 utils.py, and by the token count after the premise offset for GPT-3 utils.py. Unc is argmin of the domain cross-entropy utils.py. Discrepancies:

• HellaSwag domain premise. The paper's template table marks no domain premise for HellaSwag tab. 7. The code uses the last space-delimited word of the context data_loaders.py. §3.2 says the domain premise is "usually just the ending of the conditional premise" §3.2, which matches the code but not the table.
• Four-shot CommonsenseQA is not in the shipped script. tab. 4 reports CQA few-shot results, but the CQA branch of score.py never passes --n-shot score.py, and the few-shot prefix is built only in the SST-2 loader data_loaders.py. The CQA few-shot numbers cannot be reproduced from the repository as it stands.
• Few-shot sets. The paper averages over 5 random sets §4.5 tab. 4. The script runs one seed per invocation score.py score.py, so the 5-set mean and std need five runs with different seeds.
• Whitespace in domain premises. Some code domain premises include a newline before the last word where tab. 7 shows a space: RTE data_loaders.py, BoolQ data_loaders.py and AGN data_loaders.py.
• Extra score not in the paper. The code also computes an unconditional score using the ":" token as premise, under the key pmi utils.py utils.py. It is printed by score.py score.py but is not in the tables.
• GPT-2 XL size label. The paper labels GPT-2 XL as 1.6B tab. 6. The code loads the checkpoint by name gpt2-xl score.py. The size label is not checked by the code.
• GPT-3 model sizes. The code takes API engine names (ada, babbage, curie, davinci) score.py. Their mapping to the paper's 2.7B, 6.7B, 13B and 175B tab. 1 is not in the code.
• Split for the other datasets. The paper names dev splits for five datasets only §4.2. The script defaults to dev for all score.py.
• Dataset list. The paper says 16 splits of 13 datasets §4.2. The code's dispatch covers the same datasets score.py.

No discrepancy found in the templates for COPA data_loaders.py, StoryCloze data_loaders.py, ARC data_loaders.py, OpenBookQA data_loaders.py, CB data_loaders.py, CommonsenseQA data_loaders.py, SST-2 data_loaders.py, SST-5 data_loaders.py, AGN data_loaders.py and TREC data_loaders.py.

Pitfalls

• The domain premise is the method. Each task needs a hand-written domain premise tab. 7. The LaTeX has no ablation over alternative premises, so their effect is unknown from the paper. Small wording changes (for example a newline before the last word) change the score.
• Exact strings matter. Options carry a leading space and some premises end in a colon or newline data_loaders.py data_loaders.py. Reproducing without these leads to different numbers.
• Cost is about three passes per option. The code runs the conditional, the domain and the extra unconditional pass for every option utils.py. DC-PMI needs only the first two. Premises are short but the cost still scales with options and datasets.
• Left truncation on GPT-2. A premise that does not fit in 1024 tokens together with the option is cut from the start, keeping the end utils.py utils.py. The paper does not mention this. Long RACE passages are most at risk.
• Where the method loses. HellaSwag is best with Avg, and ARC-Easy is best with LM §6. On BoolQ the unconditional baseline is best at GPT-3 2.7B and 13B tab. 1; the paper says only DC-PMI at GPT-3 175B beats the majority baseline there §6.
• Few-shot is not uniformly better. On SST-2 at GPT-3 6.7B, LM scores 92.9 and DC-PMI 79.8 tab. 4. The paper notes LM wins for two models on SST-2 §4.5.
• Hidden infrastructure for GPT-3. Scoring GPT-3 needs the legacy OpenAI completion endpoint with logprobs and echo utils.py. The README describes the beta programme as limited-access README.md. The GPT-3 rows cannot be reproduced without that access.
• Bold marks in tab. 1. The SST-5 row at 13B bolds both the Avg (29.6) and the PMI_DC (19.1) cells. The values are reproduced as printed; the bolding may be a typesetting error.
• Ties count in the summary. tab. 2 counts ties as wins, so a method that ties on many datasets gets a high percentage.
• Symmetry is a theory result. The paper argues the DC-PMI argmax equals the premise-given-option score §5.1, and notes some differences in practice from "so" and "because" not being exact inverses §5.1. Do not assume identical picks in both directions.
• Concept-level only. The paper's lead argument is that surface form competition is the main cause of the gap §5.1. The COPA Flipped experiment supports this for COPA, but the paper does not test it on other datasets.

Lineage

• Builds on: the standard zero-shot scoring of GPT-3 (Brown et al., 2020), which uses length-normalised log-likelihood and a PMI-like score on three datasets §2 §3.1. No card in this wiki.
• Compared against: Unc, LM, Avg, and Contextual Calibration (Zhao et al., 2021), with CC numbers copied from that paper for five datasets §3.3 tab. 2 tab. 1. No card in this wiki.
• Superseded by: not stated in this wiki.