Likelihood-based text evaluation

Overview

Reference-free and reference-based metrics for generated text usually count word overlap or need human ratings to fit. This family scores a text by the probability a pre-trained generation model assigns to it, given another text. The invariant all variants share: the score uses the model's existing output layer, needs no human judgements to fit and adds no parameters 2106.11520 §1 2106.11520 §3.2.

The source text, a reference, or the hypothesis itself can play the conditioning role, which gives four directions. Each direction answers a different question about the candidate 2106.11520 §3.2.

A second use appears in language-model papers, which report perplexity rather than a metric between two texts. Here the likelihood is either measured with a fixed external model over a set of samples, or bounded because the model's own likelihood is hard to compute 2604.11748 §5 2604.11748 thm. 3.1. These numbers measure different things from BARTScore, so compare them only within their own family.

A third use is benchmark probing. The model is shown a question with several candidate answers, and the candidate with the highest conditional likelihood is taken as its answer. The score is then an accuracy, not a similarity 2210.13312 app. C. Multiple-choice scoring has a known bias: an answer with many valid wordings splits probability across them, so it can lose to a rarer correct wording 2104.08315 §1. The domain-conditional variant below corrects for this.

Variants

Vanilla BARTScore

Average token log-likelihood under the pre-trained BART encoder-decoder, with every token weighted equally. Three weighting schemes tried by the authors (skipping stop words, IDF, and the target's own token prior) did not beat uniform weights 2106.11520 §3.2. The official code divides the summed log-likelihood by target length, which the printed equation does not do bart_score.py. Card: BARTScore: Evaluating Generated Text as Text Generation.

BARTScore-cnn

The same scoring on BART fine-tuned on CNN/DailyMail, using the off-the-shelf checkpoint 2106.11520 §3.3.2. In the authors' tables it is the top BARTScore row in several summarization and factuality settings, without prompts 2106.11520 tab. 4 2106.11520 tab. 5.

BARTScore-cnn-Para

The CNN/DailyMail model fine-tuned further on paraphrase pairs from ParaBank2. The paper uses a random subset of 30,000 pairs for one epoch, learning rate 5e-5 and batch size 20 2106.11520 §3.3.2. The repository loads the full file and defaults to three epochs, and the fine-tuned weights are not in the repository finetune.py bart_score.py.

BARTScore-Prompt

A short phrase such as "in summary" or "in other words" is added to the source or the hypothesis. The scores for all prompts in a set are then averaged. The source-side set has 70 prompts seeded with "in summary". The reference-side set has 34 prompts seeded with "in other words" 2106.11520 tab. 2 2106.11520 §4.2.1. Each prompt's score is normalised by its own target length before the outer average 2106.11520 app. A.4. Machine translation uses a single prompt, "such as", chosen on the WMT18 development set 2106.11520 §4.2.1.

Generative perplexity (LangFlow)

Samples from a generative model are scored by the perplexity a fixed, external model assigns to them, not by the generator's own likelihood. LangFlow uses GPT-2 Large as the scorer and 1024 samples per setting 2604.11748 §5 2604.11748 tab. 2. Its author-reported values are 92.2 on LM1B and 36.5 on OpenWebText, with 128 and 1024 sampling steps 2604.11748 tab. 2 2604.11748 §5. The sampling-budget table also lists the sample entropy beside each perplexity 2604.11748 app. "NFE comparison on LM1B". On LM1B the data entropy is 4.32 and LangFlow at 128 steps gives 4.31 2604.11748 app. "NFE comparison on LM1B". Card: LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling.

Likelihood bound for a diffusion model (LangFlow)

A diffusion language model has no exact likelihood that is cheap to compute, so the paper integrates along the probability-flow ODE and gets an upper bound on the negative log-likelihood 2604.11748 thm. 3.1. The bound is evaluated with 128 Heun-2 steps and a Hutchinson trace estimate [arxiv:2604.11748 app. "Additional Evaluation Details"]. The reported PPL is 30.0 on LM1B and 24.6 on OpenWebText 2604.11748 tab. 2. The paper reports the bound for each diffusion baseline too 2604.11748 §5.

Candidate ranking by conditional likelihood (theory-of-mind probing)

Each answer option is scored by its conditional log-likelihood given a prefix of k solved examples, the context and the question. The option with the highest score is the prediction. The authors use this for GPT-3 Ada, Curie and Davinci, with k from 0 to 35 on SocialIQa and from 2 to 24 on ToMi 2210.13312 §2.1 2210.13312 §3.1 2210.13312 app. C. Their best SocialIQa result is about 55%, and mental-state questions on ToMi reach about 60%, against 90 to 100% on factual questions 2210.13312 §2.2 2210.13312 §3.2. The released code sums the answer's token log-probabilities after dropping the first token. The paper does not say whether the sum is length-normalised gpt3LMMCprobing.py. Card: Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs.

Domain-conditional PMI (zero-shot multiple choice)

Each option is scored by its log-likelihood given the question, minus its log-likelihood given a short task-specific domain premise, such as "the answer is:" 2104.08315 §3.2 2104.08315 eq. 2. The subtracted term divides out how likely the option's wording is in general. The paper's claim is that this stops answers with many valid wordings from being penalised for them 2104.08315 §1 2104.08315 §2. No training is involved. The authors use GPT-2 (125M to 1.6B) and GPT-3 (2.7B to 175B) on 16 splits of 13 datasets 2104.08315 §4.1 2104.08315 §4.2. Each dataset has a hand-written domain premise, and the LaTeX has no ablation over alternatives, so the premise is a choice that changes the result 2104.08315 tab. 7. The authors report that it does not come out best on HellaSwag, ARC-Easy and BoolQ 2104.08315 §6. Card: Surface Form Competition: Why the Highest Probability Answer Isn't Always Right.

Scoring directions

• Faithfulness: source to hypothesis. The authors suggest it for factuality and relevance 2106.11520 §3.2.
• Precision: reference to hypothesis.
• Recall: hypothesis to reference. Used for pyramid-style semantic coverage in summarization 2106.11520 §4.2.2.
• F score: arithmetic mean of precision and recall, used for semantic overlap, machine translation and data-to-text 2106.11520 §3.2.

When to use which

Results are author-reported. The paper says the best BARTScore is picked per dataset in some tables ("Ω") but does not name it 2106.11520 tab. 4 2106.11520 tab. 6.

Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a # comment line above each step (with its citation), short trailing comments only for shapes or units.

# Average token log-likelihood of target given source [arxiv:2106.11520 §3.2]
def bart_score(model, source, target):
    # Tokenise both sides; the official code truncates to 1024 tokens
    # [code: bart_score.py#L38-L51]
    src_ids = tokenize(source, max_len=1024)
    tgt_ids = tokenize(target, max_len=1024)
    # Log-probability of each target token given its prefix and the source
    # [arxiv:2106.11520 eq. 1]
    log_probs = log_softmax(model(src_ids, labels=tgt_ids))
    tok_logp = pick(log_probs, tgt_ids)
    # Sum over target tokens, then divide by length, as the code does
    # [code: bart_score.py#L67]; the printed eq. 2 has no division
    return sum(tok_logp) / count_non_pad(tgt_ids)

# The four directions and the F score [arxiv:2106.11520 §3.2]
def directions(source, reference, hypothesis, model):
    # Faithfulness: source conditions the hypothesis
    faith = bart_score(model, source, hypothesis)
    # Precision: reference conditions the hypothesis
    precision = bart_score(model, reference, hypothesis)
    # Recall: hypothesis conditions the reference
    recall = bart_score(model, hypothesis, reference)
    # F score: arithmetic mean of precision and recall [arxiv:2106.11520 §3.2]
    f_score = (precision + recall) / 2
    return faith, precision, recall, f_score

# Prompt ensembling: average over prompts, one average per side
# [arxiv:2106.11520 §3.3.1] [arxiv:2106.11520 app. A.4]
def prompted(source, hypothesis, prompts, model):
    # Append each prompt to the source (encoder side)
    enc = [bart_score(model, source + " " + p + ",", hypothesis)
           for p in prompts]
    # Prepend each prompt to the hypothesis (decoder side)
    dec = [bart_score(model, source, p + ", " + hypothesis)
           for p in prompts]
    # Punctuation is the code's choice, not the paper's [code: SUM/score.py#L328]
    return mean(enc), mean(dec)

# Pick the candidate with the highest conditional likelihood
# [arxiv:2210.13312 app. C]
def choose_by_likelihood(model, prefix, context, candidates):
    # Prefix holds the k solved shots; the item comes after them
    # [arxiv:2210.13312 §2.1]
    scores = []
    for answer in candidates:
        # Sum of token log-probs of the answer [code: gpt3LMMCprobing.py#L189-L199]
        # The paper does not state a length normalisation; this sums
        scores.append(sum(answer_logprobs(model, prefix + context, answer)))
    return argmax(scores)

# Domain-conditional PMI for zero-shot multiple choice
# [arxiv:2104.08315 §3.2] [arxiv:2104.08315 eq. 2]
def select_dcpmi(model, premise, options, domain_premise):
    scores = []
    for option in options:
        # Summed log P(option | premise): the conditional term
        cond = sum(answer_logprobs(model, premise, option))
        # Same option after only the domain premise: the in-domain prior
        dom = sum(answer_logprobs(model, domain_premise, option))
        scores.append(cond - dom)
    return argmax(scores)

# Generative perplexity of samples under a fixed scorer [arxiv:2604.11748 §5]
def gen_ppl(scorer, samples):
    # GPT-2 Large scores 1024 samples per setting [arxiv:2604.11748 §5]
    total_nll, total_tok = 0.0, 0
    for text in samples:
        ids = tokenize(text, scorer)
        # Token-level negative log-probs under the scorer
        total_nll += -sum(token_logprobs(scorer, ids))
        total_tok += len(ids)
    # How per-sample values are combined is not stated in §5; this pools tokens
    return exp(total_nll / total_tok)

# Upper bound on the NLL of a diffusion LM [arxiv:2604.11748 thm. 3.1]
def nll_upper_bound(model, x, a, b):
    # Noisy start z_a ~ N(alpha_a E^T x, sigma_a^2 I) [thm. 3.1]
    z_a = noisy_embedding(x, a)
    # Reverse ODE trajectory from a to b, 128 Heun-2 steps [app. "Additional
    # Evaluation Details"]
    z_b = ode_trajectory(model, z_a, a, b, steps=128)
    # Terms of the bound on log p(x) [thm. 3.1]: prior at the noisy end,
    # reconstruction of x at time a, and a weighted divergence integral
    prior = L * D / 2 - norm2(z_b) / (2 * sigma(b) ** 2)
    recon = sum(log(model.probs(z_a, a)[i, x[i]]) for i in range(L))
    # Divergence along the trajectory, with a Hutchinson trace probe
    div = integrate_weighted_divergence(model, z_a, a, b, probe=gaussian())
    # Negate, since the bound is on log p(x) and the NLL is its negative
    return -(prior + recon - div)

Keep the scores negative, where higher is better, as the code returns them bart_score.py. Use the arithmetic F score. The code also writes a harmonic variant that the paper does not define score.py. For candidate ranking, report the length rule you chose. Summed log-probs favour shorter candidates, and the paper's marginal-normalisation check changed little 2210.13312 app. C.

Contradictions

The papers measure different things, so there is no direct disagreement on a number. Within and between them:

• Length normalisation in the code. BARTScore's code divides the summed log-likelihood by target length bart_score.py, while the theory-of-mind code sums the answer's log-probs without dividing gpt3LMMCprobing.py. The printed BARTScore eq. 2 sums, and the theory-of-mind paper does not state a rule. The BARTScore card records the difference in its "Paper vs code" section.
• LangFlow. The same model's LM1B Gen. PPL is 81.5 in tab. 1 and 92.2 in tab. 2. The appendix NFE table ("NFE comparison on LM1B") supports 92.2. Both values are shown here as the paper prints them.
• LangFlow's self-conditioning. Earlier diffusion work reported PPL without self-conditioning 2604.11748 §4.2. On LM1B, MDLM's PPL goes from 31.0 to 32.7 with self-conditioning, while LangFlow's goes from 49.0 to 30.0 2604.11748 tab. 1. The two protocols are not the same metric.
• Probing type. The theory-of-mind paper reports different numbers for likelihood probing and multiple-choice probing on the same model 2210.13312 app. D. Treat the two as separate protocols.
• Length rule across tasks. The surface-form paper's length-normalised log-likelihood is best or tied on 18.75% of GPT-3 175B splits, against 62.50% for domain-conditional PMI 2104.08315 tab. 2. BARTScore divides its summed log-likelihood by target length bart_score.py, and the theory-of-mind code does not divide. These are different tasks, so the papers do not conflict, but the length rule should be chosen per task and reported.

Papers

• BARTScore: Evaluating Generated Text as Text Generation: the original likelihood-based metric, with four directions, a fine-tuned CNN/DM variant, a paraphrase variant and prompt ensembling.
• Surface Form Competition: Why the Highest Probability Answer Isn't Always Right: domain-conditional PMI for zero-shot multiple choice, which divides out the in-domain likelihood of each answer wording.
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs: k-shot conditional-likelihood ranking of answer candidates on SocialIQa and ToMi, used as a probe rather than a text metric.
• LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling: reports generative perplexity under GPT-2 Large and an ODE likelihood bound for a continuous diffusion language model.