--- kind: concept slug: likelihood-based-text-evaluation title: Likelihood-based text evaluation summary: >- Scoring text or answers by how likely a pre-trained language model finds them, with no human labels. BARTScore averages token log-likelihoods given a reference. Domain-conditional PMI subtracts a domain-prompt likelihood to offset answers with many wordings. LangFlow reports generative perplexity. Theory-of-mind probes pick the highest conditional likelihood. tags: [evaluation, metrics, text-generation, zero-shot-inference] papers: ["2104.08315", "2106.11520", "2210.13312", "2604.11748"] related: [llm-evaluation, llm-as-a-judge, text-summarization, neural-machine-translation, prompt-engineering, in-context-learning, theory-of-mind, hallucination-detection, human-evaluation, benchmark-validity, continuous-text-diffusion, zero-shot-learning, calibration] status: draft updated_at: 2026-11-01 --- ## Overview Reference-free and reference-based metrics for generated text usually count word overlap or need human ratings to fit. This family scores a text by the probability a pre-trained generation model assigns to it, given another text. The invariant all variants share: the score uses the model's existing output layer, needs no human judgements to fit and adds no parameters [arxiv:2106.11520 §1] [arxiv:2106.11520 §3.2]. The source text, a reference, or the hypothesis itself can play the conditioning role, which gives four directions. Each direction answers a different question about the candidate [arxiv:2106.11520 §3.2]. A second use appears in language-model papers, which report perplexity rather than a metric between two texts. Here the likelihood is either measured with a fixed external model over a set of samples, or bounded because the model's own likelihood is hard to compute [arxiv:2604.11748 §5] [arxiv:2604.11748 thm. 3.1]. These numbers measure different things from BARTScore, so compare them only within their own family. A third use is benchmark probing. The model is shown a question with several candidate answers, and the candidate with the highest conditional likelihood is taken as its answer. The score is then an accuracy, not a similarity [arxiv:2210.13312 app. C]. Multiple-choice scoring has a known bias: an answer with many valid wordings splits probability across them, so it can lose to a rarer correct wording [arxiv:2104.08315 §1]. The domain-conditional variant below corrects for this. ## Variants ### Vanilla BARTScore Average token log-likelihood under the pre-trained BART encoder-decoder, with every token weighted equally. Three weighting schemes tried by the authors (skipping stop words, IDF, and the target's own token prior) did not beat uniform weights [arxiv:2106.11520 §3.2]. The official code divides the summed log-likelihood by target length, which the printed equation does not do [code: bart_score.py#L67]. Card: [BARTScore: Evaluating Generated Text as Text Generation](/papers/2106.11520). ### BARTScore-cnn The same scoring on BART fine-tuned on CNN/DailyMail, using the off-the-shelf checkpoint [arxiv:2106.11520 §3.3.2]. In the authors' tables it is the top BARTScore row in several summarization and factuality settings, without prompts [arxiv:2106.11520 tab. 4] [arxiv:2106.11520 tab. 5]. ### BARTScore-cnn-Para The CNN/DailyMail model fine-tuned further on paraphrase pairs from ParaBank2. The paper uses a random subset of 30,000 pairs for one epoch, learning rate 5e-5 and batch size 20 [arxiv:2106.11520 §3.3.2]. The repository loads the full file and defaults to three epochs, and the fine-tuned weights are not in the repository [code: train/reproduce/finetune.py#L38] [code: bart_score.py#L24-L28]. ### BARTScore-Prompt A short phrase such as "in summary" or "in other words" is added to the source or the hypothesis. The scores for all prompts in a set are then averaged. The source-side set has 70 prompts seeded with "in summary". The reference-side set has 34 prompts seeded with "in other words" [arxiv:2106.11520 tab. 2] [arxiv:2106.11520 §4.2.1]. Each prompt's score is normalised by its own target length before the outer average [arxiv:2106.11520 app. A.4]. Machine translation uses a single prompt, "such as", chosen on the WMT18 development set [arxiv:2106.11520 §4.2.1]. ### Generative perplexity (LangFlow) Samples from a generative model are scored by the perplexity a fixed, external model assigns to them, not by the generator's own likelihood. LangFlow uses GPT-2 Large as the scorer and 1024 samples per setting [arxiv:2604.11748 §5] [arxiv:2604.11748 tab. 2]. Its author-reported values are 92.2 on LM1B and 36.5 on OpenWebText, with 128 and 1024 sampling steps [arxiv:2604.11748 tab. 2] [arxiv:2604.11748 §5]. The sampling-budget table also lists the sample entropy beside each perplexity [arxiv:2604.11748 app. "NFE comparison on LM1B"]. On LM1B the data entropy is 4.32 and LangFlow at 128 steps gives 4.31 [arxiv:2604.11748 app. "NFE comparison on LM1B"]. Card: [LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling](/papers/2604.11748). ### Likelihood bound for a diffusion model (LangFlow) A diffusion language model has no exact likelihood that is cheap to compute, so the paper integrates along the probability-flow ODE and gets an upper bound on the negative log-likelihood [arxiv:2604.11748 thm. 3.1]. The bound is evaluated with 128 Heun-2 steps and a Hutchinson trace estimate [arxiv:2604.11748 app. "Additional Evaluation Details"]. The reported PPL is 30.0 on LM1B and 24.6 on OpenWebText [arxiv:2604.11748 tab. 2]. The paper reports the bound for each diffusion baseline too [arxiv:2604.11748 §5]. ### Candidate ranking by conditional likelihood (theory-of-mind probing) Each answer option is scored by its conditional log-likelihood given a prefix of k solved examples, the context and the question. The option with the highest score is the prediction. The authors use this for GPT-3 Ada, Curie and Davinci, with k from 0 to 35 on SocialIQa and from 2 to 24 on ToMi [arxiv:2210.13312 §2.1] [arxiv:2210.13312 §3.1] [arxiv:2210.13312 app. C]. Their best SocialIQa result is about 55%, and mental-state questions on ToMi reach about 60%, against 90 to 100% on factual questions [arxiv:2210.13312 §2.2] [arxiv:2210.13312 §3.2]. The released code sums the answer's token log-probabilities after dropping the first token. The paper does not say whether the sum is length-normalised [code: gpt3LMMCprobing.py#L189-L199]. Card: [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](/papers/2210.13312). ### Domain-conditional PMI (zero-shot multiple choice) Each option is scored by its log-likelihood given the question, minus its log-likelihood given a short task-specific domain premise, such as "the answer is:" [arxiv:2104.08315 §3.2] [arxiv:2104.08315 eq. 2]. The subtracted term divides out how likely the option's wording is in general. The paper's claim is that this stops answers with many valid wordings from being penalised for them [arxiv:2104.08315 §1] [arxiv:2104.08315 §2]. No training is involved. The authors use GPT-2 (125M to 1.6B) and GPT-3 (2.7B to 175B) on 16 splits of 13 datasets [arxiv:2104.08315 §4.1] [arxiv:2104.08315 §4.2]. Each dataset has a hand-written domain premise, and the LaTeX has no ablation over alternatives, so the premise is a choice that changes the result [arxiv:2104.08315 tab. 7]. The authors report that it does not come out best on HellaSwag, ARC-Easy and BoolQ [arxiv:2104.08315 §6]. Card: [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](/papers/2104.08315). ### Scoring directions - **Faithfulness**: source to hypothesis. The authors suggest it for factuality and relevance [arxiv:2106.11520 §3.2]. - **Precision**: reference to hypothesis. - **Recall**: hypothesis to reference. Used for pyramid-style semantic coverage in summarization [arxiv:2106.11520 §4.2.2]. - **F score**: arithmetic mean of precision and recall, used for semantic overlap, machine translation and data-to-text [arxiv:2106.11520 §3.2]. ## When to use which | Situation | Use | Why | |---|---|---| | No human ratings, and you need a score that tracks the reference | Vanilla BARTScore, F score | Unsupervised; the authors compare it to supervised and unsupervised metrics [arxiv:2106.11520 §4.1.1] | | Summarization or data-to-text, and the best reported agreement matters | BARTScore-Prompt with fine-tuned variants | Highest averages on the authors' summarization and data-to-text tables [arxiv:2106.11520 tab. 4] [arxiv:2106.11520 tab. 6] | | Factuality | Vanilla or BARTScore-cnn, no prompt | Prompts did not help factuality, and paraphrase tuning lowered it on Rank19 [arxiv:2106.11520 §4.3.2] | | Machine translation, comparison with a supervised metric | Expect a close result, not a win | On the WMT19 average, the best BARTScore variant (0.337) trails COMET (0.339), which is supervised [arxiv:2106.11520 tab. 3] | | Tight compute or latency budget | Vanilla, one direction | Prompt ensembling multiplies scoring calls: the authors' summarization setup needs 276 calls per system before any multi-reference factor [code: SUM/score.py#L358-L403] | | Cannot download external paraphrase weights | Vanilla or BARTScore-cnn | The `bart.pth` file that the Para variant needs is not in the repository [code: bart_score.py#L24-L28] | | Multiple-choice benchmark on a base model that exposes log-probs | Candidate ranking by conditional likelihood | This is how the theory-of-mind authors scored GPT-3 [arxiv:2210.13312 app. C] | | Zero-shot multiple choice where answers have many valid wordings, on a base model that exposes log-probs | Domain-conditional PMI, with a premise written per task | Best or tied on 62.50% of GPT-3 175B splits, against 18.75% for length-normalised log-likelihood and 12.50% for raw probability [arxiv:2104.08315 tab. 2]. Gains depend on the premise and the dataset [arxiv:2104.08315 §6] | | Multiple-choice benchmark on a chat model with no log-probs | Letter answers, not likelihood | The authors switched chat models to letter answers, and the paper reports the two probing types give different numbers for the same model [arxiv:2210.13312 app. D] | | Sample quality from a generator with no cheap exact likelihood (e.g. a diffusion LM) | Generative perplexity under a fixed scorer, read with sample entropy | The paper pairs the two across step counts in its sampling-budget table [arxiv:2604.11748 app. "NFE comparison on LM1B"] | | Comparing a diffusion LM's likelihood with another method's | Bound, using the same ODE settings for both | Bounds differ by method, and the paper's own PPL is an upper bound [arxiv:2604.11748 §5] | Results are author-reported. The paper says the best BARTScore is picked per dataset in some tables ("Ω") but does not name it [arxiv:2106.11520 tab. 4] [arxiv:2106.11520 tab. 6]. ## Implementation sketch Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a `# comment` line above each step (with its citation), short trailing comments only for shapes or units. ```pseudocode # Average token log-likelihood of target given source [arxiv:2106.11520 §3.2] def bart_score(model, source, target): # Tokenise both sides; the official code truncates to 1024 tokens # [code: bart_score.py#L38-L51] src_ids = tokenize(source, max_len=1024) tgt_ids = tokenize(target, max_len=1024) # Log-probability of each target token given its prefix and the source # [arxiv:2106.11520 eq. 1] log_probs = log_softmax(model(src_ids, labels=tgt_ids)) tok_logp = pick(log_probs, tgt_ids) # Sum over target tokens, then divide by length, as the code does # [code: bart_score.py#L67]; the printed eq. 2 has no division return sum(tok_logp) / count_non_pad(tgt_ids) # The four directions and the F score [arxiv:2106.11520 §3.2] def directions(source, reference, hypothesis, model): # Faithfulness: source conditions the hypothesis faith = bart_score(model, source, hypothesis) # Precision: reference conditions the hypothesis precision = bart_score(model, reference, hypothesis) # Recall: hypothesis conditions the reference recall = bart_score(model, hypothesis, reference) # F score: arithmetic mean of precision and recall [arxiv:2106.11520 §3.2] f_score = (precision + recall) / 2 return faith, precision, recall, f_score # Prompt ensembling: average over prompts, one average per side # [arxiv:2106.11520 §3.3.1] [arxiv:2106.11520 app. A.4] def prompted(source, hypothesis, prompts, model): # Append each prompt to the source (encoder side) enc = [bart_score(model, source + " " + p + ",", hypothesis) for p in prompts] # Prepend each prompt to the hypothesis (decoder side) dec = [bart_score(model, source, p + ", " + hypothesis) for p in prompts] # Punctuation is the code's choice, not the paper's [code: SUM/score.py#L328] return mean(enc), mean(dec) # Pick the candidate with the highest conditional likelihood # [arxiv:2210.13312 app. C] def choose_by_likelihood(model, prefix, context, candidates): # Prefix holds the k solved shots; the item comes after them # [arxiv:2210.13312 §2.1] scores = [] for answer in candidates: # Sum of token log-probs of the answer [code: gpt3LMMCprobing.py#L189-L199] # The paper does not state a length normalisation; this sums scores.append(sum(answer_logprobs(model, prefix + context, answer))) return argmax(scores) # Domain-conditional PMI for zero-shot multiple choice # [arxiv:2104.08315 §3.2] [arxiv:2104.08315 eq. 2] def select_dcpmi(model, premise, options, domain_premise): scores = [] for option in options: # Summed log P(option | premise): the conditional term cond = sum(answer_logprobs(model, premise, option)) # Same option after only the domain premise: the in-domain prior dom = sum(answer_logprobs(model, domain_premise, option)) scores.append(cond - dom) return argmax(scores) # Generative perplexity of samples under a fixed scorer [arxiv:2604.11748 §5] def gen_ppl(scorer, samples): # GPT-2 Large scores 1024 samples per setting [arxiv:2604.11748 §5] total_nll, total_tok = 0.0, 0 for text in samples: ids = tokenize(text, scorer) # Token-level negative log-probs under the scorer total_nll += -sum(token_logprobs(scorer, ids)) total_tok += len(ids) # How per-sample values are combined is not stated in §5; this pools tokens return exp(total_nll / total_tok) # Upper bound on the NLL of a diffusion LM [arxiv:2604.11748 thm. 3.1] def nll_upper_bound(model, x, a, b): # Noisy start z_a ~ N(alpha_a E^T x, sigma_a^2 I) [thm. 3.1] z_a = noisy_embedding(x, a) # Reverse ODE trajectory from a to b, 128 Heun-2 steps [app. "Additional # Evaluation Details"] z_b = ode_trajectory(model, z_a, a, b, steps=128) # Terms of the bound on log p(x) [thm. 3.1]: prior at the noisy end, # reconstruction of x at time a, and a weighted divergence integral prior = L * D / 2 - norm2(z_b) / (2 * sigma(b) ** 2) recon = sum(log(model.probs(z_a, a)[i, x[i]]) for i in range(L)) # Divergence along the trajectory, with a Hutchinson trace probe div = integrate_weighted_divergence(model, z_a, a, b, probe=gaussian()) # Negate, since the bound is on log p(x) and the NLL is its negative return -(prior + recon - div) ``` Keep the scores negative, where higher is better, as the code returns them [code: bart_score.py#L68]. Use the arithmetic F score. The code also writes a harmonic variant that the paper does not define [code: SUM/score.py#L308]. For candidate ranking, report the length rule you chose. Summed log-probs favour shorter candidates, and the paper's marginal-normalisation check changed little [arxiv:2210.13312 app. C]. ## Contradictions The papers measure different things, so there is no direct disagreement on a number. Within and between them: - **Length normalisation in the code.** BARTScore's code divides the summed log-likelihood by target length [code: bart_score.py#L67], while the theory-of-mind code sums the answer's log-probs without dividing [code: gpt3LMMCprobing.py#L189-L199]. The printed BARTScore eq. 2 sums, and the theory-of-mind paper does not state a rule. The BARTScore card records the difference in its "Paper vs code" section. - **LangFlow.** The same model's LM1B Gen. PPL is 81.5 in [tab. 1] and 92.2 in [tab. 2]. The appendix NFE table ("NFE comparison on LM1B") supports 92.2. Both values are shown here as the paper prints them. - **LangFlow's self-conditioning.** Earlier diffusion work reported PPL without self-conditioning [arxiv:2604.11748 §4.2]. On LM1B, MDLM's PPL goes from 31.0 to 32.7 with self-conditioning, while LangFlow's goes from 49.0 to 30.0 [arxiv:2604.11748 tab. 1]. The two protocols are not the same metric. - **Probing type.** The theory-of-mind paper reports different numbers for likelihood probing and multiple-choice probing on the same model [arxiv:2210.13312 app. D]. Treat the two as separate protocols. - **Length rule across tasks.** The surface-form paper's length-normalised log-likelihood is best or tied on 18.75% of GPT-3 175B splits, against 62.50% for domain-conditional PMI [arxiv:2104.08315 tab. 2]. BARTScore divides its summed log-likelihood by target length [code: bart_score.py#L67], and the theory-of-mind code does not divide. These are different tasks, so the papers do not conflict, but the length rule should be chosen per task and reported. ## Papers - [BARTScore: Evaluating Generated Text as Text Generation](/papers/2106.11520): the original likelihood-based metric, with four directions, a fine-tuned CNN/DM variant, a paraphrase variant and prompt ensembling. - [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](/papers/2104.08315): domain-conditional PMI for zero-shot multiple choice, which divides out the in-domain likelihood of each answer wording. - [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](/papers/2210.13312): k-shot conditional-likelihood ranking of answer candidates on SocialIQa and ToMi, used as a probe rather than a text metric. - [LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling](/papers/2604.11748): reports generative perplexity under GPT-2 Large and an ODE likelihood bound for a continuous diffusion language model.