Factuality Enhanced Language Models for Open-Ended Text Generation

In plain words

Large language models write fluent text, but they often state things that are not true, such as a wrong year or the wrong author of a film. This paper first builds a way to measure that: it takes sentence starts drawn from fact-checking claims and checks whether the model's continuation names things that do not appear in the matching Wikipedia article.

The authors then find two causes that can be fixed. The first is the sampling step: the usual "top-p" way of picking the next word adds randomness at every word, and that randomness drifts away from facts. Their fix lowers the randomness as a sentence goes on and resets it at each new sentence, so the start of a sentence stays varied and its end stays anchored to what the model knows. The second is training: the model learns facts badly when a Wikipedia article is cut into chunks, so each sentence is given its article title as a prefix, and the training loss focuses on the later half of each sentence, where the facts are.

Together these changes reduce made-up names and facts in the authors' tests, at some cost in variety of wording.

TL;DR

• Introduces FactualityPrompts, a benchmark of factual and nonfactual prompts from FEVER, with Wikipedia-based metrics for named-entity error and entailment §3.
• Factual-nucleus sampling makes the top-p cut-off shrink within each sentence (decay λ), resets at sentence starts, and stops at a floor ω §5.1.
• Continued Wikipedia training with a title prefix ("TopicPrefix") and a sentence-completion loss that scores only the later part of each sentence §6.
• Author-reported: with both changes, 530B named-entity error on factual prompts drops from 33.3% to 14.5%, and entailment rises from 11.8% to 25.5% tab. 5.
• Trade-off: diversity and repetition get worse than top-p sampling §5.2 tab. 4.

Problem

Pretrained LMs produce nonfactual continuations when used for open-ended generation, and this matters for safe deployment §1. Earlier factuality work targets fine-tuned models on tasks such as summarisation, where a source text is given; the authors focus on pretrained LMs without retrieval §1 §2. Open-ended text has no reference answer, so measuring factuality needs Wikipedia as a ground-truth source §3. Top-p sampling is popular for diversity but is reported to hurt factuality §4 §5.1. Continued training on Wikipedia alone does not reliably improve factuality §1 §6.3. Retrieval-augmented models are a known alternative but add complexity and cost §1.

Method

Benchmark (FactualityPrompts). Prompts are FEVER validation claims: the SUPPORTED claims form the factual set and the REFUTED claims the nonfactual set. Very short claims are removed. Each set has 8,000 prompts, averaging 9.77 and 9.48 tokens §3.1 app. D.

Ground truth. For each prompt, the document-level reference is the Wikipedia article that FEVER annotates for the claim. The sentence-level reference is found in that article in two ways: the sentence with the highest TF-IDF cosine to the generated text, and the sentence with the highest similarity in a sentence-embedding space §3.2.

Check-worthy filter. Before scoring, a continuation's first sentence is skipped if it has no named entity, if it contains first-person pronouns (I, we, us), or if it contains a question mark app. B.

Named-entity error (NE error). For each scored continuation, the share of its named entities that do not appear in the document-level reference. Multi-word entities count as matched on partial n-gram overlap, ignoring stopwords §3.3.

Entailment ratio. A RoBERTa model fine-tuned on MNLI classifies each pair of (reference sentence, generated claim). A generation counts as factual only when it is classified as entailed; neutral does not count §3.3.

Quality. Fluency is the mean perplexity under a 1.3B LM. Diversity is the mean number of distinct 4-grams, normalised by text length, over 10 generations per prompt (160,000 generations per setting). Repetition uses the metric code of Holtzman et al. §3.3.

Human check. 200 continuations were labelled against Wikipedia by two crowd workers and one expert; majority vote is used, and the expert labels are reported separately §3.4 tab. 2.

Decoding analysis. Models of 126M to 530B parameters are compared under greedy decoding and top-p 0.9 §4 tab. 3.

Factual-nucleus sampling. The nucleus probability for the t-th token of a sentence is

p_{t} = \max\{\omega,\ p \times \lambda^{t-1}\} §5.1

with three parts: λ-decay shrinks p at each step; p-reset restores the default p at the start of each new sentence, detected when the previous token was a full stop; and the ω-bound stops p from falling below a floor §5.1. Algorithm:

# Factual-nucleus sampling for one continuation  [§5.1]
# Inputs: prompt tokens; p, lam, omega (best reported: 0.9, 0.9, 0.3)
# Output: generated tokens, stopping at end-of-text or max_len (150)

def factual_nucleus_generate(model, prompt, p, lam, omega, max_len):
    tokens = list(prompt)
    # position of the next token inside the current sentence  [§5.1]
    position = 1
    while len(tokens) < max_len:
        # the nucleus shrinks with position but never below omega  [§5.1]
        p_t = max(omega, p * lam ** (position - 1))
        # sample from the smallest set whose mass reaches p_t  [§5.1]
        next_token = sample_top_p(model.next_token_probs(tokens), p_t)
        tokens.append(next_token)
        if next_token == END_OF_TEXT:
            break
        # restart the schedule after a full stop, a new sentence  [§5.1]
        if next_token == FULL_STOP:
            position = 1
        else:
            position += 1
    return tokens

Factuality-enhanced training. Two changes are applied to continued pretraining on Wikipedia §6.

1. TopicPrefix. Each sentence is prefixed with its article title, written as "Title ==> sentence" in the paper's example §6.1 app. C.
2. Sentence-completion loss. For each sentence a pivot is chosen, and token losses before the pivot are zeroed out. Training stays autoregressive. Three pivot strategies are compared: half (0.5 × sentence length), random (uniform between 0.25 and 0.75 × sentence length), and root (the position of the dependency root) §6.2. The paper recommends half as the default §6.2.

# Sentence-completion loss over one training document  [§6.2]
# Sentences were already prefixed with their article title  [§6.1]
# Normalisation of the mean is not stated in the paper

def sentence_completion_loss(model, sentences, strategy):
    scored_losses = []
    for sentence in sentences:
        tokens = tokenize(sentence)
        # one negative log-likelihood per token  # (T,)
        token_losses = next_token_loss(model, tokens)
        # first token that counts toward the loss  [§6.2]
        start = pivot_token_index(sentence, strategy)
        # earlier tokens get zero weight; decoding stays autoregressive  [§6.2]
        scored_losses.extend(token_losses[start:])
    return mean(scored_losses)

# Where the loss starts inside one sentence  [§6.2]

def pivot_token_index(sentence, strategy):
    words = dependency_words(sentence)
    if strategy == "half":
        # the middle word  [§6.2]
        word_index = len(words) // 2
    elif strategy == "random":
        # uniform between 25% and 75% of the sentence  [§6.2]
        word_index = int(len(words) * uniform(0.25, 0.75))
    else:
        # the dependency root  [§6.2]
        word_index = index_of_root(words)
    # map the chosen word to its first subword token
    return first_token_of_word(sentence, word_index)

Figures: fig. 1 shows the evaluation pipeline (continuation, check-worthy filter, evidence retrieval, metrics). fig. 2 plots diversity and repetition against NE error for top-p sampling and factual-nucleus sampling across several settings; the caption gives 0.767 as the diversity of 5,000 random Wikipedia documents for reference. tab. 1 gives example continuations under greedy, top-p and factual-nucleus decoding.

Hyperparameters

Results

Author-reported. Setup: FactualityPrompts (8,000 factual and 8,000 nonfactual prompts, Wikipedia references, metrics in §3). Decoding is compared on 1.3B §5.2; the final rows use 530B §6.3. Baselines: greedy decoding and top-p 0.9 §4 §5.2. Lower is better for NE error and repetition; higher is better for entailment and diversity.

Table 3 (model size, §4). F = factual prompts, NF = nonfactual prompts.

Table 4 (decoding on 1.3B, §5.2). Columns as in Table 3; the row label is p | λ | ω.

Table 5 (factuality-enhanced training, §6.3). TP = TopicPrefix; SC = sentence-completion loss. Rows marked "0.9 | 0.9 | 0.3" use factual-nucleus decoding.

Human validation (Table 2, §3.4). Pearson correlation with human labels.

Other reported findings: larger models give fewer nonfactual entities on factual prompts §4; greedy decoding is more factual than top-p but much less diverse and more repetitive §4 tab. 3; the 530B model with both changes has lower diversity than the 1.3B version §6.3. Perplexity stays no higher than top-p's §5.2 app. F.

Paper vs code

Official code: nayeon7lee/FactualityPrompt at b9424e838e17c4943ed1161b7993b9220d8ff593 (Apache-2.0). The abstract links it, and its owner is an author of the paper. The repository holds the evaluation metrics, prompts and training-data preprocessing. The factual-nucleus sampler and the Megatron training loop are not in the repository.

• Entailment ratio denominator. Paper: divided by all generations §3.3. Code: the class counts are taken only over generations classified as having a checkable fact, so the denominator excludes the others [code: src/evaluate_v3_final.py#L206-L212] evaluate_v3_final.py.
• Which sentence is scored. Paper: check-worthy continuations, with the first-sentence rule not stated §3.1 app. B. Code: only the first sentence of each continuation is scored evaluate_v3_final.py.
• NE matching. Paper: partial n-gram overlap for multi-word entities §3.3. Code: exact substring match for all types, and word-level partial match only for PERSON factuality_metric.py.
• NE detection. Paper: spaCy NE detection §3.3. Code: ORG, PERSON, WORK_OF_ART, PRODUCT and EVENT count as "important"; other types except cardinal, ordinal and some dates count as "unimportant"; both sets are checked. If spaCy finds no entities, a regex collects capitalised words that do not start the sentence claim_handling.py claim_handling.py evaluate_v3_final.py.
• First-person filter. Paper: any I, we or us app. B. Code: checks only subject tokens, and also flags "you" claim_handling.py.
• Evidence. Paper: one TF-IDF sentence and one sentence-embedding sentence §3.2. Code: the top-1 from each, plus NLI on both, keeping the evidence with the higher entailment probability evaluate_v3_final.py evaluate_v3_final.py. Evidence sentences of 4 words or fewer, and those containing "Section::::", are removed [code: src/retriever.py#L193-L201]. The embedding model is all-MiniLM-L6-v2 [code: src/retriever.py#L204]; the paper does not name it.
• Factual-nucleus sampler. Paper: §5.1. Code: the README points to Megatron-LM's nucleus sampler for this README.md. The λ-decay, p-reset and ω-bound logic does not appear in this repository.
• TopicPrefix. Paper: "Title ==> sentence" format app. C. Code: the training-data prefixing is not in this repository. The "<== section ==>" strings in the retriever are headers for evidence retrieval, not the training prefix retriever.py.
• Pivot masking. Paper: losses before t are zeroed §6.2. Code: words are split with spaCy and the mask covers words up to and including the pivot word. Half pivot is len // 2 preprocess_data_megatron_lm.py preprocess_data_megatron_lm.py.
• Random pivot. Paper: uniform [0.25, 0.75] × length §6.2. Code: the same, truncated with int() preprocess_data_megatron_lm.py.
• End-of-text masking. Code also masks the final end-of-text token from the loss preprocess_data_megatron_lm.py. Not mentioned in the paper.
• Mask variants. The code contains two extra masking variants, v1 (NE-only) and v4 (NE after ROOT), that are not in the paper [code: src/preprocess_data_megatron_lm.py#L115-L118] [code: src/preprocess_data_megatron_lm.py#L126-L128].
• Repetition. Paper: Holtzman et al.'s code §3.3. Code: window up to 90 tokens repetition.py.
• Diversity n-gram. Paper: 4-grams §3.3. Code loops over n in 4, 3, 2 distinct_n.py.

The NLI model is described only as "a RoBERTa model fine-tuned on MNLI" §3.3; the code loads roberta.large.mnli factuality_metric.py.

Pitfalls

• Paper text contradicts its table. §4 says the NE error "drops from 63.99% to 33.30%" for 126M to 530B, but Table 3 and the introduction give 63.69% §4 tab. 3. Use the table value.
• Commented-out LaTeX is not the paper. The source includes notes on 500 or 250 training steps, a checklist that gives 8 A100 GPUs, and a "~6M documents" corpus size. All are commented out of the compiled paper. Report them as "not stated".
• The factual-nucleus schedule is ambiguous. The paper says p_t uses the t-th token "within each sentence" §5.1. The reset is triggered by a full stop in the previous token §5.1. The paper does not say how the full stop is identified in a subword vocabulary. The pseudocode above is one reading.
• The floor is reached quickly. Derived from the formula §5.1: with λ = 0.9 and ω = 0.3, the floor applies from token 12 of each sentence on. With λ = 0.5 it applies from token 3. So λ = 0.5 behaves almost like ω for most of a sentence.
• Pivot details matter. Word-level pivots from spaCy are not subword positions. The code masks the pivot word itself; the paper says "before t". Implementers should follow the code if they want to reproduce the numbers.
• Metrics depend on Wikipedia coverage. A correct claim absent from the reference article counts as an error. Document-level references come from FEVER's annotations, not from a full search §3.2. The paper validates the metrics on 200 labelled continuations only §3.4.
• Factual prompts are not verified facts. The "factual" label is a property of the FEVER claim. Its truth was judged by FEVER annotators, not by this work app. D.
• Diversity is traded for factuality. Factual-nucleus sampling gives much lower diversity than top-p tab. 4 and the 530B result is less diverse than the 1.3B one §6.3. Budget for this when choosing a setting.
• Replication with public weights will not match. The README notes the public Megatron checkpoint (345M) is smaller than the paper's models, so the same numbers will not be reproduced README.md.
• Sentence-completion normalisation is not stated. The paper does not say whether the loss is averaged over scored tokens or over all tokens §6.2.
• Ablations use the 1.3B model. Most decoding and training ablations are on 1.3B; only the best setting is shown on 530B §5.2 §6.3. Do not assume the same ranking holds at scale.
• Evidence aggregation is code-only. Choosing the evidence with the highest entailment probability is not described in the paper [code: src/evaluate_v3_final.py#L111-L114].

Lineage

• Builds on: Megatron-LM, whose experimental setup the LMs follow app. C [arXiv:1909.08053]; Smith et al. 2022, whose pretraining corpus the LMs use §4 [arXiv:2201.11990]; nucleus (top-p) sampling (Holtzman et al., ICLR 2020) §4 §5.1; continued domain-adaptive training (Gururangan et al., ACL 2020), which the paper treats as the baseline §6.1; FEVER (Thorne et al., NAACL 2018) for prompts and document references §3.1.
• Compared against: greedy decoding and top-p sampling §4; the Wikipedia-only continued-training baseline §6.3. Retrieval-based systems are discussed as a costlier alternative but not compared §1.
• Related metric: the NE-based error follows Nan et al. (EACL 2021) §3.3.
• Contrast: TruthfulQA (Lin et al., 2021) reports that larger models give more misconceptions, which the paper contrasts with its factual-knowledge results §2 §4.
• Superseded by: none known from content already in this wiki.