BARTScore: Evaluating Generated Text as Text Generation

In plain words

Judging generated text is hard. The usual options are to compare it with a reference by counting overlapping words, or to train a judge on human ratings. This paper asks a different question: how easily could a text-generation model write one text, given the other? If the model finds a candidate easy to produce from the source or from a reference, the candidate probably matches them. The score needs no human ratings and no extra parts, because it reuses the pre-trained model's normal output layer. Small changes, such as adding a short phrase before scoring, or training the model further on summaries or paraphrases, change how well the score agrees with human judgements.

TL;DR

BARTScore scores a text by the average log-likelihood that the pre-trained BART encoder-decoder assigns to it, in one of several directions: source to hypothesis, reference to hypothesis, or hypothesis to reference. The F variant averages the two reference directions. It needs no human labels and no extra parameters. The authors report gains from prompts and from fine-tuning on summarization or paraphrase data on most settings, but not for factuality. The official code departs from the paper in a few details (length normalisation, default checkpoint, paraphrase training), so read the Pitfalls section before reproducing numbers.

Problem

Generated text in machine translation, summarization, dialogue and data-to-text must be judged for perspectives such as informativeness, fluency, coherence and factuality §2.2. Most existing metrics cover only a few of these. BLEU and ROUGE mainly capture overlap, while trained metrics such as BLEURT and COMET need human judgements to fit §2.3. The authors also note that pre-trained models are usually used as feature extractors, which leaves their output prediction layer unused §1. The method assumes a pre-trained sequence-to-sequence model, here BART, is available §3.1.

Method

Given a seq2seq model with parameters θ, a source x of n tokens and a target y of m tokens, the generation probability factorises token by token eq. 1 §3.1. BARTScore is the sum of the log-probabilities of the target tokens given the source and the preceding target tokens, with a weight per token eq. 2 §3.2. The weights emphasise some tokens (IDF is given as one example). The paper weights every token equally. Three other schemes (uniform weights that skip stop words, IDF, and the target's own token prior) did not beat uniform weighting §3.2, footnote.

The score is unsupervised: no human judgements are needed to fit it. Architecturally it adds no parameters beyond those of pre-training §1.

Four directions are defined §3.2:

1. Faithfulness, source to hypothesis: the probability of the hypothesis given the source. The authors suggest it for factuality and relevance, and say it can also estimate coherence and fluency §3.2.
2. Precision, reference to hypothesis.
3. Recall, hypothesis to reference. The authors use it for pyramid-style semantic coverage in summarization §4.2.2.
4. F score: the arithmetic mean of precision and recall. The authors use it for semantic overlap and for machine translation and data-to-text §4.2.2.

Variants §3.3, §4.2.2:

• BARTScore: vanilla BART.
• BARTScore-cnn: BART fine-tuned on CNN/DailyMail, using the off-the-shelf Hugging Face checkpoint §3.3.2.
• BARTScore-cnn-Para: the CNN/DM model fine-tuned further on paraphrase pairs from ParaBank2 §3.3.2.
• BARTScore-Prompt: a short phrase is added to the source (appended) or to the hypothesis (prepended), and the scores of all prompts in a set are averaged §3.3.1, §4.2.1, app. A.4.

The prompt sets were built from seed phrases and their paraphrases. There are 70 prompts for the source-to-hypothesis direction, seeded with "in summary", and 34 for the reference/hypothesis directions, seeded with "in other words" tab. 2 §4.2.1. The full lists are in tab. 9 app. A.3. For machine translation the authors chose one prompt, "such as", on the WMT18 development set §4.2.1.

Key equations, as printed:

p(\mathbf{y}\mid\mathbf{x},\theta) = \prod_{t=1}^{m} p(\mathbf{y}_t \mid \mathbf{y}_{<t}, \mathbf{x}, \theta) eq. 1

\text{BARTScore} = \sum_{t=1}^{m} \omega_t \log p(\mathbf{y}_t \mid \mathbf{y}_{<t}, \mathbf{x}, \theta) eq. 2

\text{BARTScore-Prompt} = \frac{1}{n}\sum_{i=1}^{n}\frac{1}{m_i}\sum_{t=1}^{m_i}\log p([\mathbf{y}:\mathbf{z}_i]_t \mid [\mathbf{y}:\mathbf{z}_i]_{<t}, \mathbf{x}, \theta) eq. 3 app. A.4

Eq. 3 is the prompt average. Each prompt's score is normalised by its own target length m_i before the outer average over the n prompts.

The scoring function in code, from the official repository bart_score.py:

# Average token log-likelihood of target given source  [eq. 2] [§3.2]
def bart_score(model, source, target):
    # Tokenise both texts, truncating each to 1024 tokens
    # [code: bart_score.py#L38-L51]
    source_ids = tokenize(source, max_length=1024)
    target_ids = tokenize(target, max_length=1024)
    # Log-probability of each target token, given its prefix and the source
    # [eq. 1]
    log_probs = log_softmax(model(source_ids, labels=target_ids))
    token_log_probs = pick(log_probs, target_ids)
    # Paper eq. 2 sums these with uniform weights; the code divides by the
    # number of non-pad target tokens instead  [code: bart_score.py#L67]
    total = sum(token_log_probs)
    return total / count_non_pad(target_ids)

Then the four directions and the F score score.py:

# The four directions and the F score  [§3.2]
def bartscore_directions(source, reference, hypothesis):
    # Faithfulness: source to hypothesis  [§3.2]
    faithfulness = bart_score(source, hypothesis)
    # Precision: reference to hypothesis  [§3.2]
    precision = bart_score(reference, hypothesis)
    # Recall: hypothesis to reference  [§3.2]
    recall = bart_score(hypothesis, reference)
    # F score: arithmetic mean of precision and recall  [§3.2]
    f_score = (precision + recall) / 2
    return faithfulness, precision, recall, f_score

And prompt ensembling score.py analysis.py:

# Average of prompted scores, one average per side  [§3.3.1] [app. A.4]
def bartscore_prompt(source, hypothesis, prompts):
    # Append each prompt to the source (encoder side)  [§3.3.1]
    encoder_side = [
        bart_score(source + " " + prompt + ",", hypothesis)
        for prompt in prompts
    ]
    # Prepend each prompt to the hypothesis (decoder side)  [§3.3.1]
    decoder_side = [
        bart_score(source, prompt + ", " + hypothesis)
        for prompt in prompts
    ]
    # Average over the prompts on each side  [app. A.4]
    return mean(encoder_side), mean(decoder_side)

Hyperparameters

Results

Author-reported. The experiments cover 16 datasets from machine translation, summarization, factuality and data-to-text, spanning 7 perspectives §1 tab. 1 §4.1.3. Machine translation uses Kendall's tau on WMT19 §4.3.1. Summarization and data-to-text use Spearman correlation, factuality uses accuracy on Rank19 and Pearson on QAGS §4.1.2. Baselines are BLEU, ROUGE, CHRF, BERTScore, MoverScore, PRISM, BLEURT and COMET, plus FactCC and QAGS for factuality §4.1.1 tab. 5.

The headline claim is that BARTScore outperforms the top-scoring existing metrics in 16 of 22 test settings [abstract] §5. Prompts did not help factuality and paraphrase fine-tuning lowered factuality on Rank19 §4.3.2. Table 4 and Table 6 label the best-performing BARTScore per dataset as Ω; the paper does not list which variant that is for each dataset.

Paper vs code

Checked against the official repository at commit 248f511cb34ae3753fc81f7d7a945de5bfe33458 (Apache-2.0).

1. Equation 2 is a sum; the code is an average. The printed equation sums over target tokens eq. 2. The code divides the summed log-likelihood by the non-pad target length bart_score.py. The README calls this the "average log-likelihood" README.md.
2. Default checkpoint. BARTScorer defaults to facebook/bart-large-cnn bart_score.py. The task scripts use facebook/bart-large for the vanilla variant score.py §4.2.2. Only the script path matches the paper's vanilla BARTScore.
3. Paraphrase fine-tuning. The paper uses a random 30,000-pair subset for one epoch §3.3.2. The reproduction script loads the whole data/parabank2.json finetune.py and defaults to three epochs args.py. The repository has no sampling step. The batch size args.py and learning rate args.py match the paper.
4. Released paraphrase weights. load() reads models/bart.pth bart_score.py. The README links that file on Google Drive README.md. The repository does not contain it, so the paper's BARTScore-cnn-Para cannot be reproduced exactly from the repository alone.
5. Prompt sets. The file sizes match the paper: 70 source-side and 34 reference-side prompts src_hypo_prompt.txt ref_hypo_prompt.txt. Machine translation uses a single prompt, "Such as" tiny_ref_hypo_prompt.txt score.py, which matches the paper. The WMT18 development-set search that picked it is not in the repository.
6. Prompt punctuation. The code prepends prompt + ", " and appends " " + prompt + "," score.py score.py. The paper does not give any punctuation.
7. Prompt averaging. Eq. 3 normalises each prompt by its own length and averages over prompts. The code averages the per-prompt scores separately for the encoder side (_en_) and the decoder side (_de_) analysis.py. The paper does not say which of the two averages the reported BARTScore-Prompt uses.
8. Harmonic-style score. The code also writes harm_f = (a*b)/(a+b) score.py. The paper defines only the arithmetic F score §3.2. The code's avg_f matches the paper.
9. Multi-reference handling. The code averages each direction over references score.py and defaults multi_ref_score to agg="mean" bart_score.py. The paper says nothing about multiple references.
10. Detokenisation. The summarization and data-to-text scripts detokenise inputs with Moses before scoring score.py score.py score.py. The WMT BARTScore path does not score.py. The paper does not mention detokenisation.
11. Prompt gain for de-en. The text reports a 0.033 gain from "such as" on de-en §4.3.1. Table 3 gives 0.205 and 0.238, a difference of 0.033 tab. 3. These match.

No other discrepancy found in the scoring function or the four directions. The F score and the directions match the paper §3.2 score.py.

Pitfalls

• Direction names. Faithfulness is source to hypothesis. Precision is reference to hypothesis. Recall is hypothesis to reference. In the code, ref_hypo is precision and hypo_ref is recall §3.2 score.py. Scores are negative and higher is better bart_score.py.
• Length normalisation. Summing over tokens, as the printed equation does, makes the score grow with length. The code's per-token average is what the reported numbers rely on. Implement the average, not eq. 2 as printed. Length normalisation is an inference from the code, since the paper's text does not state it.
• Special tokens. tgt_len counts every non-pad token in the target encoding bart_score.py. That includes any special tokens the BART tokenizer adds. The paper does not mention them. This is inferred from the code; check the tokenizer's output before reproducing exact values.
• Silent truncation. Source and target are truncated to 1024 tokens each bart_score.py bart_score.py. The paper gives no limit. Long sources lose their tail without a warning.
• Wrong default checkpoint. Calling BARTScorer() with no arguments gives the CNN/DM variant, not vanilla BART bart_score.py.
• Paraphrase weights are external. bart.pth is not in the repository, and the ParaBank2 data must be downloaded and reformatted by hand README.md.
• Silent exit on errors. On a RuntimeError, score prints the traceback and calls exit(0) bart_score.py. A batch job can then look successful. Check the number of returned scores.
• Prompt cost. If both prompt sets run on single-reference summarization data, each system needs 70 source prompts with 2 scoring calls each and 34 reference prompts with 4 calls each score.py. That is 276 calls per system before any multi-reference multiplier. Budget for it.
• Prompt punctuation. The added comma and space are not in the paper. Keep the code's exact string if you want to match its numbers score.py score.py.
• Ω is undefined in the tables. "Ω + Prompt" rows in Tables 4 and 6 use the best BARTScore per dataset, but the paper does not list which one it was. State the variant you used.
• Do not use harm_f. (a*b)/(a+b) is not the paper's F score §3.2 score.py. Use avg_f.
• Multi-reference and detokenisation change the number. The mean over references and the Moses detokenisation are implementation choices that the paper does not report score.py score.py. Keep them if you are comparing against the tables.
• Factuality is weaker with prompts and paraphrase tuning. Both lowered factuality on Rank19 in the authors' results §4.3.2. Use the vanilla or CNN variant for factuality unless you test otherwise.
• Author-reported results. Significance is by bootstrap at p < 0.05 §4.2.2. The repository does not contain an independent replication.

Lineage

• Builds on: BART (Lewis et al., 2019) as the backbone, the off-the-shelf CNN/DM fine-tune from Hugging Face §3.3.2, ParaBank2 (Hu et al., 2019) for paraphrase fine-tuning §3.3.2, and prompt-based learning (Jiang et al., 2020) for the prompt ensembling §4.2.2.
• Compared against: BLEU, ROUGE, CHRF, BERTScore, MoverScore, PRISM, BLEURT and COMET §4.1.1 tab. 3 tab. 4; FactCC and QAGS for factuality tab. 5. PRISM is the closest prior work, a paraphrase-based metric §2.3.
• Superseded by: not recorded in this wiki.