Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

In plain words

Summarization models often write fluent summaries that state things the source article does not say, or get a detail wrong. Many automatic metrics try to score how factual a summary is, but each one gives a single number or yes/no, and there was no common test set to compare them on. So it was hard to say which metric to trust.

This paper proposes a finer way to label mistakes. Each sentence of a summary is judged as correct or wrong, and a wrong sentence is tagged with the kind of error: a wrong relation, a wrong entity, a wrong time or place, a pronoun with no clear referent, a wrong link between statements, information not in the article, or text that is too garbled to check. The authors had crowd workers label about 2250 summaries from nine systems on two news datasets.

With those labels, the authors measure how well each metric agrees with human judgement, and which kinds of mistakes each metric can catch. The result is a shared benchmark for comparing factuality metrics, plus a picture of which systems make which mistakes. Most metrics agree with humans only weakly.

TL;DR

• FRANK is a benchmark of about 2250 summaries (CNN/DM and XSum) from nine summarization systems, with sentence-level factuality labels §3, §5.1.
• Errors are labelled with a typology of seven error types plus "Other" and "Not an error" §2, tab. 1.
• Metric correlations with human judgement are low. The best Pearson is 0.27 (BERTScore P) and the best Spearman is 0.30 (FactCC), both on all data tab. 2.
• Entailment-based metrics do best on CNN/DM; their correlation on XSum is about a quarter of that on CNN/DM §5.3, fig. 4.
• The released code is a small evaluation script for partial correlations, the Williams test and error ablations, with the data README.md.

Problem

Summaries can be fluent and still wrong. The authors cite prior work finding that about 30% of summaries have factual inconsistencies §1. N-gram metrics such as BLEU, ROUGE and METEOR correlate poorly with human factuality judgements §1, §6. Several factuality metrics appeared, but they could not be compared because there was no common benchmark, and earlier annotated sets were small: 200 and 503 sentences in two papers and 470 summaries in another §6. Earlier annotation also treated factuality as one binary label, so it could not show what kind of error a system makes §1, §6. Crowd-expert agreement on binary factuality was low in prior work; the paper reports kappa 0.65 for three crowd workers and at most 0.74 with more than nine §3, Agreement with Domain Expert.

Method

The paper has three parts: a typology of errors, a human annotation study that applies it, and an evaluation of metrics against the labels.

1. Typology of factual errors §2, tab. 1. The errors are grouped into three levels.
   • Semantic frame errors §2.1: PredE (the relation or predicate, including negation, does not match the article), EntE (a core argument or its attribute is wrong, including swapped agent and patient), CircE (a non-core detail such as time, place or manner is wrong).
   • Discourse errors §2.2: CorefE (a pronoun or reference is wrong or has no clear antecedent), LinkE (a temporal, causal or other discourse link between statements is wrong).
   • Content verifiability errors §2.3: OutE (information not in the article), GramE (a sentence is so ungrammatical that its meaning is unclear; minor errors are accepted).
   • Two more labels complete the scheme: OthE for errors outside the typology, and NE for no error §2.
2. Source summaries §3, app. A.1. The authors used public outputs of existing models, not new runs. CNN/DM: S2S, PGN, BUS, BertSum, BART (five systems). XSum: PGN, TConvS2S, TranS2S, BERTS2S (four systems). The paper uses 250 articles per dataset, giving 1250 CNN/DM and 1000 XSum summaries §3. The model list, with the system names as used in the released data, is in app. A.1.
3. Sentence-level annotation §3, Annotation Collection. Each sentence of a summary is shown in the context of the full summary. For each sentence the annotator first says whether it is factual. If not, the annotator picks one or more error categories, because one sentence can contain several errors. Three crowd annotators label each item on Amazon Mechanical Turk. The authors tried a Likert rating of whole summaries in a pilot and dropped it because it gave less precise and less consistent judgements §3, footnote.
4. Quality control §3, app. A.6. Instructions use examples, not linguistic jargon. A tutorial gives feedback to new workers. A qualification test requires at least 85%. During the task, workers are checked with an entity question about the article and with artificially altered sentences; workers below 50% correct on the altered sentences are blocked. Pay is 50% above the average US minimum wage, with bonuses for a continuous-evaluation score of 60% or more and for sets of 10 annotations §3, app. A.6, §8.
5. Summary-level factuality score §5.1, code: README.md. Each summary gets a score from 0 to 1, the mean of its sentence-level factual judgements. The released field is named Factuality.
6. Metrics evaluated §5.1, app. A.2. Four general metrics: BLEU, METEOR, ROUGE (1, 2 and L), all n-gram based and computed against the reference summary, and BERTScore P, R and F1 between the summary and the article. Five factuality metrics: an OpenIE triple-matching baseline, FactCC (entailment on sentences), DAE (dependency-level entailment), FEQA (question generation and answering) and QAGS (question generation and answering). Each metric's implementation details are in app. A.2.
7. Correlation with human judgement §5.2, app. A.7. Each metric is correlated with the human score after removing the effect of the system that produced the summary. The paper says the control variable should be both system and dataset §5.2, app. A.7. The released code controls for one column only, by default model_name evaluate.py. The correlation is computed on the residuals of both variables after regressing each on the control, as the appendix describes:

\Delta M_k = M_k - \hat{w}_{M_k} C app. A.7

\Delta H = H - \hat{w}_{H} C app. A.7; the paper prints M_k on the right of the second equation, see Paper vs code

# Partial correlation of a metric with human judgement  [§5.2] [app. A.7]
# Input:  metric (N,), human (N,), system (N,) labels
# Output: partial Pearson r, partial Spearman rho

def partial_correlation(metric, human, system):
    # Dummy-code the system  [code: evaluation/evaluate.py#L127-L129]
    controls = one_hot(system)
    # Subtract the system effect  [code: evaluation/evaluate.py#L130-L133]
    metric_residual = metric - fitted(controls, metric)
    human_residual = human - fitted(controls, human)
    # Correlate residuals  [code: evaluation/evaluate.py#L135-L137]
    r = pearson(metric_residual, human_residual)
    rho = spearman(metric_residual, human_residual)
    return r, rho

8. Comparing two metrics app. A.4, code: evaluation/evaluate.py#L70-L98, L146-L201. The paper compares metric pairs with the Williams test, which uses the correlation between the two metrics as well as their correlations with human judgement. The authors call it the Hotelling-Williams test. The code implements the Williams (1959) formula with n set to the smaller sample size, after the partial residualisation above.
9. Error-group ablation §5.4, code: evaluation/evaluate.py#L203-L229. Errors are grouped into three levels from step 1. For each group the released data holds a Flip_ field with flipped labels. The metric's partial correlation with the flipped judgement is subtracted from its correlation with the true judgement. A large positive value means that group carries much of the metric's correlation.

# Error-group ablation  [§5.4] [code: evaluation/evaluate.py#L221-L227]
# Input:  metric (N,), human (N,), human_flipped (N,), system (N,)
# Output: drop in partial Pearson r when one error group is flipped

def ablation_drop(metric, human, human_flipped, system):
    # Full judgement  [code: evaluation/evaluate.py#L221-L222]
    full = partial_pearson(metric, human, system)
    # Flipped labels  [code: evaluation/evaluate.py#L224-L227]
    flipped = partial_pearson(metric, human_flipped, system)
    # Positive: the metric relies on this error group  [§5.4]
    return full - flipped

Hyperparameters

Results

Author-reported. Setup: 2250 summaries (as stated in the paper) from nine systems on CNN/DM and XSum, human judgements as the reference, partial correlations controlling for system and, per the paper's appendix, dataset §5.1–§5.3, app. A.7. No baseline is defined; the table lists every metric the paper evaluates. Cells are partial Pearson and Spearman correlations with human judgement tab. 2. Significance p-values are in tab. 2. Examples: FEQA on all data is not significant (Pearson p = 0.83, Spearman p = 0.60), and OpenIE on all data is not significant under Spearman (p = 0.36).

Source: tab. 2. The paper bolds the best value in several columns; the table above keeps the values only.

Share of summaries with at least one factual error, and the error breakdown per system §4, tab. 3. A sentence counts as wrong if the majority of its annotators mark it, and a summary counts as wrong if any sentence does §4. The caption describes the columns as proportions of summaries; the paper does not say whether the denominator is all summaries or only the incorrect ones. A summary can have several error types.

Source: tab. 3; the text gives the same breakdown in §4. Figure 2 shows the same proportions as bars fig. 2.

Other headline results, all author-reported:

• Factuality of summaries: 60% of annotated summaries contain at least one error. XSum has 92% and CNN/DM 43%. On XSum every system is above 80% §4.
• Annotation agreement: Fleiss kappa 0.58 for factual versus not, with 91% majority agreement (the paper compares with 76.7% in prior work). Kappa 0.39 on the eight-way category choice when all three annotators agree a sentence is not factual §3, Inter-Annotator Agreement. Against one expert on 201 items, the majority label of the three crowd annotators reaches Cohen's kappa 0.86 §3, Agreement with Domain Expert.
• Metric pattern: entailment metrics do best on CNN/DM and their partial correlation drops by about a factor of four on XSum §5.3, fig. 4. Figure 3 shows that for FEQA and FactCC the slope fitted on all data is much steeper than the slopes within system-dataset subsets, which the paper says can produce false high correlations when not controlled §5.2, fig. 3. The paper gives no numbers for the uncontrolled correlations.
• Error ablations: BERTScore correlates mainly with content verifiability errors §5.4. DAE has the highest correlation with discourse errors. FactCC correlates little with discourse errors §5.4, fig. 5.

Paper vs code

The code is at github.com/artidoro/frank at 80a88fb, under the MIT license with copyright held by Artidoro Pagnoni, the first author LICENSE. It contains the released data and evaluation/evaluate.py only. It does not contain implementations of the factuality metrics, which were run by their authors, and the annotation platform is a separate repository README.md. Differences:

• Control variable. The paper says the partial correlation controls for both system and dataset §5.2, app. A.7. The script defaults to model_name only evaluate.py, and it accepts one column, so the paper's two-variable control needs a combined column that the script does not build. The README also says only the system is controlled README.md.
• Human residual. The appendix writes the human residual as \Delta H = M_k - \hat{w}_H C app. A.7. The script residualises the human score evaluate.py. Use H, as this card does.
• Split. The paper reports results on all of FRANK §5.3, tab. 2. The script defaults to the validation split evaluate.py. To match the table use --split all; the README says tuning should use the validation split and reporting the test split README.md.
• Metric list. The script's default baseline list has ten entries: Bleu, Meteor, Rouge 1, Rouge 2, Rouge L, BertScore P Art, FEQA, QAGS, Dep Entail and FactCC evaluate.py. OpenIE is commented out in the code evaluate.py, and the released baseline file has no OpenIE column baseline_factuality_metrics_outputs.json. The OpenIE row of tab. 2 cannot be reproduced from the repository. Dep Entail is the DAE metric and BertScore P Art is BERTScore P against the article.
• Ablation groups. The code uses the three broad groups from §5.4 evaluate.py. The finer flips are commented out. This matches the paper.
• BERTScore fix. The README records a fix to BERTScore results after the paper's first release, and says the baseline file was updated README.md. The paper's table is presumed to match the fixed numbers, but this was not checked against the file, and the published values were not reproduced here.
• Reproduction. Not run. The evaluation script needs pandas, scipy, scikit-learn and matplotlib, and the run was not approved in this session, so the numbers above are the paper's.

Pitfalls

• Dataset size. The paper says 2250 summaries and 1000 XSum summaries §3. The released data has 2246 records: 671 in the validation split and 1575 in the test split human_annotations.json; code: README.md. It has 1250 CNN/DM and 996 XSum records. The four-summary gap is not explained in the paper or the README.
• Text and table disagree. §4 says BERTSum has 23% and BART 27% incorrect summaries, but tab. 3 gives BERTSum 27.2% and BART 23.8%. The two values are swapped between text and table.
• Overall rate. tab. 3 gives "All models" 60.0%. The nine per-system rows average to about 64.7% when weighted equally, and this card did not find a weighting that gives 60.0%. Treat the overall figure as unverified.
• BERTScore Spearman. The text in §5.3 gives BERTScore P a Spearman of 0.35 §5.3, but tab. 2 gives 0.24 for all data. The table is the source for this card.
• Green cells in tab. 4. The caption's note on green cells is ambiguous about which way the test points. This card reads a green cell as a significant difference between the two metrics at 0.05. The released code returns p-values, not colours evaluate.py.
• Confounder choice changes the numbers. Partial correlations depend on the control. Controlling only for system, as the code does, and controlling for system and dataset, as the paper's text says, give different results. The paper's own plot shows that uncontrolled correlations can be inflated §5.2, fig. 3. Check which control your reproduction uses before comparing numbers.
• Small correlations. The largest all-data correlation is 0.27 Pearson and 0.30 Spearman tab. 2. A metric that looks useful on one subset can have a near-zero correlation on another, as the XSum column shows.
• DAE truncation. DAE was trained with a maximum length of 128 after the article and summary are concatenated. On CNN/DM the paper's numbers come from truncated articles app. A.2.
• OpenIE threshold. The 0.6 cut-off was chosen on a pilot set that is not released app. A.2. It cannot be re-tuned from the repository.
• Annotation reliability. Agreement is moderate for the category choice (kappa 0.39). Coreference is the weakest category: when two annotators agreed on CorefE, the third chose "Not an error" 69.7% of the time app. A.5. Read category-level results with that in mind.
• Flipped-label semantics. The Flip_ fields are described only as "flipped labels for each category" README.md. The paper does not say how they are built. The ablation uses them as given.
• Appendix numbering. Appendix subsections are cited as app. A.n from the LaTeX section order, since the source uses \appendix. Figure and table numbers follow source order. The compiled PDF was not checked against these numbers.

Lineage

• Builds on: the typology uses frame semantics and rhetorical structure theory as the paper cites them §2. It has no arXiv ids in the paper source, so none are linked here.
• Compared against: BLEU, ROUGE-1/2/L and METEOR; BERTScore P, R and F1; an OpenIE triple-matching baseline with Sentence-BERT; FactCC; DAE; FEQA; QAGS §5.1, app. A.2. Their arXiv ids are not in the paper source, so none are linked.
• Prior factuality annotations: Fabbri et al. 2020 (SummEval) on CNN/DM, and an XSum annotation set the paper cites under the key factentailment §6. Neither has a card here.
• Superseded by: not known from content in this wiki.