News Summarization and Evaluation in the Era of GPT-3

In plain words

Most news summarizers are trained on large piles of article and summary pairs, so they copy the style of those pairs. This paper asks what happens if you skip the training and just ask a large instruction-following language model, GPT-3, to summarize an article in a given number of sentences. That prompt is enough to control the length of the output.

The authors ran a blind comparison with people. Readers mostly liked the GPT-3 summaries better than those from fine-tuned models. The same readers' preferences did not match the usual automatic scores, which rated the GPT-3 summaries lower. So the usual test-set scores cannot be trusted for this kind of system.

The same prompting idea also works for keyword-focused summaries, where a summary has to center on a named entity in the article. Readers preferred the GPT-3 keyword summaries over a fine-tuned baseline for about 70% of article-keyword pairs.

TL;DR

• Prompt-only GPT-3 (text-davinci-002) summaries of news beat fine-tuned BRIO, PEGASUS and T0 in a crowd preference study on CNN and BBC articles §3.2 tab. 3.
• Reference-based metrics (ROUGE, BERTScore, MoverScore, QAEval) and reference-free metrics (SUPERT, BLANC, QuestEval, QAFactEval, FactCC, DAE, SummaC) score GPT-3 lower and do not follow human rankings §4.1 §4.2 tab. 4 tab. 5.
• Keyword-focused GPT-3 is preferred over CTRLSum in roughly 70% of article-keyword pairs by majority vote §5.2 fig. 7.
• All numbers are author-reported on 100-article human studies and 500-article metric sets.

Problem

Fine-tuned summarizers imitate their training references. Those references (CNN/DM, XSum) are highly extractive or highly abstractive, so a model trained on them inherits that style and needs task-specific data for any new setting §1 §2.1 tab. 1. Automatic metrics are validated against those same references, and the authors argue that this validation does not hold for summaries produced by prompting §1 §4.

Method

The paper is an empirical study, not a new model. It has four parts.

1. Prompted summarizer. The model is text-davinci-002, a 175B instruct-tuned GPT-3, used as a black box through its API §1 §2.2. The article is placed in the prompt and an instruction fixes the number of sentences. The authors match the sentence count to each dataset's reference style §2.2 app. A.

2. Human preference study (A/B-style, best and worst). Annotators see the summaries of three systems for one article and pick the most and least preferred. Multiple choices are allowed. Annotators are not given a definition of quality §3.1.

3. Automatic metrics. The GPT-3 outputs and two fine-tuned baselines (BRIO and PEGASUS, plus T0) are scored with reference-based and reference-free metrics §4 §4.1 §4.2.

4. Keyword-focused summarization. The prompt names a keyword, which is a named entity from the article, and GPT-3 is compared with the CTRLSum fine-tuned baseline in a second human study §5 §5.2.

Prompt for generic summaries, as quoted from the paper, with N the sentence count §2.2:

• "Article: {{article}}"
• "Summarize the above article in {{N}} sentences."

The single-sentence XSum setting changes the second line to "Summarize the above article briefly in 1 sentence." app. A. The paper does not give the keyword prompt wording in the text; it is shown in the qualitative figure fig. 6.

# Prompted news summary with a sentence-count instruction  [§2.2] [app. A]
# Input:  article (str); sentence_count (int), e.g. 3 for CNN, 1 for XSum
# Output: summary (str)

def summarize_with_prompt(article, sentence_count):
    # Put the article first, then the length instruction  [§2.2]
    prompt = "Article: " + article + "\n\n"
    prompt += "Summarize the above article in "
    prompt += str(sentence_count) + " sentences."
    # Decoding settings are not stated in the paper  [app. A]
    return gpt3_text_davinci_002(prompt)

The human scores are the share of articles on which a system is selected as best (or worst) by majority vote over the three annotators, and ties are possible §3.2 tab. 3.

Data release: a corpus of about 10K generated summaries, and 1K human judgments, from the authors' project page §1 §4 Experimental Setup. Only the text data is released. No code is linked there.

Hyperparameters

Results

Author-reported. Human study on 100 CNN-2022 and 100 BBC-2022 articles, 3 annotators each, majority vote §3.2. Metric study on 500 test articles per dataset §4 Experimental Setup.

Human majority-vote shares (percent of articles):

Paired bootstrap significance for the GPT-3 lead over the next best system: CNN p = 2e-3, BBC p = 6e-4 §3.2. Only 2% of CNN and 7% of BBC articles have more than one best summary after majority vote §3.2 footnote.

Keyword-focused human study: GPT-3 is preferred over CTRLSum by a majority of annotators for nearly 70% of article-keyword pairs, and unanimously for more than half §5.2 fig. 7.

Reference-based automatic metrics tab. 4. Rows with "-" are not available (BRIO for Newsroom):

Reference-free automatic metrics tab. 5:

Summary statistics of the human-study outputs (dataset, system, sentences, words per sentence, novel 1-grams and 2-grams in percent, named entities per 100 words) tab. 2:

Dataset statistics tab. 1: CNN article 760.5 words, summary 45.7 words; DailyMail 653.3 / 54.6; XSum 431.1 / 23.2; Newsroom 658.6 / 26.7.

The paper also reports that the GPT-3 summaries follow the sentence-count instruction in 98% of the human-study test instances §2.2 fig. 3.

Paper vs code

No official code. Searched in order: (1) the paper text, abstract and appendix, which link only the data page and third-party model pages (BRIO, promptsource, CTRLSum, and the DAE repository); (2) the authors' project page, which links the data files and no repository; (3) GitHub repository search for the title's data-page name (no results) and for user:tagoyal (9 repositories, none for this study). The appendix links the DAE code (github.com/tagoyal/factuality-datasets) app. A. That repository is a metric dependency, not this paper's method, and it lists no license, so we did not clone or quote it. The CTRLSum, BRIO and promptsource code is third-party and was not used.

Pitfalls

• XSum baseline was manually cleaned. About 30% of BRIO-XSum outputs were broken on the authors' articles. The authors picked a better output from beam size 10 by hand, and where none was acceptable, removed the first sentence of the article and re-sampled §3.1. This per-article manual step is not reproducible from the text, and it changes the strongest baseline in the human study.
• Length and style are confounded. GPT-3 summaries are longer (23.4 vs 15.8 words per sentence for BRIO on CNN tab. 2). The authors report Pearson correlations between length difference and annotator votes of 0.17 (CNN) and 0.02 (BBC), so length explains little of the preference, but they do not rule out other factors app. B.
• Quality is undefined for annotators. Annotators were told to pick the summary they would like to see while browsing §3.1. Multiple selections were allowed, so "best" can be a set. After majority vote, only 2% of CNN and 7% of BBC articles have more than one best summary §3.2 footnote.
• Agreement is low. Krippendorff's alpha with MASI distance is named but its value is not stated in the text §3.2. The vote distribution shows GPT-3 unanimously best for under 30% of articles §3.2 fig. 5.
• Study date is invalid. The collection window is written as "March 1, 2022 and June 31, 2022" §3.1. June has 30 days. Use "spring 2022, end date unclear" when citing it.
• Possible test-set overlap. GPT-3's training data is unknown. The authors note that older test articles may overlap with it but see no qualitative difference §4 Experimental Setup footnote. The model is a closed API. Decoding parameters, the snapshot date and max output length are not stated app. A, so exact reruns are not guaranteed.
• ROUGE-L gap is quoted two ways. The introduction says GPT-3 scores about 7 ROUGE-L points lower on average §1. Table 4 gives the gap to BRIO as 6.7 (CNN), 11.1 (DailyMail) and 20.4 (XSum). The averaging method is not stated, so use the per-dataset numbers from tab. 4.
• Newsroom is incomplete. BRIO has no Newsroom checkpoint, so its row is empty in tab. 4 and tab. 5. GPT-3 uses N=2 on Newsroom, a setting that appears only in the appendix app. A.
• Keyword study is narrow. It uses CNN articles only, 2 named entities per article, and a lead-biased first keyword app. A §5.2. The NER tool is not named.
• Factuality scores are not reliable here. FactCC and DAE were trained with reference summaries as positives §4.2 Discussion. The entailment models are used as classifiers, scored by P(factual | article, summary) §4.2. SummaC uses a sentence-level setting and QuestEval a precision-only setting app. A.
• Long documents do not work by default. The authors' segment-then-summarize run on one CNN article gives coherent per-segment summaries but a repetitive, incoherent concatenation app. E fig. 10.
• Costs. About 2600 GPT-3 summaries were generated for about USD 150 app. D. The combined human-study cost was about USD 1020 app. D.

Lineage

• Compared against: BRIO (fine-tuned, arXiv:2203.16804, linked from the authors' project page); T0 (prompted and multi-task fine-tuned, arXiv:2110.08207, linked from the same page); PEGASUS (fine-tuned) §2.1 §3.
• Keyword baseline: CTRLSum, a fine-tuned BART with a prompt in the encoder §5.1. Its arXiv id is not stated in the paper, so it has no card here.
• Superseded by: not known from content in this wiki.