Benchmarking Large Language Models for News Summarization

In plain words

Large language models can write a news summary when simply asked to, with no training for that job. The authors wanted to know what makes them good at it, and whether their summaries are as good as those a person writes. They had people rate summaries from ten different language models on two standard news datasets.

The biggest factor was whether the model had been trained to follow instructions. The number of parameters mattered much less. The authors also found that the human-written "reference" summaries in those datasets are poor, so they paid freelance writers to write better ones. Against those, the best model's summaries were judged about as good as the writers', even though the model copied far more words straight from the article. The people judging also differed a lot from one another in which summaries they preferred.

TL;DR

Human ratings of ten LLMs on CNN/DM and XSUM find that instruction tuning, not model size, decides zero-shot summary quality §3.2. The reference summaries in both datasets are rated below most systems, so reference-based metrics and comparisons built on them are unreliable §3.2 §3.3. In a blinded pairwise test, freelance-writer summaries and zero-shot Davinci Instruct v2 summaries were preferred about equally overall, but the model's summaries are much more extractive §4.2. Read this card for the benchmark design and its caveats; the paper proposes no training method.

Problem

Earlier work showed that a single zero-shot LLM can beat fine-tuned summarizers in human judgement, but did not show which design choice caused this: model scale, in-context examples or instruction tuning §2.2. Automatic metrics are not trusted for this comparison, so the paper relies on human evaluation §2.2. The standard benchmarks, CNN/DM and XSUM, use reference summaries that were created from article highlights or bolded sentences, and these references have known quality problems §2.1. The authors argue that these problems make it hard to compare LLMs with fine-tuned models and with humans §3.2.

Method

The paper is an evaluation study, not a new model. It has three parts.

1. Benchmark. Ten LLMs (Table 1) plus Pegasus and BRIO are run on 100 validation articles per dataset. LLMs are tested zero-shot (three OpenAI GPT-3 models and three OpenAI instruction-tuned models) or five-shot (all models) §3.1. The existing reference summaries are rated with the same protocol §3.1. Each summary is rated for faithfulness, coherence and relevance by three annotators §3.1.
2. Metric check. Five reference-based metrics are correlated with the human ratings at the system level using Kendall's tau §3.3 tab. 3.
3. Writers vs. best LLM. Six freelance writers, selected by a qualification round, write about 50 words per article for 50 articles from each dataset, each article by three writers §4.1. Davinci Instruct v2 is prompted with a 25-word limit to match the writers' length §4.1. Annotators then pick the better summary of each blinded pair, or call them equal, for overall preference and for informativeness §4.2.

Model list tab. 1:

tab. 1 Caption: "List of large language models we benchmarked with human evaluation." Parameter counts and instruction-tuning flags are as the table states them. The footnote on instruction-tuned GPT-3 says their training details are inferred from the API naming scheme §3.1.

Prompt and decoding, as stated in the text §3.1:

• CNN/DM: the article followed by a request to summarize it in three sentences, ending with "Summary:". Quoted: "Article: [article]. Summarize the article in three sentences. Summary:".
• XSUM: the same format, asking for one sentence ("summarize in one sentence").
• Five-shot: five demonstration articles from the training set, each 50 to 150 GPT-2 tokens long. XSUM demonstrations were chosen by hand §3.1.
• Sampling temperature 0.3 for all LLMs §3.1.
• Pegasus and BRIO decode with beam size 5 §3.1.

Protocol for a rated summary §3.1:

# Human rating of one summary  [§3.1]
# Input:  article (str), summary (str)
# Output: faithfulness (0 or 1), coherence (1 to 5), relevance (1 to 5)

def rate_summary(article, summary):
    # Three annotators rate each summary independently  [§3.1]
    ratings = [ask_annotator(article, summary) for _ in range(3)]
    # Faithfulness is binary, so its mean is a proportion  [§3.1]
    faithfulness = mean([r.faithfulness for r in ratings])
    # Coherence and relevance are 1 to 5 Likert ratings  [§3.1]
    coherence = mean([r.coherence for r in ratings])
    relevance = mean([r.relevance for r in ratings])
    return faithfulness, coherence, relevance

Prompt construction and sampling §2.2 §3.1:

# Build a zero-shot or five-shot prompt  [§2.2] [§3.1]
# Input:  article (str), demos (list of 5 article-summary pairs, or empty)
# Output: prompt (str)

def build_prompt(article, demos, instruction):
    # Each demonstration repeats the instruction, article and summary  [§2.2]
    shots = [format_demo(demo, instruction) for demo in demos]
    # The test article comes last, and the model completes the summary  [§3.1]
    return "".join(shots) + format_query(article, instruction)

def generate(model, prompt):
    # Temperature 0.3; the paper states no max length or stop rule  [§3.1]
    return model.sample(prompt, temperature=0.3)

System-level agreement with humans §3.3 tab. 3:

# Kendall's tau between a metric and human ratings  [§3.3]
# Input:  per system: metric score, mean human rating for one aspect
# Output: tau in [-1, 1]

def system_tau(metric_scores, human_scores):
    # One point per system, not per summary; the system count is not stated
    return kendall_tau(metric_scores, human_scores)

Extractiveness of a summary, using the coverage and density measures §4.2:

# Coverage and density of a summary  [§4.2]
# Input:  summary (list of tokens), article (list of tokens)
# Output: coverage (fraction), density (mean copied-span length)

def extractiveness(summary, article):
    # Coverage: share of summary words that also occur in the article  [§4.2]
    coverage = share_of_words_in_article(summary, article)
    # Density: mean length of continuous spans copied from the article  [§4.2]
    density = mean_copied_span_length(summary, article)
    return coverage, density

The paper contains no equations and no algorithm blocks.

Hyperparameters

Results

Author-reported. Setup: CNN/DM and XSUM, 100 validation articles each, three annotators per summary, scores averaged §3.1. Bold marks entries the authors call not statistically significantly different from the best in each column tab. 2. Full table as printed:

tab. 2 Rows marked * are the instruction-tuned models of tab. 1; the asterisk is not explained in the text.

Kendall's tau between system-level metric scores and mean human ratings tab. 3 (the paper does not state which systems are included or how many):

Quality control: Mechanical Turk ratings of 100 summaries tab. 4:

Other headline numbers, from the text:

• Paired comparison, Davinci Instruct v2 vs. freelance writers, 100 pairs per annotator §4.2 fig. 5: overall preference is equal in aggregate. The more abstractive summary is judged more informative 51.1% of the time, which the authors do not treat as significant §4.2. Krippendorff's alpha is 0.32 for informativeness and 0.07 for overall preference §4.2.
• One annotator who writes abstractive summaries prefers Davinci Instruct v2 57% of the time §4.2 fig. 5.
• Extractiveness §4.2: Davinci Instruct v2 coverage 0.92 and density 12.1; writers 0.81 and 2.07.
• Five-shot with the current reference summaries as demonstrations lowers Davinci Instruct v2 faithfulness on XSUM from 0.97 to 0.77 §3.2 tab. 2.
• Reference-based metrics: Rouge-L on CNN/DM relevance reaches 0.72 tab. 3. On XSUM, replacing references with freelance-writer summaries makes the Rouge-L faithfulness correlation "much more positive" §4.3 fig. 6. The new value is shown only in the figure and is not stated in the text.

Paper vs code

The repository is a data release, not a code release. Its README describes three result files and a summary file README.md [code: README.md#L18-L19] README.md. It contains no model, prompt generation, decoding, or statistics code. Prompts, decoding limits, the significance test and the annotation guidelines cannot be checked against code. The paper says the guidelines are in the code release §3.1, but no guideline file is in the repository. The repository is the authors' own (owner Tiiiger, the first author's GitHub handle, linked from the README).

Points that do differ or cannot be confirmed:

• Model naming. The result files use keys such as openai_text-davinci-002 and openai_text-ada-001 likert_evaluation_results.json. The README calls the Instruct Davinci summaries text-davinci-002 README.md. The paper calls the same model InstructGPT davinci v2 tab. 1. The mapping is consistent, but the identifier is never stated in the paper text.
• GPT-3 350M. The paper's footnote says it was omitted from the human evaluation because it "consistently generates empty outputs" §3.2. The table still reports it for zero-shot and five-shot on CNN/DM tab. 2, and the result file has 600 rows for the zero-shot key and 300 for the five-shot key likert_evaluation_results.json. The 300 rows fit CNN/DM only, which agrees with the dash in the five-shot XSUM cells (inferred from the row count). The footnote contradicts both the table and the data.
• Rating scales match: faithfulness is binary (0 or 1); coherence and relevance run from 1 to 5 README.md.
• Pairwise labels are True, False or Equally Good for overall and informativeness README.md.
• File format. The README names the files .jsonl but the files in the repository are single JSON arrays README.md.
• Licence. The repository has no LICENSE file, so code_license is not stated. Nothing from the repository is quoted on this card.

Pitfalls

• Reference quality drives the conclusions. Most systems outscore the existing references on every aspect, and the authors judge those references as "extremely low quality" §3.2. Scores against them, and comparisons with fine-tuned models or humans that use them, are biased §3.2 §3.3.
• Do not put references in the prompt. The paper finds that five reference demonstrations reduce Davinci Instruct v2 faithfulness on XSUM from 0.97 to 0.77 §3.2 tab. 2.
• Five-shot demonstrations are under-specified. Only length (50 to 150 GPT-2 tokens) and XSUM hand-picking are stated. The separator format between demonstrations is not stated. Random uniform sampling produced unreadable XSUM articles, so the authors picked them by hand §3.1.
• Length is a confound. Davinci Instruct v2 overshoots its word limit, so the writer comparison uses a 25-word instruction to reach about 50 words. The exact instruction text is not stated. Mean lengths are 53.2 words for writers and 52.0 for the model §4.1 §4.2.
• XSUM article handling differs. The writer study uses full XSUM articles with the first bolded sentence restored. The benchmark uses the standard preprocessed version §4.1.
• Text and table disagree on Pegasus. The text gives Pegasus a CNN/DM relevance of 4.40 §3.2. Table 2 gives Pegasus 4.38 and BRIO 4.40 tab. 2.
• Metric count is inconsistent. The text says six metrics are computed §3.3, but five are listed, and Table 3 has five rows tab. 3.
• Table 4 does not match its caption. The caption says the zero-shot Davinci Instruct v2 and reference rows were taken from Table 2 after averaging CNN/DM and XSUM. Simple averages of the Table 2 values differ, for example reference coherence (3.20 + 4.13) / 2 = 3.67 against 3.59 printed, and Davinci Instruct v2 coherence (4.15 + 4.41) / 2 = 4.28 against 4.26 printed. The sample or weighting used is not stated tab. 4.
• Significance test not named. Table 2 bolds entries that are "not statistically significantly different", but no test is named tab. 2.
• Model identity is partly inferred. Instruction tuning of the GPT-3 models is inferred from the API names §3.1. The Pegasus and BRIO checkpoints are not named in the text.
• Annotator agreement is low. Krippendorff's alpha is 0.07 for overall preference and 0.32 for informativeness, which the authors call moderate §4.2. A single preference number hides large differences between annotators.
• System count and sample size. The number of systems behind each Kendall tau is not stated, so the significance of the correlations cannot be judged tab. 3. The 100-article human sample is small for system-level claims.
• Crowd pay and quality. Annotators were Mechanical Turk workers paid $15.00 per hour. The paper does not describe worker filtering against spam §3.1.
• Writer quality is not a gold standard. Writers were chosen by a qualification round and are paid $4 per article. The paper calls the writer summaries higher quality than the references on the Mechanical Turk check §4.1 tab. 4, but this is the authors' own judgement.

Lineage

• Builds on: the fine-tuned summarizers it benchmarks against, Pegasus (Zhang et al.) and BRIO (Liu et al.) §3.1, and the instruction-tuning work of Ouyang et al. (bib key instruct-gpt) §5.
• Compared against: Goyal, Li and Durrett, "News Summarization and Evaluation in the Era of GPT-3" (bib key gpt3-era). The paper calls it the closest prior work, since it also shows zero-shot GPT-3 beating fine-tuned models in human judgement, but it studies only one LLM §5.
• Related: Liu et al., "Revisiting the Gold Standard" (bib key Liu2022RevisitingTG), which the paper cites for length as a confound §4.1 and for the difficulty of human evaluation §5.
• Citation check: the paper cites goyal-durrett-2020-evaluating for a finding that an instruction-tuned GPT-3 model outperforms fine-tuned summarizers §2.2. The bib entry for that key is "Evaluating Factuality in Generation with Dependency-level Entailment", a 2020 paper that predates that model. That is probably a wrong key, since the gpt3-era paper is the likely intended source. Verify before citing.