Text summarization

Overview

Text summarization turns a long text, or a collection of texts, into a shorter one that keeps the main content. Ten cards cover it from different angles: a review of NLP for student feedback, which sorts the methods into three families 2301.08826 §2.4.1; a dialogue dataset with fine-tuned baselines 2105.06762 §3.1; a study of prompted GPT-3 on news 2209.12356 §2.2; a human study of ten LLMs on news, which finds that instruction tuning matters more than model size 2301.13848 §3.2; a study that adapts eight LLMs to clinical notes 2309.07430 §3.2; BARTScore, a metric that scores generated text, summaries included, from a pre-trained model's likelihood 2106.11520 §3.2; BRIO, a fine-tuning method that adds a ranking loss over candidate summaries 2203.16804 §3; and a measurement of the energy each inference uses, which covers summarization models among ten tasks 2311.16863 §3.1. Two more cover the edges of the task: FRANK, a benchmark of sentence-level factuality labels for about 2,250 CNN/DM and XSum summaries 2104.13346 §3, and LongT5, an encoder that handles inputs of up to 16k tokens 2112.07916 §3.1. The review motivates the task with student feedback: reading comments by hand is slow, and a summary is easier to digest than the raw corpus 2301.08826 §1 §2.4.1.

The shared idea is a summary that keeps "critical information" from the source 2301.08826 §2.4.1. The families differ in where the summary's sentences come from. Extractive methods only select sentences that already exist. Abstractive methods write new sentences, either fine-tuned on target data or prompted from a large instruction-following model. Hybrid methods select first and then paraphrase.

Evidence is uneven. The review runs no experiments and compares no two families on the same data. The experimental cards each cover a different setting: English spoken dialogue (DialogSum), English news (CNN, BBC and keyword-focused CNN, and CNN/DM and XSUM with human ratings of ten LLMs), clinical notes (four tasks, with a reader study on three), factuality labels on CNN/DM and XSum (FRANK), candidate-ranking fine-tuning (BRIO, on CNN/DailyMail, XSum and NYT), and long documents (LongT5, on six summarization sets). BARTScore's experiments test metrics, not summarizers. The energy study measures cost, not quality, on SAMSum, CNN/DM and XSum inputs 2311.16863 tab. 1. Their numbers are not comparable with each other or with the review's figures. Treat the variants as a map of options, and check the choice on your own data before you commit to it.

Variants

Extractive

The method picks a subset of the source sentences and ranks them by importance. It builds an intermediate representation with features such as bag of words, n-grams or graphs 2301.08826 §2.4.1 fig. 4. Several extractive systems are named in the review, none of them with a card on wikillm:

• Statistical frequency weighting (Madhuri et al.). Tokens are weighted by their frequency in the document, sentences are scored by summing those weights, and the top sentences are kept. The review reports higher accuracy against human-written summaries, with no value given 2301.08826 §2.4.1.
• Phrase clustering (CourseMIRROR, Fan et al.). Phrases are extracted from student reflections, grouped with k-medoids clustering and re-ranked by how many students they cover. The review reports this as better than LexRank, with no value given 2301.08826 §2.4.1.
• Graph and latent-semantic summarizers (TextRank and LSA, used by Gottipati et al. on forum posts). The review reports that the LSA summary gave more additional recommendations than TextRank, a qualitative comparison only 2301.08826 §2.4.1.
• Learned sentence selection (Mutlu et al.). An LSTM scores each sentence's salience from semantic and other features. The review reports that it outperformed hierarchical Bi-GRU, CNN, Bi-GRU, SummaRuNNer and BanditSum baselines 2301.08826 §2.4.1.

The review credits extractive methods with language independence, because they neither build nor paraphrase sentences 2301.08826 §2.4.1.

On DialogSum, the lead baseline reaches ROUGE-1 27.52 and the ext-oracle baseline 37.90, both with n = 2 2105.06762 tab. 4. The oracle chooses its sentences using the reference summaries, so it marks a ceiling for extracted output on that data, not a method anyone can run at inference time.

Abstractive, fine-tuned

The method is a sequence-to-sequence model trained on pairs of source and summary. The review's one abstractive result is from Song et al., an LSTM-CNN that switches between a generating mode and a copy mode. On ROUGE it reports 34.9 for ROUGE-1 (4.4 points above CNN-based systems) and 17.8 for ROUGE-2 (1.6 points above) 2301.08826 §2.4.1. The review does not name the dataset or the baselines' setup, so these numbers cannot be read against the extractive results above.

DialogSum is a 13,460-dialogue English dataset of daily-life conversations with one human summary each, split 12,460 / 500 / 500 2105.06762 §2.1. Each model is fine-tuned on that data alone. Author-reported, ROUGE-1 reaches 47.28 for BART-large, 47.04 for UniLMv2-base and 35.91 for a Transformer trained from scratch, on the test set with three references per dialogue 2105.06762 tab. 5. The paper says that spoken dialogue is harder than chat or news, because the relevant information is spread over many turns, with coreference and discourse relations that are not explicit [arxiv:2105.06762 §5.2]. Its error analysis on 50 summaries finds incorrect coreference in 94% of the Transformer's outputs and 60% of UniLMv2's 2105.06762 tab. 11.

The paper says its sources are freely accessible for academic use 2105.06762 §7, but its data repository declares a non-commercial licence. Check that before any commercial use of the data.

The clinical study fine-tunes open models with QLoRA, which trains small low-rank adapters on a frozen 4-bit base model 2309.07430 §3.2. In that study FLAN-T5 (2.7B) beat FLAN-UL2 (20B) under QLoRA 2309.07430 §5.1.2. That is one result in one setup, not a general claim about model size.

Abstractive, fine-tuned with candidate ranking

The method fine-tunes a pre-trained generator with two losses: cross-entropy on the reference summary, and a pairwise margin loss. The margin loss makes the model's length-normalised log-likelihood order a set of candidate summaries the same way their ROUGE against the reference does 2203.16804 §3 eq. 8 eq. 10. The candidates come from an offline pass before training: 16 per training document, from diverse beam search 2203.16804 app. B. The paper calls this pass a non-trivial overhead §5.5. On CNN/DailyMail, one epoch takes about 20 hours on four RTX 3090 GPUs app. B. Card: BRIO: Bringing Order to Abstractive Summarization.

The generator variant, BRIO-Mul, keeps the cross-entropy term. Its author-reported ROUGE-1 is 47.78 on CNN/DailyMail, against 44.29 for the authors' own BART-large evaluation, and 49.07 on XSum, against 47.46 for PEGASUS 2203.16804 tab. 2. The contrastive-only variant, BRIO-Ctr, scores lower and cannot generate §3 tab. 2. Ordering the candidates by BERTScore instead of ROUGE gave 47.53 ROUGE-1 against 47.78 tab. 6. The paper sets the contrastive weight to 100 on all three datasets tab. 13. By hand, not run, the BRIO card finds that the released XSum configuration weights the ranking term about 1 against cross-entropy, so check that setting before you reproduce the XSum result 2203.16804 tab. 13.

Abstractive, long input

The method is a fine-tuned encoder-decoder built for long inputs. Full self-attention grows quadratically with input length, which limits T5 to short inputs 2112.07916 §2. LongT5 replaces the encoder's attention with a local window of radius 127 plus one global token per block of 16 tokens, built on the fly in each layer, so the global tokens need no extra input slots 2112.07916 §3.1.1 §3.1.2. Its pretraining hides whole important sentences and asks the model to write them back, instead of hiding random spans 2112.07916 §3.2. Card: LongT5.

Author-reported, LongT5 xl with 16k input reaches ROUGE-1 48.35, ROUGE-2 21.92 and ROUGE-L 44.27 on arXiv, above PRIMER's 47.6, 20.8 and 42.6 2112.07916 tab. 2. It is not the best on every set. On Multi-News, PRIMER scores higher, and on CNN/DM, HAT-BART has higher ROUGE-1 and ROUGE-L (44.48 and 41.52, against 43.94 and 41.28 for LongT5 xl at 4k) tab. 2. Each dataset uses its own input length, so the rows are not controlled for length tab. 2.

The cost is the main constraint. The paper's complexity for this attention is O(l × (r + l/k)) for input length l 2112.07916 §3.1.2. On a TPU slice, the longest input before running out of memory was 6k for T5.1.1 base, 12k for LongT5 base with global tokens and 36k for LongT5 base with local attention 2112.07916 §5.1. Local attention reaches longer inputs, but on NQ question answering it trails the global variant 2112.07916 §5.2.

Abstractive, prompted LLM (zero-shot)

The method sends the article and an instruction to a large instruction-tuned model, with no training on news data. The study uses GPT-3 text-davinci-002, a 175B model accessed through an API, with the prompt "Summarize the above article in N sentences" 2209.12356 §1 2209.12356 §2.2. The instruction was followed in 98% of the test instances 2209.12356 §2.2 fig. 3.

In a crowd study on 100 CNN and 100 BBC articles, with three annotators per article and majority vote, annotators picked the prompted summary as best for 58% of CNN articles and 57% of BBC articles. The fine-tuned BRIO baseline got 36% and 20% 2209.12356 tab. 3. The gap is significant at p = 2e-3 (CNN) and p = 6e-4 (BBC) 2209.12356 §3.2. The authors estimate about 2,600 summaries cost about USD 150 at the API price of the time 2209.12356 app. D.

The same prompting works for keyword-focused summaries. Given a named entity from the article as the keyword, GPT-3 was preferred to CTRLSum by a majority of annotators for nearly 70% of article-keyword pairs 2209.12356 §5.2 fig. 7. That study uses 100 CNN articles only 2209.12356 app. A.

A second news study rates zero-shot LLMs with human annotators, using 100 validation articles each from CNN/DM and XSUM 2301.13848 §3.1. Among the ten models, the instruction-tuned ones scored highest without examples. On CNN/DM, Davinci Instruct v2 (175B) reached faithfulness 0.99 and coherence 4.15 on a 1-to-5 scale; GPT-3 175B without instruction tuning reached 0.76 and 2.65 2301.13848 tab. 2. The 6.7B instruction-tuned Curie model came close to the 175B one on the same measures tab. 2. The paper infers the instruction-tuning flag for the GPT-3 models from their API names §3.1. Its CNN/DM prompt asks for three sentences and ends with "Summary:"; XSUM asks for one sentence. Sampling uses temperature 0.3, and the paper states no maximum length or stop rule §3.1.

Do not put reference summaries in the examples. With the current references as five-shot demonstrations, Davinci Instruct v2's XSUM faithfulness fell from 0.97 to 0.77 2301.13848 §3.2 tab. 2.

Prompted LLM with retrieved examples

The method adds labelled examples to the prompt. For each test input, the system embeds the input, finds the nearest labelled training inputs and places them, each with its target summary, before the query 2309.07430 §3.2 §4.1.1. The number of examples is chosen so that at most 1% of test inputs exceed the context length 2309.07430 §3.2.

With GPT-4 and the largest number of examples the paper's setup allows 2309.07430 §5.1.4, ten physicians compared the model's summaries with expert summaries in a blinded pairwise study. The study covers radiology reports, patient questions and progress notes; dialogue was excluded 2309.07430 §4.2. The model's summary was preferred in 36% of comparisons, rated equal in 45%, and the expert's was preferred in 19% 2309.07430 §5.2 fig. 7. The ratings cover completeness, correctness and conciseness. Proprietary models could only use this method; open models were also run with it, and some were trained with QLoRA 2309.07430 §3.2.

Hybrid

The method selects the top-ranked sentences extractively, then paraphrases them into a summary 2301.08826 §2.4.1 fig. 6. The review describes this family but gives no specific system and no number for it.

When to use which

Constraints separate the families more than quality does. The cards measure different things, so each row below rests only on properties the cards state.

Do not choose between a prompted and a fine-tuned system on ROUGE alone. In the prompted study, GPT-3 scored 24.71 ROUGE-L on CNN against 31.44 for BRIO, yet annotators preferred GPT-3 2209.12356 tab. 4 tab. 3.

Evaluating summaries

The cards use three kinds of score. Their numbers are author-reported and come from different settings, so compare them only within one study.

• Overlap and model-based text scores. ROUGE is the main score in the DialogSum card 2105.06762 tab. 5. The clinical study uses BLEU, ROUGE-L, BERTScore and MEDCON, a score over UMLS concept types 2309.07430 §4.1.3. The news study adds MoverScore, QAEval and several reference-free metrics 2209.12356 tab. 4 tab. 5. The CNN/DM and XSUM study correlates ROUGE-L, METEOR, BERTScore, BARTScore and BLEURT with human ratings at system level, using Kendall's tau. On CNN/DM, BLEURT reaches 0.81 for relevance, while ROUGE-L reaches 0.54 for faithfulness. On XSUM, all five metrics correlate negatively with faithfulness [arxiv:2301.13848 tab. 3]. The paper does not say how many systems each tau uses §3.3.
• Likelihood-based scores. BARTScore scores a text by the average log-likelihood a pre-trained BART assigns to it, in one of four directions, and needs no human labels 2106.11520 §3.2. On summarization it reaches a Spearman correlation of 0.448 on SummEval coherence, against 0.284 for BERTScore and 0.249 for PRISM, but ROUGE-1 is higher on REALSumm coverage (0.498 against 0.475) 2106.11520 tab. 4. In the authors' results, prompts did not help factuality, and paraphrase tuning lowered it on Rank19 2106.11520 §4.3.2, so use the vanilla or CNN variant for factuality unless you test otherwise. Check the card's Paper vs code section before reproducing its numbers.
• Energy per query. The energy study reports kWh and CO2eq per 1,000 inferences, with one input per call, on A100 GPUs in one AWS region 2311.16863 §3.2. Summarization averages 0.049 kWh per 1,000 inferences, against 0.002 for text classification 2311.16863 tab. 2. Its paper states a grid intensity of 297.6 g CO2eq/kWh §3.2, but the study's own Table 3 ratios come to about 139 g/kWh, so quote energy and recompute carbon for your grid tab. 3. Table 3 averages are not stated per task, so use the per-task figures for summarization claims.
• Factuality labels. FRANK asks crowd workers to label each summary sentence as factual or not, and to tag each error with one of seven types (plus "Other" and "Not an error") 2104.13346 §2 §3 tab. 1. Its correlations with those labels are low. Among common and factuality metrics, the best partial Pearson correlation is 0.27 (BERTScore P) and the best Spearman is 0.30 (FactCC), both on all data, with the summarizing system partialled out 2104.13346 tab. 2 §5.2. The released script partials out only that one column, not the dataset as the paper's text also says 2104.13346 §5.2. The authors report at least one error in 92% of XSum summaries and 43% of CNN/DM summaries 2104.13346 §4. The released data has 996 XSum records, not the 1,000 stated, so recount before reusing those shares 2104.13346 §3.
• Human judgement. In the news study, annotators' preferences did not follow the reference-based and reference-free metrics 2209.12356 §4.1 §4.2. In the other news study, zero-shot Davinci Instruct v2 and freelance writers were preferred about equally overall. The more abstractive summary, which here is the writers' (the model is the more extractive of the two), was judged more informative in 51.1% of pairs, which the authors do not treat as significant 2301.13848 §4.2. Krippendorff's alpha was 0.07 for overall preference, so one preference number hides large differences between annotators §4.2. In the clinical study, the correlation between NLP metrics and reader scores was weak, around 0.2 in magnitude 2309.07430 §5.4 fig. 9.

Implementation sketch

Common core of the extractive family. The review gives the steps, not code, so this is a generic outline of them.

Input: sentences s[1..n] from the source, budget k (sentences to keep)
1. represent each sentence (bag of words, n-grams or graph node)      # [arxiv:2301.08826 §2.4.1]
2. score each sentence for importance     # frequency, TextRank, LSA or learned salience
3. rank the sentences by score and keep the top k                     # [arxiv:2301.08826 §2.4.1]
4. return the kept sentences; their output order is not stated        # [arxiv:2301.08826 §2.4.1]

Abstractive variants replace step 4 with a generator that writes new sentences from the selected content, and may copy source words when needed 2301.08826 §2.4.1. Hybrid variants run steps 1 to 3 first, then pass the top sentences to that generator.

The prompted variant has no selection step. It is one call with the article and the length instruction:

# Prompted abstractive summary  [arxiv:2209.12356 §2.2]
# Input:  article (str), sentence_count (int)
# Output: summary (str)

def summarize_prompted(article, sentence_count):
    # Article first, then the length instruction  [arxiv:2209.12356 §2.2]
    # The paper shows the two lines, not the separator between them
    prompt = "Article: " + article + "\n\n"
    prompt += f"Summarize the above article in {sentence_count} sentences."
    # Decoding settings are not stated in the paper  [arxiv:2209.12356 app. A]
    return llm(prompt)

The retrieved-example variant changes only the prompt. Embed the input, take the nearest labelled training inputs, write each as an input and its target, and keep the query last 2309.07430 §3.2 §4.1.1.

The ranking variant adds one term to the fine-tuning loss. Its core is a pairwise margin over candidates sorted by ROUGE, best first. The score f of a candidate is its summed token log-probabilities divided by its length to the power alpha 2203.16804 eq. 9.

# Pairwise ranking term for one document  [arxiv:2203.16804 eq. 8]
# Input:  f: score of each candidate, best ROUGE first  [arxiv:2203.16804 eq. 9]
#         lam: margin per rank step
# Output: L_ctr

def ranking_loss(f, lam):
    loss = 0
    for i in range(len(f)):
        for j in range(i + 1, len(f)):
            # The lower-ranked candidate must score lower by a margin
            margin = (j - i) * lam
            loss += max(0, f[j] - f[i] + margin)
    return loss

The total loss is the cross-entropy on the reference plus gamma times this term 2203.16804 eq. 10. Training on the ranking term alone gives a scorer that cannot generate §3.

Contradictions

• Automatic metrics against human preference. In the prompted news study, ROUGE, BERTScore, MoverScore, QAEval and most reference-free metrics rank GPT-3 below the fine-tuned systems on CNN and DailyMail, while annotators prefer GPT-3 2209.12356 §4.1 §4.2 tab. 3 tab. 4 tab. 5. The exception on CNN is FactCC, where GPT-3 scores 0.2428 against 0.1817 for BRIO 2209.12356 tab. 5. The DialogSum card uses ROUGE as its main score and reports ROUGE-based gains for fine-tuned models 2105.06762 tab. 5. The clinical study finds NLP metrics only weakly correlated with reader scores 2309.07430 §5.4. BARTScore's authors report that it outperforms the top-scoring existing metrics in 16 of 22 test settings 2106.11520 §5. These papers test different systems, data and metrics, so this is a conflict about what the metrics mean, not a measured disagreement on one system. Both sides are stated; neither is resolved here.
• Factuality scores across benchmarks. BARTScore reports a Pearson correlation of 0.735 with factuality on QAGS-CNN and 0.184 on QAGS-XSUM, for BARTScore+CNN 2106.11520 tab. 5. Those QAGS rows are plain Pearson correlations. FRANK does not test BARTScore. On its own labels, the best correlations over all data are 0.27 and 0.30, and on XSum the best Pearson is 0.18 2104.13346 tab. 2. The benchmarks differ, and FRANK's figures are partial correlations controlling for the system. Both sides are cited; neither is resolved here.
• Reference-based metrics on faithfulness. The CNN/DM and XSUM study finds all five reference-based metrics it tests correlate positively with human faithfulness on CNN/DM (0.54 to 0.58) and negatively on XSUM (-0.08 to -0.27) 2301.13848 tab. 3. The authors say the XSUM references are low quality and bias such comparisons §3.2 §3.3. The paper does not state the number of systems behind each tau, so neither side can be checked for significance §3.3. Both sides are cited; neither is resolved here.
• DialogSum scoring setup. The paper averages ROUGE over three references per test dialogue 2105.06762 tab. 5 §2.3. The released baseline script scores against a single reference instead. The paper does not explain how that changes the numbers, so compare DialogSum results only within one setup.
• Scope of the review. The review has no experiments and compares no two families on the same data 2301.08826 §2.4.1. Its figures cannot be ranked against the other cards.
• BRIO's ROUGE-1 on CNN/DailyMail. The BRIO paper reports 47.78 for BRIO-Mul 2203.16804 tab. 2. The GPT-3 study reports 38.49 ROUGE-1 and 31.44 ROUGE-L for the public BRIO CNN/DM checkpoint, on its 100 sampled CNN articles 2209.12356 tab. 4 §3.1. Both sides are cited; neither is resolved here.

Papers

• NLP for education feedback analysis: a review (arXiv:2301.08826): reviews text summarization for student feedback, classifies it as extractive, abstractive or hybrid, and reports the cited studies' results.
• DialogSum: A Real-Life Scenario Dialogue Summarization Dataset (arXiv:2105.06762): a 13,460-dialogue spoken-dialogue dataset with fine-tuned BART-large, UniLMv2-base and Transformer baselines, and an error analysis.
• News Summarization and Evaluation in the Era of GPT-3 (arXiv:2209.12356): prompts GPT-3 for news summaries, compares them with fine-tuned systems in a human study, and tests automatic metrics against human preference.
• Benchmarking Large Language Models for News Summarization (arXiv:2301.13848): rates ten LLMs with human annotators on CNN/DM and XSUM, compares zero-shot models with freelance writers, and tests five reference-based metrics against human ratings.
• BARTScore: Evaluating Generated Text as Text Generation (arXiv:2106.11520): scores generated text by a pre-trained BART's log-likelihood, with four directions and optional prompts and fine-tuning, and reports its agreement with human judgements.
• Power Hungry Processing: Watts Driving the Cost of AI Deployment? (arXiv:2311.16863): measures energy and CO2eq per 1,000 inferences for 88 models on ten tasks, including summarization, with task-specific and zero-shot multi-purpose models compared.
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization (arXiv:2309.07430): adapts eight LLMs to four clinical summarization tasks with retrieved examples or QLoRA, and reports a ten-physician reader study.
• Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (arXiv:2104.13346): labels sentences of about 2,250 CNN/DM and XSum summaries from nine systems for factual errors, with an error typology, and tests common and factuality metrics against the labels.
• LongT5: Efficient Text-To-Text Transformer for Long Sequences (arXiv:2112.07916): extends the T5 encoder to inputs of up to 16k tokens with transient global attention, and reports summarization results on arXiv, PubMed, BigPatent, MediaSum, Multi-News and CNN/DM.
• BRIO: Bringing Order to Abstractive Summarization (arXiv:2203.16804): fine-tunes BART or PEGASUS with cross-entropy plus a pairwise ranking loss over 16 candidate summaries, with a generator and a scorer variant, and reports ROUGE on CNN/DailyMail, XSum and NYT.