--- kind: concept slug: text-summarization title: Text summarization summary: >- Shorten a text, such as news, spoken dialogue or clinical notes, into a summary that keeps the key content. Extractive methods copy sentences, abstractive methods write new ones, and hybrid methods do both. Fine-tuning needs labelled pairs; a prompted LLM needs none. Long inputs need a long-context model. Automatic scores can disagree with human ratings. tags: [nlp, summarization, education-feedback, clinical-nlp, evaluation] papers: ["2105.06762", "2106.11520", "2209.12356", "2301.08826", "2301.13848", "2309.07430", "2311.16863", "2104.13346", "2112.07916", "2203.16804"] related: [sentiment-analysis, feature-selection, dialogue-generation, dialogue-evaluation, in-context-learning, hallucination-detection, human-evaluation, instruction-tuning, likelihood-based-text-evaluation, low-rank-adaptation, llm-evaluation, llm-serving, dataset-curation, local-global-attention, language-model-pretraining, contrastive-learning, learning-to-rank, decoding-strategies] status: draft updated_at: 2026-11-01 --- ## Overview Text summarization turns a long text, or a collection of texts, into a shorter one that keeps the main content. Ten cards cover it from different angles: a review of NLP for student feedback, which sorts the methods into three families [arxiv:2301.08826 §2.4.1]; a dialogue dataset with fine-tuned baselines [arxiv:2105.06762 §3.1]; a study of prompted GPT-3 on news [arxiv:2209.12356 §2.2]; a human study of ten LLMs on news, which finds that instruction tuning matters more than model size [arxiv:2301.13848 §3.2]; a study that adapts eight LLMs to clinical notes [arxiv:2309.07430 §3.2]; BARTScore, a metric that scores generated text, summaries included, from a pre-trained model's likelihood [arxiv:2106.11520 §3.2]; BRIO, a fine-tuning method that adds a ranking loss over candidate summaries [arxiv:2203.16804 §3]; and a measurement of the energy each inference uses, which covers summarization models among ten tasks [arxiv:2311.16863 §3.1]. Two more cover the edges of the task: FRANK, a benchmark of sentence-level factuality labels for about 2,250 CNN/DM and XSum summaries [arxiv:2104.13346 §3], and LongT5, an encoder that handles inputs of up to 16k tokens [arxiv:2112.07916 §3.1]. The review motivates the task with student feedback: reading comments by hand is slow, and a summary is easier to digest than the raw corpus [arxiv:2301.08826 §1] [§2.4.1]. The shared idea is a summary that keeps "critical information" from the source [arxiv:2301.08826 §2.4.1]. The families differ in where the summary's sentences come from. Extractive methods only select sentences that already exist. Abstractive methods write new sentences, either fine-tuned on target data or prompted from a large instruction-following model. Hybrid methods select first and then paraphrase. **Evidence is uneven.** The review runs no experiments and compares no two families on the same data. The experimental cards each cover a different setting: English spoken dialogue (DialogSum), English news (CNN, BBC and keyword-focused CNN, and CNN/DM and XSUM with human ratings of ten LLMs), clinical notes (four tasks, with a reader study on three), factuality labels on CNN/DM and XSum (FRANK), candidate-ranking fine-tuning (BRIO, on CNN/DailyMail, XSum and NYT), and long documents (LongT5, on six summarization sets). BARTScore's experiments test metrics, not summarizers. The energy study measures cost, not quality, on SAMSum, CNN/DM and XSum inputs [arxiv:2311.16863 tab. 1]. Their numbers are not comparable with each other or with the review's figures. Treat the variants as a map of options, and check the choice on your own data before you commit to it. ## Variants ### Extractive The method picks a subset of the source sentences and ranks them by importance. It builds an intermediate representation with features such as bag of words, n-grams or graphs [arxiv:2301.08826 §2.4.1] [fig. 4]. Several extractive systems are named in the review, none of them with a card on wikillm: - **Statistical frequency weighting** (Madhuri et al.). Tokens are weighted by their frequency in the document, sentences are scored by summing those weights, and the top sentences are kept. The review reports higher accuracy against human-written summaries, with no value given [arxiv:2301.08826 §2.4.1]. - **Phrase clustering** (CourseMIRROR, Fan et al.). Phrases are extracted from student reflections, grouped with k-medoids clustering and re-ranked by how many students they cover. The review reports this as better than LexRank, with no value given [arxiv:2301.08826 §2.4.1]. - **Graph and latent-semantic summarizers** (TextRank and LSA, used by Gottipati et al. on forum posts). The review reports that the LSA summary gave more additional recommendations than TextRank, a qualitative comparison only [arxiv:2301.08826 §2.4.1]. - **Learned sentence selection** (Mutlu et al.). An LSTM scores each sentence's salience from semantic and other features. The review reports that it outperformed hierarchical Bi-GRU, CNN, Bi-GRU, SummaRuNNer and BanditSum baselines [arxiv:2301.08826 §2.4.1]. The review credits extractive methods with language independence, because they neither build nor paraphrase sentences [arxiv:2301.08826 §2.4.1]. On DialogSum, the lead baseline reaches ROUGE-1 27.52 and the ext-oracle baseline 37.90, both with n = 2 [arxiv:2105.06762 tab. 4]. The oracle chooses its sentences using the reference summaries, so it marks a ceiling for extracted output on that data, not a method anyone can run at inference time. ### Abstractive, fine-tuned The method is a sequence-to-sequence model trained on pairs of source and summary. The review's one abstractive result is from Song et al., an LSTM-CNN that switches between a generating mode and a copy mode. On ROUGE it reports 34.9 for ROUGE-1 (4.4 points above CNN-based systems) and 17.8 for ROUGE-2 (1.6 points above) [arxiv:2301.08826 §2.4.1]. The review does not name the dataset or the baselines' setup, so these numbers cannot be read against the extractive results above. DialogSum is a 13,460-dialogue English dataset of daily-life conversations with one human summary each, split 12,460 / 500 / 500 [arxiv:2105.06762 §2.1]. Each model is fine-tuned on that data alone. Author-reported, ROUGE-1 reaches 47.28 for BART-large, 47.04 for UniLMv2-base and 35.91 for a Transformer trained from scratch, on the test set with three references per dialogue [arxiv:2105.06762 tab. 5]. The paper says that spoken dialogue is harder than chat or news, because the relevant information is spread over many turns, with coreference and discourse relations that are not explicit [arxiv:2105.06762 §5.2]. Its error analysis on 50 summaries finds incorrect coreference in 94% of the Transformer's outputs and 60% of UniLMv2's [arxiv:2105.06762 tab. 11]. The paper says its sources are freely accessible for academic use [arxiv:2105.06762 §7], but its data repository declares a non-commercial licence. Check that before any commercial use of the data. The clinical study fine-tunes open models with QLoRA, which trains small low-rank adapters on a frozen 4-bit base model [arxiv:2309.07430 §3.2]. In that study FLAN-T5 (2.7B) beat FLAN-UL2 (20B) under QLoRA [arxiv:2309.07430 §5.1.2]. That is one result in one setup, not a general claim about model size. ### Abstractive, fine-tuned with candidate ranking The method fine-tunes a pre-trained generator with two losses: cross-entropy on the reference summary, and a pairwise margin loss. The margin loss makes the model's length-normalised log-likelihood order a set of candidate summaries the same way their ROUGE against the reference does [arxiv:2203.16804 §3] [eq. 8] [eq. 10]. The candidates come from an offline pass before training: 16 per training document, from diverse beam search [arxiv:2203.16804 app. B]. The paper calls this pass a non-trivial overhead [§5.5]. On CNN/DailyMail, one epoch takes about 20 hours on four RTX 3090 GPUs [app. B]. Card: [BRIO: Bringing Order to Abstractive Summarization](/papers/2203.16804). The generator variant, BRIO-Mul, keeps the cross-entropy term. Its author-reported ROUGE-1 is 47.78 on CNN/DailyMail, against 44.29 for the authors' own BART-large evaluation, and 49.07 on XSum, against 47.46 for PEGASUS [arxiv:2203.16804 tab. 2]. The contrastive-only variant, BRIO-Ctr, scores lower and cannot generate [§3] [tab. 2]. Ordering the candidates by BERTScore instead of ROUGE gave 47.53 ROUGE-1 against 47.78 [tab. 6]. The paper sets the contrastive weight to 100 on all three datasets [tab. 13]. By hand, not run, the BRIO card finds that the released XSum configuration weights the ranking term about 1 against cross-entropy, so check that setting before you reproduce the XSum result [arxiv:2203.16804 tab. 13]. ### Abstractive, long input The method is a fine-tuned encoder-decoder built for long inputs. Full self-attention grows quadratically with input length, which limits T5 to short inputs [arxiv:2112.07916 §2]. LongT5 replaces the encoder's attention with a local window of radius 127 plus one global token per block of 16 tokens, built on the fly in each layer, so the global tokens need no extra input slots [arxiv:2112.07916 §3.1.1] [§3.1.2]. Its pretraining hides whole important sentences and asks the model to write them back, instead of hiding random spans [arxiv:2112.07916 §3.2]. Card: [LongT5](/papers/2112.07916). Author-reported, LongT5 xl with 16k input reaches ROUGE-1 48.35, ROUGE-2 21.92 and ROUGE-L 44.27 on arXiv, above PRIMER's 47.6, 20.8 and 42.6 [arxiv:2112.07916 tab. 2]. It is not the best on every set. On Multi-News, PRIMER scores higher, and on CNN/DM, HAT-BART has higher ROUGE-1 and ROUGE-L (44.48 and 41.52, against 43.94 and 41.28 for LongT5 xl at 4k) [tab. 2]. Each dataset uses its own input length, so the rows are not controlled for length [tab. 2]. The cost is the main constraint. The paper's complexity for this attention is O(l × (r + l/k)) for input length l [arxiv:2112.07916 §3.1.2]. On a TPU slice, the longest input before running out of memory was 6k for T5.1.1 base, 12k for LongT5 base with global tokens and 36k for LongT5 base with local attention [arxiv:2112.07916 §5.1]. Local attention reaches longer inputs, but on NQ question answering it trails the global variant [arxiv:2112.07916 §5.2]. ### Abstractive, prompted LLM (zero-shot) The method sends the article and an instruction to a large instruction-tuned model, with no training on news data. The study uses GPT-3 text-davinci-002, a 175B model accessed through an API, with the prompt "Summarize the above article in N sentences" [arxiv:2209.12356 §1] [arxiv:2209.12356 §2.2]. The instruction was followed in 98% of the test instances [arxiv:2209.12356 §2.2] [fig. 3]. In a crowd study on 100 CNN and 100 BBC articles, with three annotators per article and majority vote, annotators picked the prompted summary as best for 58% of CNN articles and 57% of BBC articles. The fine-tuned BRIO baseline got 36% and 20% [arxiv:2209.12356 tab. 3]. The gap is significant at p = 2e-3 (CNN) and p = 6e-4 (BBC) [arxiv:2209.12356 §3.2]. The authors estimate about 2,600 summaries cost about USD 150 at the API price of the time [arxiv:2209.12356 app. D]. The same prompting works for keyword-focused summaries. Given a named entity from the article as the keyword, GPT-3 was preferred to CTRLSum by a majority of annotators for nearly 70% of article-keyword pairs [arxiv:2209.12356 §5.2] [fig. 7]. That study uses 100 CNN articles only [arxiv:2209.12356 app. A]. A second news study rates zero-shot LLMs with human annotators, using 100 validation articles each from CNN/DM and XSUM [arxiv:2301.13848 §3.1]. Among the ten models, the instruction-tuned ones scored highest without examples. On CNN/DM, Davinci Instruct v2 (175B) reached faithfulness 0.99 and coherence 4.15 on a 1-to-5 scale; GPT-3 175B without instruction tuning reached 0.76 and 2.65 [arxiv:2301.13848 tab. 2]. The 6.7B instruction-tuned Curie model came close to the 175B one on the same measures [tab. 2]. The paper infers the instruction-tuning flag for the GPT-3 models from their API names [§3.1]. Its CNN/DM prompt asks for three sentences and ends with "Summary:"; XSUM asks for one sentence. Sampling uses temperature 0.3, and the paper states no maximum length or stop rule [§3.1]. Do not put reference summaries in the examples. With the current references as five-shot demonstrations, Davinci Instruct v2's XSUM faithfulness fell from 0.97 to 0.77 [arxiv:2301.13848 §3.2] [tab. 2]. ### Prompted LLM with retrieved examples The method adds labelled examples to the prompt. For each test input, the system embeds the input, finds the nearest labelled training inputs and places them, each with its target summary, before the query [arxiv:2309.07430 §3.2] [§4.1.1]. The number of examples is chosen so that at most 1% of test inputs exceed the context length [arxiv:2309.07430 §3.2]. With GPT-4 and the largest number of examples the paper's setup allows [arxiv:2309.07430 §5.1.4], ten physicians compared the model's summaries with expert summaries in a blinded pairwise study. The study covers radiology reports, patient questions and progress notes; dialogue was excluded [arxiv:2309.07430 §4.2]. The model's summary was preferred in 36% of comparisons, rated equal in 45%, and the expert's was preferred in 19% [arxiv:2309.07430 §5.2] [fig. 7]. The ratings cover completeness, correctness and conciseness. Proprietary models could only use this method; open models were also run with it, and some were trained with QLoRA [arxiv:2309.07430 §3.2]. ### Hybrid The method selects the top-ranked sentences extractively, then paraphrases them into a summary [arxiv:2301.08826 §2.4.1] [fig. 6]. The review describes this family but gives no specific system and no number for it. ## When to use which Constraints separate the families more than quality does. The cards measure different things, so each row below rests only on properties the cards state. | Situation | Use | Why | |---|---|---| | The summary must reuse the source's exact sentences | Extractive | Output is a subset of the original sentences; the review credits this with language independence [arxiv:2301.08826 §2.4.1] | | Labelled summaries exist for your domain (thousands of pairs) and you can fine-tune | Fine-tuned abstractive | On DialogSum, BART-large beat both baselines on ROUGE, by 0.24 over UniLMv2 [arxiv:2105.06762 tab. 5]. Author-reported, and the 0.24 gap is small | | Labelled pairs exist, you can afford an offline candidate pass, and you need higher ROUGE from a fine-tuned generator | Fine-tuned with candidate ranking (BRIO-Mul) | Author-reported ROUGE-1 of 47.78 on CNN/DailyMail and 49.07 on XSum, against the authors' BART and PEGASUS baselines [arxiv:2203.16804 tab. 2]. The offline pass and two-loss training add cost [app. B] [§5.5]. The gains are ROUGE-based; see Contradictions | | Labelled examples exist and the data may go to a hosted API | Prompted LLM with retrieved examples | Best clinical setup: model preferred 36%, equal 45%, expert preferred 19%, with ten readers on three tasks [arxiv:2309.07430 §5.2]. That study used a proprietary model, and its examples were retrieved per input [§3.2] | | Labelled examples exist, but only open models are allowed | QLoRA on a small open model | FLAN-T5 (2.7B) beat FLAN-UL2 (20B) under QLoRA [arxiv:2309.07430 §5.1.2]. GPT-4 generally scored best overall [§5.1.4] | | No labelled data, news-style text, and a hosted LLM API is acceptable | Prompted LLM, zero-shot | Annotators preferred it to fine-tuned BRIO and T0 on CNN and BBC, in one study with one model [arxiv:2209.12356 tab. 3]. Costs and model versions change, and decoding settings are not stated [arxiv:2209.12356 app. A] | | Spoken dialogue, not news | Fine-tuned abstractive, with human checks on coreference and intent | Baselines on DialogSum made coreference and intent errors often [arxiv:2105.06762 tab. 9] [tab. 10] [tab. 11] | | Zero-shot, and you choose a model by quality, not size | Instruction-tuned LLM | Instruction tuning separated the zero-shot models more than parameter count did; the 6.7B instruction-tuned Curie model came close to the 175B Davinci Instruct v2 [arxiv:2301.13848 tab. 2] [§3.2]. Instruction-tuning flags for GPT-3 are inferred from API names [§3.1] | | Summaries will be checked against news reference summaries | Human ratings; treat the references as a weak baseline | Existing CNN/DM and XSUM references scored 0.84 and 0.37 on faithfulness, and the authors call them extremely low quality [arxiv:2301.13848 tab. 2] [§3.2] | | Zero-shot news summaries must read like writer output and copying is a concern | Zero-shot instruction-tuned LLM, with a copy-rate check | Judged equal to freelance writers overall, but copied far more: coverage 0.92 and density 12.1, against 0.81 and 2.07 for writers [arxiv:2301.13848 §4.2]. Annotator agreement was low [§4.2] | | You want fluent prose but still anchored to source sentences | Hybrid | Selects, then paraphrases [arxiv:2301.08826 §2.4.1]. No reported numbers, so untested in the cards | | Many short forum or course comments, with a topic-based view wanted | Extractive, topic-based (TextRank or LSA) | Topic summaries from forum posts; LSA gave more extra recommendations than TextRank [arxiv:2301.08826 §2.4.1] | | Clinical or other safety-critical use | Not supported by these results alone | The reader study had ten physicians, three tasks, no dialogue, and progress notes from one ICU centre [arxiv:2309.07430 §4.2] [§5.5]. The safety analysis covers 27 radiology and 44 progress-note pairs [§4.3] | | You must trust the output's accuracy | Human evaluation, not ROUGE alone | In the prompted study, reference-based and factuality metrics did not follow human preference [arxiv:2209.12356 §4.1] [§4.2] | | Inputs of several thousand tokens, such as reports, papers or many documents | Long-input fine-tuned model (LongT5 with global tokens), tested at your input lengths | Reported gains on arXiv and PubMed at 16k input [arxiv:2112.07916 tab. 2]. On Multi-News and CNN/DM the gains are smaller or absent [tab. 2]. The global-token cost grows with input length [arxiv:2112.07916 §3.1.2] | | Checking whether a summary's sentences are supported by the source | Sentence-level human labels with an error typology, not one automatic score | Best correlations with human factuality on FRANK were 0.27 (BERTScore P, Pearson) and 0.30 (FactCC, Spearman), on all data [arxiv:2104.13346 tab. 2]. Labels show which error types each system makes [tab. 3] | | Summaries at high query volume, where energy per query counts | Task-specific summarizer, if its quality on your data is adequate | Summarization by task-specific models cost 4–10 g CO2eq per 1,000 inferences, against 20–30 g for multi-purpose zero-shot models [arxiv:2311.16863 §4.2.1] [fig. 3]. The energy study does not match quality across these two families [§4.2] | Do not choose between a prompted and a fine-tuned system on ROUGE alone. In the prompted study, GPT-3 scored 24.71 ROUGE-L on CNN against 31.44 for BRIO, yet annotators preferred GPT-3 [arxiv:2209.12356 tab. 4] [tab. 3]. ## Evaluating summaries The cards use three kinds of score. Their numbers are author-reported and come from different settings, so compare them only within one study. - **Overlap and model-based text scores.** ROUGE is the main score in the DialogSum card [arxiv:2105.06762 tab. 5]. The clinical study uses BLEU, ROUGE-L, BERTScore and MEDCON, a score over UMLS concept types [arxiv:2309.07430 §4.1.3]. The news study adds MoverScore, QAEval and several reference-free metrics [arxiv:2209.12356 tab. 4] [tab. 5]. The CNN/DM and XSUM study correlates ROUGE-L, METEOR, BERTScore, BARTScore and BLEURT with human ratings at system level, using Kendall's tau. On CNN/DM, BLEURT reaches 0.81 for relevance, while ROUGE-L reaches 0.54 for faithfulness. On XSUM, all five metrics correlate negatively with faithfulness [arxiv:2301.13848 tab. 3]. The paper does not say how many systems each tau uses [§3.3]. - **Likelihood-based scores.** BARTScore scores a text by the average log-likelihood a pre-trained BART assigns to it, in one of four directions, and needs no human labels [arxiv:2106.11520 §3.2]. On summarization it reaches a Spearman correlation of 0.448 on SummEval coherence, against 0.284 for BERTScore and 0.249 for PRISM, but ROUGE-1 is higher on REALSumm coverage (0.498 against 0.475) [arxiv:2106.11520 tab. 4]. In the authors' results, prompts did not help factuality, and paraphrase tuning lowered it on Rank19 [arxiv:2106.11520 §4.3.2], so use the vanilla or CNN variant for factuality unless you test otherwise. Check the card's Paper vs code section before reproducing its numbers. - **Energy per query.** The energy study reports kWh and CO2eq per 1,000 inferences, with one input per call, on A100 GPUs in one AWS region [arxiv:2311.16863 §3.2]. Summarization averages 0.049 kWh per 1,000 inferences, against 0.002 for text classification [arxiv:2311.16863 tab. 2]. Its paper states a grid intensity of 297.6 g CO2eq/kWh [§3.2], but the study's own Table 3 ratios come to about 139 g/kWh, so quote energy and recompute carbon for your grid [tab. 3]. Table 3 averages are not stated per task, so use the per-task figures for summarization claims. - **Factuality labels.** FRANK asks crowd workers to label each summary sentence as factual or not, and to tag each error with one of seven types (plus "Other" and "Not an error") [arxiv:2104.13346 §2] [§3] [tab. 1]. Its correlations with those labels are low. Among common and factuality metrics, the best partial Pearson correlation is 0.27 (BERTScore P) and the best Spearman is 0.30 (FactCC), both on all data, with the summarizing system partialled out [arxiv:2104.13346 tab. 2] [§5.2]. The released script partials out only that one column, not the dataset as the paper's text also says [arxiv:2104.13346 §5.2]. The authors report at least one error in 92% of XSum summaries and 43% of CNN/DM summaries [arxiv:2104.13346 §4]. The released data has 996 XSum records, not the 1,000 stated, so recount before reusing those shares [arxiv:2104.13346 §3]. - **Human judgement.** In the news study, annotators' preferences did not follow the reference-based and reference-free metrics [arxiv:2209.12356 §4.1] [§4.2]. In the other news study, zero-shot Davinci Instruct v2 and freelance writers were preferred about equally overall. The more abstractive summary, which here is the writers' (the model is the more extractive of the two), was judged more informative in 51.1% of pairs, which the authors do not treat as significant [arxiv:2301.13848 §4.2]. Krippendorff's alpha was 0.07 for overall preference, so one preference number hides large differences between annotators [§4.2]. In the clinical study, the correlation between NLP metrics and reader scores was weak, around 0.2 in magnitude [arxiv:2309.07430 §5.4] [fig. 9]. ## Implementation sketch Common core of the extractive family. The review gives the steps, not code, so this is a generic outline of them. ```text Input: sentences s[1..n] from the source, budget k (sentences to keep) 1. represent each sentence (bag of words, n-grams or graph node) # [arxiv:2301.08826 §2.4.1] 2. score each sentence for importance # frequency, TextRank, LSA or learned salience 3. rank the sentences by score and keep the top k # [arxiv:2301.08826 §2.4.1] 4. return the kept sentences; their output order is not stated # [arxiv:2301.08826 §2.4.1] ``` Abstractive variants replace step 4 with a generator that writes new sentences from the selected content, and may copy source words when needed [arxiv:2301.08826 §2.4.1]. Hybrid variants run steps 1 to 3 first, then pass the top sentences to that generator. The prompted variant has no selection step. It is one call with the article and the length instruction: ```pseudocode # Prompted abstractive summary [arxiv:2209.12356 §2.2] # Input: article (str), sentence_count (int) # Output: summary (str) def summarize_prompted(article, sentence_count): # Article first, then the length instruction [arxiv:2209.12356 §2.2] # The paper shows the two lines, not the separator between them prompt = "Article: " + article + "\n\n" prompt += f"Summarize the above article in {sentence_count} sentences." # Decoding settings are not stated in the paper [arxiv:2209.12356 app. A] return llm(prompt) ``` The retrieved-example variant changes only the prompt. Embed the input, take the nearest labelled training inputs, write each as an input and its target, and keep the query last [arxiv:2309.07430 §3.2] [§4.1.1]. The ranking variant adds one term to the fine-tuning loss. Its core is a pairwise margin over candidates sorted by ROUGE, best first. The score f of a candidate is its summed token log-probabilities divided by its length to the power alpha [arxiv:2203.16804 eq. 9]. ```pseudocode # Pairwise ranking term for one document [arxiv:2203.16804 eq. 8] # Input: f: score of each candidate, best ROUGE first [arxiv:2203.16804 eq. 9] # lam: margin per rank step # Output: L_ctr def ranking_loss(f, lam): loss = 0 for i in range(len(f)): for j in range(i + 1, len(f)): # The lower-ranked candidate must score lower by a margin margin = (j - i) * lam loss += max(0, f[j] - f[i] + margin) return loss ``` The total loss is the cross-entropy on the reference plus gamma times this term [arxiv:2203.16804 eq. 10]. Training on the ranking term alone gives a scorer that cannot generate [§3]. ## Contradictions - **Automatic metrics against human preference.** In the prompted news study, ROUGE, BERTScore, MoverScore, QAEval and most reference-free metrics rank GPT-3 below the fine-tuned systems on CNN and DailyMail, while annotators prefer GPT-3 [arxiv:2209.12356 §4.1] [§4.2] [tab. 3] [tab. 4] [tab. 5]. The exception on CNN is FactCC, where GPT-3 scores 0.2428 against 0.1817 for BRIO [arxiv:2209.12356 tab. 5]. The DialogSum card uses ROUGE as its main score and reports ROUGE-based gains for fine-tuned models [arxiv:2105.06762 tab. 5]. The clinical study finds NLP metrics only weakly correlated with reader scores [arxiv:2309.07430 §5.4]. BARTScore's authors report that it outperforms the top-scoring existing metrics in 16 of 22 test settings [arxiv:2106.11520 §5]. These papers test different systems, data and metrics, so this is a conflict about what the metrics mean, not a measured disagreement on one system. Both sides are stated; neither is resolved here. - **Factuality scores across benchmarks.** BARTScore reports a Pearson correlation of 0.735 with factuality on QAGS-CNN and 0.184 on QAGS-XSUM, for BARTScore+CNN [arxiv:2106.11520 tab. 5]. Those QAGS rows are plain Pearson correlations. FRANK does not test BARTScore. On its own labels, the best correlations over all data are 0.27 and 0.30, and on XSum the best Pearson is 0.18 [arxiv:2104.13346 tab. 2]. The benchmarks differ, and FRANK's figures are partial correlations controlling for the system. Both sides are cited; neither is resolved here. - **Reference-based metrics on faithfulness.** The CNN/DM and XSUM study finds all five reference-based metrics it tests correlate positively with human faithfulness on CNN/DM (0.54 to 0.58) and negatively on XSUM (-0.08 to -0.27) [arxiv:2301.13848 tab. 3]. The authors say the XSUM references are low quality and bias such comparisons [§3.2] [§3.3]. The paper does not state the number of systems behind each tau, so neither side can be checked for significance [§3.3]. Both sides are cited; neither is resolved here. - **DialogSum scoring setup.** The paper averages ROUGE over three references per test dialogue [arxiv:2105.06762 tab. 5] [§2.3]. The released baseline script scores against a single reference instead. The paper does not explain how that changes the numbers, so compare DialogSum results only within one setup. - **Scope of the review.** The review has no experiments and compares no two families on the same data [arxiv:2301.08826 §2.4.1]. Its figures cannot be ranked against the other cards. - **BRIO's ROUGE-1 on CNN/DailyMail.** The BRIO paper reports 47.78 for BRIO-Mul [arxiv:2203.16804 tab. 2]. The GPT-3 study reports 38.49 ROUGE-1 and 31.44 ROUGE-L for the public BRIO CNN/DM checkpoint, on its 100 sampled CNN articles [arxiv:2209.12356 tab. 4] [§3.1]. Both sides are cited; neither is resolved here. ## Papers - [NLP for education feedback analysis: a review (arXiv:2301.08826)](/papers/2301.08826): reviews text summarization for student feedback, classifies it as extractive, abstractive or hybrid, and reports the cited studies' results. - [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset (arXiv:2105.06762)](/papers/2105.06762): a 13,460-dialogue spoken-dialogue dataset with fine-tuned BART-large, UniLMv2-base and Transformer baselines, and an error analysis. - [News Summarization and Evaluation in the Era of GPT-3 (arXiv:2209.12356)](/papers/2209.12356): prompts GPT-3 for news summaries, compares them with fine-tuned systems in a human study, and tests automatic metrics against human preference. - [Benchmarking Large Language Models for News Summarization (arXiv:2301.13848)](/papers/2301.13848): rates ten LLMs with human annotators on CNN/DM and XSUM, compares zero-shot models with freelance writers, and tests five reference-based metrics against human ratings. - [BARTScore: Evaluating Generated Text as Text Generation (arXiv:2106.11520)](/papers/2106.11520): scores generated text by a pre-trained BART's log-likelihood, with four directions and optional prompts and fine-tuning, and reports its agreement with human judgements. - [Power Hungry Processing: Watts Driving the Cost of AI Deployment? (arXiv:2311.16863)](/papers/2311.16863): measures energy and CO2eq per 1,000 inferences for 88 models on ten tasks, including summarization, with task-specific and zero-shot multi-purpose models compared. - [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization (arXiv:2309.07430)](/papers/2309.07430): adapts eight LLMs to four clinical summarization tasks with retrieved examples or QLoRA, and reports a ten-physician reader study. - [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (arXiv:2104.13346)](/papers/2104.13346): labels sentences of about 2,250 CNN/DM and XSum summaries from nine systems for factual errors, with an error typology, and tests common and factuality metrics against the labels. - [LongT5: Efficient Text-To-Text Transformer for Long Sequences (arXiv:2112.07916)](/papers/2112.07916): extends the T5 encoder to inputs of up to 16k tokens with transient global attention, and reports summarization results on arXiv, PubMed, BigPatent, MediaSum, Multi-News and CNN/DM. - [BRIO: Bringing Order to Abstractive Summarization (arXiv:2203.16804)](/papers/2203.16804): fine-tunes BART or PEGASUS with cross-entropy plus a pairwise ranking loss over 16 candidate summaries, with a generator and a scorer variant, and reports ROUGE on CNN/DailyMail, XSum and NYT.