TLDR: Extreme Summarization of Scientific Documents

In plain words

Abstracts are long and formal. This paper asks for something much shorter: a TLDR, a single sentence that says what a research paper is about and why it matters. The authors built SciTLDR, a dataset of about 5,400 such summaries for about 3,200 computer science papers. The papers in the test and validation splits have several summaries each. Some were written by the paper's authors. Others were written by annotators who rewrote the summary paragraph from a peer review of the paper.

The method is called CATTS. It starts from BART, a language model that already knows how to summarize, and also trains it to write paper titles. Each training example carries a short tag that tells the model which job to do. Every paper has a title, so this adds training signal without any extra annotation. On the authors' tests the result writes slightly better one-sentence summaries than plain BART. The gains are small, and the numbers are the authors' own.

TL;DR

• Defines TLDR generation: one-sentence, highly compressed summaries of a whole scientific paper §1.
• SciTLDR: 5,411 summaries for 3,229 CS papers, with author-written and peer-review-derived summaries; dev and test papers have several summaries each §1 §2.
• CATTS: BART trained on summaries plus paper-to-title pairs, separated by control codes; the same hyperparameters as the baseline §4 app. D.
• Author-reported: CATTS beats BART on ROUGE-1 in both input settings, by +0.5 (abstract-only) and +2.0 (AIC); only the AIC difference is marked significant tab. 3.
• Code (Apache-2.0) releases data preprocessing, generation and evaluation, but not the fine-tuning or title-data scripts (see Paper vs code).

Problem

Existing scientific summarization produces abstract-length summaries or complements abstracts §1. An extreme single-sentence summary of a whole paper requires heavy compression, domain knowledge and faithfulness §1. Existing datasets give one gold summary per document, and human summaries vary enough that single-reference ROUGE can misjudge system quality §2. Expert summaries are expensive to collect for scientific papers, and the amount of labelled data is small §4. The source papers are also long: the AIC input averages 993 words §5.2.

Method

Dataset (SciTLDR)

1. Author summaries: collect OpenReview submissions that have an author-written TLDR, plus the PDF, converted to full text with the S2ORC pipeline. Papers without a PDF (under 5%) are kept, with the abstract as input §2.
2. Peer-review summaries: annotators (28 recruited, 20 kept after a first hour of training) read the title and the first 128 words of a sampled review, and rewrite its summary if one exists, in 15 to 25 words and in one sentence. Annotators keep the reviewer's wording where possible. They enter "None" when the review is only opinion, when its summary is heavily negative, or when the paper is outside their expertise §2 app. F.
3. Splits: the papers are randomly split into train, dev and test §2. Train papers have one gold summary; dev and test papers also have peer-review summaries §2.

Input space

• Abstract-only (AO): the abstract, mean 159 words §5.2.
• AIC: abstract, introduction and conclusion, mean 993 words §5.2. The paper does not state how AIC is extracted from full text beyond naming the sections.

CATTS training

1. Build a title scaffold: about 20K paper-title pairs from arXiv computer science papers that have identified introduction and conclusion sections §5.3 [fn. in §5.3].
2. Append a control code to each source: the TLDR code for summaries and a TITLE code for titles §4. In the released code the code is a separate final list item of the source build_ctrl_datasets.py.
3. Up-sample the TLDR instances to the size of the title data §5.3.
4. Shuffle the two sources together and train BART-large, or BART-large fine-tuned on XSUM (BART_XSUM), on the shuffled data §4 §5.3.
5. Use the same hyperparameters as the baseline BART, with no separate tuning for CATTS. Step counts differ from BART (see Hyperparameters) app. D.

# Add a control code to a source document  [§4]
# Input:  source (list of sentences), code (string)
# Output: source with the code as one more item

def add_control_code(source, code):
    return source + [code]

# Build the shuffled training mix  [§4] [§5.3]
# Input:  tldr_pairs (paper, TLDR), title_pairs (paper, title)
# Output: shuffled list of (source, target)

def build_training_mix(tldr_pairs, title_pairs):
    # Tag each task with its control code  [§4]
    tldr_items = [
        (add_control_code(src, "<|TLDR|>"), tgt)
        for src, tgt in tldr_pairs
    ]
    title_items = [
        (add_control_code(src, "<|TITLE|>"), tgt)
        for src, tgt in title_pairs
    ]
    # Up-sample TLDR items to match the title scaffold size  [§5.3]
    tldr_items = upsample_to_size(tldr_items, len(title_items))
    # Shuffle both sources together  [§4]
    return shuffle(tldr_items + title_items)

Generation

1. At inference, append the TLDR control code to the paper source §4 generate.py.
2. Decode with beam search. The beam size and length penalty are tuned per model by grid search §5.3. The released defaults and the per-model values are in decoder_params.md.

# Generate one TLDR  [§4] [§5.3] [code: scripts/generate.py#L31-L35]
# Input:  paper_source (list of sentences), beam_size, length_penalty
# Output: TLDR string

def generate_tldr(paper_source, beam_size, length_penalty):
    source = add_control_code(paper_source, "<|TLDR|>")
    return beam_search(
        model,
        source,
        beam=beam_size,
        lenpen=length_penalty,
        max_len_b=30,            # code default, not stated in paper
        min_len=5,               # code default, not stated in paper
        no_repeat_ngram_size=3,  # code default, not stated in paper
    )

Evaluation

1. Automated: ROUGE-1, ROUGE-2 and ROUGE-L. Each prediction is scored against every gold summary for its paper, and the maximum is taken as the paper's score §5.4. The paper also reports means in app. E.
2. Informativeness (human): the nugget analysis scores each summary by the number of unique nuggets divided by its token count. The systems are ranked per paper, and the rankings are aggregated by mean reciprocal rank §5.4.
3. Correctness (human): the original authors of arXiv papers rate each system summary as 1 (false or misleading), 2 (partially accurate) or 3 (mostly correct) §5.4.

# Multi-target ROUGE for one paper  [§5.4]
# Input:  prediction (string), gold_tldrs (list of strings: author + PR)
# Output: ROUGE-1, ROUGE-2, ROUGE-L of the best-matching gold summary

def score_paper(prediction, gold_tldrs):
    # Score the prediction against each gold summary  [§5.4]
    scores = [rouge(prediction, gold) for gold in gold_tldrs]
    # Keep the gold with the top ROUGE-1  [code: scripts/cal-rouge.py#L76-L78]
    return max(scores, key=lambda s: s.rouge1_f)

Note: the pseudocode follows the released evaluation code; the paper text describes a maximum over gold summaries without naming the selection key (see Pitfalls).

Hyperparameters

Results

Author-reported. Setup: SciTLDR test set (618 papers). Baselines are the extractive methods PACSUM (unsupervised), BERTSumExt and MatchSum, and the abstractive methods BART and BART_XSUM §5.1. MatchSum starts from a released CNN/DM checkpoint app. D. Scores are the maximum ROUGE over each paper's gold summaries §5.4 tab. 3.

• Abstractive gains over the matching BART baseline, R1: CATTS +0.5 (AO), +2.0 (AIC); CATTS_XSUM +1.8 (AO), +0.9 (AIC) §6.1. Significance is a paired two-sided t-test with Holm-Bonferroni correction §6.1.
• Mean ROUGE (average over gold summaries) is lower for every abstractive model; for example CATTS AIC is 31.9 / 11.8 / 25.6 and BART AIC is 30.7 / 10.6 / 24.4 app. E.
• Full-text oracle (sentence-level extraction): 54.5 / 30.6 / 45.0 §5.1.

Human informativeness, mean reciprocal rank (MRR) tab. 4. Higher is better. The system rows use AIC inputs on 51 sampled papers §6.2; the paper set for the gold rows is not stated separately (see Pitfalls):

• Human informativeness: CATTS_XSUM is more informative than BART_XSUM and comparable to gold author summaries, but less informative than peer-review summaries §6.2.
• Human correctness: no difference between BART_XSUM and CATTS_XSUM. There were 42 ties, 10 cases where BART_XSUM was more correct and 12 where CATTS_XSUM was. Both average 2.5 §6.2.
• Novelty (share of summary words not in the source): BART 2.9% (AO) and 1.3% (AIC); CATTS 5.5% (AO) and 5.3% (AIC); CATTS_XSUM 5.8% (AO) and 4.5% (AIC) tab. 5.
• Oracle-input experiment (sentences chosen from the full text): CATTS_XSUM reaches 45.7 R1, 1.1 above its best standard result tab. 6.
• Dataset properties: 238.1 compression ratio, 15.2% novel words, multi-target tab. 1. Author summaries are 18.9 words on average and peer-review summaries 22.9 §3.1. Peer-review summaries have 20.2% novel words and author summaries 9.6% §3.1.

Paper vs code

Official code: allenai/scitldr at code_commit, Apache-2.0. The paper links it in the abstract. The repository contains data preprocessing, generation and evaluation scripts. It does not contain a fine-tuning script or the title-data pipeline.

• Training is not released. The README describes preprocessing, generation and evaluation only README.md. The learning rate, batch size, steps and warm-up are therefore paper-only §5.3 app. D.
• Title scaffold is not in the code. build_ctrl_datasets.py adds only the <|TLDR|> code to the source build_ctrl_datasets.py. The arXiv title pairs, the <|TITLE|> code, the shuffling and the up-sampling are absent. The paper describes them §4 §5.3.
• Decoder defaults differ from the paper. generate.py defaults to beam 6, length penalty 1.0, max_len_b 30, min_len 5 and no_repeat_ngram_size 3 generate.py. The paper states the grid (beam 2 to 8, length penalty 0.2 to 1.0) and gives no caps §5.3. The per-model settings in decoder_params.md fall inside that grid.
• Multi-target scoring picks the gold by ROUGE-1 only. The code takes the gold with the highest ROUGE-1 F-score and reports its ROUGE-2 and ROUGE-L cal-rouge.py cal-rouge.py. The paper says the maximum over gold summaries is used, without naming the key §5.4. If Table 3 reports per-metric maxima, the R2 and RL values may not match the code.
• Mean ROUGE is also computed cal-rouge.py, matching app. E.
• Data splits match. The released AO files hold 1,992 train, 619 dev and 618 test lines, equal to §2. The "val" naming in the preprocessing maps dev to valid to_stories.py.
• Dependencies are not pinned in the README. requirements.txt pins torch 1.4.0 and fairseq 0.9.0 requirements.txt requirements.txt. The README says the fairseq version is "TODO" README.md. The ROUGE tool is files2rouge, not pinned either README.md.
• Half precision at inference. On CUDA, generation calls bart.half() generate.py. The paper does not mention precision.
• Weights are linked, not used. The README links eight checkpoints README.md. They were not downloaded or checked, and no weight license is stated, so this card does not depend on them.
• No separate data license is stated in the repository. The Apache-2.0 LICENSE is the only license file LICENSE.

No discrepancies were found in the released split sizes.

Pitfalls

• Multi-target ROUGE is not comparable to single-reference ROUGE. Taking the maximum over several gold summaries raises scores. Train papers have only one gold summary, so training never sees the multi-target setting §2.
• Max-over-golds selection key. See Paper vs code. A reimplementation should use the released selection rule and report it.
• Extractive AIC results are unexplained. The paper suspects that longer inputs make it harder to learn new position embeddings from little data, but does not test this §6.1. BertSumExt used a truncation length of 1500 for AIC after testing three values app. D.
• Title-data construction is underspecified. The paper says about 20K pairs are sampled from arXiv papers with S2ORC-identified sections. It does not say how they are sampled, how the ratio of TLDR to title items is set, or whether SciTLDR test papers are excluded §5.3 §4. Up-sampling multiplies the one gold summary per training paper, and the repetition count is not stated.
• Step counts are not comparable across settings. BART uses 500 steps; CATTS uses 11,000 (AO) and 45,000 (AIC). The appendix says both are "5 epochs", but the sentence follows the AIC figure app. D.
• Input length limits are not stated. The paper does not say how BART handles an AIC input longer than its position limit, or how the AIC sections are cut from a full-text PDF §5.2.
• Generation length caps can truncate. The code caps output at 30 tokens, with a minimum of 5 generate.py. Summaries of 15 to 25 words may hit the cap. This was not checked against the released outputs.
• Human evaluation counts are inconsistent. §5.4 says informativeness uses "the same 76 gold papers" as the nugget analysis, but §6.2 uses 51 sampled papers for the system comparison. Correctness uses 29 authors and 64 arXiv papers. The paper does not reconcile the counts §5.4 §6.2.
• Significance is per pair, not per table. The dagger marks CATTS vs its own BART baseline. It does not mark differences between CATTS and CATTS_XSUM tab. 3.
• Gains are small and uneven. ROUGE-1 gains range from +0.5 to +2.0, and the human correctness result shows no difference §6.1 §6.2.
• Seed variance is not reported. The paper tests three seeds for BART but does not report their spread or the one used app. D.
• Evaluation tool details. Stemming and tokenisation are not stated. The code delegates scoring to files2rouge and calls it with ignore_empty=True cal-rouge.py.
• Peer-review summaries are not author summaries. They are rewrites of reviewer summaries, not of the paper, and they are more abstractive than author summaries (20.2% vs 9.6% novel words) §3.1.

Lineage

• Builds on: BART (Lewis et al., 2019), with the fairseq implementation and released XSUM weights §5.3; title scaffolds from Swayamdipta et al. (2018) and Cohan et al. (2019) §4; control codes from CTRL (Keskar et al., 2019) §4. arXiv ids are not given in the source and are not linked.
• Compared against: PACSUM, BertSumExt, MatchSum (extractive); BART and BART_XSUM (abstractive) §5.1.
• Superseded by: not stated in the sources used for this card.