Chunked long-input encoding

Overview

A pretrained encoder such as BERT accepts a bounded number of tokens, and its self-attention cost grows quadratically with length 1910.10781 §3.1. Chunked long-input encoding works around the limit without changing the encoder. The input is cut into segments that fit the encoder, each segment is encoded separately, and a second, small model combines the segment vectors into one document decision 1910.10781 §3.2 1910.10781 §3.3.

The shared invariant is that the encoder only ever sees one segment at a time, so its per-call cost is fixed by the segment length, not by the document length. Everything document-level is the job of the aggregator. Only one paper in this wiki covers the approach so far, so the variants below are both from the same source. Results are author-reported on three text tasks 1910.10781 §4.

Variants

Recurrent aggregator (RoBERT)

Each segment is encoded by a fine-tuned BERT-Base. The segment vectors form a sequence that a 100-unit LSTM reads. Its output feeds a 30-unit ReLU layer and a softmax over classes 1910.10781 §3.2. Card: Hierarchical Transformers for Long Document Classification. Reported accuracy: 83.65 on CSAT, 84.71 on 20 newsgroups, 82.28 on Fisher, using the segment representation H 1910.10781 tab. 3.

Transformer aggregator (ToBERT)

The LSTM is replaced by a 2-layer Transformer over the segment vectors. The authors also test learned position embeddings on those vectors, which limits the model to sequence lengths seen in training 1910.10781 §3.3. Reported accuracy with the same segment representation: 83.48 on CSAT, 85.52 on 20 newsgroups, 95.48 on Fisher 1910.10781 tab. 3. Position embeddings helped on CSAT (83.48 vs 82.84) and made almost no difference on 20 newsgroups and Fisher 1910.10781 tab. 5. Card: Hierarchical Transformers for Long Document Classification.

Prediction-level aggregation

No second neural model. Each segment gets a class posterior P from the fine-tuned BERT. The document label comes from averaging the posteriors, from the most frequent predicted segment class, or from a classifier trained on the posteriors 1910.10781 §5 1910.10781 tab. 4. With P, averaging reaches 82.84 on CSAT and 88.48 on Fisher; the learned heads reach 83.54 and 91.18 (RoBERT) or 81.48 and 94.16 (ToBERT) 1910.10781 tab. 4. Card: Hierarchical Transformers for Long Document Classification.

When to use which

The source paper tests three text classification tasks with long inputs. The statements below apply to those tasks and nothing wider.

Cost: every segment is a full encoder pass, so the number of passes grows with document length. With the stated 200-token segments and 50-token shift, each pair of consecutive segments shares 150 tokens, which is a derived reading and not a figure the paper reports 1910.10781 §4.5.

Implementation sketch

Language-agnostic outline of the common core. The variants differ only in the last two steps. Values are those the paper reports for its experiments.

# Chunked encoding of one document  [arxiv:1910.10781 §3.2] [§3.3]
# Input:  tokens: subword tokens of one document  (N,)
# Output: class probabilities  (C,)

def classify_document(tokens, aggregator):
    # Cut into segments of 200 tokens with a shift of 50  [§4.5]
    # Trailing segment and padding are not specified by the paper
    segments = cut_with_shift(tokens, segment_len=200, shift=50)
    # Encode each segment with the fine-tuned BERT-Base; keep pooled output  [§3.1]
    segment_vectors = [bert_pooled(s) for s in segments]
    if aggregator == "lstm":
        # RoBERT: 100-unit LSTM; the paper does not name which step is output  [§3.2]
        doc = lstm_100(segment_vectors)
    elif aggregator == "transformer":
        # ToBERT: 2 Transformer layers; how the output becomes one vector is not stated  [§3.3]
        doc = transformer_2_layers(segment_vectors)
    else:
        # Prediction-level: average (or vote over) per-segment posteriors  [§5]
        posteriors = [bert_posterior(s) for s in segments]
        return mean(posteriors)
    # Dense ReLU layer of 30 units, then softmax over classes  [§3.2]
    hidden = relu(dense_30(doc))
    return softmax(dense_classes(hidden))

Training, as reported: BERT is fine-tuned on segment-level labels, each segment inheriting its document's label 1910.10781 §3.1. The RoBERT head uses Adam with learning rate 0.001 and a decay by a factor of 0.95 after three epochs without validation-loss improvement; ToBERT uses learning rate 5e-5 1910.10781 §4.5. Batch size, epoch count and dropout are not stated.

Contradictions

• Fisher gain between heads. The text gives ToBERT's margin over RoBERT on Fisher as 13.63 points 1910.10781 §5. Table 3 gives 95.48 − 82.28 = 13.20 points 1910.10781 tab. 3. This page uses the table values.
• "All tasks" claim. The conclusion says ToBERT beats RoBERT on all tasks 1910.10781 §6. On CSAT, RoBERT is higher with H (83.65 vs 83.48) 1910.10781 tab. 3 and with P (83.54 vs 81.48) 1910.10781 tab. 4. The results text calls ToBERT "slightly worse" on CSAT 1910.10781 §5.
• Metric name on CSAT. The text calls a CSAT gain an improvement in F1 1910.10781 §5, while the paper states all results are accuracy 1910.10781 §4.5. The table difference (83.48 − 82.84 = 0.64) is an accuracy difference 1910.10781 tab. 5.
• 20 newsgroups baseline. The MS-CNN baseline scores 86.12 against ToBERT's 85.52 on 20 newsgroups 1910.10781 tab. 6, so the approach does not beat every baseline on every task.
• Position embeddings on Fisher. The table bolds the "Yes" entry (95.48), which is lower than "No" (95.84) 1910.10781 tab. 5. The text says embeddings did not significantly affect Fisher 1910.10781 §5. No significance test is reported.
• Training length. The abstract says models converge after as little as one epoch on a small domain dataset 1910.10781 abstract. The schedule that produced this is not stated 1910.10781 §4.5.
• Framework. The experiments cite TensorFlow 1.14.0 1910.10781 §4.5, while the BERT footnote points to a PyTorch implementation 1910.10781 §3.1.

Papers

• Hierarchical Transformers for Long Document Classification: splits long documents into 200-token segments with a 50-token shift, encodes them with BERT-Base, and aggregates them with an LSTM (RoBERT) or a 2-layer Transformer (ToBERT). Author-reported results on CSAT, 20 newsgroups and Fisher.