---
kind: concept
slug: chunked-long-input-encoding
title: Chunked long-input encoding
summary: >-
  Classify a document longer than the encoder's limit by cutting it into overlapping fixed-size segments, encoding each with BERT-Base, and pooling the segment vectors with a small LSTM or a 2-layer Transformer. Author-reported: ToBERT reaches 95.48% on Fisher topic ID against 92.93% for an MS-CNN baseline, while on CSAT the two heads are roughly tied. Segment-level fine-tuning is required.
tags: [long-documents, text-classification, bert, fine-tuning, hierarchical-encoding]
papers: ["1910.10781"]
related: [language-model-pretraining, text-classification, long-short-term-memory, self-attention, sliding-window-attention, local-global-attention]
status: draft
updated_at: 2026-11-03
---

## Overview

A pretrained encoder such as BERT accepts a bounded number of tokens, and its
self-attention cost grows quadratically with length [arxiv:1910.10781 §3.1].
Chunked long-input encoding works around the limit without changing the
encoder. The input is cut into segments that fit the encoder, each segment is
encoded separately, and a second, small model combines the segment vectors
into one document decision [arxiv:1910.10781 §3.2] [arxiv:1910.10781 §3.3].

The shared invariant is that the encoder only ever sees one segment at a time,
so its per-call cost is fixed by the segment length, not by the document
length. Everything document-level is the job of the aggregator. Only one paper
in this wiki covers the approach so far, so the variants below are both from
the same source. Results are author-reported on three text tasks [arxiv:1910.10781 §4].

## Variants

### Recurrent aggregator (RoBERT)

Each segment is encoded by a fine-tuned BERT-Base. The segment vectors form a
sequence that a 100-unit LSTM reads. Its output feeds a 30-unit ReLU layer and
a softmax over classes [arxiv:1910.10781 §3.2]. Card:
[Hierarchical Transformers for Long Document Classification](/papers/1910.10781).
Reported accuracy: 83.65 on CSAT, 84.71 on 20 newsgroups, 82.28 on Fisher, using
the segment representation H [arxiv:1910.10781 tab. 3].

### Transformer aggregator (ToBERT)

The LSTM is replaced by a 2-layer Transformer over the segment vectors. The
authors also test learned position embeddings on those vectors, which limits
the model to sequence lengths seen in training [arxiv:1910.10781 §3.3].
Reported accuracy with the same segment representation: 83.48 on CSAT, 85.52 on
20 newsgroups, 95.48 on Fisher [arxiv:1910.10781 tab. 3]. Position embeddings
helped on CSAT (83.48 vs 82.84) and made almost no difference on 20 newsgroups
and Fisher [arxiv:1910.10781 tab. 5].
Card: [Hierarchical Transformers for Long Document Classification](/papers/1910.10781).

### Prediction-level aggregation

No second neural model. Each segment gets a class posterior P from the
fine-tuned BERT. The document label comes from averaging the posteriors, from
the most frequent predicted segment class, or from a classifier trained on the
posteriors [arxiv:1910.10781 §5] [arxiv:1910.10781 tab. 4]. With P, averaging
reaches 82.84 on CSAT and 88.48 on Fisher; the learned heads reach 83.54 and
91.18 (RoBERT) or 81.48 and 94.16 (ToBERT) [arxiv:1910.10781 tab. 4]. Card:
[Hierarchical Transformers for Long Document Classification](/papers/1910.10781).

## When to use which

The source paper tests three text classification tasks with long inputs. The
statements below apply to those tasks and nothing wider.

| Situation | Use | Why |
|---|---|---|
| Long documents where segment order may matter (conversations, transcripts) | RoBERT or ToBERT over segment vectors | The aggregator sees the segments in order. On CSAT with P, RoBERT beats averaging (83.54 vs 82.84) and ToBERT does not (81.48). The paper calls the CSAT gap between the two heads not statistically significant [arxiv:1910.10781 tab. 4] [arxiv:1910.10781 §5] |
| Long documents with many classes and ample labelled data, such as topic labelling | ToBERT on segment vectors H | Fisher accuracy 95.48 in the main table, with 40 classes and 90% of 1374 training documents used for training [arxiv:1910.10781 tab. 3] [arxiv:1910.10781 §4.3] |
| No budget to fine-tune the encoder | Not this approach as tested | Pretrained BERT features without fine-tuning score far lower, for example 38.04 for RoBERT on Fisher [arxiv:1910.10781 tab. 2] |
| Small labelled set, or a need to keep the pipeline simple | Prediction-level averaging over posteriors P | Needs no second model. On CSAT with P, averaging (82.84) beats ToBERT (81.48) [arxiv:1910.10781 tab. 4] |
| Need to handle sequences longer than those in training | Avoid learned position embeddings in the aggregator | The authors state the learned variant cannot exceed training lengths [arxiv:1910.10781 §3.3] |
| Every document is short enough for the encoder | Plain encoder, no chunking | The paper's motivation is the length limit. For shorter inputs it notes a prior BERT approach without chunking [arxiv:1910.10781 §2] |

Cost: every segment is a full encoder pass, so the number of passes grows with
document length. With the stated 200-token segments and 50-token shift, each
pair of consecutive segments shares 150 tokens, which is a derived reading and
not a figure the paper reports [arxiv:1910.10781 §4.5].

## Implementation sketch

Language-agnostic outline of the common core. The variants differ only in the
last two steps. Values are those the paper reports for its experiments.

```pseudocode
# Chunked encoding of one document  [arxiv:1910.10781 §3.2] [§3.3]
# Input:  tokens: subword tokens of one document  (N,)
# Output: class probabilities  (C,)

def classify_document(tokens, aggregator):
    # Cut into segments of 200 tokens with a shift of 50  [§4.5]
    # Trailing segment and padding are not specified by the paper
    segments = cut_with_shift(tokens, segment_len=200, shift=50)
    # Encode each segment with the fine-tuned BERT-Base; keep pooled output  [§3.1]
    segment_vectors = [bert_pooled(s) for s in segments]
    if aggregator == "lstm":
        # RoBERT: 100-unit LSTM; the paper does not name which step is output  [§3.2]
        doc = lstm_100(segment_vectors)
    elif aggregator == "transformer":
        # ToBERT: 2 Transformer layers; how the output becomes one vector is not stated  [§3.3]
        doc = transformer_2_layers(segment_vectors)
    else:
        # Prediction-level: average (or vote over) per-segment posteriors  [§5]
        posteriors = [bert_posterior(s) for s in segments]
        return mean(posteriors)
    # Dense ReLU layer of 30 units, then softmax over classes  [§3.2]
    hidden = relu(dense_30(doc))
    return softmax(dense_classes(hidden))
```

Training, as reported: BERT is fine-tuned on segment-level labels, each segment
inheriting its document's label [arxiv:1910.10781 §3.1]. The RoBERT head uses
Adam with learning rate 0.001 and a decay by a factor of 0.95 after three epochs
without validation-loss improvement; ToBERT uses learning rate 5e-5
[arxiv:1910.10781 §4.5]. Batch size, epoch count and dropout are not stated.

## Contradictions

- **Fisher gain between heads.** The text gives ToBERT's margin over RoBERT on
  Fisher as 13.63 points [arxiv:1910.10781 §5]. Table 3 gives 95.48 − 82.28 =
  13.20 points [arxiv:1910.10781 tab. 3]. This page uses the table values.
- **"All tasks" claim.** The conclusion says ToBERT beats RoBERT on all tasks
  [arxiv:1910.10781 §6]. On CSAT, RoBERT is higher with H (83.65 vs 83.48)
  [arxiv:1910.10781 tab. 3] and with P (83.54 vs 81.48) [arxiv:1910.10781 tab. 4].
  The results text calls ToBERT "slightly worse" on CSAT [arxiv:1910.10781 §5].
- **Metric name on CSAT.** The text calls a CSAT gain an improvement in F1
  [arxiv:1910.10781 §5], while the paper states all results are accuracy
  [arxiv:1910.10781 §4.5]. The table difference (83.48 − 82.84 = 0.64) is an
  accuracy difference [arxiv:1910.10781 tab. 5].
- **20 newsgroups baseline.** The MS-CNN baseline scores 86.12 against ToBERT's
  85.52 on 20 newsgroups [arxiv:1910.10781 tab. 6], so the approach does not beat
  every baseline on every task.
- **Position embeddings on Fisher.** The table bolds the "Yes" entry (95.48),
  which is lower than "No" (95.84) [arxiv:1910.10781 tab. 5]. The text says
  embeddings did not significantly affect Fisher [arxiv:1910.10781 §5]. No
  significance test is reported.
- **Training length.** The abstract says models converge after as little as one
  epoch on a small domain dataset [arxiv:1910.10781 abstract]. The schedule that
  produced this is not stated [arxiv:1910.10781 §4.5].
- **Framework.** The experiments cite TensorFlow 1.14.0 [arxiv:1910.10781 §4.5],
  while the BERT footnote points to a PyTorch implementation [arxiv:1910.10781 §3.1].

## Papers

- [Hierarchical Transformers for Long Document Classification](/papers/1910.10781):
  splits long documents into 200-token segments with a 50-token shift, encodes
  them with BERT-Base, and aggregates them with an LSTM (RoBERT) or a 2-layer
  Transformer (ToBERT). Author-reported results on CSAT, 20 newsgroups and Fisher.