wikillm

Paper · 2020-05-26 · cs.CV

Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition

Handwritten text-line recognizer with no recurrence: a ResNet50 encoder plus Transformer self-attention on the image and on the character sequence, with a cross-attention step that aligns them. Trained on synthetic line images, then fine-tuned on IAM. Author-reported 4.67% CER and 15.45% WER on the IAM line-level test set, without a closed lexicon.

Optical character recognitionSelf-attentionCross-attentionSynthetic training dataocrtransformersequence-recognition

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Reading handwriting line by line used to be done with recurrent networks, which read a line one step at a time, left to right. That order makes training slow, because each step waits for the previous one. This paper replaces the recurrence with attention, the same building block used in modern language models, so the whole line can be processed at once.

The model first turns the picture of a line into a sequence of visual features, and marks each position with its place in the line so that repeated letters are told apart. A second stream reads the target text, one character at a time, and attends over the characters that came before it. A final attention step lets each character being written look at the matching part of the picture. Because the model works on single characters rather than whole words, it can write words it never saw in training.

Training uses a large set of machine-generated handwriting-like lines first, then real annotated lines. The paper reports that this pre-training helps most when little real data is available.

TL;DR

A non-recurrent Transformer for offline handwritten text-line recognition at character level. A ResNet50 feature encoder, four self-attention blocks on the image and on the text, and a text-to-image cross-attention step, all trained end to end. Pre-training on synthetic lines gives the best reported result: 4.67% CER and 15.45% WER on IAM line level, author-reported tab. 5. Relevant if you need open-vocabulary line recognition and cannot use recurrent training.

Problem

Recurrent encoder-decoder recognizers for handwritten lines are the standard approach. Their sequential computation blocks parallel training and makes long sequences memory-heavy §1. Attention-based decoders still ran on top of recurrent networks §1. Scene-text Transformers exist, but they assume fixed image size and word-level input, which does not fit variable-length handwritten lines §2. Real annotated handwriting is expensive, which motivates the synthetic pre-training §1 §4.3. The experiments assume the IAM line-level partition §4.1.

Method

The model has two parts: a visual feature encoder and a text transcriber. Both are trained together §3.1.

  1. Visual feature encoder §3.2. A ResNet50 backbone, with its last convolution stride changed from 2 to 1, produces a feature map F_c of size f §3.2.1 §4.4.1 tab. 1. The height and channel axes are merged and a fully connected layer reduces them back to f, giving a width-indexed sequence §3.2.2. Sinusoidal positional encodings over the width axis are added §3.2.2 eq. 1. A final fully connected layer gives the abscissa-aware visual features F̄_c, shape (f, w) §3.2.2.
  2. Visual self-attention §3.2.3. Four self-attention blocks run on F̄_c, each with eight heads, with Q = K = V = F̄_c §3.2.3 §4.2.1 eq. 2.
  3. Text encoding §3.3.1. The target string is wrapped with a start symbol and an end symbol, then padded with a pad symbol to N = 89 characters §4.2.1. Each character is embedded into f dimensions and the same positional encoding is added §3.3.1 eq. 3. Future positions are masked, so the character at position j depends only on earlier positions §3.3.1.
  4. Language self-attention §3.3.2. Self-attention blocks run on the text sequence. Four blocks are reported §4.2.1. The paper says this stage learns character-level language structure, and that it takes the place of a separate language model §4.4.3 §4.7.
  5. Mutual (cross-) attention §3.3.3. Queries come from the text features, keys and values from the visual features §3.3.3 eq. 4. The output has one vector per output position.
  6. Output. A linear layer followed by softmax over the alphabet of 83 symbols gives a prediction for each position §3.3.3 §4.2.1. Training uses label smoothing with ε = 0.4 §4.2.2.

The whole model is trained end to end §3.1. Training feeds all output positions in one pass, using the mask instead of step-by-step feedback §3.3.1. Decoding at test time is sequential, see the pseudocode below §3.4.

Pseudocode
# Training step for one batch  [§3.1] [§3.2] [§3.3]
# Input:  images (B, C, 64, 2227); targets: strings, at most 89 characters
# Output: scalar loss
def training_step(images, targets):
    # ResNet50 features, last stride 1  [§3.2.1] [§4.4.1]
    features = resnet50_modified(images)            # (B, f_c, h, w)
    # Merge height into features, reduce to f  [§3.2.2]
    features = fc_reduce(flatten_height(features))  # (B, f, w)
    # Add the sinusoidal encoding over width  [§3.2.2] [eq. 1]
    features = fc_pos(features + positional_encoding(w))  # (B, f, w)
    # Four visual self-attention blocks, 8 heads  [§3.2.3] [§4.2.1]
    visual = features
    for block in visual_blocks:
        visual = self_attention(visual, visual, visual)   # (B, f, w)
    # Pad the target to N = 89 with start and end symbols  [§3.3.1] [§4.2.1]
    tokens = pad_to_length(add_start_end(targets), 89)  # (B, N)
    # Embed characters, add the same positional encoding  [§3.3.1] [eq. 3]
    text = embedding(tokens) + positional_encoding(N)   # (B, f, N)
    # Causal self-attention: position j sees only positions before j
    # [§3.3.1] [§3.3.2]
    text = causal_self_attention(text)                  # (B, f, N)
    # Four language self-attention blocks  [§3.3.2] [§4.2.1]
    for block in language_blocks:
        text = causal_self_attention(text)
    # Cross-attention: text queries, visual keys and values  [§3.3.3] [eq. 4]
    fused = cross_attention(text, visual, visual)       # (B, f, N)
    # Project to the 83-symbol alphabet  [§3.3.3] [§4.2.1]
    logits = linear(fused)                              # (B, N, 83)
    # Label smoothing with epsilon = 0.4  [§4.2.2]
    # Alignment of outputs with targets is not stated  [§3.3.1]
    loss = label_smoothed_cross_entropy(logits, tokens, eps=0.4)
    return loss
Pseudocode
# Greedy decoding at test time  [§3.4]
# Input:  image (B = 1, 64 px height); maximum length N = 89  [§4.1]
# Output: character string
def transcribe(image):
    # Encode the image once  [§3.4] [§3.2]
    visual = encode_image(image)
    # Start from the start symbol  [§3.4]
    tokens = [START]
    for position in range(89):
        # Score the next character given the tokens so far  [§3.4] [§3.3]
        logits = predict(tokens, visual)
        # Take the most probable symbol  [§3.4]
        next_symbol = argmax(logits[-1])
        # Stop at the end symbol  [§3.4]
        if next_symbol == END:
            break
        tokens.append(next_symbol)
    return tokens[1:]

Key equations, as written in the paper:

TE(pos, 2i) = \sin\left(\frac{pos}{10000^{2i/f}}\right), \quad TE(pos, 2i+1) = \cos\left(\frac{pos}{10000^{2i/f}}\right) eq. 1

\hat{v}_c^{i} = \operatorname{Softmax}\left(\frac{q_c^{i} K_c}{\sqrt{f}}\right) V_c eq. 2

F_t = \operatorname{Embedding}(y) + TE eq. 3

\hat{v}_{ct}^{i} = \operatorname{Softmax}\left(\frac{q_t^{j} K_c}{\sqrt{f}}\right) V_c eq. 4

Label smoothing §4.2.2 is written in prose, not as a numbered equation. Each hard target 0 becomes ε/|A| and each target 1 becomes 1 − (|A| − 1)ε/|A|, with |A| = 83 and ε = 0.4.

Hyperparameters

NameValueWhereNotes
Input text-line height64 px§4.1aspect ratio kept, so width varies
Padded image width2227 px§4.1blank padding to pack mini-batches; handling of wider lines not stated
Alphabet size |A|83§4.1 §4.2.1includes special symbols; softmax output size 83
Max output length N89§4.1 §4.2.1start and end symbols included in the padding
Feature size f1024§4.2.1
Visual self-attention blocks4§3.2.3 §4.2.1
Language self-attention blocks4§4.2.1
Heads per self-attention module8§3.2.3 §4.2.1
Dropout0.1§4.2.1on every dropout layer
Label smoothing ε0.4§4.2.2
OptimizerAdam§4.2.2betas, weight decay: not stated
Initial learning rate2 · 10⁻⁴§4.2.2
Learning-rate schedulehalved every 20 epochs§4.2.2total epochs, warm-up: not stated
Mini-batch sizenot stated
CNN backboneResNet50, last stride 2 → 1§3.2.1 §4.4.1 tab. 1ImageNet initialisation: not stated
Early stoppingvalidation CER§4.4used to choose hyperparameters
Synthetic pre-training data138,000 text lines§4.3from over 130,000 corpus lines, 387 cursive fonts
Real data split (IAM RWTH)6,482 train / 976 val / 2,914 test lines§4.1
Seq2Seq baseline encoderVGG19-BN + 2-layer BGRU, 512§4.5authors' re-implementation
Seq2Seq baseline decoder2-layer GRU, 512, location attention§4.5dropout 0.5
LM for shallow fusion2,400 LSTM units, weight 0.2§4.7trained on 130,000 text lines

Results

Author-reported. Setup: IAM handwritten English, RWTH line-level partition, greedy decoding, PyTorch on an NVIDIA cluster (GPU model not stated) §4.1 §4.2.2 §4.6. CER and WER are the error rates defined in §4.1.

Main comparison against the authors' own Seq2Seq baseline tab. 5:

SettingCER (%)WER (%)Time (s)Params (M)Source
Seq2Seq11.9137.39338.737tab. 5
Seq2Seq + synth10.6433.64338.737tab. 5
This paper, IAM only7.6224.54202.5100tab. 5
This paper, + synth4.6715.45202.5100tab. 5

The time column is labelled "Time(s)", and the text describes it as time per epoch §4.5.

State of the art on IAM line level tab. 7: the paper's model is best among methods without a closed lexicon, at 4.67 CER and 15.45 WER. The best closed-lexicon method in the table is Bluche et al. at 3.20 CER and 10.50 WER tab. 7 §4.8.

Ablations, all trained on IAM only §4.4:

SettingCER (%)WER (%)Source
Without temporal encoding (image, text)6.3321.64tab. 2
Temporal encoding on image only7.7224.70tab. 2
Temporal encoding on text only5.5220.72tab. 2
Temporal encoding on both4.8618.65tab. 2
No self-attention7.7829.78tab. 4
Visual self-attention only7.7128.50tab. 4
Language self-attention only4.8918.57tab. 4
Both self-attention modules4.8618.65tab. 4

Modified ResNet50 gives the best CER among the backbones tried (4.86 vs 5.49 for unmodified ResNet50) tab. 1. The ablation baseline (4.86) is the model without synthetic pre-training, not the 4.67 result above.

Low-data results, fine-tuning on random subsets of real IAM training data tab. 3 §4.6:

Real dataSeq2Seq CER / WERSeq2Seq + synthThis paperThis paper + synth
20%20.61 / 56.5018.64 / 51.7773.81 / 132.746.51 / 20.53
40%16.15 / 46.9713.01 / 39.7217.34 / 42.576.20 / 19.69
60%15.61 / 46.0113.00 / 39.3410.14 / 30.345.54 / 17.71
80%12.18 / 38.1112.15 / 37.4310.11 / 29.904.90 / 16.44
100%11.91 / 37.3910.64 / 33.647.62 / 24.544.67 / 15.45

Language-model post-processing tab. 6 §4.7: 4.67 / 15.45 without the shallow-fusion LM, and 4.66 / 15.47 with it. The paper reads this as the language self-attention already capturing the same information.

Paper vs code

No official code. The paper says the code will be publicly available §4.2.2, but gives no repository link in the text, abstract or source. Searches made: the paper's own text and comments; GitHub repository search for "transformer handwritten text line", "non-recurrent handwritten" and "htr transformer" plus the author lab's organisation (omni-us, a co-author's company); and a web search for the title and author names. The only repository found from that organisation, research-seq2seq-HTR, implements a different paper (Convolve, Attend and Spell) and was not used. Re-run these searches before implementing the paper, since the release could have appeared after this card was compiled.

Pitfalls

  • Ablation numbers differ from headline numbers. Ablation tables tab. 2, tab. 4 are trained on IAM only and early-stopped on validation CER §4.4. The 4.67 CER headline needs synthetic pre-training tab. 5. Mixing them up changes the reference point by about 0.2 CER.
  • Equation 2 is ambiguous. It writes q_c^i K_c with no transpose and no stated softmax axis, while K_c has shape (f, w) §3.2.3. Multi-head splitting and the output projection are described in prose only §3.2.3.
  • Equation 4 mixes indices. The output index is i, but the query is q_t^j with j ∈ {0, …, N−1} §3.3.3. The text calls this a "mutual self-attention", but the query comes from text and keys and values from the image, which is cross-attention. The text also says the keys and values come from F̂_c, but the equation writes K_c and V_c.
  • Causal masking is implied, not written out. The text says the language self-attention sees only earlier characters §3.3.2, and the mask is described for the decoding step §3.3.1. Whether the mask also applies to the cross-attention step §3.3.3 is not stated.
  • Input-target alignment is not stated. The paper does not say whether the decoder input is the target shifted right, or whether the start symbol is predicted from the image. Check against the code if one is released.
  • Channel count of the CNN output is not stated. The paper gives f = 1024 as the feature size §4.2.1, but not the number of channels leaving ResNet50. Input channel count C is not stated either.
  • Wide lines. All lines are padded to 2227 px and keep their aspect ratio §4.1. The paper does not say what happens to lines wider than 2227 px.
  • Synthetic data is hard to reproduce. The corpus source is "online e-books" and the 387 fonts are "freely available" §4.3. Their licences and the exact augmentation parameters are not stated. Do not assume they can be redistributed.
  • Training length and batch size are missing. Total epochs, batch size, warm-up, weight decay and the GPU used are not stated §4.2.2. The only schedule given is the halving every 20 epochs.
  • Decoding cost. Training runs all positions in one pass, but inference is sequential, one character per step, up to N = 89 §3.4. Whether a key/value cache is used is not stated.
  • Greedy decoding only. Results use greedy decoding; beam search is not reported §3.4 §4.6.
  • Label smoothing values. With |A| = 83 and ε = 0.4, the hard-target replacements are about 0.0048 for a zero and about 0.6048 for a one (derived from the paper's formula, not stated in it); the two sum to 1 across the alphabet.
  • Model size is not iso-compute. The transformer has about 100 M parameters against 37 M for the Seq2Seq baseline tab. 5, so the comparison is not parameter-matched.
  • Few-shot subsets are random, with no seed stated §4.6. Results at 20% for the transformer without synthetic data (73.81 CER, 132.74 WER) are very high and are reported as is tab. 3.
  • Language-model baseline is the authors' own. The shallow-fusion LM is trained on 130,000 lines, also the size of the synthetic corpus §4.7. Only the LM's weight (0.2) is given; its training setup is not.

Lineage

  • Builds on: the Transformer of Vaswani et al. (2017), cited as the basis for the attention blocks and positional encoding §1 §3.2.2.
  • Builds on: the authors' earlier attention-based recurrent recognizer, which the paper uses as its Seq2Seq comparison, with a re-implementation §4.5.
  • Compared against: a recurrent Seq2Seq recognizer with VGG19-BN and BGRU, trained under the same conditions §4.5 tab. 5; HMM, RNN+CTC and closed-lexicon systems on IAM line level tab. 7.
  • Superseded by: not stated in the paper; not established from other wiki content.

Source and license

Adapted from “Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition” by Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, Mauricio Villegas (arXiv:2005.13044), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0.