SVTR: Scene Text Recognition with a Single Visual Model

In plain words

Reading text in a photo, such as a sign or a shop name, is usually done with two parts: one part looks at the image, and a second part turns what it saw into a sequence of letters. The second part is often slow, because it reads one letter at a time, or it needs a large language-style model to fix mistakes.

SVTR does the job with a single visual network. It cuts the word image into small patches, where each patch holds a piece of a letter. Blocks of attention then let neighbouring patches share information, which picks up the strokes that make up one letter. Other blocks let patches far apart talk to each other, which helps tell neighbouring letters apart. The image is squeezed shorter at each stage, so the network works at several scales. At the end, one simple linear layer predicts a letter for each position, and repeated letters are merged.

The authors report that this is accurate enough to compete with the best, heavier systems on English text, and clearly better on Chinese text, while running faster. A smaller version, SVTR-T, is much lighter and faster still.

TL;DR

• SVTR is a single visual model for scene text recognition: no separate sequence or decoder model is used §1.
• It tokenises the image into character components, runs three height-shrinking stages of local and global self-attention blocks, and predicts with one linear layer §2.1.
• Author-reported: SVTR-L is competitive on six English benchmarks and reaches 72.1% on the Chinese Scene dataset; SVTR-T has 6.03M parameters and 4.5 ms per image on a GTX 1080 Ti §3.4 tab. 4.
• The official code is in PaddleOCR (Apache-2.0). It ships a Tiny English config and a Chinese config, which differ from the paper in several details (see Paper vs code).

Problem

Recognisers with a CNN or RNN sequence model are fast but fragile: reshaping the feature map into a sequence is sensitive to deformation and occlusion §1. Encoder- decoder models with autoregressive decoding are accurate but slow, because they transcribe one character at a time §1. Vision-language pipelines add a language model and a complex decoding step, which costs efficiency §1. Single-visual-model designs with a plain CNN or ViT backbone were reported as less accurate, because CNNs capture local patterns but not long-range dependence, and general-purpose transformers do not favour local character strokes §1. The paper assumes that a single visual model works only if it captures both intra-character strokes and inter-character dependence §1.

Method

Overview

SVTR is a three-stage network whose height is reduced at each stage fig. 2. A text image of size H x W x 3 is converted into H/4 x W/4 patches of dimension D0, called character components §2.1. Each stage applies a series of mixing blocks, then a merging operation (stages 1 and 2) or a combining operation (stage 3) §2.1 fig. 2. The output is a representation C of size 1 x W/4 x D3, followed by a parallel linear prediction with de-duplication §2.1.

Progressive overlapping patch embedding

The patch embedding uses two consecutive 3 x 3 convolutions with stride 2, each followed by batch normalization. The dimension grows progressively across the two convolutions, which the paper says helps feature fusion §2.2 fig. 3. The paper compares this with a 4 x 4 disjoint linear projection and a 7 x 7 convolution with stride 4 §2.2 fig. 3. Ablation in tab. 2 §3.3.

Mixing blocks

Each mixing block is a self-attention layer, with a layer norm before it and a residual connection, followed by a layer norm, an MLP and a second residual connection §2.3.

• Global mixing attends over all character components of the stage §2.3 fig. 4.
• Local mixing attends only to components inside a window of 7 x 11 that slides over the component grid §2.3 fig. 4. The paper does not say which axis is 7 and which is 11 (see Pitfalls).

Within a stage, blocks are repeated. The paper names the arrangement by counts, for example [L]6[G]6 for SVTR-T, meaning six local blocks followed by six global blocks §2.3 tab. 1.

# Mixing block  [§2.3] [code: ppocr/modeling/backbones/rec_svtrnet.py#L271-L278]
# Input: tokens (B, HW, d); mixer is "local" or "global"

def mixing_block(tokens, mixer):
    # Normalise, mix tokens with self-attention, add the shortcut  [§2.3]
    tokens = tokens + attention(layer_norm(tokens), mixer)  # (B, HW, d)
    # Normalise, expand with an MLP, add the shortcut  [§2.3]
    tokens = tokens + mlp(layer_norm(tokens))              # (B, HW, d)
    return tokens


# Self-attention with an optional local window mask  [§2.3]
# [code: ppocr/modeling/backbones/rec_svtrnet.py#L196-L212]

def attention(tokens, mixer):
    # Project to queries, keys and values, split into heads  [§2.3]
    queries, keys, values = qkv_projection(tokens)  # (B, heads, HW, d_head)
    scores = queries @ keys.transpose() / sqrt(d_head)
    # Local mixing: keep only positions inside the 7 x 11 window  [§2.3]
    if mixer == "local":
        scores = scores + window_mask  # -inf outside the window
    weights = softmax(scores, axis=-1)
    return output_projection(weights @ values)

Merging and combining

• Merging (stages 1 and 2): reshape the tokens to a grid of height h and width w, apply a 3 x 3 convolution with stride 2 in height and 1 in width, then a layer norm. The height is halved and the width is kept §2.4. The channel count increases from d(i-1) to d(i) §2.4.
• Combining (stage 3): pool the height to 1, then a fully connected layer, a non-linear activation and dropout §2.5. The paper does not name the activation or the dropout rate.

Prediction

A linear classifier with N outputs per position gives a sequence of length W/4. Ideally components of one character repeat the character and non-text components map to a blank; the sequence is then condensed §2.5. N is 37 for English (36 case-insensitive alphanumerics plus blank) and 6625 for Chinese §2.5 §3.2. The paper does not name the training loss.

# SVTR forward pass  [§2] [fig. 2] [tab. 1]
# Input: image (B, 3, H, W), H and W fixed by the configuration
# Output: per-position logits (B, W/4, N)

def svtr_forward(image):
    # Progressive overlapping patch embedding  [§2.2] [fig. 3]
    components = patch_embed(image)                 # (B, H/4 * W/4, D0)
    # Stage 1 mixing blocks at full height  [§2.3] [tab. 1]
    for block in stage1_blocks:
        components = mixing_block(components, block.mixer)
    # Merge: halve the height, widen channels to D1  [§2.4]
    components = merge(components)                  # (B, H/8 * W/4, D1)
    # Stage 2 mixing blocks  [§2.3] [tab. 1]
    for block in stage2_blocks:
        components = mixing_block(components, block.mixer)
    # Merge again: halve the height, widen channels to D2  [§2.4]
    components = merge(components)                  # (B, H/16 * W/4, D2)
    # Stage 3 mixing blocks  [§2.3] [tab. 1]
    for block in stage3_blocks:
        components = mixing_block(components, block.mixer)
    # Combine: pool height to 1, then FC, activation, dropout  [§2.5]
    features = combine(components)                  # (B, W/4, D3)
    # Linear prediction, one logit vector per position  [§2.5]
    return linear(features)                         # (B, W/4, N)

Architecture variants

Four variants differ in channel widths, heads, block counts and mixer order tab. 1.

Table 1 excludes the rectification module and the linear classifier tab. 1.

Hyperparameters

The paper's training recipe is in §3.2. The table shows the paper value, then the value the shipped config uses where it differs. "Not stated" means neither the paper nor the cited config gives it.

Results

Author-reported. Setup: MJSynth and SynthText for training; six English test sets (IC13, SVT, IIIT5K, IC15, SVTP, CUTE) and the Chinese Scene dataset §3.1. All models trained on 4 Tesla V100 GPUs with PaddlePaddle §3.2. Word accuracy §3.2. Speed is inference time on one NVIDIA 1080Ti averaged over 3000 English images tab. 4. The Chinese results compare against six existing methods, whose accuracies the paper takes from the Chinese Scene benchmark it cites §3.4.

Ablation setting §3.3: SVTR-T without rectification and augmentation, measured on IC13 and IC15.

Patch embedding and merging tab. 2:

Mixing-block permutation tab. 3:

Comparison with state of the art tab. 4. Language-free ("Lan-free") rows marked

• use the language-free version of a method. Speed is in ms on one 1080Ti.

Headline claims from the text §3.4: SVTR-T is highly competitive among language-free methods; the other SVTR variants are new best results on most datasets. Against language-aware methods, SVTR-L is said to be best on IIIT, IC15 and CUTE, but tab. 4 shows ties on IIIT (96.3 with VST) and CUTE (95.1 with VST). On Chinese, SVTR-B and SVTR-L are reported 5.4% to 9.6% above SAR (the best listed baseline) §3.4. Speed and parameter comparisons are in fig. 5, which the text describes but does not tabulate.

Paper vs code

Code: PaddleOCR, commit dab3fe35379033fdcb2d0e9572fac0b36c9a9ebf, Apache-2.0 (LICENSE file). The paper links this repository in its abstract. The repository is a rolling project, so line numbers cite the pinned commit, not later code. The repository also ships SVTRv2, RepSVTR and CPPD configs that are not part of this paper.

Only the Tiny English config (rec_svtrnet.yml) and a Chinese config (rec_svtrnet_ch.yml) are named for this method. No config is named for the S, B or L variants, so their settings are not checked.

• Epochs: paper 21 for English §3.2; config epoch_num: 20 rec_svtrnet.yml.
• Chinese warm-up: paper 5 epochs §3.2; config 2 epochs rec_svtrnet_ch.yml.
• Chinese learning rate: paper 3e-4 x batch/512 §3.2; config 5e-4 rec_svtrnet_ch.yml. Config batch is 256 per card rec_svtrnet_ch.yml; GPU count not given there.
• Chinese augmentation: paper says none §3.2; config enables RecConAug with probability 0.5 rec_svtrnet_ch.yml.
• Chinese output length: paper says sequence length W/4 §2.5 and max predicted length 40 §3.2; config out_char_num: 40 on a 32 x 320 input gives W/8 rec_svtrnet_ch.yml, and max_text_length: 25 rec_svtrnet_ch.yml.
• Chinese rectification: paper describes rectification as part of SVTR §3.2; Chinese config disables it rec_svtrnet_ch.yml. Chinese output channels are 96 in config rec_svtrnet_ch.yml, against D3 = 192 in tab. 1. The paper does not state D3 for Chinese.
• Combining width: the code pools the width to out_char_num with an adaptive pool rec_svtrnet.py, which equals W/4 only when out_char_num matches the input width.
• Activation after the combining FC: paper says non-linear, unnamed §2.5; code uses Hardswish rec_svtrnet.py.
• Augmentation: paper lists rotation, perspective, motion blur and Gaussian noise §3.2; code uses geometric, deterioration and colour-jitter operators with their own parameters rec_img_aug.py.
• Decoder: paper describes linear prediction with de-duplication §2.5; code uses CTCLoss rec_svtrnet.yml with a blank token at index 0 rec_postprocess.py.
• Normalisation order: the config sets prenorm: false rec_svtrnet.yml. The flag name is inverted: prenorm=False selects the pre-norm residual form (norm before mixing), which matches the paper's description rec_svtrnet.py §2.3. The default prenorm=True (line 232) applies the norm after the residual add. A final LayerNorm is applied when prenorm is False rec_svtrnet.py.
• Positional embedding: the code adds a learned position embedding after patch embedding rec_svtrnet.py; the paper does not mention one.
• Mixer schedule: the code applies a single list of 12 mixer types across the whole network, sliced per stage rec_svtrnet.py rec_svtrnet.py. This matches the network-wide totals in tab. 1, not the per-stage wording in §3.3.

No discrepancies found in: patch embedding stride and channel plan, the 7 x 11 local window size, the merging operation (3 x 3, stride 2 in height), the stage depths and heads for Tiny, and the English alphabet size of 37 (see Hyperparameters).

Pitfalls

• Mixer notation. §3.3 says "for each stage, six global mixing blocks ... then six local", but tab. 1 counts are network-wide: SVTR-S [L]8 [G]7 sums to 15 blocks, matching depths 3 + 6 + 6. The code uses the network-wide reading. Implement a single sequence of 12 mixer types for Tiny.
• Window axes. "7 x 11" is not oriented §2.3. The code applies 7 along height and 11 along width rec_svtrnet.yml rec_svtrnet.py.
• Input size is fixed. The local-window mask and position embedding are built once for the configured input size, and the forward pass asserts the input shape rec_svtrnet.py rec_svtrnet.py. Changing width means rebuilding the model.
• Ablation text vs table. §3.3 says progressive embedding beats the two defaults by 0.75% and 2.8% "on average on the two datasets". tab. 2 gives averages of 0.7 points over overlap ((0.5 + 0.9) / 2) and 1.9 points over linear ((1.0 + 2.8) / 2). The 2.8 figure is the IC15 gap to linear. Use the table values.
• Two parameter counts. SVTR-T is 4.15M in tab. 1 (no rectifier, no classifier) and 6.03M in tab. 4 (full model, as quoted in §1). Use the matching figure for each comparison.
• Speed. Latencies are for a 1080 Ti on English images only tab. 4. The paper gives no Chinese speed. Figure 5 is only described in words fig. 5 §3.4.
• Tie claims. The text says SVTR-L is best on IIIT, IC15 and CUTE among the English benchmarks §3.4. The table shows ties on IIIT and CUTE with VST tab. 4.
• Rectification size. The paper resizes to 32 x 64 for rectification §3.2. The config feeds the backbone 32 x 100 after the TPS warp and resizes the dataset to 64 x 256 first rec_svtrnet.yml rec_svtrnet.yml. Implement the code's sizes if matching the released model.
• Training data. The paper names MJSynth and SynthText §3.1. The English config reads an LMDB folder whose contents are not listed in the repository files checked; treat the exact mixture as not stated.
• Learning rate scaling. The English formula is linear in batch size §3.2. The config's 512 per card matches batch 2048 only on 4 cards. Scale the rate with your own total batch.
• Unstated settings. Optimizer betas, epsilon, dropout rates, stochastic depth, attention bias, MLP ratio, LayerNorm epsilon and the position embedding all come from the code, not the paper (see Hyperparameters).
• Chinese setup. The paper gives no Chinese input size, rectification setting or D3 §3.2 tab. 1. The only Chinese recipe in the repository has a 40-step output but max_text_length 25 and no rectifier rec_svtrnet_ch.yml rec_svtrnet_ch.yml. Results for SVTR on Chinese cannot be reproduced exactly from the paper alone.

Lineage

• Builds on: vision transformer patch tokenisation and self-attention §1 (ViT and Swin, cited by name in the paper, no arXiv ids in the source); rectification module from ASTER §3.2.
• Compared against: CRNN, Rosetta, SRN, PREN, ViTSTR, ABINet, VST, ASTER, MORAN, NRTR, SAR, AutoSTR, PREN2D, VisionLAN tab. 4.
• Superseded by: not known from wiki content. The repository ships SVTRv2 and RepSVTR configs, which are separate methods and are not described here .