Focusing Attention: Towards Accurate Text Recognition in Natural Images

In plain words

Reading words in photos is harder than reading scanned pages. Lighting, blur, odd fonts and busy backgrounds can make a model look at the wrong spot while it reads each letter. The authors call this "attention drift": the model is looking at a neighbouring patch, so it reads the wrong letter or skips one.

Their fix is to add a second small network that checks where the reader is looking for each letter and nudges that focus back onto the letter. Some training images also mark where each letter sits, which tells the checker where it should have looked. The encoder that turns the image into features is a deeper residual network than earlier work used.

TL;DR

Scene text recognition with an attention-based encoder-decoder fails on hard images because the attention for each character drifts off the character §1. FAN adds a focusing network (FN) that scores where the attention falls and trains the attention to land on each character, using pixel-level character labels for about 30% of the training words §3.2 §4.2. The paper reports gains over the same model without FN on every benchmark it tests tab. 2.

Problem

Attention-based recognizers learn the alignment between output characters and image regions without supervision on where each character is. On complicated or low-quality photos these alignments are often wrong, and the wrong regions produce wrong characters §1 §3.1. The authors name this failure "attention drift" and say it is the main reason for poor results on such images §1. Training such models on large synthetic sets is also hard §3.1.

Method

The model has three parts: a feature encoder, an attention network (AN) that reads the features one character at a time, and a focusing network (FN) that is trained only to check AN's attention §3 fig. 4.

Encoder. A ResNet-style CNN turns the image, resized to 256 wide by 32 high, into a sequence of 65 column features. A bidirectional LSTM runs over that sequence §4.2 tab. 1. Each feature corresponds to an image region called an attention region §1.

Attention network (AN). A standard attention decoder. At each output step it scores every feature against the decoder state, normalises the scores with a softmax to get alignment factors, and forms a glimpse vector as the weighted sum of the features. The decoder LSTM then emits a character from the glimpse and its state. An end-of-sequence symbol ends the output §3.1 eq. 1 eq. 2 eq. 3 eq. 4 eq. 5. The AN loss is the negative log-likelihood of the true characters eq. 6.

Focusing network (FN). For output step t the FN does two things §3.2:

1. It computes an attention centre. Each feature has a centre, which is the centre of its receptive field in the input image, found by applying the convolution geometry layer by layer eq. 7 eq. 8. The attention centre for step t is the alignment-weighted average of these centres eq. 9.
2. It crops a fixed-size patch of feature maps around that centre eq. 10. A small head scores every position of the patch against the glimpse vector and the cropped feature, and a softmax over the K label classes gives a distribution for each position eq. 11 eq. 12. Each position therefore predicts a label, as if the patch were a pixel-level segmentation.

The FN loss is the negative log-likelihood of the labelled positions, summed over steps, over the patch width and over the patch height, and it is applied only to images that carry character annotations eq. 13 §3.2.

Joint objective. The total loss mixes the two losses with a weight λ, L = (1 - λ) L_Att + λ L_focus, where 0 ≤ λ < 1 eq. 14 §3.3. The network is trained by back-propagation §3.3.

Decoding. For unconstrained recognition the most probable character is taken at each step. For constrained recognition, the conditional probability of each lexicon word is computed and the best word is returned §3.4.

# Focusing attention, one training pass over a word image  [§3.1] [§3.2]
# Input:  image (H, W, C); labels y_1..y_M; pixel labels (optional)
# Output: scalar loss  [eq. 14]

def fan_loss(image, labels, pixel_labels, lam):
    # Encode the image into T column features  [§4.2] [tab. 1]
    features = encode_cnn_blstm(image)               # (T, feat)
    # Centre of the receptive field of each feature  [eq. 7] [eq. 8]
    centres = receptive_field_centres(features)      # (T, 2)
    state = initial_decoder_state()
    loss_att = 0
    loss_focus = 0
    for t in range(M):
        # Alignment factors over the T features  [eq. 4] [eq. 5]
        alpha = attention_weights(state, features)   # (T,)
        # Glimpse vector: weighted sum of features  [eq. 3]
        glimpse = weighted_sum(alpha, features)
        # Attention centre: weighted average of the centres  [eq. 9]
        centre = weighted_sum(alpha, centres)        # (2,)
        # Crop a P_H x P_W patch of feature maps at the centre  [eq. 10]
        patch = crop(feature_map, centre, P_H, P_W)
        # Decode the character for step t  [eq. 1] [eq. 2]
        label_probs, state = decode_step(state, glimpse, labels[t-1])
        # AN loss: negative log-likelihood of the true character  [eq. 6]
        loss_att -= log(label_probs[labels[t]])
        # FN: per-position label distribution over the patch  [eq. 11] [eq. 12]
        pixel_probs = focus_head(glimpse, patch)     # (P_W, P_H, K)
        # FN loss only where pixel labels exist  [eq. 13] [§3.2]
        if pixel_labels is not None:
            loss_focus -= sum_log_prob(pixel_probs, pixel_labels[t])
    # Trade off the two losses with lambda  [eq. 14] [§3.3]
    return (1 - lam) * loss_att + lam * loss_focus

Key equations, as written in the paper:

\mathcal{L} = (1 - \lambda)\,\mathcal{L}_{Att} + \lambda\,\mathcal{L}_{focus} eq. 14

c_t = \sum_{j=1}^{T} \alpha_{t,j}\, c_{t,j} eq. 9

\mathcal{L}_{focus} = - \sum_t^M \sum_i^{\mathcal{P}_W} \sum_j^{\mathcal{P}_H} \log P(\hat{y}_{t}^{(i,j)} \mid \mathcal{I}, \omega) eq. 13

Hyperparameters

Values the paper states are cited. Anything not stated is marked.

Results

Author-reported. Setup: the model is trained on the 8M plus 4M synthetic words above, with no fine-tuning on the ICDAR 2015 training set §4.3. The baseline is the same ResNet-based AN without FN, so the difference isolates FN §4.3. Accuracy is word accuracy. Tables are copied from the paper with the authors' notation; bold marks the best entry in each column in the paper.

Word accuracy on general benchmarks tab. 2

The paper's table labels some rows only by author name; two rows cite the same 2014 work and are kept in the order printed. The paper does not say which of the two is which.

Total normalized edit distance (NED), lower is better tab. 3. NED per image is edit distance divided by ground-truth length; the totals are summed over images §4.3.

FAN vs AN, both with the Shi et al. 2016 image encoder (unconstrained; no ResNet) tab. 4 tab. 5

Effect of λ (unconstrained) tab. 6 tab. 7

Effect of the share of pixel-labelled training samples (unconstrained) tab. 8 tab. 9

The 0% row equals the baseline row in tab. 2 and tab. 3.

Paper vs code

No official code. The paper gives no code link, and the source has no URL to a repository. The arXiv metadata held in the wiki has no comment field to check. GitHub searches for "focusing attention text recognition" and "attention drift text recognition" found no repository from the authors. "FAN scene text recognition" found only Lornatang/FAN-PyTorch, a third-party PyTorch reimplementation under Apache-2.0. It is not used as code for this card.

Pitfalls

• Focus loss normalisation is not stated. eq. 13 is a plain sum over steps and patch positions, with no division by batch size or count. An earlier version in the source had a 1/N factor and a count of pixel-labelled samples; that text is commented out §3.2. With λ = 0.01 the relative scale of the two losses depends on this choice, which the paper does not state.
• Patch size is not a number. P_H and P_W are "the max-size of ground-truth regions in the input image" §3.2. The paper gives no fixed value, and it does not say how variable-size patches are batched into one tensor.
• Centre location is undefined. eq. 8 uses location(j) for "the centre of the receptive field". The paper does not give a formula. The natural reading is the midpoint of eq. 7's bounds, but that is an inference.
• Crop source is ambiguous. Section 3.2 says the patch comes from "the input image or a convolution output". The implementation section says "we crop feature maps from the input image" §4.2. The paper does not say how image coordinates map to feature-map coordinates after the ResNet strides.
• FN parameter shapes are missing. R, S and b in eq. 11 have no dimensions. The softmax in eq. 12 over K needs e to have K entries, and no output projection is mentioned. Implementers must pick shapes that make this consistent.
• K is inconsistent. eq. 12 says K is "the number of label classes", and §4.2 gives 37 output units. Using 37 is the only reading that matches §4.2.
• Feature count and layer table. In the encoder table each layer spec lists width before height for kernel, stride and padding §4.2. The paper calls the network 32 layers §4.2; the layer count is not checked against the table here. The width of 65 follows from the pool and conv entries in the table under that reading.
• Pixel labels are a cost. About 30% of 4 million synthetic instances carry character positions §4.2. The sweep tab. 8 tab. 9 shows that 1% already gives most of the gain, but the 30% setting is what the headline numbers use. The 0% row matches the baseline, so the paper treats "no pixel labels" as "no focusing loss".
• λ was picked on the evaluation benchmarks. The λ sweep tab. 6 tab. 7 reports test-benchmark results and no separate validation split is named §4.4. The selected λ = 0.01 is therefore tuned on the reported test sets.
• Not every comparison is a win. Baidu IDL is higher on IC15 (77.0 vs 70.6) tab. 2, and Jaderberg et al. 2016 is higher on IC03 Full (98.6 vs 97.3). The paper states both exceptions §4.3.
• Lexicon scoring is underspecified. For constrained decoding the paper says it computes "conditional probability distributions for all lexicon words" and takes the highest §3.4. It does not say whether this is a sum of log-probabilities, nor how the cost grows with lexicon size.
• Dataset filtering. IC03 keeps 867 images after dropping non-alphanumeric labels and words under three characters §4.1. IC15 keeps 1811 of 2077 after dropping non-alphanumeric and irregular text §4.1. Compare numbers only on the same subsets.
• Throughput. About 90 samples per second and about 5 days for about 3 epochs on one M40 GPU §4.2. The paper gives no cost for other hardware.

Lineage

• Builds on: attention alignment of Bahdanau et al. (2014) §3.1; end-of-sequence handling following Sutskever et al. (2014) §3.1; the image encoder released by Shi et al. (2016, robust) §4.2 tab. 4; the end-to-end CNN-RNN work of Shi et al. (2016) §2; the ResNet design of He et al. (2016) §1.
• Related: Kim et al. (2016), joint CTC and attention for speech, where attention drift was also noted; the authors tried CTC with attention for scene text and dropped it §2.
• Compared against: 18 published methods plus a baseline without FN tab. 2.
• Superseded by: not stated in the card's sources.