AON: Towards Arbitrarily-Oriented Text Recognition

In plain words

Reading text in a photo is harder when the letters are not lined up horizontally: they may be tilted, seen at an angle, or bent around a curve. Most text readers assume the letters run from left to right, so they fail on such images.

This paper's idea is to look at the image in four ways at once: along the rows in both directions, and along the columns in both directions. A small learned network then decides, for each character position, which of those four views to trust. The combined features go to an attention-based decoder that writes out the letters one by one.

The model is trained only from images paired with the word they show. It needs no boxes around each character, which makes it cheaper to train than some earlier methods for irregular text. The authors report that it does well on curved and angled text and stays competitive on ordinary text.

TL;DR

• Input: a cropped word image; output: the character string, with optional lexicon constraint §3.5.
• Core change: four directional feature sequences, weighted per position by predicted "placement clues" and fused with a filter gate §3.2, §4.1, §4.2.
• Trained end-to-end from word-level labels only, with an attention LSTM decoder §3.4.
• Author-reported: best lexicon-free result on CUTE80 and ICDAR15 among the compared methods, and close to the field on regular benchmarks §5.3, §5.4, tab. 1, tab. 2.

Problem

Earlier attention-based recognisers read a text image as one left-to-right sequence of features §2. The authors report that such models trained directly on irregular text are hard to optimise because character placement varies §1. Rectifying the text first with a thin-plate-spline spatial transformer is the alternative they test, and they report that it is hard to train without geometric labels and does not rectify strongly oriented or curved text well §1, §5.3, fig. 5. The method by Yang et al. handles irregular text but, the authors state, needs character-level boxes §2, §5.3.

Method

The network has four parts §3, fig. 3:

1. A basal CNN of four convolution blocks produces a square feature map §3.1.
2. The arbitrary orientation network (AON) builds four sequences of features from that map and a set of placement clues §3.2, §4.1. The horizontal network (HN) pools the height to 1 with five shared convolution blocks, then runs a BiLSTM. Its reversed order gives the second horizontal sequence §4.1, eq. 6. The vertical network (VN) rotates the square map by 90 degrees and repeats the HN procedure §4.1, eq. 7. The horizontal and vertical branches share their convolution filters, which the authors found necessary for training §4.1, fig. 4. The character placement network (CN) predicts, for each of the L positions, four weights that sum to 1 [§4.1, eq. 8].
3. The filter gate (FG) combines the four sequences at each position with that position's four weights, then applies tanh §4.2, eq. 9, eq. 10.
4. An attention-based LSTM decoder predicts one character per step, ending with an end-of-sequence symbol §3.3, eq. 1–4.

Pseudocode of the forward pass and loss. The shapes are the reader's interpretation of the text, not values stated in the paper.

# AON forward pass and training loss  [§3] [§4]
# Input:  image (3, 100, 100); target characters y_1 .. y_M
# Output: negative log-likelihood loss

def aon_loss(image, target_chars):
    # Basal CNN: four conv blocks, square output map  [§3.1]
    base = basal_cnn(image)                              # (C, S, S)

    # Horizontal branch: shared convs pool height to 1, then BiLSTM  [§4.1]
    horizontal = bilstm(pool_height_to_one(shared_convs(base)))  # (L, D)

    # Reversed horizontal sequence: same features in reverse order  [eq. 6]
    horizontal_rev = reverse_order(horizontal)           # (L, D)

    # Vertical branch: rotate the square map 90 degrees, then same steps  [§4.1]
    rotated = rotate_90(base)                            # (C, S, S)
    vertical = bilstm(pool_height_to_one(shared_convs(rotated)))  # (L, D)

    # Reversed vertical sequence  [eq. 7]
    vertical_rev = reverse_order(vertical)               # (L, D)

    # Placement clues: four weights per position, summing to 1  [§4.1] [eq. 8]
    clues = placement_net(base)                          # (L, 4)

    # Filter gate: weight the four directions per position, then tanh  [§4.2]
    for each position i:
        mixed = stack_four(horizontal[i], horizontal_rev[i],
                           vertical[i], vertical_rev[i]) @ clues[i]
        fused[i] = tanh(mixed)                           # (D,)  [eq. 9, 10]

    # Attention decoder: one character per step, stop at EOS  [§3.3] [eq. 1–4]
    state = initial_state()
    loss = 0
    for t in range(len(target_chars)):
        # Attention weights from the previous decoder state  [eq. 4]
        weights = attend(state, fused)                   # (L,)
        # Glimpse: weighted sum of the fused features  [eq. 3]
        glimpse = weights @ fused                        # (D,)
        # Decoder recurrence  [eq. 2]
        state = lstm_step(previous_char, glimpse, state)
        # Character distribution over 37 symbols  [eq. 1] [§5.2]
        probs = softmax(output_layer(state))
        # Accumulate negative log-likelihood of the true character  [eq. 5]
        loss = loss - log(probs[target_chars[t]])
        previous_char = target_chars[t]

    return loss

Loss, as written eq. 5: the sum over output steps of the negative log probability of each ground-truth character. No normalisation by length is stated.

Decoding §3.5:

• Unconstrained: take the most probable character at each step.
• Constrained: score every word in the lexicon by its conditional probability and output the highest-scoring one. Lexicon sizes are 50, 1k and "full" tab. 1, tab. 2. The size of the "full" lexicon is not stated in the text.

Training §5.2: ADADELTA optimiser. Training data is 8 million synthetic images from Jaderberg et al. and 4 million from Gupta et al.'s set of 80 thousand scene images, both restricted to alphanumeric labels. Each image is rotated by a random angle between 0° and 360° once as augmentation.

Hyperparameters

Results

Author-reported. Setup: models trained on the synthetic data above, tested on the listed benchmarks, against Naive_base (the HN branch alone) and STN_base (HN with a TPS spatial transformer), both trained by the authors on similar data §5. Other rows are numbers the authors took from the literature, which used their own training data §5.3.

Irregular benchmarks tab. 1. Columns: SVT-Perspective (50 / full / none lexicon), CUTE80 (none), ICDAR15 (none). Accuracy in %.

Regular benchmarks tab. 2. Columns: IIIT5K (50 / 1k / none), SVT (50 / none), ICDAR03 (50 / full / none). Accuracy in %.

Headline reading of the tables, as stated by the authors:

• Irregular sets: AON is best on most columns, but not all. Yang et al. is higher on SVT-Perspective with no lexicon, and STN_base is higher on SVT-Perspective with the 50-word lexicon §5.3, tab. 1.
• Regular sets: the authors describe results as "comparable". They are behind Cheng et al. (FAN) on the three unconstrained columns, and behind Jaderberg et al. (2016) on ICDAR03 unconstrained §5.4.
• The authors say AON needs only word-level labels. FAN and Yang et al. use character-level boxes §5.3.

Further results: removing the convolutional placement network (CN) and concatenating the four sequences in a different way gives slower convergence and worse accuracy. Concatenating the horizontal and vertical sequences along the time axis gives results about 4% lower on average than AON §6. These are described in text only; no table is given.

Qualitative results: fig. 6 shows the learned placement clues and fig. 7 shows the placement trends derived from them, both on example images §5.5.

Paper vs code

No official code.

Search, in order:

1. The paper's source (LaTeX, raw/src/1711.04226) and bibliography contain no repository link. The abstract as stored in the metadata contains none either.
2. The paper links no project page.
3. GitHub repository search via the API, for "arbitrary orientation network text recognition" and "AON arbitrarily-oriented text recognition". The first returned no results. The second returned one repository, huizhang0110/AON (MIT), described as a TensorFlow implementation of this paper. Its owner is not an author of the paper, so it is a third-party reimplementation and is not used here.

No code is cited, so no discrepancy with code can be checked.

Pitfalls

• Equation and figure numbers. The equations eq. 1–eq. 10 and figures fig. 1–fig. 7 are numbered in the order they appear in the LaTeX source. The compiled PDF was not checked. Two equations share one LaTeX label (eq:py), so label-based cross-references in the source are unreliable.
• Missing layer table. The per-block channel counts and pooling strides of the convolution stacks are shown only in fig. 3, which is a diagram. Implementers need them and should read the figure.
• Shared convolutions and the 90° rotation. The horizontal and vertical branches share filters, and the vertical branch rotates a square map §4.1. The text requires the BCNN output to be square §3.1 but does not say what happens if it is not. Inputs are resized to 100 × 100 §5.2, so aspect ratio is not kept. The authors say they saw no clear harm on wide text §6, but this is not tested in a table.
• Reversal. The reversed sequences are the same features in reverse order eq. 6, eq. 7. The paper does not say whether they are computed by a separate pass.
• Placement clues. Each position's four weights must sum to 1 eq. 8. The paper does not state how this is enforced, for example by a softmax.
• Filter gate shape. The paper writes the gate as the four directional vectors, stacked, multiplied by the four clue weights eq. 9. The stacking axis and the resulting dimension are not spelled out. The implementation reading above is an interpretation.
• Attention scoring. The attention function is cited to a general reference and its scoring form is not stated eq. 4. Implementers must choose one, for example additive or dot-product.
• Teacher forcing. The paper does not say whether the previous character fed into the decoder during training is the ground truth or the model's own prediction.
• Optimiser settings. The learning rate, ADADELTA constants, batch size, number of iterations and weight decay are not stated in the published text. Do not copy values from other papers.
• Alphabet. The decoder has 37 outputs: 26 letters, 10 digits and EOS §5.2. Whether letters are case-folded is not stated, and training data was filtered to alphanumeric labels.
• Lexicon decoding cost. Constrained decoding scores every lexicon word §3.5. Cost grows with lexicon size, and the size of the "full" lexicon is not given.
• Comparison fairness. Most rows in the tables are numbers taken from other papers, trained on other data §5.3. Only Naive_base, STN_base and the AON model share training conditions.
• Abstract versus tables. The abstract and the text claim state-of-the-art on irregular data [abstract, §5.3], but tab. 1 shows STN_base ahead on one column and Yang et al. ahead on another. Read the tables, not the summary.
• Cost. The authors report AON has about twice the parameters and compute of Naive_base, and that the STN variant has about three times §6. Training throughput is about 190 samples per second on one Tesla P40 §5.2. These numbers are author-reported.
• Pretraining. Use of ImageNet or other pretrained weights is not stated.

Lineage

• Builds on: attention-based sequence decoding of Bahdanau et al. and the end-of-sequence convention of Sutskever et al. §3.3. Rectification with a thin-plate-spline spatial transformer, from Shi et al. (2016), is the alternative it compares against §2, §5.3.
• Compared against: ABBYY, Mishra et al., Wang et al., Phan et al., Shi et al. (two variants), Yang et al., Cheng et al. (FAN), Jaderberg et al., Lee et al., and the authors' own Naive_base and STN_base controls tab. 1, tab. 2.
• Superseded by: not stated in this wiki.