Optical character recognition

Overview

Optical character recognition (OCR) is the task of reading the text that appears in an image, whether it is a printed line, a scene sign, a receipt or a handwritten formula. The sources here show two strands. One builds a recognizer that reads word-sized crops of natural scenes 1709.02054 §1. The other treats text reading as one skill that large multimodal models (LMMs) must show when they answer questions about pictures, and measures it 2305.07895 §1.

The scene-text paper names a failure it calls attention drift: an attention-based reader looks at a neighbouring patch and emits the wrong character 1709.02054 §1. The LMM paper notes that such models could read text without training on OCR data, yet it was unclear how well they did so, which is the gap its benchmark targets 2305.07895 §1.

The one invariant shared by the evaluation here: a reply is correct when the ground-truth string appears somewhere in it 2305.07895 §3.1.

Variants

The first variant is a system you can train. The rest measure reading ability and do not compete with it: they are task families and a diagnostic probe inside one benchmark.

Attention with a focusing network (FAN)

A scene-text recognizer with three parts: a ResNet-style encoder followed by a bidirectional LSTM, an attention decoder that emits one character per step, and a focusing network that is trained only to check where the attention falls 1709.02054 §3 1709.02054 §4.2. The focusing network predicts a label for each position in a patch around the attention centre, and its loss applies only to images that carry character positions 1709.02054 §3.2. The paper trains on 8 million synthetic words without positions and 4 million with them, about 30% of the set 1709.02054 §4.2. On IIIT5k with no lexicon, the same model without the focusing branch reaches 83.7% word accuracy and with it 87.4% 1709.02054 tab. 2. Card: Focusing Attention: Towards Accurate Text Recognition in Natural Images.

Benchmark task families (OCRBench)

The benchmark has 1000 question-answer pairs in five families: text recognition (300 items), scene text-centric VQA (200), document-oriented VQA (200), key information extraction (200) and handwritten maths (100) 2305.07895 §3.3 2305.07895 tab. 5. Card: OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models.

Semantic-reliance probe

To test whether a model reads letters or guesses from the word's shape, the authors pair a set of real dictionary words with the same words with their characters shuffled 2305.07895 §3.1. The paper uses this contrast to ask whether models read letters or guess words, and it labels its own explanation of the effect as not conclusive 2305.07895 §3.3. The full set is described in the paper, but the released benchmark file holds only a small part of it (checked against the released code, not the paper), so confirm the data you have before using it as a probe.

When to use which

Implementation sketch

Two short sketches. The first is the scoring rule from the benchmark. The paper defines only the substring check; the released scorer also normalises case and, for one family, removes spaces, so a reimplementation must choose whether to follow the paper or the code 2305.07895 §3.1.

# Score one item: 1 if an accepted answer occurs in the reply, else 0
# Input:  answers (list of strings), reply (string or None)
# Output: 1 or 0

def score_item(answers, reply):
    # A missing reply is never correct
    if reply is None:
        return 0
    # Any accepted alternative answer is enough
    for answer in answers:
        # Substring match: extra words around the answer still pass
        if answer in reply:
            return 1
    return 0

# Final score: the sum over all items, so the maximum is the item count
# Input:  items (list of answers and reply pairs)
# Output: total (0 to 1000 for OCRBench)

def final_score(items):
    total = 0
    for answers, reply in items:
        total += score_item(answers, reply)
    return total

The second is the joint training loss of the focusing variant. The paper weights the attention loss and the focusing loss with one factor λ, and λ = 0 switches the focusing term off 1709.02054 §3.3 1709.02054 tab. 6. The paper does not state how the focusing loss is normalised, so the scale of the second term relative to the first is open (see Pitfalls).

# Joint loss for a recognizer with a focusing branch [arxiv:1709.02054 §3.3]
# Input:  att_loss (scalar), focus_loss (scalar), lam (float, 0 to < 1)
# Output: total loss (scalar)

def fan_total_loss(att_loss, focus_loss, lam):
    # Weight the attention term by (1 - lam) and the focusing term by lam;
    # the paper's selected value is 0.01 [arxiv:1709.02054 §4.2]
    return (1 - lam) * att_loss + lam * focus_loss

Contradictions

Across the two papers, no metric and benchmark pair appears in both, so no two numbers conflict. They measure different things: FAN reports word accuracy for a recognizer trained on synthetic words 1709.02054 §4.3, and OCRBench reports zero-shot final scores for LMMs 2305.07895 tab. 3. OCRBench's supervised references, such as a PARSeq-style system for each dataset, are a different protocol 2305.07895 app. C. The cards do not tie those references to FAN, so do not read them as FAN's numbers 2305.07895 tab. 1.

Within OCRBench, three points disagree or do not line up. The Table 2 caption gives the ESTVQA supervised reference as 42.3, but its cells show 43.3 2305.07895 tab. 2. The text gives Monkey's input as 1344×896, while the resolution table lists 896 2305.07895 §3.2 [arxiv:2305.07895 tab. 6]. The paper evaluates 14 LMMs, but no single table reports all 14 2305.07895 §3.2 2305.07895 tab. 1 2305.07895 tab. 3.

Pitfalls

• Focusing loss scale is not stated. The focusing loss is written as a plain sum over steps and patch positions, with no division by count [arxiv:1709.02054 eq. 13]. A commented-out passage in the paper's source had a 1/N factor 1709.02054 §3.2. With λ = 0.01, the balance between the two terms depends on this choice.
• Patch size is not a number. The patch is sized by the largest ground-truth region in the image, with no fixed value and no account of how patches of different sizes are batched 1709.02054 §3.2.
• Centre location and crop source are not fully defined. The receptive-field centre is named but not given a formula 1709.02054 eq. 8. The text says the patch comes from the input image or a convolution output, and the implementation section says feature maps; the mapping between the two is not stated 1709.02054 §3.2 1709.02054 §4.2.
• Focusing-head shapes are missing. The parameters in the scoring head have no dimensions, and no output projection is named 1709.02054 eq. 11.
• Class count is ambiguous. The paper calls K "the number of label classes" and gives 37 output units. Using 37 is the only reading that matches the implementation section 1709.02054 eq. 12 1709.02054 §4.2.
• λ was chosen on the test sets. The λ sweep reports results on the evaluation benchmarks, and no separate validation split is named, so the selected value is tuned on the reported test data 1709.02054 §4.4.
• Not every comparison is a win. Baidu IDL is higher on IC15 (77.0 against 70.6), and Jaderberg et al. 2016 is higher on IC03 Full (98.6 against 97.3) 1709.02054 tab. 2 1709.02054 §4.3.
• Lexicon scoring is underspecified. The paper picks the lexicon word with the highest conditional probability, but does not say whether this is a sum of log-probabilities, nor how cost grows with lexicon size 1709.02054 §3.4.
• Filtered datasets. IC03 keeps 867 images and IC15 keeps 1811 of 2077 after filtering 1709.02054 §4.1. Compare numbers only on the same subsets.
• No official FAN code. The paper gives no repository. Any implementation is written from the paper, so the gaps above must be filled by the implementer 1709.02054 §4.2.
• Cost. About 3 epochs took about 5 days on one M40 GPU, at about 90 samples per second 1709.02054 §4.2. The paper gives no figures for other hardware.
• Substring scoring. A long reply that contains the right string passes the check, even when the surrounding words make it wrong 2305.07895 §3.1 (the effect follows from the rule; the paper does not discuss it). The four-symbol answer filter reduces false positives but does not remove this effect 2305.07895 §3.1.
• Format mismatches score as wrong. The paper reports Gemini converting "02/02/2018" to "2 February 2018" even when asked to copy the text. The rule does no date normalisation, so this counts as an error 2305.07895 §3.3.
• Refusals score as wrong. GPT4V refused 84 images on safety grounds, and those count as 0 2305.07895 §3.3.
• Settings not stated. The paper does not give per-model decoding or token limits 2305.07895 §3.2. Compare models only when their settings match, and record them.
• Author-reported numbers. Every score is the authors' own run under their prompts and settings. None has been independently reproduced here.

Papers

• Focusing Attention: Towards Accurate Text Recognition in Natural Images: an attention-based scene-text recognizer plus a focusing network that pulls each character's attention back onto the character.
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models: a 1000-question zero-shot benchmark and substring scoring rule for reading text in large multimodal models, with results for 14 models.