---
kind: concept
slug: optical-character-recognition
title: Optical character recognition
summary: >-
  Reading the text that appears in an image. Two strands: FAN, a scene-text recognizer whose focusing network pulls attention back onto each character, and OCRBench, a 1000-question zero-shot test of text reading in large multimodal models, scored by whether the ground-truth string appears in the reply. Every reported score is the authors' own.
tags: [ocr, text-recognition, scene-text-recognition, document-understanding, benchmarks, vision-language]
papers: ["1709.02054", "2305.07895"]
related: [multimodal-language-model, llm-evaluation, benchmark-validity, chart-data-extraction, attention-mechanism, convolutional-recurrent-network, residual-learning]
status: draft
updated_at: 2026-10-24
---

## Overview

Optical character recognition (OCR) is the task of reading the text that
appears in an image, whether it is a printed line, a scene sign, a receipt or
a handwritten formula. The sources here show two strands. One builds a
recognizer that reads word-sized crops of natural scenes [arxiv:1709.02054 §1].
The other treats text reading as one skill that large multimodal models (LMMs)
must show when they answer questions about pictures, and measures it
[arxiv:2305.07895 §1].

The scene-text paper names a failure it calls attention drift: an
attention-based reader looks at a neighbouring patch and emits the wrong
character [arxiv:1709.02054 §1]. The LMM paper notes that such models could
read text without training on OCR data, yet it was unclear how well they did
so, which is the gap its benchmark targets [arxiv:2305.07895 §1].

The one invariant shared by the evaluation here: a reply is correct when the
ground-truth string appears somewhere in it [arxiv:2305.07895 §3.1].

## Variants

The first variant is a system you can train. The rest measure reading ability
and do not compete with it: they are task families and a diagnostic probe
inside one benchmark.

### Attention with a focusing network (FAN)

A scene-text recognizer with three parts: a ResNet-style encoder followed by
a bidirectional LSTM, an attention decoder that emits one character per step,
and a focusing network that is trained only to check where the attention
falls [arxiv:1709.02054 §3] [arxiv:1709.02054 §4.2]. The focusing network
predicts a label for each position in a patch around the attention centre, and
its loss applies only to images that carry character positions
[arxiv:1709.02054 §3.2]. The paper trains on 8 million synthetic words without
positions and 4 million with them, about 30% of the set [arxiv:1709.02054 §4.2].
On IIIT5k with no lexicon, the same model without the focusing branch reaches
83.7% word accuracy and with it 87.4% [arxiv:1709.02054 tab. 2]. Card:
[Focusing Attention: Towards Accurate Text Recognition in Natural Images](/papers/1709.02054).

### Benchmark task families (OCRBench)

The benchmark has 1000 question-answer pairs in five families: text
recognition (300 items), scene text-centric VQA (200), document-oriented VQA
(200), key information extraction (200) and handwritten maths (100)
[arxiv:2305.07895 §3.3] [arxiv:2305.07895 tab. 5]. Card:
[OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](/papers/2305.07895).

### Semantic-reliance probe

To test whether a model reads letters or guesses from the word's shape, the
authors pair a set of real dictionary words with the same words with their
characters shuffled [arxiv:2305.07895 §3.1]. The paper uses this contrast to
ask whether models read letters or guess words, and it labels its own
explanation of the effect as not conclusive [arxiv:2305.07895 §3.3]. The full set is
described in the paper, but the released benchmark file holds only a small
part of it (checked against the released code, not the paper), so confirm the
data you have before using it as a probe.

## When to use which

| Situation | Use | Why |
|---|---|---|
| You must train a recognizer for cropped scene-text words, and you can label some character positions | Attention with a focusing network (FAN) | The focusing loss needs character positions [arxiv:1709.02054 §3.2]. Labelling about 1% of the synthetic set already recovers most of the gain (IIIT5k 86.2%, against 83.7% without the branch and 87.4% at 30%) [arxiv:1709.02054 tab. 8] [arxiv:1709.02054 tab. 9] |
| You can label no character positions at all | The attention baseline (FAN without its focusing branch) | The paper's 0% row is the baseline, so it gives no result for the branch without labels [arxiv:1709.02054 tab. 8] |
| The words come from a known list, such as a closed vocabulary | FAN with lexicon decoding (50, 1k or full lexicon) | The paper scores lexicon words with the highest conditional probability [arxiv:1709.02054 §3.4]; its constrained results are in [arxiv:1709.02054 tab. 2] |
| You want one score for how well an LMM reads text, zero-shot, with free-text answers | Benchmark task families (OCRBench) | The protocol is zero-shot and free-text, with one rule for every family [arxiv:2305.07895 §3.1] |
| You want to know whether an LMM reads each character or guesses the word | Semantic-reliance probe | The probe is built to separate the two cases [arxiv:2305.07895 §3.1] |
| You need to compare LMMs with specialised OCR systems | Use the supervised reference numbers only as a loose ceiling | Those systems are trained on each dataset's training data, while LMMs are zero-shot, so the comparison is not like-for-like [arxiv:2305.07895 app. C] |
| You test handwritten maths | HMER family, prompt for LaTeX | The paper asks for LaTeX output [arxiv:2305.07895 §3.1]; in its table, only Gemini (8) and GPT4V (9) score above 0 of 100 [arxiv:2305.07895 tab. 3] |
| You need the scores to hold for a model you can only reach through an API | Treat the API snapshot as a dated result | The paper names model snapshots such as `gpt-4-vision-preview` [arxiv:2305.07895 tab. 6]; a later rerun may not match (inferred, not stated by the paper) |
| Your input is full pages or documents rather than scene-text crops | No card covers a document OCR system | FAN is evaluated on word images [arxiv:1709.02054 §4.1], and OCRBench only scores final answers. Document VQA is a task family there, not a method [arxiv:2305.07895 §3.3] |

## Implementation sketch

Two short sketches. The first is the scoring rule from the benchmark. The paper
defines only the substring check; the released scorer also normalises case and,
for one family, removes spaces, so a reimplementation must choose whether to
follow the paper or the code [arxiv:2305.07895 §3.1].

```pseudocode
# Score one item: 1 if an accepted answer occurs in the reply, else 0
# Input:  answers (list of strings), reply (string or None)
# Output: 1 or 0

def score_item(answers, reply):
    # A missing reply is never correct
    if reply is None:
        return 0
    # Any accepted alternative answer is enough
    for answer in answers:
        # Substring match: extra words around the answer still pass
        if answer in reply:
            return 1
    return 0

# Final score: the sum over all items, so the maximum is the item count
# Input:  items (list of answers and reply pairs)
# Output: total (0 to 1000 for OCRBench)

def final_score(items):
    total = 0
    for answers, reply in items:
        total += score_item(answers, reply)
    return total
```

The second is the joint training loss of the focusing variant. The paper
weights the attention loss and the focusing loss with one factor λ, and
λ = 0 switches the focusing term off [arxiv:1709.02054 §3.3] [arxiv:1709.02054 tab. 6].
The paper does not state how the focusing loss is normalised, so the scale of
the second term relative to the first is open (see Pitfalls).

```pseudocode
# Joint loss for a recognizer with a focusing branch [arxiv:1709.02054 §3.3]
# Input:  att_loss (scalar), focus_loss (scalar), lam (float, 0 to < 1)
# Output: total loss (scalar)

def fan_total_loss(att_loss, focus_loss, lam):
    # Weight the attention term by (1 - lam) and the focusing term by lam;
    # the paper's selected value is 0.01 [arxiv:1709.02054 §4.2]
    return (1 - lam) * att_loss + lam * focus_loss
```

## Contradictions

Across the two papers, no metric and benchmark pair appears in both, so no two
numbers conflict. They measure different things: FAN reports word accuracy for
a recognizer trained on synthetic words [arxiv:1709.02054 §4.3], and OCRBench
reports zero-shot final scores for LMMs [arxiv:2305.07895 tab. 3]. OCRBench's
supervised references, such as a PARSeq-style system for each dataset, are
a different protocol [arxiv:2305.07895 app. C]. The cards do not tie those
references to FAN, so do not read them as FAN's numbers [arxiv:2305.07895 tab. 1].

Within OCRBench, three points disagree or do not line up. The Table 2 caption
gives the ESTVQA supervised reference as 42.3, but its cells show 43.3
[arxiv:2305.07895 tab. 2]. The text gives Monkey's input as 1344×896, while
the resolution table lists 896 [arxiv:2305.07895 §3.2] [arxiv:2305.07895
tab. 6]. The paper evaluates 14 LMMs, but no single table reports all 14
[arxiv:2305.07895 §3.2] [arxiv:2305.07895 tab. 1] [arxiv:2305.07895 tab. 3].

## Pitfalls

- **Focusing loss scale is not stated.** The focusing loss is written as a plain
  sum over steps and patch positions, with no division by count [arxiv:1709.02054
  eq. 13]. A commented-out passage in the paper's source had a 1/N factor
  [arxiv:1709.02054 §3.2]. With λ = 0.01, the balance between the two terms
  depends on this choice.
- **Patch size is not a number.** The patch is sized by the largest
  ground-truth region in the image, with no fixed value and no account of how
  patches of different sizes are batched [arxiv:1709.02054 §3.2].
- **Centre location and crop source are not fully defined.** The receptive-field
  centre is named but not given a formula [arxiv:1709.02054 eq. 8]. The text
  says the patch comes from the input image or a convolution output, and the
  implementation section says feature maps; the mapping between the two is not
  stated [arxiv:1709.02054 §3.2] [arxiv:1709.02054 §4.2].
- **Focusing-head shapes are missing.** The parameters in the scoring head have
  no dimensions, and no output projection is named [arxiv:1709.02054 eq. 11].
- **Class count is ambiguous.** The paper calls K "the number of label classes"
  and gives 37 output units. Using 37 is the only reading that matches the
  implementation section [arxiv:1709.02054 eq. 12] [arxiv:1709.02054 §4.2].
- **λ was chosen on the test sets.** The λ sweep reports results on the
  evaluation benchmarks, and no separate validation split is named, so the
  selected value is tuned on the reported test data [arxiv:1709.02054 §4.4].
- **Not every comparison is a win.** Baidu IDL is higher on IC15 (77.0 against
  70.6), and Jaderberg et al. 2016 is higher on IC03 Full (98.6 against 97.3)
  [arxiv:1709.02054 tab. 2] [arxiv:1709.02054 §4.3].
- **Lexicon scoring is underspecified.** The paper picks the lexicon word with
  the highest conditional probability, but does not say whether this is a sum of
  log-probabilities, nor how cost grows with lexicon size [arxiv:1709.02054 §3.4].
- **Filtered datasets.** IC03 keeps 867 images and IC15 keeps 1811 of 2077
  after filtering [arxiv:1709.02054 §4.1]. Compare numbers only on the same subsets.
- **No official FAN code.** The paper gives no repository. Any implementation
  is written from the paper, so the gaps above must be filled by the implementer
  [arxiv:1709.02054 §4.2].
- **Cost.** About 3 epochs took about 5 days on one M40 GPU, at about 90 samples
  per second [arxiv:1709.02054 §4.2]. The paper gives no figures for other hardware.
- **Substring scoring.** A long reply that contains the right string passes
  the check, even when the surrounding words make it wrong
  [arxiv:2305.07895 §3.1] (the effect follows from the rule; the paper does not discuss it). The four-symbol
  answer filter reduces false positives but does not remove this effect
  [arxiv:2305.07895 §3.1].
- **Format mismatches score as wrong.** The paper reports Gemini converting
  "02/02/2018" to "2 February 2018" even when asked to copy the text. The
  rule does no date normalisation, so this counts as an error
  [arxiv:2305.07895 §3.3].
- **Refusals score as wrong.** GPT4V refused 84 images on safety grounds, and
  those count as 0 [arxiv:2305.07895 §3.3].
- **Settings not stated.** The paper does not give per-model decoding or
  token limits [arxiv:2305.07895 §3.2]. Compare models only when their
  settings match, and record them.
- **Author-reported numbers.** Every score is the authors' own run under
  their prompts and settings. None has been independently reproduced here.

## Papers

- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](/papers/1709.02054): an attention-based scene-text recognizer plus a focusing network that pulls each character's attention back onto the character.
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](/papers/2305.07895): a 1000-question zero-shot benchmark and substring scoring rule for reading text in large multimodal models, with results for 14 models.