OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models

In plain words

Large multimodal models read pictures and answer in text. Many can describe a scene, but reading the words inside a photo, a document or a receipt is a separate skill, and the older tests for it were slow to run or were not built for free-form answers. This paper builds a small, checked test called OCRBench: 1000 questions across five jobs. The jobs are reading text, answering questions about text in photos, answering questions about documents and charts, pulling out fields such as a receipt total, and reading handwritten maths.

The scoring is deliberately simple. A model answers in its own words, and the answer counts as correct if the right string appears somewhere in that answer. Run on 14 models, the two strongest, Gemini and GPT4V, still struggle with handwritten maths, blurry images and text that does not form real words. The numbers come from the authors' own runs, so treat them as a snapshot.

TL;DR

OCRBench is a 1000-item benchmark that tests OCR-style reading in large multimodal models across five tasks. Each item is scored 1 if a ground-truth string appears in the model's reply, and 0 otherwise, and the final score is the sum out of 1000. The paper reports zero-shot results for 14 models, with Gemini (659) and GPT4V (645) highest. Every number is author-reported.

Problem

Large multimodal models (LMMs) were reported to read text in images without training on OCR data §1. Existing OCR evaluations did not fit this use well. Exact-match and ANLS scoring are unsuitable when a model writes explanatory text around the answer §3.1. True/False and multiple-choice questions cannot test whether a model recognises each word §2.2. Running many datasets takes a long time, and some of their annotations are inaccurate §3.3. The paper assumes zero-shot evaluation with free-text replies §3.1.

Method

Benchmark composition. OCRBench has 1000 question-answer pairs over five tasks §3.3. Text Recognition has 300 items, 50 each from six sets: regular, irregular, artistic, handwriting, digit strings and non-semantic text app. A tab. 5. Scene Text-centric VQA has 200 items, 50 each from STVQA, TextVQA, ESTVQA (English) and OCRVQA app. A tab. 5. Document-oriented VQA has 200 items, 50 each from DocVQA, ChartQA (augmented), ChartQA (human) and InfoVQA app. A tab. 5. Key Information Extraction has 200 items: 67 SROIE, 66 FUNSD and 67 POIE app. A tab. 5. Handwritten maths has 100 items from HME100k app. A tab. 5. The paper does not say how the 1000 items were chosen from the source sets.

Filtering and checking. Answers with fewer than four symbols are removed from every dataset to reduce false positives §3.1. The authors then checked all 1000 answers by hand and corrected wrong ones. Where more than one answer is correct, they added the alternatives §3.3.

Prompts. Each item stores its own question text. English words use "What is written in the image", Chinese words "What are the Chinese characters in the image", and digit strings "What is the number in the image?" §3.1. Handwritten maths asks for LaTeX output §3.1. For key information extraction, the question is followed by "Answer this question using the text in the image directly." so the answer format is restricted §3.3.

Evaluation. Each model gets one image and one question and returns free text. The same rule scores every dataset: the ground truth must appear in the reply §3.1. The released code adds two details the paper does not state, a case-insensitive match and a space-free match for HME100k. The pseudocode below follows the released scorer.

# Normalise text before matching  [code: OCRBench/example.py#L128-L151]
# Input:  text (string), dataset_name (string)
# Output: normalised string

def normalise(text, dataset_name):
    # Strip the ends and turn newlines into spaces  [code: example.py#L143]
    text = text.strip().replace("\n", " ")
    if dataset_name == "HME100k":
        # HME100k: also drop every space  [code: example.py#L131-L132]
        return text.replace(" ", "")
    # Other datasets: compare case-insensitively  [code: example.py#L143-L144]
    return text.lower()

# Score one record, 1 if an accepted answer occurs in the reply
# [code: OCRBench/example.py#L124-L151]
# Input:  answers (string or list), predict (string or None)
# Output: 1 (correct) or 0 (wrong)

def score_record(answers, predict, dataset_name):
    # A record without a reply is never scored and adds 0
    # [code: OCRBench/example.py#L124-L125]
    if predict is None:
        return 0
    # Accept a single string or a list of alternative answers
    # [code: OCRBench/example.py#L129]
    accepted = answers if isinstance(answers, list) else [answers]
    reply = normalise(predict, dataset_name)
    for answer in accepted:
        # Substring match: extra words in the reply still count
        # [code: OCRBench/example.py#L141-L151]
        if normalise(answer, dataset_name) in reply:
            return 1
    return 0

# Final Score: the sum of the 1000 binary results
# [code: OCRBench/example.py#L153-L159]
# Input:  records (the 1000 scored items)
# Output: final_score (0 to 1000)

def final_score(records):
    # Add each record's result to the total
    # [code: OCRBench/example.py#L157]
    total = 0
    for record in records:
        total += score_record(
            record["answers"],
            record.get("predict"),
            record["dataset_name"],
        )
    return total

Semantic-reliance probe. To test whether LMMs read letters or guess words, the authors built a semantic set (ST, 3000 images of IIIT5K dictionary words) and a non-semantic set (NST, the same words with characters shuffled) §3.1. The OCRBench file holds 50 NST items, not the 3000-image set [code: OCRBench/OCRBench/OCRBench.json].

Scoring total. The Final Score is the sum of the 1000 results, so its maximum is 1000 §3.3 tab. 3.

Supervised references. For each dataset the paper names a supervised state-of-the-art system for comparison, such as PARSeq trained on the ST and MJ synthetic sets and tested on the real datasets app. C.

Hyperparameters

The paper gives few numeric settings. The rows below combine what it states with what the released scripts use, and say which is which.

Results

Author-reported. Setup: zero-shot, 14 LMMs in total, 1000-item OCRBench for Table 3 and the wider test sets for Tables 1 and 2 §3.2 §3.3. The paper gives no supervised baseline for Table 3; the supervised references are for Tables 1 and 2 only app. C.

Headline (OCRBench, Final Score out of 1000) tab. 3. Gemini 659, GPT4V 645, Monkey 514, and MiniGPT4V2 157 lowest of the twelve rows. GPT4V refused to answer 84 images on safety grounds §3.3; the scorer counts those as wrong.

Source: tab. 3. Recog. = text recognition (out of 300); VQA^S = scene text-centric VQA (out of 200); VQA^D = document-oriented VQA (out of 200); KIE (out of 200); HMER (out of 100).

Text recognition, accuracy in % tab. 1. Supervised SOTA row is the reference from app. C. Bold and blue highlights are dropped here.

Source: tab. 1. Columns: IIIT5K, SVT, IC13, IC15, SVTP, CT80, COCO-Text, CTW, TT, HOST and WOST (occluded), WordArt, IAM, ReCTS, ORAND, NST and ST.

Scene text, document, KIE and HMER, accuracy in % tab. 2. Results are author-reported.

Source: tab. 2. The Supervised-SOTA ESTVQA cells show 43.3, but the caption says the reference average is 42.3 (see Pitfalls).

Paper vs code

Repository: Yuliang-Liu/MultimodalOCR, MIT licence, commit 4e363b3f499811b43d792ef0bb33932ef05eca27. The owner is the first author, and the abstract links the repository. The code is for OCRBench v1 only, under OCRBench/. The repo also has OCRBench_v2/ and MDPBench/, which this card does not cover.

• Matching rule. The paper states only "the ground truth is present in the output" §3.1. The scorer also lowercases non-HME text and removes every space from HME100k text before matching example.py. Those details change which replies pass.
• Missing replies. The scorer skips records with no predict field, so they add 0 and are not reported [code: OCRBench/example.py#L124-L125, L155-L156]. The Gemini script prints the failure and moves on [code: OCRBench/scripts/Genimi.py#L56-L57]. The paper says nothing about this case.
• GPT4V refusals. The paper reports 84 refused images §3.3. The scorer has no refusal handling, so a refusal scores 0 [inferred from code].
• Prompt wording. The paper quotes "What is written in the image" §3.1. The stored question is "what is written in the image?", lowercase with a question mark OCRBench.json.
• Dataset counts. Per-task counts match the paper: 50 per recognition set, 200 each for the VQA and KIE tasks, 100 for HMER tab. 5. The KIE per-dataset counts also match (SROIE 67, FUNSD 66, POIE 67). No discrepancy.
• Dataset count. The abstract says 29 datasets [code: OCRBench/README.md#L4]. The 1000-item file lists 25 distinct dataset names OCRBench.json. The paper does not explain the gap.
• Monkey resolution. The text says 1344×896 §3.2; the table says 896 tab. 6. The script does not set a resolution [grep: no resolution setting in OCRBench/scripts/monkey.py]. The paper is inconsistent.
• Results files. The Table 3 numbers come from result files that the README links on Google Drive README.md. They are not in the repository, so the table cannot be re-derived from it.
• Example script. OCRBench/example.py has its model call commented out, and line 84 assigns an undefined response example.py. Run a per-model script instead, such as OCRBench/scripts/GPT4V.py.

Pitfalls

• Substring scoring. A reply passes if the answer appears anywhere in it, so a long answer that contains the right string, or a negated one, counts as correct [inferred from code: OCRBench/example.py#L141-L151]. The four-symbol filter §3.1 reduces false positives, but it does not remove this effect.
• Format mismatches score as wrong. The paper reports that Gemini reads "02/02/2018" as "2 February 2018" even when told to copy the text §3.3. The rule has no date normalisation, so this counts as an error.
• Token caps can cut answers. A long reply, such as a HMER LaTeX answer, can be cut off at the script's max_new_tokens or max_tokens. These caps differ between scripts [code: OCRBench/scripts/monkey.py#L70, OCRBench/scripts/GPT4V.py#L82]. The paper does not state them. Compare models only when the caps match.
• Decoding is not uniform. Most scripts decode greedily, but the BLIP2 Vicuna variant uses 5 beams blip2_vicuna_instruct.py. The mPLUG-DocOwl15 script sets temperature=1.0 while do_sample=False. In common generate() implementations the temperature then has no effect [code: OCRBench/scripts/mPLUG-DocOwl15.py#L185-L187] [general knowledge; not checked against this repo's pinned library version].
• Time-bound API models. The GPT4V and Gemini entries name gpt-4-vision-preview and gemini-pro-vision tab. 6. Those snapshots may no longer be served, so a rerun may not reproduce Table 3 [inferred].
• Two different model sets. The paper evaluates 14 LMMs §1. Table 3 covers 12 of them and omits UniDoc and DocPedia; Tables 1 and 2 omit Gemini, GPT4V, LLaVA1.5-13B and MiniGPT4V2 tab. 1 tab. 2 tab. 3. No table compares all 14.
• Reference inconsistency. Table 2's caption gives the ESTVQA supervised reference as 42.3, but the cells show 43.3 for both EN and CN tab. 2.
• Supervised references differ in protocol. Supervised systems are trained on each dataset's training data or on synthetic sets app. C. LMMs are zero-shot. The gap is not a like-for-like comparison.
• Authors' explanation is open. The paper's account of why LMMs read text is labelled as not conclusive §3.3. Do not treat it as established.
• Author-reported. Every number is the authors' run under their prompts and caps. Nothing here has been independently reproduced.

Lineage

• Builds on: the public OCR datasets it draws from, such as IIIT5K, SVT, TextVQA and DocVQA §3.1. It does not describe a new model.
• Compared against: the 14 LMMs in its tables, including LLaVA, BLIP2, InstructBLIP, mPLUG-Owl, Monkey and Gemini and GPT4V §3.2 tab. 3. It also cites VLMEvalKit and lmms-eval as other evaluation systems §2.2. The supervised reference per dataset is in app. C.
• Superseded by: not stated in this card's sources. The repository's OCRBench_v2/ folder is a separate release that this card does not cover.