ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction

In plain words

Reading a shop receipt by machine has two steps. First the system finds where the text is and reads it, which is OCR. Then it pulls out the few values that matter, such as the shop name, the date, the address and the total. This report describes a public contest that tested both steps on a new set of scanned receipts.

The organisers built a dataset of 1000 receipt images, some of them poor scans with folded paper or low resolution, and set three tasks. Task 1 asks for the location of every text line or word. Task 2 asks for the words that appear in the receipt, with no locations. Task 3 asks for the values of a few named fields, saved as a structured file. Each task was scored with a single F1 number, so entries could be ranked.

The report says little about how the entries worked. It describes the top three entries for each task in a sentence or two, and many of the strongest entries combined several models. The authors' main message is that detecting and reading text on receipts works well, while pulling out the right fields is still far from solved.

TL;DR

• A competition report, not a method: it defines three tasks, a 1000-image dataset and the scoring rules, then ranks the submissions §I §II §III.
• Dataset: 1000 scanned receipts, 600 trainval (with annotations) and 400 test §II.
• Ranking metric is F1 for every task §III-A2 §III-B2 §III-C2.
• Author-reported top scores: Task 1 Hmean 98.59% fig. 3, Task 2 Hmean 96.43% fig. 4, Task 3 Hmean 90.49% fig. 5.
• The authors conclude that Task 3 is the open problem: only one entry exceeds 90% and many fall below 80% §VI.

Problem

Receipt OCR needs higher accuracy than general OCR for commercial use, and it gets harder with poor-quality scans §I. Receipts also contain many unneeded texts in complex layouts, long lines and small fonts §I. The authors say no existing research or past ICDAR competition fully covers these problems, and that named-entity methods do not transfer directly to extracting key fields from receipts §I. The competition was set up to create a public benchmark and draw research attention to this problem §I §VI.

Method

There is no model in this report. The contribution is the dataset, the three task definitions and the evaluation protocols.

Dataset and annotations §II

1. 1000 whole scanned receipt images are used for all three tasks, with different annotations for each §II.
2. Each receipt carries roughly four target fields, for example an item name, a unit price and a total. The text is mainly digits and English characters §II. The paper's own figure with example receipts is fig. 2, although the prose says "Figure 1" §II.
3. Split: 600 "trainval" images with annotations released to participants; 400 "test" images released a few weeks before the submission deadline §II.
4. Task 1 and Task 2 annotations: one line per text box, with four corner points given clockwise starting from the top, followed by the transcript. The file has the same name as the image §II fig. 1(a).
5. Task 3 annotations: one JSON-like file per image holding key-value pairs. The example in fig. 1(b) has four keys: company, date, address and total. The prose does not list the required fields §II §V-C.
6. Sensitive fields such as name, address and phone number were blurred for privacy §I.

Task 1: scanned receipt text localisation §III-A

1. Task: localise every text item with four vertices. Ground truth is at least word level. Results are submitted as a zip file for all test images §III-A1.
2. Evaluation: a DetVal-based methodology that partly handles one-to-many and many-to-one matches. The report computes mean average precision and average recall, then F1, and ranks by F1 §III-A2. The matching thresholds and the exact computation are not stated.

Task 2: scanned receipt OCR §III-B

1. Task: list the words recognised in each image, with no locations. Only Latin letters and digits are used §III-B1.
2. Ground truth: split every transcription on spaces. "Date: 12/3/56" gives "Date:" and "12/3/56". "Date: 12 / 3 / 56" gives "Date:", "12", "/", "3", "/", "56" §III-B1.
3. Evaluation: predicted words are matched to ground-truth words. Repeated words must be repeated in the submission. Precision is correct matches over detected words, recall is correct matches over ground-truth words, and F1 is the ranking metric §III-B2.

# Task 2 word-level F1  [§III-B2]
# Input:  predicted, truth: per image, a list of word strings
# Output: F1 over the whole test set

def task2_f1(predicted, truth):
    # Count correct matches, detected words and truth words  [§III-B2]
    correct = 0
    detected = 0
    truth_count = 0
    for each image i:
        # Repeated words count as repeated matches  [§III-B2]
        correct += multiset_matches(predicted[i], truth[i])
        detected += length(predicted[i])
        truth_count += length(truth[i])
    precision = correct / detected
    recall = correct / truth_count
    # F1 is the ranking metric  [§III-B2]
    return 2 * precision * recall / (precision + recall)

The paper does not state the word-matching criterion (exact string match or otherwise) or whether scores are summed over the set or averaged per image. The multiset_matches and aggregation lines above are an interpretation of the "repeated words" rule, not something the paper specifies.

Task 3: key information extraction §III-C

1. Task: extract the text of named key fields from each receipt and save it as a JSON file per image. Results are submitted as a zip file §III-C1.
2. Correctness: an extracted text counts as correct only when both its content and its category match the ground truth §III-C2.
3. Scoring: mean average precision is computed over all extracted texts in all test images. F1 is then computed from that mAP and recall, and F1 ranks the entries §III-C2. The formula is not given.

Hyperparameters

This report trains nothing, so there are no model hyperparameters. The table lists the dataset and evaluation settings an implementer would need.

Results

Author-reported. The scores are the competition's own, computed by scripts written by the RRC web team on the test set §V. The tables show the top ten entries from each ranking figure. Team names are copied from the figures. The paper's prose describes only the top three per task §V-A §V-B §V-C.

Task 1, text localisation fig. 3:

Task 2, scanned receipt OCR fig. 4:

Task 3, key information extraction fig. 5. In the figure, recall, precision and Hmean are equal for every row except ranks 6, 8 and 9:

Headline numbers in the prose §VI: 16 teams above 90% Hmean on Task 1, 7 on Task 2, and one (90.49%) on Task 3. The figures show only the top ten, so the Task 1 count of 16 cannot be checked from them. Unlike Task 1 and Task 2, the prose does not give a figure for how many Task 3 entries fell below 80%.

Paper vs code

No official code.

The paper gives no repository or project URL. It names the RRC web portal §IV as the competition site, without a link. Searches run for this card: GitHub repositories for "SROIE scanned receipt OCR information extraction" returned two repositories (csangwar/SROIE-2019 and Abdelrahman2572002/SmartReceipt-OCR), and "ICDAR2019 scanned receipt OCR competition" returned none. Neither owner is an author or an author's lab, so neither is used. No code, configs or evaluation scripts are referenced in the paper.

Pitfalls

• Figure numbers. The text says "Figure 1 shows some example scanned receipt images" §II, but in the compiled paper those examples are fig. 2; fig. 1 is the annotation format.
• Dates. The paper gives the competition open date as 10 Feb 2019 §I, but the timeline lists "Website online: February 15, 2019" §IV.
• Metric names. The Task 1 protocol says mAP and average recall, then F1 for ranking §III-A2. The Task 1 figure reports recall, precision and "Hmean" fig. 3, and the discussion calls the ranking value "Hmean" §VI. The paper never says whether Hmean and F1 are the same quantity, although they normally are.
• Task 3 formula. "F1 computed based on mAP and recall" §III-C2 gives no formula. Matching of content and category is described, but the mAP procedure is not. Do not reimplement Task 3 scoring from this card alone.
• Task 3 figure. For 7 of the 10 Task 3 entries, recall, precision and Hmean are identical fig. 5. The paper does not explain this.
• Task 3 fields. The Ping An entry is described as using keywords "company, cash, date, and address" §V-C, but the annotation example has "total" where that list has "cash" fig. 1(b). The required field list is not stated.
• Team names. The second Task 3 entry is called "Enetity detection" in the text §V-C and "EAST det + Multi-class classification" in the figure fig. 5.
• Word matching. Task 2 does not define the matching criterion and does not say how scores are aggregated across images §III-B2.
• Citation numbers. Bracketed references do not match the bibliography. "[1][2]" for the RRC competitions points at the first two entries, which are a DSRC congestion-control paper and the ICDAR 2013 RRC. "[4]" for the RRC web portal points at the Pascal VOC retrospective §III-A1 §IV [bib].
• Terminology. The text sometimes says "invoice" where it means receipt, for example for the Task 3 input §I §III-C1.
• Data licence and access. The dataset "will be made available after the event" §VII, and the paper states no licence or download location. Do not redistribute the images or annotations from this card. The arXiv metadata spells one author "Shjian Lu"; the paper's author block says "Shijian Lu" [raw metadata].
• Blurring. Sensitive fields are blurred §I. The paper does not say how blurred fields are annotated or scored.
• Top-three methods. The method descriptions for the top three entries per task are one or two sentences each, self-reported in the report. They are not verified against any paper or code.
• Sample size. Only the top ten of each ranking are shown, and the paper reports no baselines or variance across runs.

Lineage

• Builds on: the ICDAR Robust Reading Competition series (2013, 2015, 2017) for text localisation and recognition §I §III-A1. The paper's own bibliography entries for these are not clearly matched to the citations; see Pitfalls.
• Compared against: no baseline is reported.
• Superseded by: not stated in this card's sources.