GLM-OCR Technical Report

In plain words

Reading a scanned page is harder than reading one line of text. A page can hold headings, columns, tables, formulas and stamps, all in different places. GLM-OCR is a small model built for this job. It first finds the blocks on the page, then reads each block separately, and the blocks can be read at the same time. Its output is text in a structured form, such as Markdown or JSON.

The model also changes how it writes its answer. A normal language model writes one token at a time. OCR output is very predictable, so the model is trained to guess several tokens ahead and writes them in bigger steps. The paper reports that this makes the output faster to produce while keeping the accuracy, and that the extra guessing does not add much memory because the guessing parts share their weights.

The paper's point is that a compact, specialised model with this layout-first design can match or beat much larger general models on document tasks, which matters when the model has to run on modest hardware or handle many pages.

TL;DR

• A 0.9B model: a 0.4B CogViT vision encoder and a 0.5B GLM decoder §2.1.
• Multi-token prediction is used in training and inference; the paper reports 5.2 tokens per decoding step on average and about 50% more throughput §1.
• A two-stage pipeline: PP-DocLayout-V3 finds regions, then regions are recognised in parallel. Key information extraction skips the layout step §2.1.
• Author-reported: 94.6 on OmniDocBench v1.5 (94.62 overall), ahead of the models compared tab. 3 tab. 4.
• The official repository holds the inference SDK, serving instructions and LoRA/full SFT examples. The training code (pretraining, MTP loss, RL) is not in it.

Problem

Small models are reported to hallucinate and repeat text on complex layouts §2.1. General multimodal LLMs read such pages well but are large, and their token-by-token decoding is slow and memory-hungry under high concurrency or on edge devices §1. The authors also name three production needs: strong results on tables, formulas, code and seals; high throughput and low latency; and easy domain adaptation §1. The paper does not give a measured failure rate for single-model OCR that motivates the layout stage.

Method

Model. GLM-OCR has a vision encoder (CogViT, about 400M parameters), a lightweight cross-modal connector, and a language decoder (GLM, about 500M parameters). Visual features are projected into the decoder's embedding space and fed in as prefix tokens. The decoder then writes Markdown or JSON autoregressively §2.1 §1. The paper gives no downsampling factor for the connector §1.

Multi-token prediction (MTP). In addition to the main head, the model has k auxiliary heads that predict the next k tokens. The heads share parameters and are trained for different future offsets §2.1. The paper says training predicts ten tokens per step §1. The number k and the way an offset enters the shared head are not stated §2.1.

Tasks. Two tasks share one generative formulation and differ in preprocessing and prompt §2.1:

1. Document parsing. Layout analysis splits the page into regions such as paragraphs, tables and formulas. Each region goes through the model on its own. A merge-and-post-process step restores reading order and writes Markdown and JSON §2.1.
2. Key information extraction. The full page goes to the model with a prompt that gives a JSON schema. There is no layout crop; the model attends to regions implicitly §2.1 §5.3.4.

Training stages §2.2 tab. 1:

• Stage 1, vision encoder: image-text and grounding data, described as tens of billions of pairs, with a MIM and CLIP objective and distillation from a larger in-house ViT.
• Stage 2.1, vision-language pretraining: the encoder with GLM-0.5B on image-text, document parsing, grounding and VQA data.
• Stage 2.2, pretraining with MTP.
• Stage 3, supervised fine-tuning with MTP on text, formula, table and KIE data.
• Stage 4, reinforcement learning with GRPO §2.2, using rollouts from the SFT model and rewards per task tab. 2:

# Stage 4 reward per task  [tab. 2]
# Input:  task name, model output, reference
# Output: scalar reward

def reward(task, output, reference):
    # Primary accuracy metric per task  [tab. 2]
    # text: normalized edit distance; formula: CDM; table: TEDS; KIE: field F1
    accuracy = accuracy_metric(task, output, reference)
    # Repetition and malformed-structure penalties  [tab. 2]
    penalties = repetition_penalty(output) + structure_penalty(output, task)
    # How accuracy and penalties are combined is not stated  [tab. 2]
    return combine(accuracy, penalties)

The table lists the metrics and the penalties tab. 2. The paper does not say how they are combined into one reward, what the sign or scale of each penalty is, or how the normalized edit distance maps to a reward. The function above shows the structure only.

Inference pipeline. The shipped SDK does the following [code: glmocr/pipeline/_workers.py#L322-L369]:

# Two-stage document parsing with parallel region OCR  [§2.1]
# Input:  page_image (one page)
# Output: region texts in reading order, as Markdown and JSON

def parse_page(page_image):
    # 1. Detect layout regions and their labels with PP-DocLayout-V3  [§1]
    regions = detect_layout(page_image, threshold=0.3)
    # 2. Keep text, table and formula regions; drop the rest
    # [code: glmocr/config.yaml#L252-L275]
    jobs = [crop(page_image, r) for r in regions if needs_ocr(r.label)]
    # 3. Recognise regions in parallel with a label-specific prompt  [§5.3]
    results = parallel_map(recognize_region, jobs)
    # 4. Merge and restore reading order  [§2.1]; the order rule is not stated
    return merge_and_order(results)

# One decoding step with multi-token drafts  [§1] [code: README.md#L130]
# Input:  prefix tokens
# Output: prefix extended by one or more tokens

def decode_step(prefix, draft_count):
    # Auxiliary heads propose the next tokens after the prefix  [§2.1]
    drafts = propose_tokens(prefix, draft_count)
    # The main model checks the drafts; the paper does not describe this rule,
    # so the serving stack's standard speculative verification is assumed
    accepted = verify(prefix, drafts)
    return prefix + accepted

Hyperparameters

Results

Author-reported. Setup: OmniDocBench v1.5, OCRBench (Text), UniMERNet, PubTabNet, TEDS_TEST, Nanonets-KIE and Handwritten-KIE; internal benchmarks for six scenarios. Hardware and evaluation scripts for the competitor numbers are not stated. Gray entries (Gemini-3-Pro, GPT-5.2) are reference only and excluded from the ranking tab. 3 tab. 5.

Public benchmarks tab. 3. GLM-OCR compared with PaddleOCR-VL-1.5, DeepSeek-OCR2, MinerU 2.5 and dots.ocr:

The paper's text says GLM-OCR is highest on OCRBench (Text), UniMERNet, TEDS_TEST and OmniDocBench, and second to MinerU 2.5 on PubTabNet §3.1.

OmniDocBench v1.5 breakdown tab. 4. Lower is better for Text Edit and Reading Order Edit; higher is better for the other columns:

The paper's text says PaddleOCR-VL-1.5 is slightly better on Text Edit and Formula CDM §3.1 tab. 4. The full table has more rows than shown; the rows above are the most relevant ones.

Internal scenarios tab. 5. Six scenarios, compared with the same open-weight models plus gray references:

Gemini-3-Pro (gray) scores 97.3 on Receipt KIE, and GPT-5.2 (gray) scores 83.5 tab. 5.

Throughput tab. 6. Single replica, single concurrency, same hardware (not named), pages per second:

Cost claim §4.2. The paper states a hosted price of 0.2 RMB per million tokens and says 1 RMB processes about 2,000 A4 scans. This is the authors' claim and is not checked here.

Paper vs code

Code: zai-org/GLM-OCR at 0288f96bb9e78cb439883eaa725eeae90bd177c5 (Apache-2.0 LICENSE). The repository is linked from the paper's first page. Its scope: the inference SDK, serving instructions and fine-tuning examples. It does not contain the model definition, the MTP loss, the pretraining or RL code, or the benchmark scripts.

• MTP at serving time. The paper trains for 10 tokens per step §1. The shipped vLLM command uses num_speculative_tokens 3 README.md, and the SGLang command uses 3 steps with 4 draft tokens README.md. The multi-GPU example uses 1 engine.py. The paper's 5.2 tokens per step is not tied to any of these settings.
• Layout. The paper says PP-DocLayout-V3 is used §1 §2.1. The code adds a detection threshold of 0.3 config.yaml, NMS on config.yaml and an optional polygon mode that is off by default config.yaml.
• Parallelism. The paper describes parallel recognition §2.1. The code submits each region to a thread pool _workers.py, with 32 workers by default config.yaml.
• Prompts. The paper's use-case prompts are "Text Recognition:", "Table Recognition:" and "Formula Recognition:" §5.3. The code uses the same strings config.yaml. The KIE prompt example is commented out in the paper source; no KIE prompt is in the code.
• Decoding. The code sends temperature 0, top_k 1 and top_p 0.00001 config.yaml. The paper does not state its decoding settings for the benchmark numbers.
• Fine-tuning. The paper points to LLaMA-Factory §1 §4.3. The example configs freeze the vision tower in full SFT [code: examples/finetune/glm_ocr_full_sft.yaml#L9] and train on a 1000-sample set glm_ocr_full_sft.yaml. These are examples, not the paper's training recipe.
• MaaS default. maas.enabled defaults to true, so the SDK calls the hosted API unless it is set to false config.yaml. Self-hosted use needs that flag set to false.
• License. The code is Apache-2.0 LICENSE. The README says the model weights are MIT README.md, and it notes that PP-DocLayout-V3 is Apache-2.0 README.md.

Pitfalls

• 5.2 tokens per step is not reachable with the shipped serving flags. With standard speculative verification, three draft tokens give at most four accepted tokens per step, and the SGLang flags give four draft tokens. The paper does not give the serving configuration used to measure 5.2, so it is unclear how that figure was produced.
• Training versus serving counts differ. The paper trains for ten predicted tokens §1, but serves with three or fewer drafts README.md. An implementer who copies the training setting into serving will get different behaviour.
• MTP head details are missing. The number of heads k, how the offset enters the shared head, and the loss weighting are not stated §2.1.
• The recipe table is incomplete. The text says Table 1 gives learning rates, but it lists data types only §2.2 tab. 1. Batch sizes, steps, and the pretraining data mixture are not stated.
• Reward details are ambiguous. The text reward term is written as "normalized edit distance" without saying how it maps to a reward tab. 2. GRPO group size and clipping are not stated §2.2.
• Config default mismatch for pixel limits. config.py sets max_pixels to 14 * 14 * 4 * 1280, which is 1,003,520 config.py. The shipped config.yaml sets 71,372,800 with a comment that gives the first formula config.yaml. Pick one and check it for your images.
• Two-stage errors propagate. The paper names error propagation from layout mistakes, poor reading order across pages or in multi-column layouts, and low-resolution or distorted inputs §6. Key information extraction bypasses layout, so its failure modes differ §2.1.
• Benchmark protocol is thin. The in-house datasets are not described (size, source, release). Hardware for the throughput test is not named, and the competitor versions and runs are not stated §3.2 §4.1.
• Licence of the weights. The paper is CC BY 4.0, which is this card's licence field. The README gives MIT for the model weights README.md, a separate licence that this card does not cover.

Lineage

• Builds on: GLM-V encoder-decoder framework and CogViT §1 (named only, not compiled); multi-token prediction, cited to DeepSeek-V3 §1; GRPO, cited to DeepSeekMath §2.2; PP-DocLayout-V3 for layout, cited as PaddleOCR-VL-1.5 §1.
• Compared against: PaddleOCR-VL-1.5, MinerU 2.5, DeepSeek-OCR2, dots.ocr, PaddleOCR-VL, Gemini-3 Pro, GPT-5.2, Qwen3-VL, GPT-4o, InternVL3 and others listed in tab. 3 tab. 4 tab. 5.
• Superseded by: not known from content already in this wiki.