ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools

In plain words

This report describes a family of language models called ChatGLM, made by Team GLM at Zhipu AI and Tsinghua University. It focuses on the newest series, GLM-4. The models first learn from a huge pile of mostly Chinese and English text, then are tuned to follow instructions and to match what people prefer.

One variant, GLM-4 All Tools, is trained to decide on its own when a question needs an outside tool: a web browser, a Python interpreter, an image generator, or a function a developer has defined. It then uses the tool's output to continue.

Most of the report is a list of benchmark results. The authors say GLM-4 is close to the strongest commercial models on common tests and ahead of them on Chinese alignment tests. Those are the authors' own numbers. The report gives few training details, so use it to judge whether the models are relevant, not to rebuild them.

TL;DR

The report describes the ChatGLM family from GLM-130B to GLM-4 and GLM-4 All Tools, by Team GLM at Zhipu AI and Tsinghua University p. 1. GLM-4 is pre-trained on about ten trillion tokens, mostly Chinese and English, and aligned with supervised fine-tuning and RLHF [abstract] §2 Alignment. GLM-4 All Tools is further trained to choose among a web browser, a Python interpreter, a text-to-image model and user-defined functions §2 GLM-4 All Tools. Results are author-reported. Training hyperparameters, the data mixture and the All Tools training method are largely not stated, so this card supports relevance checks, not reproduction.

Problem

The report has no single research question. Its goal is a family of general models that match top commercial systems on English benchmarks and lead on Chinese, with open releases alongside §1. Three concrete pressures shape it:

• Inference cost: the team switched to grouped-query attention to shrink the key-value cache during inference §2 Architecture.
• Context length: moving from 2K (ChatGLM-6B) to 32K and then to 128K and 1M needed position-encoding extension, continual training on long text and long-context alignment §2 Architecture.
• Tool use: the model has to decide when a task needs an external tool and to fold the tool's result back into the answer §2 GLM-4 All Tools.

Assumptions: the models are pre-trained mostly on Chinese and English, and aligned primarily for those two languages [abstract] §2 Pre-Training Data. Other languages get only a small share of the pre-training data and are not the alignment focus [abstract].

Method

The paper has no equations and no algorithm box. The steps below follow its prose.

1. Pre-training data. Multilingual documents (mostly English and Chinese) from web pages, Wikipedia, books, code and research papers. The pipeline runs deduplication (exact and fuzzy), filtering of noisy web pages, and tokenization §2 Pre-Training Data. Sources are re-weighted to favour books and Wikipedia §2 Pre-Training Data. Mixture ratios are not stated.
2. Tokenizer. Byte-level BPE learns separate Chinese and multilingual vocabularies, which are merged with the cl100k_base tokenizer of tiktoken into one vocabulary of 150,000 tokens §2 Pre-Training Data.
3. Architecture changes from GLM-130B to GLM-4 §2 Architecture:
   1. Remove all bias terms except those of the query, key and value projections.
   2. Use RMSNorm instead of LayerNorm, and SwiGLU instead of ReLU in the feed-forward block.
   3. Extend rotary positional embeddings to two dimensions.
   4. Replace multi-head attention with grouped-query attention; set the feed-forward width to 10/3 of the hidden size so the parameter count stays the same.
4. Context extension §2 Architecture. Position-encoding extension, continual training on long text, and long-context alignment. The paper defers the long-context recipe to LongAlign §2 ChatGLM Techniques.
5. Alignment §2 Alignment. Supervised fine-tuning on authentic human prompts (the paper says template-based or model-written prompts were worse), then RLHF to reduce refusals, unsafe outputs, mixed-language output and multi-turn drift. Annotators score each response for safety, factuality, relevance, helpfulness and preference. Early models used annotations by the developers; later models used in-house annotation and purchased third-party data under quality control.
6. All Tools §2 GLM-4 All Tools. The model analyses a request, plans steps, calls one or more tools, and uses their intermediate results. The tools named are a web browser, a Python interpreter and a text-to-image model; the GLMs platform adds user-defined functions, APIs and knowledge bases §2 GLM-4 All Tools.

Sketch of the architecture block as the paper describes it. Norm placement relative to the residual connection is not stated.

# Transformer block for GLM-4  [§2 Architecture]
# Input and output: hidden (B, T, d); cache holds grouped keys and values
def transformer_block(hidden, positions, cache):
    # RMSNorm replaces LayerNorm  [§2 Architecture]
    normed = rms_norm(hidden)                    # (B, T, d)

    # Only the Q, K and V projections carry a bias  [§2 Architecture]
    # Keys and values use fewer heads than queries (GQA)  [§2 Architecture]
    queries, keys, values = qkv_projection(normed)

    # Two-dimensional rotary positional encoding  [§2 Architecture]
    queries, keys = rope_2d(queries, keys, positions)

    # Keys and values are cached in the grouped (smaller) form  [§2]
    cache.append(keys, values)
    attended = attention(queries, cache.keys, cache.values)
    hidden = hidden + output_projection(attended)

    # SwiGLU feed-forward; width d_ffn = 10/3 * d  [§2 Architecture]
    normed = rms_norm(hidden)
    hidden = hidden + swiglu_ffn(normed)
    return hidden

Sketch of the All Tools loop, written from the prose. The paper gives no algorithm and no step limit §2 GLM-4 All Tools.

# Tool loop for an All Tools model  [§2 GLM-4 All Tools]
# Tools: browser, Python, text-to-image, user-defined functions
def all_tools_loop(conversation, tools):
    while True:
        # The model plans and may request one tool  [§2 GLM-4 All Tools]
        reply = model.generate(conversation, tools)
        if reply.tool_name is None:
            return reply.text
        # Run the tool, then feed its result back  [§2 GLM-4 All Tools]
        result = dispatch_tool(reply.tool_name, reply.arguments)
        conversation.append(reply)
        conversation.append(result)

Hyperparameters

Values from the paper unless the notes say otherwise. Repository defaults are marked as such, because they are not the paper's training settings.

Results

Author-reported throughout. Baselines come from public technical reports or from public APIs, which the paper says are the sources §3 intro. Table numbers follow source order in the LaTeX, because the compiled PDF is not in the repository.

Open models and earlier generations §1

Setup: ChatGLM-6B (2023-03-14), ChatGLM2-6B (2023-06-25), ChatGLM3-6B-Base (2023-10-27) and GLM-4-9B (2024-06-05). Evaluation settings are not stated tab. 1.

Academic benchmarks §3.1 tab. 2

Setup: MMLU, GSM8K, MATH, BBH, GPQA and HumanEval, with chain-of-thought for GSM8K, MATH and BBH §3.1.

Instruction following, IFEval §3.2 tab. 3

Setup: 541 prompts; L = loose, S = strict, P = prompt-level, I = instruction-level. The Chinese columns use a translated set with inapplicable instructions removed §3.2.

Alignment in Chinese, AlignBench v1.1 §3.3 tab. 4

Setup: 683 queries, 8 categories, GPT-4-based multidimensional scoring §3.3.

Long context, LongBench-Chat §3.4 tab. 5

Setup: context lengths from 10k to 100k (unit not stated); GPT-4 scores, few-shot prompting, averaged over repeated runs §3.4.

Real-world coding, NaturalCodeBench §3.5 tab. 6

Setup: real user prompts in Python and Java, in English and Chinese §3.5.

Function calling, Berkeley Function Call Leaderboard §3.6 tab. 7

Setup: AST summary, execution summary and relevance detection §3.6.

Agents, AgentBench §3.7 tab. 8

Setup: seven environments; overall score uses AgentBench's per-dataset weights §3.7.

OS = Operating System, DB = Database, KG = Knowledge Graph, LTP = Lateral Thinking Puzzles, HH = House Holding, WS = Web Shopping, WB = Web Browsing.

GLM-4 All Tools §3.8 tab. 9

Setup: the All Tools system on the web, GLM-4 All Tools (Web, 0116), compared with GPT-4 (Web, 0110). Python tasks are GSM8K, MATH and Math23K; the browser task is "Information Seeking", which the paper does not define §3.8.

Safety, SafetyBench (Chinese subset) §4 tab. 10

Setup: seven categories; Chinese subset with highly sensitive questions removed §4.

Paper vs code

Code: github.com/THUDM/GLM-4 at commit 024d8fd, Apache-2.0 LICENSE. The README cites this arXiv id README.md, and the owner is the THUDM organisation of the authors' lab. The clone is the repository's current main branch, which is later than the paper. Its README lists releases after June 2024 README.md. The repository holds no training code, no model architecture code and no benchmark harness. The architecture lives in checkpoint configs loaded with trust_remote_code trans_cli_demo.py. I searched the repository for AlignBench, IFEval, LongBench, NaturalCodeBench and AgentBench and found only README mentions.

• Checkpoint: the CLI demo loads zai-org/GLM-4-9B-0414, not the GLM-4 (0520) or GLM-4-Air (0605) checkpoints evaluated in the paper trans_cli_demo.py §3 intro. Numbers from the paper cannot be reproduced from this code.
• Decoding: the paper gives no evaluation decoding settings §3 intro. The repository's two demos disagree: the CLI uses temperature 0.6 and repetition penalty 1.2 trans_cli_demo.py trans_cli_demo.py; the web demo defaults to temperature 0.95 and repetition penalty 1.0 main.py.
• Precision: the paper says BFloat16 at evaluation §3 intro. The CLI demo also loads in bfloat16 trans_cli_demo.py. These agree.
• Tool use: the paper says the model chooses tools itself §2 GLM-4 All Tools. The repository only contains the client-side dispatch. ALL_TOOLS maps simple_browser, python and cogview tool_registry.py. dispatch_tool runs a predefined tool or a registered custom tool whose arguments are JSON tool_registry.py. The learned choice of tool is not in the repository.
• Architecture: grouped-query attention, RMSNorm, SwiGLU and 2D RoPE cannot be checked against the repository. The one architecture log is from an Intel demo README and lists multi_query_group_num = 2 and ffn_hidden_size = 13696 README.md. It gives no hidden size, so the 10/3 width ratio §2 Architecture cannot be checked.
• Fine-tuning defaults: the repository's SFT and LoRA configs use learning rates of 5e-5 and 5e-4 sft.yaml lora.yaml. The paper gives no SFT learning rate §2 Alignment, so these are user defaults and do not describe the paper's alignment.

Pitfalls

• Baseline labels disagree. The text compares GLM-4 with GPT-4 (0613) at 86.4 MMLU and Gemini 1.5 Pro at 83.7 §1. Table 2 labels 86.4 as GPT-4 (0314) and lists Gemini 1.5 Pro at 85.9 tab. 2. Use the table values and name the GPT-4 build.
• Percentage claim. The text says GLM-4 reaches 96.3% of GPT-4's MMLU accuracy §3.1. Dividing the table values 83.3 by 86.4 gives 96.4%. The difference is rounding or a different base.
• AlignBench wording. The text says GLM-4 outperforms GPT-4 Turbo overall §3.3. The table shows both at 8.00 for GLM-4 (0520) and GPT-4 Turbo (2024-04-09) tab. 4.
• Suspicious duplicate. The GLM-4-9B-Chat IFEval rows are identical for English and Chinese tab. 3. This looks like a copy error, so do not read it as a cross-language result.
• All Tools comparison. The text compares with "ChatGPT-4 (Web)" §3.8, but the table column is "GPT-4 (Web, 0110)" tab. 9. The browser benchmark "Information Seeking" is not defined, and no sample size is given. Its numbers cannot be re-run from the paper.
• Context length. GLM-4 is said to support 128K §1, but the only stated pre-training length is 8,192, for GLM-4-9B §1. The pre-training length for the 128K models is not stated, so long-context ability depends on the long-context alignment step §2 Architecture.
• Baseline sourcing. Baselines come from technical reports or public APIs, and the paper does not say which for each cell §3 intro. API results depend on each provider's safety policy, which the paper says affected its Chinese SafetyBench subset §4.
• Judge dependence. AlignBench and LongBench-Chat are scored by a GPT-4-based judge §3.3 §3.4. The judge prompt, judge version and the number of repeated runs are not stated.
• Benchmark edits. The SafetyBench results use a Chinese subset with highly sensitive questions removed §4, so they are not comparable with the full benchmark.
• Missing training detail. Learning rates, batch sizes, step counts, optimizer settings and the SFT and RLHF data sizes are not stated §2 Alignment. The RLHF methods are described in other reports the paper cites §2 ChatGLM Techniques.
• Norm placement and bias details. The paper does not say whether RMSNorm is applied before or after each residual branch, nor the RMSNorm epsilon. The sketch above leaves placement open.
• Tool loop limits. The paper gives no step limit, stopping rule or error handling for tool calls §2 GLM-4 All Tools. The repository's dispatcher returns a system error when a tool call fails tool_registry.py.
• Cost hidden in agents. The paper gives no latency or cost figures for the tool calls behind the All Tools results §3.8.

Lineage

• Builds on: GLM-130B, arXiv:2210.02414. This report keeps the transformer base but changes the normalization, feed-forward activation, bias setup and attention design §2 Architecture.
• Alignment work used or cited by the authors: ChatGLM-RLHF, arXiv:2404.00934 §2 ChatGLM Techniques; ChatGLM-Math, arXiv:2404.02893 §2 ChatGLM Techniques; Self-Contrast, arXiv:2404.00604 §2 ChatGLM Techniques. AgentTuning and LongAlign are cited in the paper, but their arXiv ids are not checked on this card.
• Compared against: GPT-4 (0314, 0613, Turbo variants), GPT-4o, Claude 2, Claude 3 Opus, Gemini 1.5 Pro and Llama-3-8B-Instruct tab. 2 to tab. 9.
• Superseded by: none known from content already in this wiki.