GLM-130B: An Open Bilingual Pre-trained Model

In plain words

Most of the very large language models people use read text in one direction, left to right, and learn by guessing the next word. GLM-130B takes a different route. It looks at the whole context in both directions and is trained to fill in blanks, somewhat like a fill-in-the-word exercise. The model has 130 billion parameters and reads both English and Chinese.

Training a model this big is hard. The loss can suddenly spike or the run can blow up, and the authors spent months finding out why. Their most useful fix was to shrink the learning signal that flows into the word-embedding layer, and they changed how the layer normalization is set up so the numbers stay in a safe range. They also kept the training logs and code public.

The practical point is about running the model. The authors compressed the weights to 4 bits, which keeps quality close to the original, so the model fits on a handful of consumer graphics cards instead of a large data-center server.

TL;DR

GLM-130B is an open, 130B-parameter English and Chinese model trained with a bidirectional blank-infilling objective on 400B tokens §1. The paper is mostly an engineering report: DeepNorm Post-LN, embedding gradient shrink, FP16 training, and weight-only INT4 quantization that needs no retraining §2.1, §3, §4. Its reported results beat GPT-3 175B on several English benchmarks and ERNIE Titan 3.0 260B on Chinese zero-shot tasks, all author-reported §5.

Problem

Models of 100B scale were then mostly closed (GPT-3) or open but hard to train and serve (OPT-175B, BLOOM-176B) §1. Pre-training at this size produced loss spikes and divergence in FP16, and the usual remedies did not work here: Pre-LN, Sandwich-LN and PB-Relax all failed in the authors' runs §3 app. A. Serving these models also needed 8 A100 (80 GB) servers for the comparison models §4. The paper asks whether a 100B-scale model can be trained stably, released with its training process, and run on affordable hardware §1.

Method

The architecture is a GLM with 70 transformer layers, hidden size 12,288, 96 attention heads and a feed-forward block of width 32,768 §2.1 app. A model_glm_130b.sh. The head size of 128 follows from these numbers (derived). The feed-forward block is a GeGLU: the paper sets the width to 8/3 of the hidden size to keep the parameter count of a plain FFN (derived 32,768 = 8/3 × 12,288) §2.1. Positions use one-dimensional RoPE with base 10000, and the two-dimensional positional encoding of the original GLM is dropped §2.1 app. A.

Training corrupts each sequence by replacing spans with mask tokens and asks the model to write the spans back. Spans are visible to each other in an order drawn at random §2.1. Two mask tokens are used. [MASK] marks short spans spread through the text, and [gMASK] marks one long span at the end of a prefix that the model keeps as context §2.1.

# Build one self-supervised sample (95% of training tokens)  [§2.2]
def make_blank_infilling_sample(tokens):
    # 30% of sequences: several short spans marked with [MASK]  [§2.2]
    if random() < 0.3:
        # span lengths ~ Poisson(3), together about 15% of the input
        spans = sample_spans(tokens, poisson_mean=3, fraction=0.15)
        return corrupt(tokens, spans, mask_token="[MASK]")
    # 70% of sequences: keep a prefix, predict the rest  [§2.2]
    # the length of the masked tail is drawn uniformly  [§2.2]
    cut = sample_uniform_cut(tokens)
    return corrupt_tail(tokens, cut, mask_token="[gMASK]")

The remaining 5% of tokens come from a multi-task instruction mix (MIP) of 74 prompted datasets, which is part of the pre-training corpus and not a fine-tuning stage §2.2 app. B. Each sample gets one of the two mask types §2.2.

Normalization is Post-LN with DeepNorm. The residual branch is scaled by alpha = (2N)^(1/2) before the layer norm, where N is the number of layers. The ffn, v_proj and out_proj weights use Xavier normal initialization scaled by (2N)^(-1/2), and all biases start at zero §2.1.

# DeepNorm residual sublayer, Post-LN form  [§2.1]
def deepnorm_sublayer(hidden, sublayer, num_layers):
    # alpha = (2N)^(1/2); N = 70 gives alpha of about 11.83 (derived)
    alpha = (2 * num_layers) ** 0.5
    return layer_norm(alpha * hidden + sublayer(hidden))

Training stability has three parts. The paper uses FP16 mixed precision, with FP16 forward and backward passes and FP32 master weights and optimizer states §3. DeepNorm keeps the scale of the main branch bounded §3. The embedding layer's gradient is shrunk by a factor alpha = 0.1 §3.

# Embedding gradient shrink with factor alpha = 0.1  [§3]
def shrunk_embedding(word_embedding, alpha=0.1):
    # forward value equals word_embedding (derived from the formula [§3])
    # gradient to the table is scaled by alpha
    return (word_embedding * alpha
            + word_embedding.detach() * (1 - alpha))

Parallelism is three-dimensional: 4-way tensor, 8-way pipeline and 24-way data parallel, which is 768 GPUs on 96 nodes of 8 A100 (40 GB) §2.3 app. A. The global batch of 4,224 sequences is 176 micro-batches per data-parallel rank (derived from the config app. A). With p = 8 stages and m = 176 micro-batches, the pipeline bubble ratio (p − 1)/(m + p − 1) is about 3.8% app. A.

Inference quantizes only the weights, to INT4, and keeps activations in FP16. Each output row of a linear layer gets its own scale from its absolute maximum divided by 7, and the weights are dequantized to FP16 at run time §4 app. A. The quantized weights take about 70 GB, which is what allows 4 × RTX 3090 (24 GB) §4.

# Weight-only INT4 quantization, one absmax scale per output row  [§4]
def quantize_linear_int4(weight):
    # weight: (out_features, in_features), FP16; biases and embeddings kept
    # scale = absmax of the row / 7  [app. A] [code: quantization/layers.py#L28]
    scale = max(abs(weight), dim=1) / 7
    # round to integers in [-7, 7]  [code: quantization/layers.py#L29]
    # two INT4 values are packed per byte  [code: quantization/layers.py#L31]
    quant = round(weight / scale[:, None])
    return quant, scale

def linear_forward(inputs, quant, scale):
    # dequantize at run time; activations stay FP16  [§4]
    weight = quant * scale[:, None]
    return inputs @ weight.T

Hyperparameters

Results

Author-reported. Setups: LAMBADA and Pile are zero-shot language modeling; MMLU is 5-shot; BIG-bench-lite (24 tasks) is 0, 1 and 3-shot; CLUE and FewCLUE are zero-shot on dev sets. Baselines are taken from the paper's own tables and cited works §5.

The speed comparison in the appendix is against an unoptimized PyTorch implementation of BLOOM-176B, not against other optimized inference stacks.

Paper vs code

The official repository is THUDM/GLM-130B, linked from the abstract. I cloned it at commit 212215c and read the inference code, configs, quantization code and training logs. It is Apache-2.0 for code; the model weights are under a separate non-commercial model license README.md MODEL_LICENSE.

• The repository contains inference and quantization code only. The model class comes from the external SwissArmyTransformer package, so DeepNorm, the Xavier scaling, the embedding gradient shrink and the training loop cannot be verified here initialize.py. The training notes confirm DeepNorm, GeGLU, RoPE, FP32 softmax and alpha 0.1 main-log-en.md.
• Embedding gradient shrink: the paper says alpha = 0.1 "wipes out most spikes" §3. The logs show alpha was raised to 1 from step 6,000 with a 500-step warm-up main-log-en.md, lowered to 0.5 at about step 33,600 main-log-en.md, and set to 0.1 at step 35,200 main-log-en.md. The reported value is the final one.
• Minimum learning rate: the config says 8e-6 app. A. The logs record a change to 4e-6 on 2022-06-22 main-log-en.md. The paper does not mention this change.
• Quantization scale: the appendix writes the quantization range as [-(2^b − 1), 2^b − 1] but defines the scale with 2^(b−1) − 1 app. A. The code uses absmax / (2^(b−1) − 1), so for INT4 it is absmax / 7, computed per output row in FP16 layers.py. Rounding and int8 packing follow layers.py.
• Quantized layers: the code quantizes the QKV projection, the attention output projection and both MLP projections in every layer [code: quantization/init.py#L14-L50]. Embeddings, norms and biases stay FP16, as the paper says app. A.
• Positions at generation time: with [gMASK], positions run 0, 1, 2, ... across the context and the generated tokens generate.py, which matches §2.1. With [MASK], the code sets every position after the context to the mask position generate.py. The paper does not state this.
• Initialization: the text describes Xavier normal scaled by (2N)^(-1/2) §2.1, but the config sets init_method_std 0.0052 and init_method_xavier_uniform False app. A. The code does not show which applies.
• Parallel layout: the FP16 inference config uses 8-way model parallelism model_glm_130b.sh; the INT4 config uses 4-way model_glm_130b_int4.sh. Training used 4-way tensor and 8-way pipeline §2.3.
• Vocabulary: the text says the icetk vocabulary has 150,000 entries app. A; the configs use 150,528 model_glm_130b.sh.

Pitfalls

• Licensing of the released weights: the code is Apache-2.0, but the model weights may be used only for non-commercial research MODEL_LICENSE. Implementing the method from this card is not affected; using the released checkpoint is.
• Training was not a single clean run. The paper mentions three late-stage loss divergences in the final run §3, but the logs record many restarts, data-skipping ranges and changes to alpha, the learning rate and the MIP data main-log-en.md. Anyone reproducing the schedule should follow the logs, not the one-line summary.
• MIP data changed during training. The logs record restarts at about step 22,800 and later replacements of the MIP data main-log-en.md main-log-en.md. The visible text describes only the final mix.
• Zero-shot claims depend on MIP. The paper asks users not to report zero-shot or few-shot results on the 74 MIP datasets §2.2. For LAMBADA and CLUE it reports minimal 13-gram overlap with the training data §5.
• FP16 is the hard part. BF16 avoids the overflow but uses about 15% more GPU memory in the authors' tests and is not supported on V100 §3. PB-Relax did not fix the overflow §3.
• The INT4 result is specific to weight-only quantization. Activations contain outlier dimensions, reported at up to about 30% of dimensions §4 app. A, so activation quantization is not used. The scheme worked for GLM but failed for BLOOM at INT4 app. A table.
• The row-wise absmax scheme is not a generic recipe. In the appendix table, the zero-point INT4 variant scores 80.63% on LAMBADA against 79.47% for absmax app. A table. The paper chose absmax because it kept performance and is cheaper to compute than zero-point app. A.
• Naming is inconsistent: the text says RTX 3090 Ti in one place §4 and RTX 3090 in the speed table §4.
• The pipeline-bubble ratio is a calculation from the paper's model, not a measurement app. A.
• The config has min_gmask_ratio 0.2 app. A, which the text does not mention. The exact [gMASK] tail distribution therefore depends on the code.

Lineage

• Builds on: GLM, the blank-infilling objective and bidirectional backbone (Du et al., ACL 2022; no arXiv id in the paper's bibliography). DeepNorm (arXiv:2203.00555) for the Post-LN scaling. RoFormer (arXiv:2104.09864) for RoPE.
• Compared against: GPT-3 175B, OPT-175B, BLOOM-176B, PaLM 540B, ERNIE Titan 3.0 260B. Quantization is compared with LLM.int8() (/papers/2208.07339), which keeps outlier features in higher precision; GLM-130B instead keeps activations in FP16 and quantizes weights only §4.
• Superseded by: not known from content already in this wiki.