CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X

In plain words

CodeGeeX is a language model trained mostly on source code, so it can write and translate programs from a description. It has 13 billion parameters. The authors report that a compressed version runs on one consumer graphics card (Method, step 9).

The original HumanEval test set has only Python problems. The authors wrote a new test set that asks for the same small programming tasks in five languages: Python, C++, Java, JavaScript and Go. Answers are scored by running them against unit tests, not by comparing text. On that test set, the authors report that the model beats other open models of similar size.

The authors also built editor plug-ins on top of the model and ran a user survey. The survey is self-reported, so it shows how people felt, not how much time they saved. The released model weights are for non-commercial research only, so check that before using them in a product.

TL;DR

CodeGeeX is a 13B decoder-only Transformer pretrained on 850B tokens across 23 programming languages §2.1 §2.3. The authors also built HumanEval-X, 164 hand-translated problems in five languages, scored by unit tests with pass@k §3. Author-reported: average pass@1 is 18.40% on HumanEval-X generation, ahead of CodeGen-Multi-16B at 16.73% tab. 5. Read the Pitfalls before implementing: the released training script and the paper disagree on several hyperparameters, and the weights are non-commercial.

Problem

HumanEval, the benchmark the paper builds on, has only Python problems. Earlier multilingual datasets scored output by string similarity (BLEU and CodeBLEU) rather than by running it §1 §3.1 §3.2. The authors want a model that works across several languages, a benchmark that checks functional correctness in each, and open weights and code §1.

Method

The model is a left-to-right Transformer decoder trained with next-token prediction on code. Training data, tokenizer and training recipe are described in §2; the benchmark in §3; the evaluation protocol in §4.

1. Architecture. A 39-layer decoder with hidden size 5120 and 40 attention heads, using multi-head self-attention, MLP blocks, layer norm and residual connections §2.1 tab. 2 fig. 2. The activation is FastGELU, an approximation of GELU chosen for speed on Ascend 910 §2.1 eq. 1.
2. Top query layer. One extra attention layer sits above the 39 blocks. Its query is the embedding of position n+1, so the output at that position is a prediction for the next token. The output is multiplied by the transpose of the word-embedding matrix to get token probabilities §2.1 fig. 2.
3. Training corpus. Files come from The Pile (repositories with more than 100 GitHub stars, filtered to 23 languages by file suffix and repository language) and CodeParrot (Python), plus Python, Java and C++ files scraped from GitHub under explicit size and quality filters. Python is formatted to PEP 8 §2.2 fig. 3 tab. 8. The training data is 158B tokens §2.2.
4. Language tags. The corpus is cut into equal-length segments. Each segment gets a tag such as # language: Python in the language's own comment syntax §2.2 tab. 8.
5. Tokenizer. The GPT-2 byte-pair-encoding tokenizer is applied to code as if it were text. The initial vocabulary is 50,000 tokens. A run of L spaces becomes one extra token with index 8 + L. The final vocabulary size is 52,224 §2.2 app. A.2 fig. 12.
6. Embeddings. Learned token embeddings and learned position embeddings, with 2048 positions. The two are summed §2.2.
7. Objective. Cross-entropy on the next token, summed over positions §2.1 eq. 2. The paper writes a sum; the reduction used in code is not stated in the paper.
8. Optimization. Adam, with a 2000-step warm-up and then cosine decay §2.3 tab. 2 eq. 3. The paper does not say the warm-up is linear; the code makes it linear. The model is 8-way model-parallel and 192-way data-parallel, with ZeRO stage 2 for optimizer state §2.3. Weights are FP16, with layer norm and softmax in FP32 §2.3.
9. Quantization for inference. Each linear layer's weights go from FP16 to INT8 with absolute-maximum scaling at b = 8 bits §2.4 eq. 4. A custom kernel multiplies INT8 weights with FP16 activations §2.4. The authors report INT8 inference on one RTX 3090, where FP16 runs out of memory §2.4 tab. 4.
10. Decoding. Greedy, temperature, top-k, top-p and beam search are supported §2.1. Experiments use temperature and nucleus sampling §4.1.
11. Benchmark construction. Each HumanEval problem is rewritten by hand in C++, Java, JavaScript and Go, giving 164 × 5 = 820 pairs. Each pair has a task id, declaration, docstring, prompt, canonical solution and test program. Naming follows each language's style, docstrings are moved before declarations where the language expects that, and test cases are changed where language behaviour differs (for example Python's round-half-to-even) §3.1 fig. 5.
12. Two tasks. Generation: the input is the declaration and docstring; the model writes the body. Translation: the input is the declaration in both languages and the canonical solution in the source language, with no docstring, so the model cannot simply solve the problem again §3.2 fig. 5. There are 20 ordered language pairs §3.2.
13. Metric. pass@k with n = 200 samples per problem and k in {1, 10, 100}, using the unbiased estimator in eq. 5. The paper also defines a multilingual pass@k in which a total budget k is split across languages §3.2 eq. 6 eq. 7.
14. Deployment (not evaluated in the paper's tables). The authors built extensions for Visual Studio Code, JetBrains and Tencent Cloud Studio supporting completion, generation, translation and explanation §1 §5 fig. 1.

Pseudocode for the training objective (step 7):

# Next-token training step for a decoder-only model  [§2.1] [eq. 2]
# Input:  token_ids (B, n), language-tagged segments  [§2.2]
# Output: scalar loss

def training_step(token_ids, model):
    # Predict token t+1 from tokens 1..t  [§2.1]
    inputs = token_ids[:, :-1]                  # (B, n - 1)
    targets = token_ids[:, 1:]                  # (B, n - 1)
    # Decoder gives a distribution over the vocabulary  [§2.1] [fig. 2]
    logits = model(inputs)                      # (B, n - 1, v)
    # Cross-entropy against the true next token  [§2.1] [eq. 2]
    loss = cross_entropy(logits, targets)
    return loss

Pseudocode for the unbiased pass@k estimate used on every problem (step 13):

# Unbiased pass@k for one problem  [§3.2] [eq. 5]
# Input:  n samples, c of them pass all tests, budget k
# Output: estimated probability that at least one of k samples passes

def pass_at_k(n, c, k):
    # Fewer than k failures: some drawn sample must pass  [§3.2]
    # [code: codegeex/benchmark/metric.py#L40-L41]
    if n - c < k:
        return 1.0
    # One minus the probability that all k draws fail  [§3.2] [eq. 5]
    return 1.0 - comb(n - c, k) / comb(n, k)

Multilingual budget split (step 13), where budgets sums to k:

# pass@k for a split of the budget across languages  [§3.2] [eq. 7]
# Input:  budgets[i] samples for language i, correct[i] of n passing
# Output: estimated probability that the problem is solved

def pass_at_k_split(budgets, correct, n):
    # Probability that every language's draws all fail  [§3.2] [eq. 7]
    all_fail = 1.0
    for language in range(len(budgets)):
        all_fail *= comb(n - correct[language], budgets[language])
        all_fail /= comb(n, budgets[language])
    return 1.0 - all_fail

Hyperparameters

Results

Author-reported. Setup: HumanEval-X, 164 problems in five languages, n = 200 samples per problem, A100 GPUs for generation, each baseline run from its HuggingFace checkpoint §4.1. Baselines: GPT-J-6B, GPT-NeoX-20B, InCoder-6.7B and CodeGen-Multi-6B and 16B §4.1.

The authors describe these as the best results among open multilingual models of similar scale §1 §4.2. Translation (Table 6): CodeGeeX-13B-FT has the best pass@100 on 11 of 20 language pairs; CodeGen-Multi-16B is best on 7 §4.2 tab. 6. Generation results are broken down by error type in fig. 7: wrong answers dominate for most languages, and Go has a high share of compile errors §4.2.

Paper vs code

The official repository is THUDM/CodeGeeX, linked from the abstract. The authors are at Tsinghua, Zhipu.AI and Huawei p. 1; the acknowledgements name Tsinghua KEG. Its LICENSE is Apache-2.0; MODEL_LICENSE is a separate custom licence (see Pitfalls). Checked at commit 2838420.

• Pretraining script differs from the paper. Peak learning rate 2e-4 pretrain_codegeex.sh (paper 1e-4 tab. 2); minimum 1e-7 [L95] (paper 1e-6); warm-up 1500 steps [L98] (paper 2000); decay 100000 iterations [L96] (paper 200,000). Global batch 560 and micro-batch 10 pretrain_codegeex.sh (paper 3072 and 16 §2.3). Sequence length 512 [L88] (paper 2048 tab. 2). The script's paths are placeholders, so it may be a template rather than the exact paper run.
• Adam β2. Script uses 0.95 pretrain_codegeex.sh; paper says 0.999 tab. 2.
• Gradient clipping and weight decay are set in the script [L110] [L111] (1.0 and 0.1). The paper does not state either.
• Learning-rate schedule. Code adds linear warm-up learning_rates.py, then cosine over (n − warm-up) / (decay − warm-up) [L92-L94], then returns the minimum after the decay point [L88-L89] and uses the cosine coefficient [L110]. Paper's eq. 3 uses n / n_decay with no warm-up offset.
• Quantization granularity. Code takes the absolute maximum per output row, divided by 127 quantize.py. Paper eq. 4 writes one Max(|W|) per linear layer. The two differ unless each row has the same maximum.
• Default generation config. The release script samples one solution per problem at temperature 0.8 and top-p 0.95 generate_humaneval_x.sh [L36-L37]. That is the pass@10/100 setting; the paper's pass@1 uses temperature 0.2 §4.1.
• Architecture and activation match. Layers, hidden size, heads and maximum positions codegeex_13b.sh; FastGELU utils.py eq. 1. No discrepancy found.
• Pass@k estimator matches eq. 5 metric.py. No discrepancy found.
• Whitespace tokens match the 8 + L rule: start id 10 and max length 10 tokenizer.py [L38-L39]. No discrepancy found.
• Training stack. The paper trains on Ascend with MindSpore §2.3. The release has MindSpore code (codegeex/mindspore/train.py), a Megatron/DeepSpeed path (scripts/pretrain_codegeex.sh) and a PyTorch inference path (codegeex/torch/). The repository does not ship the training corpus; the script takes a data path as a placeholder.

Pitfalls

• Weights are non-commercial. MODEL_LICENSE grants a licence "solely for your non-commercial research purposes" and prohibits commercial use, and it is governed by Chinese law MODEL_LICENSE §2-§3, §6. The repository code is Apache-2.0. Do not infer commercial rights to the weights from the paper's CC BY licence or from the code licence.
• Numbers come from the LaTeX source, not a compiled PDF. Section, table, figure and equation numbers were counted from the source. Check them against the published PDF before quoting.
• Learning-rate schedule has a warm-up offset. Paper's eq. 3 and the code differ as shown above. Implement the code's version if you want to match the released checkpoints, and say so.
• Temperature for pass@10 is inconsistent. Section 4.1 uses 0.8 §4.1. The HumanEval table caption says 0.2 / 0.6 / 0.8 for @1 / @10 / @100 app. A.4.1. Use 0.8 unless you are reproducing the caption.
• "Improvements" in §4.2 are absolute points. The 0.37% to 1.67% gain over CodeGen-Multi-16B is the difference in average pass@k, not a relative gain §4.2 tab. 5.
• Throughput wording. The text says the overall token throughput "was improved by 224%" §2.3. Table 3 gives 24.2B to 54.3B tokens per day, which is 2.24 times the original and a 124% increase.
• Weighted budgets do not reproduce from the corpus table. Weights in tab. 9 (32, 33, 20, 9, 6) do not exactly match the five-language shares in tab. 8 after normalising (about 32, 34, 19, 9, 6). The paper does not say how they were derived.
• User survey is self-reported. 168 users rated "improve or slightly increase" their efficiency §5 fig. 11. The "tens of thousands" of users and "4.7 billion tokens per week" are operational figures, not verified here §1 §5.
• Pass@k needs many samples. The estimator uses n = 200 per problem; the code only reports pass@k when every problem has at least k samples evaluate_humaneval_x.py. Running fewer samples silently changes the reported set of k values.
• Translation prompts omit docstrings on purpose §3.2. Do not add them when reproducing translation numbers.
• Few-shot hurts on MBPP. More shots lowered the score in the authors' runs app. A.4.2. The paper attributes this to limited model capacity; that is a guess, not a tested claim.
• Ascend-specific training. Fusion and graph optimisations in §2.3 are MindSpore and Ascend specific and are not portable as written §2.3.

Lineage

• Builds on: GPT-style decoder pretraining §2.1; Codex and its HumanEval benchmark, which the paper extends to multiple languages §1 §3.1; The Pile and CodeParrot as data sources §2.2; the GPT-2 tokenizer §2.2; the top query layer from PanGu-Alpha §2.1; the pass@k estimator from Codex §3.2.
• Compared against: GPT-J-6B, GPT-NeoX-20B, InCoder-6.7B, CodeGen-Multi-6B and 16B §4.1.
• Superseded by: not stated in this card.