Qwen Technical Report

In plain words

Qwen is a family of language models from Alibaba's Qwen team, released in three sizes: 1.8, 7 and 14 billion parameters. Each model first learns to predict the next token over a large mix of text, including web pages, encyclopedias, books and code, in several languages with a lot of English and Chinese.

A second stage turns the base model into a chat assistant. It is fine-tuned on conversations, then a reward model learns to score answers the way people would, and reinforcement learning pushes the model toward the answers that score well. The team also built versions specialised for code and for maths, and chat models that can call tools and run Python code through a reasoning-and-acting prompt.

The authors report that the largest model beats other open models of similar size on their benchmarks, but still trails GPT-4. A training-free set of inference-time changes lets the model read inputs longer than the length it was trained on.

TL;DR

• Qwen 1.8B, 7B and 14B base models are pretrained on up to 3T tokens; the 7B and 14B base and chat models are released §1 §2.1 tab. 1.
• Chat models get SFT and PPO-based RLHF with a learned reward model §3.1 §3.2. Code-Qwen and Math-Qwen-Chat are derived from the base models §4 §5.
• Author-reported: the 14B base scores 66.3 on MMLU (5-shot) and 72.1 on C-Eval (5-shot) tab. 2.
• The released repository differs from the paper in several training settings (see Paper vs code). Use the paper's numbers for the paper's method.

Problem

Pretrained base models do not follow human instructions and are not suited to assistant use without alignment §3. The report also names reproducibility, steerability and accessibility for service providers as open problems for LLMs in general §1. Attention cost grows quadratically with context length, so long inputs are expensive in compute and memory, and models degrade beyond the length they were trained on §2.5. Collecting human chat data is hard for groups outside large platforms §6.

Method

Overview of the pipeline, in the order the paper presents it:

1. Pretrain autoregressive Transformers on up to 3T tokens with LLaMA-style changes §2.1 §2.3 §2.4.
2. Optionally extend the context at inference time without training §2.5.
3. Supervised fine-tuning on chat-format data, with a ChatML-style template §3.1 app. A.1.
4. Train a reward model on human comparison data, then run PPO with it §3.2.
5. Specialise: continued pretraining on code and SFT for Code-Qwen §4.1 §4.2; SFT from the base model for Math-Qwen-Chat §5.1.
6. Add about 2,000 self-instructed tool-use and agent samples to the general SFT mix instead of a separate stage §3.4.

Pretraining data and tokenizer §2.1 §2.2

• Sources: public web documents, encyclopedias, books, code and other data, multilingual with a large English and Chinese share §2.1.
• Web text is extracted from HTML and language-identified. Duplicates are removed by exact match after normalization and by fuzzy matching with MinHash and LSH §2.1.
• Quality filters combine rules and model scores (language models, quality scorers, offensive-content detectors), plus manual sampling §2.1.
• Selected sources are up-sampled. Multi-task instruction data is mixed into pretraining §2.1.
• Instruction samples with a 13-gram overlap with the evaluation test sets are removed; the paper says this is done for the instruction data used in reported tasks §2.1.
• The tokenizer is byte-pair encoding starting from tiktoken's cl100k_base, extended with common Chinese characters and words and with other languages. Numbers are split into single digits. The vocabulary is about 152K §2.2.
• Compression is compared against XLM-R, LLaMA, Baichuan and InternLM tokenizers across languages fig. 3.

Architecture §2.3

• Untied input embedding and output projection, accepting the extra memory cost for better performance §2.3.
• Rotary position embeddings (RoPE). The inverse-frequency matrix is kept in FP32 §2.3.
• Biases removed from most layers, but kept in the query, key and value projections §2.3.
• Pre-normalization with RMSNorm instead of LayerNorm §2.3.
• SwiGLU activation. The feed-forward width is 8/3 of the hidden size instead of 4 times §2.3.

Model sizes are in tab. 1.

Pretraining objective §2.4

• Next-token prediction over documents that are shuffled, merged and truncated to a context of 2048 tokens §2.4.
• FlashAttention is used; the optimizer is AdamW with beta1 0.9, beta2 0.95 and epsilon 1e-8 §2.4.
• Cosine learning-rate decay from a per-size peak down to 10% of that peak §2.4.
• BFloat16 mixed precision §2.4.

Context-length extension, inference only §2.5

• NTK-aware interpolation changes the RoPE base instead of scaling every dimension equally, to keep high-frequency information §2.5.
• Dynamic NTK-aware interpolation changes the scale chunk by chunk. The paper cites a later formal treatment §2.5.
• LogN-scaling multiplies the attention term by a factor that depends on the ratio of context length to training length, to keep attention entropy stable §2.5. The paper's text says "query and value"; see Pitfalls.
• Window attention limits attention to a local window. Window sizes differ by layer: shorter in lower layers, longer in upper layers §2.5. The window values are not stated.
• Perplexity on arXiv papers stays low up to 8192 and 16384 tokens with all techniques on tab. 3.

Supervised fine-tuning §3.1 app. A.1

• Data: conversations written or annotated in several styles, deliberately without prompt-template data, with safety data on violence, bias and pornography §3.1.1. The number of conversations is not stated.
• Format: ChatML, with special tokens for turn start and end, so the template tokens never occur in pretraining data §3.1.1 app. A.1.
• Loss: next-token prediction, with the system and user turns masked §3.1.2.
• Settings are in the Hyperparameters table §3.1.2.

# SFT example construction  [§3.1.1] [§3.1.2] [app. A.1]
# Input:  conversation, a list of (role, text) turns
# Output: token ids and a loss mask of the same length

def build_example(conversation):
    # Wrap each turn in ChatML special tokens  [app. A.1]
    text = ""
    for role, content in conversation:
        text += "<|im_start|>" + role + "\n" + content + "<|im_end|>\n"
    # Mask system and user turns so only assistant text is trained  [§3.1.2]
    tokens, loss_mask = tokenize_with_roles(text)
    # Limit the sequence to 2048 tokens  [§3.1.2]
    return tokens[:2048], loss_mask[:2048]

Reward model §3.2.1

• Initialised from a same-size Qwen model. A pooling layer reads the reward at a specific end token §3.2.1.
• Preference pretraining (PMP) on a large comparison dataset, then fine-tuning on higher-quality comparisons §3.2.1.
• Prompts are chosen with a balanced sampler over about 6,600 tags, scoring both diversity and complexity §3.2.1. Responses come from several model sizes and sampling strategies, and annotators score them to form pairs §3.2.1.
• Training: one epoch, constant learning rate 3e-6, batch 64, sequence length 2048 §3.2.1.
• Test pairwise accuracy is in tab. 4.

PPO §3.2.2

• Four models are in memory: policy, value, reference and reward model §3.2.2.
• The value model is trained alone for 50 steps first §3.2.2.
• Two responses are sampled per query §3.2.2.
• Rewards are normalized by a running mean. KL coefficient 0.04 §3.2.2.
• A pretraining-data gradient is mixed in to limit the alignment tax on benchmarks that are not code or maths §3.2.2.
• Value loss is clipped at 0.15 §3.2.2.

# PPO alignment after SFT  [§3.2.2]
# Input:  SFT policy, reference copy, value model, reward model, prompts
# Output: aligned policy

def ppo_alignment(policy, reference, value, reward_model, prompts):
    # Train the value model alone for 50 steps first  [§3.2.2]
    for step in range(50):
        value.update_only(prompts)
    for batch in prompts:
        # Sample two responses per query  [§3.2.2]
        responses = policy.sample_two(batch)
        # Score each response with the reward model  [§3.2.1] [§3.2.2]
        scores = reward_model(batch, responses)
        # Normalize rewards by a running mean  [§3.2.2]
        scores = normalize_by_running_mean(scores)
        # Penalize divergence from the reference, coefficient 0.04  [§3.2.2]
        # (the per-token form of the KL term is not stated)
        penalty = 0.04 * kl(policy, reference, batch, responses)
        # Mix in a pretraining-data gradient; coefficient not stated  [§3.2.2]
        pretrain_grad = pretrain_gradient(policy)
        # Policy learning rate 1e-6, value learning rate 5e-6  [§3.2.2]
        # Value loss clipped at 0.15  [§3.2.2]
        policy.update(scores - penalty, pretrain_grad, lr=1e-6)
        value.update(scores, clip=0.15, lr=5e-6)
    return policy

Evaluation of chat models §3.3 fig. 4

• Automatic benchmarks: MMLU, C-Eval, GSM8K, HumanEval, BBH, in zero-shot and few-shot settings §3.3 tab. 5.
• Human evaluation: 300 Chinese instructions across knowledge, language, creative writing, coding and maths. Three annotators ranked the answers for helpfulness, informativeness and validity §3.3. Win, tie and loss rates are shown as bars against GPT-3.5 in fig. 4; the bar values are not given in the text. The paper says the RLHF model beats the SFT models and trails GPT-4 §3.3.
• Baselines come from official reports and OpenCompass §3.3 app. A.2.
• GPT-3.5 and GPT-4 were called through the OpenAI API with model versions gpt-3.5-turbo-0613 and gpt-4-0613 §3.3.

Tool use, code interpreter and agent §3.4

• Tool use via ReAct prompting. A new benchmark, excluding plugins seen in training, measures the choice among up to five tools, parameter plausibility and false positives §3.4 tab. 6.
• Code interpreter: a benchmark with math, visualization (easy and hard) and general tasks. Executability is measured directly. Correctness is judged by Qwen-VL, a multimodal model, for visualization tasks §3.4 tab. 7 tab. 8.
• Hugging Face agent benchmark, run mode and chat mode tab. 9.

Code models §4

• Code-Qwen: continued pretraining from the base models on about 90B code tokens, at context length 8192, with 3% warm-up and no decay §4.1.
• Code-Qwen-Chat: multi-stage SFT from Code-Qwen §4.2.
• Evaluated with pass@1 on HumanEval, MBPP and HumanEvalPack §4.3 tab. 10 tab. 11.

Math models §5

• Math-Qwen-Chat: SFT directly from the base models on an augmented maths instruction set, with inputs masked and a peak learning rate of 2e-5 for 50,000 steps at sequence length 1024 §5.1.
• Evaluated with greedy decoding on GSM8K, MATH, Math401 and Math23K §5.2 tab. 12.

Hyperparameters

Values as stated in the paper. "not stated" means the paper does not give it.

Results

Author-reported. All numbers are from the paper's own evaluation and baseline tables.

Base models, zero- or few-shot as labelled in tab. 2: MMLU 5-shot, C-Eval 5-shot, GSM8K 8-shot, MATH 4-shot, HumanEval 0-shot, MBPP 3-shot, BBH 3-shot.

Chat models, tab. 5. The paper's column headers give zero- or few-shot per benchmark. The table uses 5-shot for MMLU and C-Eval, 8-shot for GSM8K, 0-shot for HumanEval and 3-shot for BBH.

Other results from the paper:

Appendix tables with the Qwen rows (baselines in the same tables):

Paper vs code

Repository: https://github.com/QwenLM/Qwen, checked at commit 2df8e8ac450fa185c421a08b0090ef81826caa6e. The repository's README names the paper and the Qwen-7B and Qwen-14B releases. The repository is newer than the paper and also covers later models; the settings below are those in the code at that commit.

• Long-context settings. The paper describes NTK, dynamic NTK, LogN and per-layer windows §2.5. The repository's FAQ says the flags for dynamic NTK and LogN attention live in config.json and default to on FAQ.md. The FAQ spells the key use_dynamc_ntk; check the exact key in the released config. The repository's vLLM wrapper defaults to a single max_window_size of 6144 vllm_wrapper.py, not the per-layer windows of the paper. The technical memo's 7B perplexity rows match the paper's 7B rows in tab. 3 tech_memo.md.
• SFT learning rate. The paper uses peak 2e-6 with 4000 steps and batch 128 §3.1.2. The full-parameter fine-tuning recipe uses learning rate 1e-5, weight decay 0.1, warm-up ratio 0.01 and cosine decay finetune_fullparameter_single_gpu.ipynb.
• SFT sequence length. The paper uses 2048 §3.1.2. The training script defaults model_max_length to 8192 finetune.py.
• Pretraining context and tokens. The paper trains at 2048 §2.4 and gives 7B as 2.4T tokens tab. 1. The README says the original Qwen-7B used 2.2T tokens and 2048 context, and the updated 7B uses 2.4T and 8192 README.md. The paper's Table 1 matches the updated release, not the original.
• Reward model, PPO and pretraining code. Not in the repository. The repository mentions RLHF only in documentation. Neither the reward-model training nor the PPO loop is available here.
• Architecture code. The model class is not in the repository. The training script loads the config and model with trust_remote_code=True finetune.py, so the architecture comes from the model files on the model hub, which were not checked here.
• Fine-tuning defaults. The script's LoRA defaults (rank 64, alpha 16, dropout 0.05, target modules including c_attn, c_proj, w1, w2) are not in the paper finetune.py.
• Discrepancies found: the SFT learning rate, the SFT sequence length and the 7B token and context figures above. The long-context implementation is not checked against the paper's formulas because the paper gives none.

Pitfalls

• Context extension has no formulas in the paper. NTK base scaling, dynamic NTK scale, the LogN factor and per-layer window sizes are described in words only §2.5. Implementing them exactly requires the released code or the config flags, and the paper's per-layer window sizes are not stated.
• LogN wording. §2.5 says the LogN factor rescales "the dot product of the query and value". Attention logits normally come from query and key; read this as the standard query-key scaling, and confirm in the code.
• Token and context numbers differ by release. The paper's Table 1 uses 2.4T tokens for 7B; the repository's README says the original 7B used 2.2T tokens and 2048 context README.md. Use the release you run as the reference.
• Batch size unit. The pretraining batch "4M" in tab. 1 has no stated unit.
• Tokenizer details. The base is cl100k_base, extended with unspecified Chinese and multilingual tokens. Merge rules and the exact vocabulary are not in the paper. Take the tokenizer from the release.
• Pretraining data mix. Source proportions and upsampling factors are not stated. The decontamination (13-gram overlap) is described for instruction data only §2.1.
• PPO is expensive and under-specified. Four large models are held at once §3.2.2, so memory cost is high. The per-token form of the KL term, the pretraining-gradient coefficient, PPO batch size and step count, and the number of PPO updates per batch are not stated.
• Reward data cannot be reproduced. The helpful datasets are proprietary §3.2.1. Use the paper's results, not its data.
• Benchmark baselines. Baseline numbers are copied from official reports and OpenCompass §3.3 app. A.2. Prompts and shot settings may differ from those papers.
• Human evaluation is small. It uses 300 Chinese instructions and three annotators per item §3.3, and the figure values are not in the text.
• Tool-use and code-interpreter benchmarks are self-built. The tool-use set is Chinese-focused and the authors attribute part of GPT-3.5's poor score to that §3.4. Visualization correctness is judged by Qwen-VL §3.4.
• Code SFT is multi-stage §4.2, and the stages are not described. Do not assume a single-stage recipe.
• Licensing. The paper is CC BY 4.0. The repository's code is Apache-2.0 LICENSE. The model materials are under a separate Tongyi Qianwen agreement, whose commercial clause requires a licence for products with more than 100 million monthly active users Tongyi Qianwen LICENSE AGREEMENT. Check it before using the weights.

Lineage

• Builds on: LLaMA and LLaMA 2 for the architecture and training recipe §2.3 §2.1; ReAct prompting §3.4; self-instruct for tool-use data §3.4; NTK-aware interpolation, dynamic NTK, LogN scaling and window attention, all cited in §2.5 and not compiled here.
• Compared against: LLaMA, LLaMA 2, MPT, Falcon, Baichuan2, ChatGLM2, InternLM, XVERSE and StableBeluga2 §2.6; ChatGPT and GPT-4 in alignment and code evaluations §3.3 §4.3.
• Specialised siblings: Qwen-VL, a multimodal model built on these base models fig. 1. Code Llama is the main coding baseline §4.3.