Gemma: Open Models Based on Gemini Research and Technology

In plain words

Gemma is a family of language models from Google DeepMind that anyone can download and run. Two sizes are released: a model with about 2 billion parameters and one with about 7 billion. For each size there is a raw "pretrained" version, which only learned to continue text, and a version tuned to follow instructions and hold a conversation.

The authors built the models with the same ideas and training recipes they used for their larger Gemini models, but scaled down. The bigger model was trained on more text than the smaller one. Both use a large shared vocabulary, and both were tuned afterwards with example conversations and with human preference ratings. The smaller model uses a simpler attention layout in which all query heads share one key and one value head.

The point of the release is that engineers can run, fine-tune and study these models themselves. The report also checks how often the models repeat their training text word for word, and it measures behaviour on safety tests. The authors describe the release as a careful choice, because open weights cannot be taken back once published.

TL;DR

• Gemma 2B and 7B are open decoder-only Transformers, pretrained on 3T and 6T tokens respectively, with instruction-tuned and RLHF-tuned variants § Pretraining § Instruction Tuning.
• The paper is a technical report: it gives architecture sizes and benchmark results, but no training hyperparameters such as learning rate or batch size § Training Infrastructure.
• Author-reported: the 7B pretrained model averages 56.9 over 18 academic benchmarks, against 54.5 for Mistral 7B tab. 6.
• Use this card for architecture facts, baseline numbers and known gaps. The linked repository is a later library, so its code does not confirm Gemma 1 settings directly (see Paper vs code).

Problem

The paper asks how to release a strong, small, openly available language model family. It targets the 2B and 7B scales for deployment on GPU, TPU, CPU and on-device hardware § Introduction. Its claim is that the models outperform similarly sized open models on 11 of 18 text-based tasks [abstract]. The paper's scope assumes primarily English text; the models are not multimodal and are not tuned for multilingual tasks § Training Data.

Because weights are published, the paper also treats misuse as a problem it cannot prevent: it notes that bad actors can fine-tune the models for harmful ends. It states that users should run safety testing specific to their use case before deployment § Introduction § Risks.

Method

The paper is a technical report with no algorithm box and no numbered equations. Sections are unnumbered in the compiled paper, so citations below use section titles instead of numbers.

Architecture § Model Architecture tab. 1. Gemma is a Transformer decoder with a context length of 8192 tokens. Relative to the original Transformer it uses:

1. Multi-query attention: the 2B model has one key/value head shared by 8 query heads; the 7B model uses full multi-head attention with 16 heads § Model Architecture tab. 1.
2. Rotary position embeddings (RoPE), applied in every layer; absolute position embeddings are not used. Input and output embeddings are shared § Model Architecture.
3. GeGLU feed-forward activations, in the approximated form § Model Architecture.
4. RMSNorm applied to the input of the attention and feed-forward sublayers § Model Architecture.

Training infrastructure § Training Infrastructure. The models are trained on TPUv5e pods of 256 chips in a 2D torus of 16 by 16. The 7B model runs on 16 pods (4096 chips), with 16-way model sharding and 16-way data replication inside a pod. The 2B model runs on 2 pods (512 chips) with 256-way data replication. Optimizer state is sharded in a way the paper describes as similar to ZeRO-3. Across pods, a data-replica reduce runs over the data-centre network, using the Pathways approach. The authors estimate about 131 tCO2eq for pretraining § Carbon Footprint.

Pretraining data § Training Data. The 2B model sees 3T tokens and the 7B model 6T tokens, primarily English web documents, mathematics and code. The tokenizer is a subset of the Gemini SentencePiece tokenizer with a 256k vocabulary. It splits digits, keeps whitespace, and falls back to byte-level encodings for unknown text. The mixture was chosen with ablations on both sizes, and training is staged so that higher-quality data gets more weight near the end § Training Data.

Filtering § Pretraining, Filtering. Pretraining data is filtered with heuristics and model-based classifiers for harmful, low-quality and personal content. Evaluation sets are removed from the mixture, and contamination checks are run § Pretraining, Filtering.

Supervised fine-tuning § Supervised Fine-Tuning § Formatting. The instruction-tuned models are fine-tuned on a mix of text-only, English-only synthetic and human-written prompt-response pairs. Mixtures are selected with side-by-side LM-based evaluations: responses from a test model and a baseline are shuffled, and a larger model picks the preferred one. The authors also filter synthetic data for personal information, unsafe outputs, self-identification errors and duplicates § Instruction Tuning, Filtering.

Formatter § Formatting tab. 3 tab. 4. Instruction-tuned models expect a control-token format. Roles are marked with user and model. Each turn starts with <start_of_turn> and ends with <end_of_turn>. The paper says output without the formatter is out of distribution and will very likely be worse.

RLHF § Reinforcement Learning from Human Feedback. A reward model is trained on human preference pairs under the Bradley-Terry model. The policy is then optimized against it with a reinforcement-learning algorithm that the paper calls novel but does not name. An automatic high-capacity rater is used to tune hyperparameters and to limit reward hacking.

The forward pass below is the paper's description, with details the paper leaves out taken from the official code, which is cited on each line.

# Gemma decoder forward pass (paper description, code-checked details)
# Input:  tokens (B, T) with ids in [0, 256128)
# Output: logits (B, T, 256128)

def forward(tokens):
    # Look up the shared embedding table and scale by sqrt(d_model)
    # [§ Model Architecture] [code: gemma/gm/nn/_modules.py#L99]
    hidden = embedding_table[tokens] * sqrt(d_model)    # (B, T, d)

    for layer in layers:
        # Normalize the attention input with RMSNorm
        # [§ Model Architecture] [code: gemma/gm/nn/_modules.py#L466]
        normed = rms_norm(hidden)                       # (B, T, d)

        # Project to queries, keys and values; 2B uses one K/V head
        # [§ Model Architecture] [tab. 1]
        # [code: gemma/gm/nn/_modules.py#L137-L142]
        queries = normed @ w_query                      # (B, T, H, dh)
        keys = normed @ w_key                           # (B, T, K, dh)
        values = normed @ w_value                       # (B, T, K, dh)

        # Apply rotary position embeddings to queries and keys
        # [§ Model Architecture] [code: gemma/gm/nn/_modules.py#L196-L208]
        queries = rope(queries, positions)
        keys = rope(keys, positions)

        # Scale queries by head_dim ** -0.5 (paper silent; code default)
        # [code: gemma/gm/nn/_config.py#L73-L75]
        # [code: gemma/gm/nn/_modules.py#L202]
        queries = queries * head_dim ** -0.5

        # Causal self-attention, grouped by key/value head
        # [§ Model Architecture]
        # [code: gemma/gm/nn/_modules.py#L240-L256]
        attn_out = causal_attention(queries, keys, values) @ w_out

        # Add the attention output back to the residual stream
        # [code: gemma/gm/nn/_modules.py#L482]
        hidden = hidden + attn_out                      # (B, T, d)

        # Normalize the feed-forward input with RMSNorm
        # [§ Model Architecture] [code: gemma/gm/nn/_modules.py#L435]
        normed = rms_norm(hidden)                       # (B, T, d)

        # GeGLU: GELU of one projection gates a second projection
        # [§ Model Architecture] [code: gemma/gm/nn/_modules.py#L380]
        gate = normed @ w_gate                          # (B, T, 2, F)
        mlp_out = (gelu(gate[..., 0, :]) * gate[..., 1, :]) @ w_down
        hidden = hidden + mlp_out                       # (B, T, d)

    # Final RMSNorm, then reuse the embedding table as the output projection
    # [§ Model Architecture] [code: gemma/gm/nn/_transformer.py#L303]
    # [code: gemma/gm/nn/_modules.py#L112]
    hidden = rms_norm(hidden)
    return hidden @ embedding_table.T                   # (B, T, vocab)

Equations. The paper contains no numbered equations. Its only formulas are the Bradley-Terry preference model named in the RLHF text and the verbatim-memorization definitions, which are stated in words.

Figures. No figure is reproduced here. fig. 1 is an average-score chart for Gemma 7B by capability. fig. 2 compares memorization across model families by size, and fig. 3 and fig. 4 show sensitive-data and approximate-memorization rates.

Hyperparameters

Only architecture values, data sizes and infrastructure are stated. Training and tuning hyperparameters are not given.

Results

Author-reported throughout. Each row below was checked against the paper's table; the paper does not state hardware for the evaluations.

Pretrained models, academic benchmarks tab. 6. Setup: each pretrained model is evaluated on 18 benchmarks with the metric and shot count listed; baselines are LLaMA-2 7B and 13B, Mistral 7B, and Gemma 2B and 7B. Values for Mistral 7B marked "(a)" were run by the authors; LLaMA-2 values are copied from the LLaMA-2 report.

Bold marks the best of the five columns as the paper prints it. The caption says Mistral reports 50.2 on a different MBPP split, and that on that split the authors' 7B model scores 54.5 tab. 6. The table lists Mistral at 40.2 (a).

Hugging Face H6 benchmarks tab. 7. Run independently by Hugging Face on the 7B models.

Human preference, instruction-tuned models tab. 5. Win rate against Mistral v0.2 7B Instruct with 95% confidence intervals and win/tie/loss breakdowns (ties split evenly). Safety prompts: about 400 held-out prompts. Instruction-following prompts: about 1000 held-out prompts covering creative writing, coding and instructions.

Safety benchmarks, Gemma 1.1 IT tab. 8. Evaluations run by the authors. Bold marks the best of the three columns as printed. The table does not give a metric direction for each row.

Appendix, Gemma 1.0 IT. The human-preference results for the first release tab. 9: Gemma 7B IT wins 58% on safety (CI 55.9–60.1) and 51.7% on instruction following (CI 49.6–53.8) against Mistral v0.2 7B Instruct. Gemma 2B IT wins 56.5% (CI 54.4–58.6) and 41.6% (CI 39.5–43.7). The safety benchmark table for 1.0 tab. 10:

Memorization § Memorization Evaluations fig. 2 fig. 3 fig. 4. The paper gives no numbers in text beyond these: about 50% more data is approximately memorized than exactly memorized fig. 4; no memorized sensitive data was found fig. 3; total memorization across the pretraining set is comparable to PaLM fig. 2. Method: 10,000 documents per corpus, first 50 tokens as prompt, exact memorization when the next 50 tokens match, approximate memorization at a 10% edit-distance threshold.

Paper vs code

Repository. The paper links no code repository. The official code was located by GitHub search in the google-deepmind organisation, whose name matches the paper's author line "Gemma Team, Google DeepMind" [title block]. The introduction mentions an "open-source codebase for inference and serving" without a link § Introduction. The code is Apache-2.0 LICENSE.

Later library. The repository at code_commit is a later version of the library. It defines Gemma 2, Gemma 3 and later models. It contains no Gemma 1 2B or 7B model class. The only Gemma 1 values are the layer counts _NUM_LAYERS_GEMMA_2B = 18 and _NUM_LAYERS_GEMMA_7B = 28 _gemma.py, which match tab. 1 but are not referenced elsewhere in the repository. Gemma 1 settings are therefore not verified against code. Several behaviours below come from the shared layer code and may differ for Gemma 1 checkpoints.

Architecture comparison.

• Attention layout: the paper uses multi-query attention for 2B and multi-head for 7B tab. 1. The code selects multi-head when key/value heads equal query heads, grouped-query attention when they lie between 1 and the query-head count, and the multi-query case otherwise _modules.py. The mechanism matches the paper; no Gemma 1 config confirms the values.
• RMSNorm: the paper does not give the form. The code uses x / rms(x) scaled by (1 + scale) with a zero-initialized scale and epsilon 1e-6 _layers.py.
• Embedding scaling: the paper is silent. The code multiplies the embedding lookup by sqrt(embed_dim) _modules.py.
• Tied embeddings: the paper says embeddings are shared between input and output § Model Architecture. The code computes logits as the hidden state times the transposed embedding table _modules.py. Consistent.
• GeGLU: the paper says "approximated" GeGLU § Model Architecture. The code computes gelu(gate_0) * gate_1 using the library's nn.gelu with its default arguments _modules.py. Whether this is the tanh approximation depends on the library default and is not checked here.
• Residual and block order: pre-normalization before attention and before the feed-forward layer, with residual additions after each _modules.py. Consistent with "normalize the input of each sublayer" § Model Architecture.
• Query scaling: the paper does not mention it. The default in the code is head_dim ** -0.5 _config.py _config.py.
• Post-normalization, soft-capping of logits or attention, and sliding-window attention exist in the code. They belong to later Gemma configs and are not in the paper; do not add them for Gemma 1 on the basis of this code.

Training. The repository contains inference, sampling and fine-tuning code. No pretraining recipe or training hyperparameters were found in the files checked, so the paper's training settings cannot be compared with code. The paper's sharding and TPU setup § Training Infrastructure has no code counterpart here.

No discrepancies found for the architecture points the code covers. Gemma 1 specifics are unverified (see above).

Pitfalls

• No training hyperparameters. Learning rate, schedule, warmup, batch size, optimizer settings, step count, SFT learning rate and epochs are all absent § Training Infrastructure § Instruction Tuning. An implementer must find them elsewhere or mark them as unknown.
• RL algorithm unnamed. The RLHF policy update is described only as a "novel" algorithm § Reinforcement Learning from Human Feedback. Clipping, KL control and rollout settings are not stated.
• Judge model unnamed. The side-by-side evaluation uses "a larger, high capability model" as judge, but its identity and prompt are not given § Instruction Tuning.
• Feed-forward width is ambiguous. The table lists 32768 and 49152 as "feedforward hidden dims" tab. 1. GeGLU uses two projections. The paper does not say whether these numbers are per branch or summed. The current code's hidden_dim is per branch _modules.py, but no Gemma 1 config confirms the convention. Check against the released checkpoint shapes before implementing.
• Code-only details. Embedding scaling by sqrt(d_model), the RMSNorm (1 + scale) form and epsilon, and the query scale come from the code, not the paper _modules.py _layers.py. Verify them against Gemma 1 weights before relying on them.
• RoPE base and scaling. Neither base frequency nor any scale factor is stated for Gemma 1. The code default is 10000 _modules.py; this is a library default, not a confirmed Gemma 1 value.
• Vocabulary numbers. Table 1 gives 256128; the text says "256k" § Training Data. Use 256128 when sizing the embedding table.
• Tokenizer reproduction. The tokenizer is a subset of Gemini's SentencePiece model with specific behaviour: digits split, whitespace kept, byte fallback § Training Data. The paper does not say whether the tokenizer model is released or how it was subset.
• Format matters. The paper states that generations without the control-token formatter are out of distribution and will very likely be worse § Formatting. Implementers must use the <start_of_turn> / <end_of_turn> format for the tuned models.
• Small model sensitivity. The HF H6 caption warns that small-model scores are sensitive to prompt changes tab. 7. Benchmark numbers are not directly reproducible without the same prompts.
• Mixed baseline methodology. LLaMA-2 values are copied from the LLaMA-2 report, not re-run; the authors could not run LLaMA-2 under its licence terms § Automated Benchmarks. Mistral values marked (a) were run by the authors. ARC, CommonsenseQA, BBH and AGIEval follow the Mistral report's methodology as closely as possible § Automated Benchmarks.
• The "11 of 18" claim. The abstract says the models outperform similarly sized open models on 11 of 18 tasks. In tab. 6, Gemma 7B is the bold best in 11 rows, which matches, but the paper does not state this rule. Check the table before quoting the count.
• "6 standard safety benchmarks". The conclusion says the models outperform competitors on 6 standard safety benchmarks § Discussion and Conclusion. The paper does not list which 6.
• Memorization numbers are plot-only. The memorization rates are given only in fig. 2–fig. 4; the text gives relative statements. Do not quote numbers beyond those in the text.
• Compute cost. The 131 tCO2eq estimate combines measured energy and a data-centre overhead the authors scale for; the method for that scaling is described only briefly § Carbon Footprint.

Lineage

• Builds on: Gemini (arXiv:2312.11805), whose architecture, data and recipes the paper says it draws on § Introduction; the Transformer decoder (arXiv:1706.03762) § Model Architecture; multi-query attention (arXiv:1911.02150), RoPE (arXiv:2104.09864), GeGLU (arXiv:2002.05202) and RMSNorm (arXiv:1910.07467), all cited in § Model Architecture.
• Compared against: Mistral 7B (arXiv:2310.06825) in tab. 5–tab. 10; LLaMA-2 (arXiv:2307.09288) in tab. 6; the paper also cites Falcon, CodeLLaMA and PaLM-family models in the text § Automated Benchmarks.
• Superseded by: not stated in this wiki. The linked repository contains later Gemma model families (see Paper vs code).