Mistral 7B

In plain words

Bigger language models answer better, but they are slower and more expensive to run. Mistral 7B is a model with 7 billion parameters that tries to be small and fast while still scoring well on common tests. The authors report that it beats a larger model from an earlier family on most of them.

Two changes make it cheaper to run. First, several query heads share one set of stored keys and values, which cuts the memory needed while the model writes text. Second, each token in a layer only looks back at a fixed window of recent tokens, so the work per token does not grow with the full length of the text. Stacking layers still lets information travel past one window. The stored history is kept in a small buffer that overwrites its oldest entries, so its size stays fixed. The authors also release a chat-tuned version of the model.

TL;DR

Mistral 7B is a 7-billion-parameter Transformer language model released under Apache 2.0 §1. It combines grouped-query attention, with 8 key/value heads shared by 32 query heads tab. 1, with sliding-window attention of width W = 4096 tab. 1 §2, and a rolling-buffer key/value cache of fixed size W §2. Author-reported results show it ahead of Llama 2 13B on every benchmark row of Table 2 except one (NQ) tab. 2. Read the Pitfalls before reproducing any number: the paper leaves several details unstated, and the code differs in places.

Problem

Scaling a model raises computing cost and inference latency, which makes deployment harder §1. The paper targets a 7B model with high accuracy and low inference cost §1. In vanilla attention, the operations grow quadratically with sequence length and the memory grows linearly with the number of tokens, which raises latency and lowers throughput when the cache is large fig. 1.

Method

The model is a Transformer §2. It keeps the general shape of Llama and changes three things: sliding-window attention, a rolling-buffer cache and chunked pre-fill §2. Grouped-query attention is also used §1 tab. 1. The paper gives no equations and no algorithm box; the pseudocode below is written from the prose and figures.

1. Backbone. 32 layers, model width 4096, 32 query heads with head width 128, feed-forward width 14336, vocabulary of 32000 tokens tab. 1. The paper does not name the feed-forward activation, the position encoding or the normalisation; see Pitfalls for what the code does [code: src/mistral_inference/transformer_layers.py#L96-L106].
2. Grouped-query attention. 8 key/value heads serve 32 query heads tab. 1 §1. The ratio of 4 query heads per key/value head is derived from these two numbers; the code repeats each key/value head to match the query heads transformer_layers.py transformer_layers.py.
3. Sliding-window attention. Position i in layer k attends to the hidden states of the previous layer at positions between i − W and i §2. With W = 4096 and 32 layers, information can move about k × W tokens, a theoretical span of roughly 131K tokens at the last layer §2 fig. 1. The paper reports a 2× speed-up over a vanilla attention baseline for a 16K sequence with this window §2.
4. Rolling-buffer cache. The cache has a fixed size W. The keys and values for token position i are stored in slot i mod W, so later entries overwrite earlier ones once i exceeds W §2 fig. 2. The paper says this cuts cache memory by 8× at 32K tokens without affecting quality §2; no table backs the quality claim.
5. Pre-fill and chunking. The known prompt is pushed into the cache before generation. A long prompt may be split into chunks, and the paper suggests using W as the chunk size §2. For each chunk, attention covers the cache through the sliding window and the chunk itself through a causal mask §2 fig. 3.

# Sliding-window attention for one layer  [§2] [fig. 1]
# Input:  hidden (T, d) from the previous layer; window W
# Output: new_hidden (T, d)

def sliding_window_attention(hidden, window):
    # Each position i reads previous-layer states from i - W up to i  [§2]
    for position i in range(T):
        # The text says positions between i - W and i; the figure says
        # "at most W tokens". Whether this slice has W or W + 1 keys
        # is not settled by the paper.  [§2] [fig. 1]
        visible = hidden[max(0, i - window) : i + 1]
        new_hidden[i] = attend(hidden[i], visible, visible)
    return new_hidden

# Rolling-buffer cache write  [§2] [fig. 2]
# Input:  cache_k, cache_v of shape (W, n_kv_heads, head_dim)
#         key, value for absolute position i
# Output: updated cache

def write_to_cache(cache_k, cache_v, i, key, value, window):
    # Position i goes to slot i mod W; older entries are overwritten  [§2]
    slot = i % window
    cache_k[slot] = key
    cache_v[slot] = value
    return cache_k, cache_v

Hyperparameters

Results

Author-reported. Setup: the authors re-ran Llama 2 7B/13B and Code Llama 7B with their own pipeline §3. Few-shot settings are listed in §3: 0-shot for commonsense and reading comprehension, 5-shot for world knowledge, 8-shot with maj@8 for GSM8K, 4-shot with maj@4 for MATH, 0-shot for HumanEval, 3-shot for MBPP, and 5-shot for MMLU §3. The model is compared with Llama 1 34B only in the bar chart, not in Table 2 fig. 4; on MBPP the paper uses the hand-verified subset and on TriviaQA it gives no Wikipedia context §3.

Source: tab. 2. Bold marks the best value per column, as in the paper.

Chat model, MT-Bench and Chatbot Arena tab. 3:

On MT-Bench, Mistral 7B Instruct scores highest among the 7B models listed and close to the 13B chat models §4 tab. 3. Human preference counts (5020 for Mistral 7B against 4143 for Llama 2 13B, as of 6 October 2023) come from an external site; the sample size is not stated §4.

System prompt effect on MT-Bench tab. 4:

Safety: with the recommended system prompt, the model declined 100% of 175 unsafe prompts §5.1. Self-reflection moderation reached 99.4% precision and 95.6% recall on the authors' own adversarial and standard prompt set §5.2. The paper does not release the prompt set.

Efficiency: a 16K sequence with W = 4096 runs about 2× faster than a vanilla attention baseline with the FlashAttention and xFormers changes §2. The authors compare to "equivalent" Llama 2 sizes: on reasoning and MMLU, above 3× its size; on knowledge benchmarks, 1.9× §3 fig. 5.

Paper vs code

The code is the official reference implementation at commit 9eaeb91c17450e09021b6065a1d5cc69876507c8, linked from the abstract and the introduction §1. Its LICENSE is Apache-2.0. The commit is newer than the 2023 paper: it also contains Mixture-of-Experts, Mamba and vision-encoder modules, which the paper does not describe moe.py mamba.py vision_encoder.py. The v0.1 model configuration (params.json) is not in the repository. The code reads the configuration at run time, so the values in tab. 1 are not verified against code.

• Window size: the paper uses W = 4096 tab. 1. The code caps each layer's cache at sliding_window from the configuration, and prefill uses a local mask of that size cache.py cache.py. The value is not in the repo.
• Rotary base: the paper does not state it. When the configuration omits rope_theta, the code uses 1e6 transformer.py. This is a fallback; the v0.1 value is not in the repo.
• Position table: the paper gives a context length of 8192 tab. 1. The code precomputes rotary frequencies for 128,000 positions transformer.py.
• Chunked pre-fill: the paper suggests chunks of size W §2. In generate, chunk_size defaults to None, and the whole prompt is then processed as one chunk generate.py generate.py. Chunking is optional in code.
• Rolling buffer: the write uses slot = position mod window, as the paper says cache.py. Before a prefill step, the code reorders the cache into time order so that attention sees the right sequence cache.py cache.py.
• Guardrail system prompt: the text in §5.1 does not occur anywhere in the repository. The README points to external guardrailing docs instead README.md.
• Evaluation: the repository has no benchmark harness; tests/ contains test_generate.py only. The numbers in tab. 2 cannot be reproduced from this repository.
• Position encoding and feed-forward: the code uses rotary embeddings rope.py and a gated SiLU feed-forward, w2(silu(w1 x) * w3 x) transformer_layers.py. The paper names neither.
• Normalisation: the code uses RMSNorm with an epsilon from the configuration transformer_layers.py. The paper does not describe the normalisation.

Pitfalls

• Window boundary. The text says a position attends to states between i − W and i §2; the figure says "at most W tokens" fig. 1. These imply W or W + 1 keys per position. The paper does not decide, and the code's boundary is not checked here. Test the exact mask before relying on it.
• Effective reach is indirect. The 131K figure is a theoretical span from stacking layers §2. Only the last W tokens are read directly at each layer; older information reaches the position through the stacked states. The paper does not measure accuracy at that length; its only speed figure is at 16K.
• Quality claim without evidence. The 8× cache saving at 32K tokens is said to leave quality unchanged §2, but no table or ablation shows this.
• Cache is preallocated and rectangular. The reference cache allocates max_batch × W × n_kv_heads × head_dim per layer up front, and its own docstring calls this wasteful cache.py cache.py. The 8× saving is a saving in the paper's analysis, not a property of this code.
• Rotary and context numbers disagree. Context 8192 tab. 1 against 128,000 precomputed positions in code transformer.py and a fallback theta of 1e6. Do not copy the code's constants into a v0.1 reimplementation without checking the configuration file.
• Abstract overstates Table 2. The abstract and §3 say Mistral 7B beats Llama 2 13B on all benchmarks §1 §3. Table 2 shows Llama 2 13B higher on NQ, 29.0% against 28.8% tab. 2.
• Code is not a 2023 snapshot. The pinned commit has MoE, Mamba and vision modules. Read only the Mistral 7B path (transformer.py, cache.py, transformer_layers.py, generate.py) when reproducing this paper.
• Pre-fill with images is not supported in generate. The chunked path asserts no images generate.py. The paper does not mention images.
• Evaluation differences. MBPP uses a hand-verified subset and TriviaQA has no Wikipedia context §3. Scores are not comparable with published Llama 2 numbers without this change.
• Unreleased evaluations. The 175-prompt safety set and the 99.4% / 95.6% moderation figures use the authors' own data, which is not released §5.1 §5.2.
• Instruction data. The chat model's datasets are not named §4. "No proprietary data or training tricks" is an author statement, not a recipe.
• Costs the abstract hides. Grouped-query attention reduces decoding memory §1. The rolling buffer bounds the cache at W per layer §2, but the paper gives no end-to-end memory or latency table beyond the 16K speed figure §2.

Lineage

• Builds on: Grouped-query attention (grouped key/value heads §1); sliding-window attention from sparse Transformers and Longformer, arXiv:1904.10509 and arXiv:2004.05150 §1; FlashAttention and xFormers, used for the attention kernels §2; vLLM (/papers/2309.06180) for serving §1.
• Compared against: Llama 2 7B and 13B, arXiv:2307.09288 tab. 2; Llama 1 34B (/papers/2302.13971) fig. 4; Code Llama 7B, arXiv:2308.12950 tab. 2; Llama 2 Chat models tab. 3. (Code Llama and FlashAttention ids are from the paper's bibliography.)
• Superseded by: not stated in this wiki's content.