Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

In plain words

Most language models get better by getting bigger and by reading more raw web text. This report shows a small model, about 3.8 billion parameters, whose benchmark scores are close to much larger models and that is small enough to run on a phone.

The main lever is the training data rather than the model's size. The web text is filtered so that only material useful for learning reasoning and general knowledge is kept. Some of the training text is written by a large language model on purpose, to teach logic and niche skills. After pre-training, the model is tuned to follow instructions and to avoid harmful replies.

The authors also make larger versions, a Mixture-of-Experts model, and a version that reads images. The report gives the benchmark results with the settings the authors used, and it is open about where the small model is weak, such as facts it cannot store and languages other than English.

TL;DR

• phi-3-mini (3.8B parameters, 3.3T training tokens) is a Llama-2-style decoder with a 4K default context, and the authors report results close to Mixtral 8x7B and GPT-3.5 §1 §2 tab. 3.
• Training uses two sequential phases: web data for general knowledge, then more heavily filtered web data mixed with synthetic reasoning data §2.
• Post-training is supervised fine-tuning followed by DPO §2. A 4-bit build of phi-3-mini runs on an iPhone 14 at more than 12 tokens per second, offline §2 fig. 2.
• Larger siblings (phi-3-small 7B, phi-3-medium 14B) and phi-3.5 models (MoE 16x3.8B, Vision 4.2B) extend the family; all numbers are author-reported §2 §4 §7.
• No official code was found, and the report leaves core training hyperparameters unstated, so an implementation cannot be checked against code §2.

Problem

Scaling laws predict better results from larger models trained on fixed data §1. The authors argue that this assumption breaks down, because frontier LLMs can now filter and generate data §1. They target a small model that matches much larger models and fits on a phone §1 fig. 2. Their prior work, the phi series, showed that LLM-filtered web data plus LLM-written synthetic data can make small models strong §1.

The report does not state a cost for training runs or the hardware used beyond the token counts §2.

Method

Model family. Four model sizes with one recipe, plus later additions §2 §4 §7.

1. phi-3-mini (3.8B). Transformer decoder with a Llama-2-like block structure, so Llama-2 tooling can be reused. Same 32064-token vocabulary as Llama-2 with BoS removed and chat-template tokens added §2, footnote. Default context 4K, extended to 128K as phi-3-mini-128K with LongRope §2 §4.
2. phi-3-small (7B). tiktoken vocabulary of 100352 (unused tokens removed), default context 8192. GEGLU activation and muP hyperparameter transfer from a small proxy model §2. Grouped-query attention with 4 queries sharing 1 key §2. Blocksparse attention, with dense and blocksparse layers alternated §2 fig. 1.
3. phi-3-medium (14B). Same tokenizer and architecture style as phi-3-mini, 40 heads and 40 layers, embedding dimension 5120, trained on 4.8T tokens §2.
4. phi-3.5-MoE (16 x 3.8B, 6.6B active). Each MoE layer replaces the feed-forward block with 16 GLU experts; each token is routed to the top 2 §2. The sparse router is trained with the SparseMixer method §2. Total 42B parameters §2.
5. phi-3.5-Vision (4.2B). CLIP ViT-L/14 image encoder feeding phi-3.5-mini, with visual and text tokens interleaved and dynamic cropping of high-resolution images §7.1.

Pre-training data (phi-3-mini). Two disjoint sequential phases §2.

• Phase 1: mostly web sources, to teach general knowledge and language §2.
• Phase 2: a more heavily filtered subset of phase-1 web data, merged with synthetic data that teaches logical reasoning and niche skills §2.
• Filtering keeps pages with the right level of "knowledge" and pages that help "reasoning", and removes facts that only frontier-scale models need (the authors' example is a single sports result) §2 "Data Optimal Regime".
• The filter criterion is described as "educational level" §2 "Training Methodology". The exact classifier is not described.

Post-training. Supervised fine-tuning, then direct preference optimization §2. SFT data covers math, coding, reasoning, conversation, model identity and safety; the SFT mix starts with English-only examples §2. DPO data covers chat, reasoning and responsible-AI cases; DPO uses unwanted behaviours as the "rejected" responses §2.

Chat template. The model is already chat-tuned. Format: <|user|> newline, question, <|end|> newline, <|assistant|> §2.

Blocksparse attention (phi-3-small). Each head applies a different sparsity pattern over the KV cache, so all tokens are reached by some head §2. Training uses a Triton kernel based on FlashAttention §2. Inference uses a custom prefill kernel and an extension of vLLM's paged attention for decoding §2. The toy example in the figure uses 2 local blocks and vertical stride 3 fig. 1.

Mixture-of-Experts routing (phi-3.5-MoE). Pseudocode restating the top-2 routing described in §2. The paper gives no formula for the router or the SparseMixer gradient.

# Top-2 mixture-of-experts feed-forward for one token  [§2]
# Input:  hidden (d,) for one token; 16 GLU experts; router weights
# Output: weighted sum of the 2 selected expert outputs (d,)

def moe_ffn(hidden, experts, router):
    # Score every expert for this token  [§2]
    scores = router(hidden)                  # (16,)
    # Keep the 2 highest-scoring experts  [§2]
    chosen = top_k_indices(scores, k=2)      # (2,)
    # Combine the chosen expert outputs; the weighting rule is not stated  [§2]
    weights = gate_weights(scores[chosen])   # (2,)
    output = zeros_like(hidden)
    for index, weight in zip(chosen, weights):
        output = output + weight * experts[index](hidden)
    return output

Phi-3.5-Vision inputs. Images are split into a 2-D grid of blocks, and the block tokens are concatenated §7.1. Multi-image input concatenates the tokens of each image §7.1. Pre-training caps image resolution at 1344 x 1344 §7.1.

Phi-3.5-Vision training. Pre-training uses 0.5T tokens of interleaved image-text documents, image-text pairs, OCR-derived PDF data, chart and table data and text-only data; the loss is applied only to text tokens §7.1. SFT includes about 33B multimodal tokens; DPO mixes text and a smaller multimodal set §7.1.

Hyperparameters

Results

Author-reported. Setup: a shared internal evaluation pipeline, few-shot prompts, temperature 0 §3. Baselines: phi-2, Mistral-7B, Gemma-7B, Llama-3-Instruct-8B, Mixtral 8x7B and GPT-3.5 §3. Every model was run through the same pipeline, so numbers can differ from published ones §3.

Academic benchmarks §3, unnumbered table in that section. Reasoning and knowledge scores, phi-3 family and comparators.

Multilingual and long context §4.

• Multilingual MMLU average: phi-3.5-mini 55.4, phi-3-mini 47.3, phi-3.5-MoE 69.9 §4 fig. 4.
• RepoQA (long code, 128k context), average of five languages: phi-3.5-MoE 85, phi-3.5-mini 77, Llama-3.1-8B-Instruct 71 tab. 1.
• RULER at 4k vs 128k: phi-3.5-MoE 94.8 vs 64.2, phi-3.5-mini 94.3 vs 63.6, Llama-3.1-8B-Instruct 95.5 vs 77.0 tab. 2. The authors call the 128K drop "significant" and attribute it to limited long-context mid-training data §4.

Phi-3.5 model quality tab. 3. Setup as in the table caption: each model run through one pipeline.

Phi-3 safety §5 tab. 4 (lower is better). Internal multi-turn RAI benchmark judged with GPT-4 §5.

Phi-3.5-Vision, single image tab. 5. Setup: LLaVA-1.5 style prompts, single letter or short answer, 0-shot, no image preprocessing; images placed first §7.2.1. Numbers are author-reported.

Phi-3.5-Vision, multi-image and video tab. 6. Setup as stated in §7.2.2: BLINK is answered by the model directly, and VideoMME uses 16 uniformly sampled frames for all models.

Phi-3.5-Vision safety tab. 7 (internal and public multimodal RAI benchmarks, higher is better, scale 0–10) §7.3.

Headline comparisons in the text: phi-3-mini "rivals" Mixtral 8x7B and GPT-3.5 [abstract] §1; phi-3.5-MoE is "on par with" Gemini-1.5-Flash and GPT-4o-mini on language benchmarks §4 tab. 3.

Paper vs code

No official code.

Searches run: the paper, abstract and arXiv metadata (raw/arxiv/2404.14219.json) name no repository, project page or code link; the LaTeX source has no URL besides a HuggingFace Chat-UI screenshot caption §6 fig. 6. A GitHub repository search for "phi-3" under the microsoft organisation and for "Phi-3-mini" returned only sample apps and unrelated projects, none of which contains the model or training code. No discrepancies can be checked against code.

Pitfalls

• Training recipe is largely unstated. No learning rate, batch size, optimizer, schedule, sequence packing, epochs for phi-3-mini, or data-mixture proportions are given §2. Phase-1 and phase-2 token counts are not given §2. An implementer cannot reproduce pre-training from this report alone.
• "Educational level" filtering is undefined. The web-filter classifier, its labels and threshold are not described §2 "Training Methodology". Synthetic data generation prompts and its volume are not described either §2.
• Abstract vs table rounding. The abstract gives 69% MMLU for phi-3-mini [abstract]; the table gives 68.8 §3. Use the table for exact numbers.
• Benchmark settings change the numbers. The authors state that using ## before the question improved phi-3-mini on many benchmarks, but they did not use it in the reported runs §3, footnote. Prompts are an internal Microsoft tool and are not released §3. Shots are listed per benchmark §3. Treat cross-paper comparisons as indicative only §3.
• Long-context drop. phi-3.5 models lose much accuracy at 128K on RULER, e.g. phi-3.5-mini 94.3 at 4K vs 63.6 at 128K tab. 2 §4. The 128K RepoQA and RULER results are the only long-context evidence in the report.
• LongRope details are not in this report. phi-3-mini-128K and phi-3.5 long context rely on LongRope and a "mixed context window" §2 §4. The rescaling procedure and the mix ratio are not stated here. Cite the LongRope reference for the method (see Lineage).
• Blocksparse attention is under-specified. The production sparsity pattern, number of local blocks, stride per head, and dense-to-sparse layer ratio are not stated §2. The only concrete values are in the toy illustration (2 local blocks, vertical stride 3) fig. 1. Kernel availability is not stated, and the authors built custom Triton and vLLM-based kernels §2.
• Hidden inference cost of the MoE model. phi-3.5-MoE has 42B total parameters with 6.6B active §2. Memory footprint is therefore the total, not the active count. The report states no on-device figure for MoE.
• 4-bit on-device claim. The 1.8 GB and >12 tokens/s figures are for phi-3-mini quantized to 4 bits on iPhone 14 with an A16 chip §2 fig. 2. Quantization scheme, runtime, context length and prompt are not stated.
• Architecture "similar" to Llama-2, not identical. The text says the block structure is "similar" to Llama-2 §2. Normalization, positional encoding and activation details are not given for phi-3-mini. The text does say phi-3-small uses GEGLU §2.
• Only phi-3-small and phi-3-medium use the 4.8T-token budget. The mini model uses 3.3T §2 [abstract]. Compare the families with that difference in mind.
• Phi-3-medium is described as "slightly more epochs". The report gives no epoch count or token-level detail for this §2.
• Phi-3-mini is mostly English. The authors state that they "mostly restricted the language to English" §6. The phi-3-small multilingual result is described as initial §6.
• Figure numbers. Figure and table numbers in this card are counted from the LaTeX source order. The source labels fig:1 for the iPhone figure, which is the second figure in source order. Check the compiled PDF before relying on a number.
• Vision: unknown visual token count. Dynamic cropping splits images into blocks, but the block size, the maximum number of blocks, and tokens per block are not stated §7.1.
• Vision: evaluation departs from published pipelines. For BLINK the authors do not use ChatGPT for answer selection, and for VideoMME they use 16 frames for all models §7.2.2. Compare to published baselines with care.
• Vision safety is partial. The authors say the model sometimes fails to refuse harmful inquiries, such as deciphering captchas or describing scam images, and they link this to OCR abilities from instruction data §7.4.
• Safety evaluation uses GPT-4 as judge and in-house data. The RAI scores come from a GPT-4-simulated multi-turn benchmark and an internal red team §5 tab. 4. They are not reproducible from the paper.

Lineage

• Builds on: the "Textbooks Are All You Need" line of work, arXiv 2306.11644 §2, and the phi-2 model §1 (a blog post in the bibliography, not an arXiv paper). Llama-2 block structure and tokenizer §2; LongRope for long context (arXiv 2402.13753) §2; FlashAttention §2 and vLLM paged attention, see /papers/2309.06180 §2.
• Compared against: phi-2, Mistral 7B, Gemma 7B, Llama-3-Instruct 8B, Mixtral 8x7B, GPT-3.5 §3; Llama-3.1-8B, Mistral-Nemo 12B, Gemma-2 9B, Gemini-1.5 Flash, GPT-4o-mini tab. 3; LLaVA-1.6, Qwen-VL-Chat, GPT-4V tab. 5 tab. 7.
• Superseded by: not known from content already in this wiki.