StarCoder: may the source be with you!

In plain words

StarCoder is a large language model that writes and completes source code. The authors trained it on a large collection of public code, using only repositories under permissive licenses, and on related material such as GitHub issues, commits and notebooks. The base model was trained on about one trillion tokens of this data. A second, shorter training run specialised one version of it for Python.

Two design choices make it practical in an editor. It can fill a gap in the middle of a file, not only continue text to the right, because part of the training data is rearranged so that the model learns to write the missing middle. It also reads a long window of 8K tokens, and it uses multi-query attention, which shares one key and value head across all query heads.

On the common Python tests the reported scores are higher than those of the other open code models in the comparison tables. The authors also report that it keeps good results in many other languages. These are the authors' own measurements, and the card lists the setup for each.

TL;DR

StarCoderBase is a 15.5B-parameter code LLM trained on 1T tokens of permissively licensed code with 8K context, fill-in-the-middle and multi-query attention. StarCoder is the same model fine-tuned on 35B Python tokens. Author-reported pass@1 is 33.6 on HumanEval and 52.7 on MBPP for StarCoder. The official repository contains only a LoRA fine-tuning example, not the pretraining or data-formatting code, so the training recipe must be taken from this paper's text.

Problem

Code LLMs were mostly closed or only partly open. The authors say that access through a paid API limits safety research, inspection and contributions §1. Open models often released weights without the training data, or they carried licence restrictions on distribution §1. The paper therefore aims at a code model whose training data is public, governed and filtered for copyright and personal data, with a capable size and context length §1.

Method

The paper trains two models with one architecture §5.4 tab. 11: StarCoderBase is pretrained on the data curated in §3 and §5.1, and StarCoder is StarCoderBase fine-tuned on Python §5.

1. Source data. Restrict to The Stack v1.2, which holds only permissively licensed GitHub repositories §3. Keep 86 of the 358 languages in The Stack by file extension: languages above 500 MB, plus languages in the top 50 of two popularity rankings, plus dialects of selected languages §3.1. Add Jupyter notebooks §3.2, GitHub issues §3.3 and Git commits §3.4.
2. Near-deduplication. MinHash with LSH over all code files and notebooks, using 5-grams and a Jaccard threshold of 0.7 §3.5. Commits and issues were not deduplicated §3.5.
3. Weighting. Sample each source in proportion to its volume, except that JSON and YAML are capped at 1 GB and CSS at 3 GB §3.6.
4. PII redaction. A separate detection model and redaction pipeline are described in §4. They are not covered in this card.
5. Decontamination. Any file holding text from an evaluation set is dropped. The matched text is HumanEval and MBPP (docstrings and solutions), APPS (docstrings), GSM8K (questions) and DS-1000 (prompts) §5.2. Python lost 558 files on this filter §5.2.
6. Formatting. Each source uses its own template built from sentinel tokens §5.1 tab. 10. Code files get repository, file and star metadata prefixes and character-level fill-in-the-middle (FIM) §5.1.
7. Tokenizer. Byte-level BPE with 49,152 tokens, including the sentinel tokens; the pre-tokenizer splits digits and uses the GPT-2 regex §5.3.
8. Architecture. Decoder-only Transformer with multi-query attention and learned absolute position embeddings. Context 8K. FlashAttention is used, and the key and value are expanded before the attention call when MQA is used in training §5.4 tab. 11.
9. Pretraining. 250k iterations, batch 4M tokens, about 1T tokens in total, with Adam and a cosine learning-rate schedule §5.5. The distributed setup is in §5.6.
10. Fine-tuning to StarCoder. About 35B Python tokens, roughly two epochs, 8,500 steps, with a lower learning rate and the same other settings §5 §5.5.

The FIM transform used for code files, written as pseudocode in our own words:

# Build the training string for one code file  [§5.1]
# Input:  code (str); repo_name, filename (str); stars (int)
# Output: training string (str)
def format_code_sample(code, repo_name, filename, stars):
    # Each metadata prefix is added independently with probability 0.2
    # Literal reading of the text; see Pitfalls  [§5.1]
    prefix = ""
    if random() < 0.2:
        prefix += "<reponame>" + repo_name
    if random() < 0.2:
        prefix += "<filename>" + filename
    if random() < 0.2:
        # Stars fall into five buckets: 0, 1-10, 10-100, 100-1000, 1000+
        prefix += "<gh_stars>" + star_bucket(stars)
    prefix += "\n"
    # Character-level FIM applied with rate 0.5  [§5.1]
    if random() < 0.5:
        # PSM and SPMv2 each chosen with probability 0.5  [§5.1]
        mode = "SPM_V2" if random() < 0.5 else "PSM"
        code = fim_character_transform(code, mode)
    # Sample ends with the end-of-text token  [§5.1] [tab. 10]
    return prefix + code + "<|endoftext|>"

The learning-rate schedule of the base model, written as pseudocode:

# Learning rate for StarCoderBase pretraining  [§5.5]
# Input:  iteration (int), counted from 0 to 250000
# Output: learning rate (float)
def learning_rate(iteration):
    # Linear warmup over the first 2,000 iterations  [§5.5]
    if iteration < 2000:
        return 3e-4 * (iteration + 1) / 2000
    # Cosine decay from 3e-4 to 3e-5; the decay horizon is not stated
    # The total of 250,000 iterations is assumed here  [§5.5]
    progress = (iteration - 2000) / (250000 - 2000)
    return 3e-5 + 0.5 * (3e-4 - 3e-5) * (1 + cos(pi * progress))

Hyperparameters

Results

Author-reported. Setup: Python benchmarks (HumanEval, MBPP, DS-1000) and 19 languages (MultiPL-E) §6.1 §6.2.1. Models are 15.5B unless noted. Pass@1 on HumanEval and MBPP uses sampling temperature 0.2 with n = 200 samples for open models and n = 20 for API models §6.1.

Full HumanEval and MBPP comparison tab. 12:

Full MultiPL-E comparison, pass@1 on HumanEval translated to each language tab. 15:

Other results in the paper, not tabled here: ODEX, FIM single-line infilling, return-type and TypeScript prediction, docstring generation, security (Asleep at the Keyboard), per-checkpoint curves fig. 2 and perplexity with long contexts (§6.3, §6.4), and natural-language tasks (§7, §8, §9) §6.2 §6.4.

Paper vs code

The official repository is bigcode-project/starcoder at d72c7fe, licensed Apache-2.0. The paper links the BigCode organisation that owns it §10. It contains an example LoRA fine-tune and a chat demo. It does not contain pretraining code, the data pipeline, the FIM data formatting, the decontamination or deduplication code, or the evaluation harness used for the paper's tables. Evaluation code lives in a separate repository §6.1.

Differences between the paper and the code in finetune/finetune.py:

• The script is a LoRA (PEFT) fine-tune: rank 16, alpha 32, dropout 0.05, targets c_proj, c_attn, q_attn finetune.py finetune.py. The paper does not state whether its StarCoder fine-tune is full-parameter or parameter-efficient §5.5.
• The base model is loaded in 8-bit finetune.py. The paper trains in BF16 §5.6.
• Default learning rate is 5e-6 finetune.py. The paper's StarCoder fine-tune uses 5e-5 decaying to 5e-6 §5.5.
• Default warmup is 100 steps finetune.py. The paper uses 1,000 for the fine-tune §5.5 and 2,000 for pretraining §5.5.
• Default weight decay is 0.05 finetune.py. The paper uses 0.1 §5.5.
• Default sequence length is 2048 finetune.py. The paper trains at 8K §5.4.
• The script sets no Adam betas, so the Trainer defaults apply finetune.py. The paper states β2 = 0.95 §5.5.
• The default model path is bigcode/large-model and the default dataset is CodeAlpaca finetune.py, not StarCoder and not the Python data of the paper §5.
• Packing appends the tokenizer's end-of-sequence token, or id 49152 when the tokenizer has none finetune.py. The paper's vocabulary is 49,152 tokens §5.3. The fallback id is outside that range (inference from the numbers; see Pitfalls).
• Precision flags. --no_fp16 is a store-false flag, and the script passes fp16 = not no_fp16, so with default arguments fp16 is off and bf16 is on finetune.py finetune.py. The paper uses BF16 §5.6.

Discrepancies the paper does not address and the code does not cover: the pretraining schedule, data pipeline and parallel layout are described only in the paper §5.5 §5.6.

Pitfalls

• Metadata prefixes. The text says each of the repository, file and star prefixes is added "with a probability of 0.2" §5.1. The stated goal is to let the model work without metadata at inference, which suggests the prefix is dropped 20% of the time, not added 20% of the time. The paper does not settle it. The pseudocode uses the literal reading.
• Star buckets. The buckets are written as "1-10", "10-100" and "100-1000" §5.1. The boundary values 10, 100 and 1000 belong to two buckets in the text. Pick one rule and test it.
• SPMv2. The SPMv2 FIM layout is named but not defined §5.1. Implementing it requires the FIM paper the authors cite (Bavarian et al., 2022).
• Unused sentinel. <fim_pad> appears in the sentinel table tab. 10, but the text does not say where it is used.
• Commit windows. The window is "0 to 32 lines around the changed lines" §5.1. The paper does not say how the window is positioned or how the changed lines are located.
• Optimizer. The paper says Adam with weight decay 0.1 §5.5. It does not say whether the decay is decoupled (AdamW). Check the training framework's default.
• Decay horizon. The cosine schedule's end point is not stated. The pseudocode assumes the total of 250,000 iterations. This is an inference.
• Parallel layout. The text gives "tensor and pipeline parallelism rank 4" with 16 GPUs per replica §5.6. The 4 x 4 split is an inference from the GPU count.
• Global batch. The 4M-token batch is 512 sequences of 8K tokens. It equals micro-batch 1 times 16 accumulation steps times 32 data-parallel ranks §5.6. Other world sizes need a different accumulation count.
• End-of-sequence id. The paper's vocabulary is 49,152 tokens, so the ids run from 0 to 49,151 §5.3. The fallback id in the code is 49,152 finetune.py finetune.py. It is only used if the tokenizer has no end-of-sequence token, but it would be out of range for the paper's vocabulary.
• Claims versus table. The abstract says StarCoderBase matches or beats code-cushman-001. On HumanEval, StarCoderBase scores 30.4 against 33.5 tab. 12. The claim holds for StarCoder (33.6) and for MBPP. Check the table before quoting the claim.
• Languages not in training. D and Swift are reported in MultiPL-E, but the paper says they were not in the training set §3.1 tab. 15. Those columns measure transfer, not training coverage.
• MultiPL-E prompts. The translated prompts differ slightly from the original HumanEval and MBPP prompts. Three HumanEval problems are dropped §6.2.1. The effect is said to be small. Compare MultiPL-E numbers only with other MultiPL-E numbers.
• Checkpoint variance. For some low-resource languages, per-checkpoint pass rates vary in ways the authors attribute to chance §6.3. Single runs are noisy.
• Context and memory. The paper credits FlashAttention with allowing the 8K window §5.4. The text gives no memory figures for other attention kernels.
• Cost. Pretraining used 320,256 GPU hours on A100 80 GB GPUs, 512 of them across 64 nodes §5.6 §5.7.
• Precision default. Because of the inverted --no_fp16 flag, the script does not run fp16 unless --no_fp16 is passed. Check the precision that actually runs before comparing loss curves with the paper §5.6.
• Weights licence. The weights are released under an OpenRAIL-M-type agreement, not under this paper's CC BY-SA licence §1. Check the model licence before using the weights.
• Out of scope here. PII redaction (§4), the attribution tool (§9) and the demo are not covered in this card.

Lineage

• Builds on: SantaCoder (Allal et al., 2023), which the paper says it follows for architecture and tokenizer design §5.3 §5.4 and for deduplication §3.5. The Stack v1.2 (Kocetkov et al., 2022) is the source dataset §3. FIM (Bavarian et al., 2022) §5.1; multi-query attention (Shazeer, 2019) §5.4; FlashAttention (Dao et al., 2022) §5.4.
• Compared against: CodeGen-Multi and CodeGen-Mono, CodeGeeX, LLaMA, LaMDA, PaLM, code-cushman-001 and code-davinci-002 §6.1 tab. 12.
• Superseded by: none recorded in this wiki yet.