Code completion

Overview

Code completion covers two kinds of help while programming. The first is a model that proposes code at the cursor, including code that must fit between text already written before and after it. The second is a chat assistant that answers questions about the task. Five cards cover the topic from different angles. The Code Llama card describes models trained to fill a gap in code and measures them on benchmarks 2308.12950 §2.3 tab. 2. The StarCoder card describes a 15.5B model trained on permissively licensed code, with fill-in-the-middle, and reports HumanEval and MBPP pass@1 2305.06161 §3 §5.1 tab. 12. The CodeGeeX card describes a 13B left-to-right model, a multilingual HumanEval variant, and editor plug-ins built on the model 2303.17568 §1 §3.1. The phi-1 card describes a small model trained on filtered and generated Python data, and it reports HumanEval function completion from docstrings 2306.11644 §1 tab. 1. The Widening Gap study describes how 21 CS1 students used GitHub Copilot in VS Code and ChatGPT in a browser tab, and it does not measure the tools themselves 2405.17739 §3.3.

Autoregressive training alone does not let a model fill a missing span while using the text on both sides. The infilling objective is meant to close that gap 2308.12950 §1.

The main finding of the user study is that students who struggled accepted more Copilot suggestions on average, and often reached a fix through ChatGPT. The authors add three difficulties to their earlier taxonomy: Progression, Interruption and Mislead 2405.17739 tab. 2.

Variants

Infilling (fill-in-the-middle)

The model sees a prefix and a suffix and writes the middle. In training, 90% of documents are cut this way and the rest are left whole 2308.12950 §2.3. Two layouts are used in equal share: prefix-suffix-middle (PSM) and suffix-prefix-middle (SPM) 2308.12950 §2.3. The objective costs on average 0.6 points at 7B and 1.1 points at 13B across HumanEval and MBPP pass@1, pass@10 and pass@100, and the autoregressive test loss is essentially unchanged 2308.12950 §3.2. Single-line HumanEval infilling in PSM reaches 77.0 pass@1 at 7B and 80.7 at 13B, before long-context tuning 2308.12950 app. E.

The released scope is narrower than the paper's claim. The abstract and §2.1 say 70B models support infilling, but the repository README lists 7B and 13B only [Code Llama card, Paper vs code]. See Contradictions.

StarCoder also trains with fill-in-the-middle, applied at a rate of 0.5 and at character level 2305.06161 §5.1. Its two layouts are PSM and a second one called SPMv2, which the paper names but does not define 2305.06161 §5.1. The paper reports single-line infilling results that this page does not tabulate 2305.06161 §6.2.

Base and Python models (left-to-right)

The base model continues code from the cursor to the end. The Python variant gets 100B more tokens, 75% of them Python 2308.12950 §2.1 tab. 1. The Python models at 7B, 13B and 34B are trained without infilling 2308.12950 §2.1. HumanEval pass@1, greedy and zero-shot, for base and Python: 33.5 and 38.4 at 7B, 36.0 and 43.3 at 13B, 48.8 and 53.7 at 34B, and 53.0 and 57.3 at 70B 2308.12950 tab. 2.

Instruct (plain-language requests)

The instruct variant answers requests written in plain language 2308.12950 §2.5. Its training mix has three parts: a proprietary Llama 2 instruction set, a self-instruct set of questions with unit-tested solutions, and a small rehearsal mix 2308.12950 §2.5. The 70B Instruct model starts from Python 70B 2308.12950 §2.1. HumanEval pass@1 is 34.8, 42.7, 41.5 and 67.8 at 7B, 13B, 34B and 70B 2308.12950 tab. 2. At 34B this is below the base model's 48.8, and the paper attributes part of that cost to safety tuning 2308.12950 §4 §6. In the paper's ablation, adding the self-instruct data raised the 7B HumanEval score from 30.5 to 34.8 2308.12950 §3.4.

Long context

Long-context fine-tuning (LCFT) trains on 16,384-token sequences and raises the RoPE base from 10^4 to 10^6 2308.12950 §2.4. Single-line completion on LCC-balanced improves on exact match: 39.23 against 36.86 at 7B, 41.06 against 37.96 at 13B, and 44.89 against 42.52 at 34B 2308.12950 §3.3. The cost is an average drop of 0.52 points on HumanEval pass@1 and 1.9 points on MBPP pass@1 2308.12950 §3.3. Retrieval is strong within 16K tokens and weaker beyond it 2308.12950 §3.3. Which sizes received LCFT is inconsistent in the card; see Contradictions.

Permissively trained model (StarCoder)

StarCoderBase is a 15.5B decoder-only model with 8K context and multi-query attention. It was trained on 1T tokens of code from repositories under permissive licences 2305.06161 §3 §5.4 §5.5. StarCoder is the same architecture fine-tuned on 35B Python tokens 2305.06161 §5. The author-reported pass@1 is 33.6 on HumanEval and 52.7 on MBPP for StarCoder, and 30.4 and 49.0 for StarCoderBase 2305.06161 tab. 12. The training data is restricted to 86 of 358 languages, with near-duplicate removal at a Jaccard threshold of 0.7 2305.06161 §3.1 §3.5.

The weights are not under the paper's CC BY-SA licence. They use an OpenRAIL-M-type agreement 2305.06161 §1, so check that licence before use. The official repository holds an example LoRA fine-tune, not the pretraining or data pipeline [StarCoder card, Paper vs code].

Left-to-right model with editor plug-ins (CodeGeeX)

A 13B decoder-only model trained with next-token prediction on 850B tokens in 23 languages 2303.17568 §2.1 §2.3. The card records no infilling objective for it. The authors built extensions for VS Code, JetBrains and Tencent Cloud Studio that offer completion, generation, translation and explanation 2303.17568 §1 §5 fig. 1. The paper's tables do not evaluate those extensions. In a survey of 168 users, 83.4% said their coding efficiency improved or slightly increased. The survey is self-reported 2303.17568 §5 fig. 11.

On HumanEval (Python), pass@1 is 22.89% against 19.22% for CodeGen-Multi-16B, with pass@1 sampled at temperature 0.2 2303.17568 tab. 10 §4.1. Across five languages in HumanEval-X, the average pass@1 is 18.40% against 16.73% 2303.17568 tab. 5. The INT8 weights cut inference memory from 26.9 GB to 14.7 GB for 128-token output on an A100 2303.17568 tab. 4.

The weights are licensed for non-commercial research only. The repository code is Apache-2.0 MODEL_LICENSE §2-§3, §6. Do not use the weights in a commercial product on the strength of the paper's licence or the code licence.

Small model on curated data (phi-1)

A 1.3B decoder-only Transformer for Python. It is pretrained on about 7B tokens of filtered web code and generated textbook text, then fine-tuned on under 180M tokens of generated exercises, each one a docstring that the model completes 2306.11644 §2 §2.2. The model is Python-only §6. It is trained without fill-in-the-middle, which the authors deliberately leave out 2306.11644 §2.3. Author-reported: 50.6% HumanEval pass@1 and 55.5% MBPP pass@1 2306.11644 tab. 1. The filter threshold, the generation prompts and the decoding setup are not stated §2.1 §2.2 tab. 1.

The paper's own contamination check splits HumanEval by whether a close match exists in the exercises. At τ = 0.95, pass@1 is 81.7% on the similar subset and 26.9% on the rest 2306.11644 tab. 3. Retraining on pruned data keeps the total at 50.6% for that τ, and the authors read this as evidence that the gain is not only contamination §5 tab. 3.

Chat assistance (ChatGPT)

The student asks questions in a browser tab. The paper does not report how often students used it or which version they had 2405.17739 §3.3. Nine of the ten struggling students reached a solution with GenAI help, which here includes both tools 2405.17739 §5.2.

Inline suggestions (GitHub Copilot)

The tool suggests code in the editor. Acceptance is the share of suggestions a student keeps. Across students it ranges from 10% to 53% [arxiv:2405.17739 tab. 3]. The students who struggled had a mean acceptance of 34.1%, against 24.5% for the others 2405.17739 tab. 4. No test is reported for that difference, and the paper does not show that difficulty causes higher acceptance 2405.17739 tab. 4. The paper gives no model, prompt format or version for Copilot 2405.17739 §3.3.

When to use which

The corpus has no latency, memory, cost or head-to-head figures for code completion tools, so this table covers only what the sources support.

Implementation sketch

The infilling objective is the only part of this topic with an algorithm in the sources. The training transform splits a document and reorders the parts:

# Training: split one document and reorder it  [arxiv:2308.12950 §2.3]
# Input: document (str); output: token list in PSM or SPM order

def infilling_example(document, rng):
    # About 10% of documents stay whole  [§2.3]
    if rng.uniform() >= 0.9:
        return encode(document)
    # Cut the text into prefix, middle and suffix  [§2.3]
    prefix, middle, suffix = split_at_two_points(document, rng)
    # Half the examples use PSM and half SPM  [§2.3]
    if rng.uniform() < 0.5:
        return psm(prefix, middle, suffix)
    return spm(prefix, middle, suffix)

# Inference: the editor supplies the text around the cursor
def infilling_prompt(prefix, suffix):
    # PSM layout: prefix first, then suffix, then the middle marker  [§2.3]
    # The suffix is encoded without its implicit leading space
    # [Code Llama card, Paper vs code]
    return ([BOS, PRE] + encode(prefix) + [SUF]
            + encode_no_lead(suffix) + [MID])
    # Generation runs until a fourth special token ends the span  [§2.3]

Variants differ in the infilling layout and in how training text is formatted. StarCoder uses character-level FIM at rate 0.5, with PSM or SPMv2 layouts 2305.06161 §5.1. The paper names SPMv2 without defining it, so a reproduction must take the layout from the FIM paper the authors cite [StarCoder card, Pitfalls].

Decoding must match the metric. pass@1 is greedy. pass@10 and pass@100 use nucleus sampling with p = 0.95 and temperature 0.8, and APPS uses temperature 0.6 2308.12950 tab. 2 tab. 3. Do not compare greedy pass@1 with sampled pass@k.

Contradictions

• Infilling at 70B. The abstract and §2.1 say the 70B base and Instruct models support infilling. The repository README lists infilling for 7B and 13B only 2308.12950 §2.1 [Code Llama card, Paper vs code]. The paper does not say whether 70B Python has infilling 2308.12950 §2.1.
• Which sizes got long-context tuning. The card's Method and the summary of §2.1 say only the base 70B model received LCFT 2308.12950 §2.1. The card's hyperparameter and results sections give LCFT settings and LCC results for 7B, 13B and 34B 2308.12950 §2.4 §3.3. The README says all models except 70B Python and Instruct support up to 100,000 tokens. The card says this agrees with §2.1, but on its face it does not [Code Llama card, Paper vs code]. This page cannot settle it.
• RoPE base. The paper gives 10^6 for LCFT 2308.12950 §2.4. The repository default is 10000, the Llama 2 value. The 10^6 setting is not in the repository, so it can be checked only in the checkpoint parameters [Code Llama card, Paper vs code].
• Sampling defaults. The repository's generate defaults to temperature 0.6 and top-p 0.9. The paper's pass@10 and pass@100 setting is p = 0.95 and temperature 0.8 2308.12950 tab. 2 [Code Llama card, Paper vs code].
• StarCoderBase against code-cushman-001. The paper's abstract says StarCoderBase matches or beats code-cushman-001. On HumanEval it scores 30.4 against 33.5, while on MBPP it scores 49.0 against 45.9 [arxiv:2305.06161 tab. 12]. The claim holds for StarCoder (33.6 on HumanEval) but not for StarCoderBase on HumanEval [StarCoder card, Pitfalls].
• Across papers. No direct conflict. Code Llama, CodeGeeX, StarCoder and phi-1 all report HumanEval pass@1, but under different setups. Code Llama's HumanEval number is greedy and zero-shot 2308.12950 tab. 2. CodeGeeX samples at temperature 0.2 2303.17568 §4.1. StarCoder samples at temperature 0.2 with 200 samples for open models 2305.06161 §6.1. phi-1's decoding is not stated 2306.11644 tab. 1. CodeGeeX's HumanEval number is Python-only, and the paper's main benchmark is HumanEval-X 2303.17568 tab. 10 tab. 5. The Widening Gap study measures student behaviour, so its numbers are not comparable with either. Treat each number as the authors' claim, not as a ranking.
• Within the Widening Gap card. In six of the seven struggling students whose narrative labels differ from Table 3, the two sources swap Progression and Interruption 2405.17739 §4.1 tab. 3. Acceptance means computed from the Table 3 rows, 32.5% and 25.1%, do not match Table 4's 34.1% and 24.5% 2405.17739 tab. 3 tab. 4. Use the table values, and check the per-student labels before relying on them.

Papers

• Textbooks Are All You Need (2023, cs.CL, cs.AI, cs.LG): phi-1, a 1.3B Python model trained on filtered web code, generated textbooks and generated exercises. Its HumanEval and MBPP results are author-reported.
• Code Llama (2023, cs.CL): Llama 2 further trained on code, with infilling, long-context tuning and instruction tuning, at 7B to 70B. The paper is CC BY 4.0; the card records the repository under the Llama 2 Community License, which must be read before reuse.
• The Widening Gap (2024, cs.AI and cs.HC): a lab study of 21 novices using Copilot and ChatGPT on one C++ problem, coded for metacognitive difficulties.
• CodeGeeX (2023, cs.LG, cs.AI, cs.SE): a 13B left-to-right code model in 23 languages, with the HumanEval-X benchmark and editor plug-ins. The weights are non-commercial; the repository code is Apache-2.0.
• StarCoder (2023, cs.CL, cs.AI, cs.PL, cs.SE): a 15.5B code model trained on permissively licensed code, with fill-in-the-middle and multi-query attention. Its benchmark numbers are author-reported.