---
kind: concept
slug: code-completion
title: Code completion
summary: >-
  Suggesting code while typing, and chat help with coding tasks. Covered by Code Llama and StarCoder (infilling, benchmark pass rates), CodeGeeX (13B model with editor plug-ins), phi-1 (1.3B model on curated data) and a student study of Copilot and ChatGPT. No head-to-head comparison exists, so no tool is ranked. Numbers are author-reported.
tags: [code, infilling, education, human-factors]
papers: ["2303.17568", "2305.06161", "2306.11644", "2308.12950", "2405.17739"]
related: [human-evaluation, instruction-tuning, positional-encoding, language-model-pretraining, decoding-strategies, dataset-curation, multilingual-evaluation, post-training-quantization, multi-query-attention]
status: draft
updated_at: 2026-10-24
---

## Overview

Code completion covers two kinds of help while programming. The first is a
model that proposes code at the cursor, including code that must fit between
text already written before and after it. The second is a chat assistant that
answers questions about the task. Five cards cover the topic from different
angles. The [Code Llama card](/papers/2308.12950) describes models trained to
fill a gap in code and measures them on benchmarks [arxiv:2308.12950 §2.3]
[tab. 2]. The [StarCoder card](/papers/2305.06161) describes a 15.5B model
trained on permissively licensed code, with fill-in-the-middle, and reports
HumanEval and MBPP pass@1 [arxiv:2305.06161 §3] [§5.1] [tab. 12]. The [CodeGeeX card](/papers/2303.17568) describes a 13B left-to-right
model, a multilingual HumanEval variant, and editor plug-ins built on the model
[arxiv:2303.17568 §1] [§3.1]. The [phi-1 card](/papers/2306.11644) describes a
small model trained on filtered and generated Python data, and it reports
HumanEval function completion from docstrings [arxiv:2306.11644 §1] [tab. 1].
The [Widening Gap study](/papers/2405.17739) describes how 21 CS1 students used
GitHub Copilot in VS Code and ChatGPT in a browser tab, and it does not measure
the tools themselves [arxiv:2405.17739 §3.3].

Autoregressive training alone does not let a model fill a missing span while
using the text on both sides. The infilling objective is meant to close that
gap [arxiv:2308.12950 §1].

The main finding of the user study is that students who struggled accepted
more Copilot suggestions on average, and often reached a fix through ChatGPT.
The authors add three difficulties to their earlier taxonomy: Progression,
Interruption and Mislead [arxiv:2405.17739 tab. 2].

## Variants

### Infilling (fill-in-the-middle)

The model sees a prefix and a suffix and writes the middle. In training, 90% of
documents are cut this way and the rest are left whole [arxiv:2308.12950 §2.3].
Two layouts are used in equal share: prefix-suffix-middle (PSM) and
suffix-prefix-middle (SPM) [arxiv:2308.12950 §2.3]. The objective costs on
average 0.6 points at 7B and 1.1 points at 13B across HumanEval and MBPP
pass@1, pass@10 and pass@100, and the autoregressive test loss is essentially
unchanged [arxiv:2308.12950 §3.2]. Single-line HumanEval infilling in PSM
reaches 77.0 pass@1 at 7B and 80.7 at 13B, before long-context tuning
[arxiv:2308.12950 app. E].

The released scope is narrower than the paper's claim. The abstract and §2.1
say 70B models support infilling, but the repository README lists 7B and 13B
only [Code Llama card, Paper vs code]. See Contradictions.

StarCoder also trains with fill-in-the-middle, applied at a rate of 0.5 and
at character level [arxiv:2305.06161 §5.1]. Its two layouts are PSM and a
second one called SPMv2, which the paper names but does not define
[arxiv:2305.06161 §5.1]. The paper reports single-line infilling results that
this page does not tabulate [arxiv:2305.06161 §6.2].

### Base and Python models (left-to-right)

The base model continues code from the cursor to the end. The Python variant
gets 100B more tokens, 75% of them Python [arxiv:2308.12950 §2.1] [tab. 1].
The Python models at 7B, 13B and 34B are trained without infilling
[arxiv:2308.12950 §2.1]. HumanEval pass@1, greedy and zero-shot, for base and
Python: 33.5 and 38.4 at 7B, 36.0 and 43.3 at 13B, 48.8 and 53.7 at 34B, and
53.0 and 57.3 at 70B [arxiv:2308.12950 tab. 2].

### Instruct (plain-language requests)

The instruct variant answers requests written in plain language
[arxiv:2308.12950 §2.5]. Its training mix has three parts: a proprietary Llama
2 instruction set, a self-instruct set of questions with unit-tested solutions,
and a small rehearsal mix [arxiv:2308.12950 §2.5]. The 70B Instruct model
starts from Python 70B [arxiv:2308.12950 §2.1]. HumanEval pass@1 is 34.8, 42.7,
41.5 and 67.8 at 7B, 13B, 34B and 70B [arxiv:2308.12950 tab. 2]. At 34B this is
below the base model's 48.8, and the paper attributes part of that cost to
safety tuning [arxiv:2308.12950 §4] [§6]. In the paper's ablation, adding the
self-instruct data raised the 7B HumanEval score from 30.5 to 34.8
[arxiv:2308.12950 §3.4].

### Long context

Long-context fine-tuning (LCFT) trains on 16,384-token sequences and raises the
RoPE base from 10^4 to 10^6 [arxiv:2308.12950 §2.4]. Single-line completion on
LCC-balanced improves on exact match: 39.23 against 36.86 at 7B, 41.06 against
37.96 at 13B, and 44.89 against 42.52 at 34B [arxiv:2308.12950 §3.3]. The cost
is an average drop of 0.52 points on HumanEval pass@1 and 1.9 points on MBPP
pass@1 [arxiv:2308.12950 §3.3]. Retrieval is strong within 16K tokens and
weaker beyond it [arxiv:2308.12950 §3.3]. Which sizes received LCFT is
inconsistent in the card; see Contradictions.

### Permissively trained model (StarCoder)

StarCoderBase is a 15.5B decoder-only model with 8K context and multi-query
attention. It was trained on 1T tokens of code from repositories under
permissive licences [arxiv:2305.06161 §3] [§5.4] [§5.5]. StarCoder is the same
architecture fine-tuned on 35B Python tokens [arxiv:2305.06161 §5]. The
author-reported pass@1 is 33.6 on HumanEval and 52.7 on MBPP for StarCoder,
and 30.4 and 49.0 for StarCoderBase [arxiv:2305.06161 tab. 12]. The training
data is restricted to 86 of 358 languages, with near-duplicate removal at a
Jaccard threshold of 0.7 [arxiv:2305.06161 §3.1] [§3.5].

The weights are not under the paper's CC BY-SA licence. They use an
OpenRAIL-M-type agreement [arxiv:2305.06161 §1], so check that licence before
use. The official repository holds an example LoRA fine-tune, not the
pretraining or data pipeline [StarCoder card, Paper vs code].

### Left-to-right model with editor plug-ins (CodeGeeX)

A 13B decoder-only model trained with next-token prediction on 850B tokens in
23 languages [arxiv:2303.17568 §2.1] [§2.3]. The card records no infilling
objective for it. The authors built extensions for VS Code, JetBrains and
Tencent Cloud Studio that offer completion, generation, translation and
explanation [arxiv:2303.17568 §1] [§5] [fig. 1]. The paper's tables do not
evaluate those extensions. In a survey of 168 users, 83.4% said their coding
efficiency improved or slightly increased. The survey is self-reported
[arxiv:2303.17568 §5] [fig. 11].

On HumanEval (Python), pass@1 is 22.89% against 19.22% for CodeGen-Multi-16B,
with pass@1 sampled at temperature 0.2 [arxiv:2303.17568 tab. 10] [§4.1]. Across
five languages in HumanEval-X, the average pass@1 is 18.40% against 16.73%
[arxiv:2303.17568 tab. 5]. The INT8 weights cut inference memory from 26.9 GB
to 14.7 GB for 128-token output on an A100 [arxiv:2303.17568 tab. 4].

The weights are licensed for non-commercial research only. The repository code
is Apache-2.0 [code: MODEL_LICENSE §2-§3, §6]. Do not use the weights in a
commercial product on the strength of the paper's licence or the code licence.

### Small model on curated data (phi-1)

A 1.3B decoder-only Transformer for Python. It is pretrained on about 7B tokens
of filtered web code and generated textbook text, then fine-tuned on under 180M
tokens of generated exercises, each one a docstring that the model completes
[arxiv:2306.11644 §2] [§2.2]. The model is Python-only [§6]. It is trained
without fill-in-the-middle, which the authors deliberately leave out
[arxiv:2306.11644 §2.3]. Author-reported: 50.6% HumanEval pass@1 and 55.5% MBPP
pass@1 [arxiv:2306.11644 tab. 1]. The filter threshold, the generation prompts
and the decoding setup are not stated [§2.1] [§2.2] [tab. 1].

The paper's own contamination check splits HumanEval by whether a close match
exists in the exercises. At τ = 0.95, pass@1 is 81.7% on the similar subset
and 26.9% on the rest [arxiv:2306.11644 tab. 3]. Retraining on pruned data
keeps the total at 50.6% for that τ, and the authors read this as evidence that
the gain is not only contamination [§5] [tab. 3].

### Chat assistance (ChatGPT)

The student asks questions in a browser tab. The paper does not report how
often students used it or which version they had [arxiv:2405.17739 §3.3]. Nine
of the ten struggling students reached a solution with GenAI help, which here
includes both tools [arxiv:2405.17739 §5.2].

### Inline suggestions (GitHub Copilot)

The tool suggests code in the editor. Acceptance is the share of suggestions a
student keeps. Across students it ranges from 10% to 53% [arxiv:2405.17739
tab. 3]. The students who struggled had a mean acceptance of 34.1%, against
24.5% for the others [arxiv:2405.17739 tab. 4]. No test is reported for that
difference, and the paper does not show that difficulty causes higher
acceptance [arxiv:2405.17739 tab. 4]. The paper gives no model, prompt format
or version for Copilot [arxiv:2405.17739 §3.3].

## When to use which

The corpus has no latency, memory, cost or head-to-head figures for code
completion tools, so this table covers only what the sources support.

| Situation | Use | Why |
|---|---|---|
| Single-line gap with code after the cursor | Infilling model in SPM layout | SPM is better on single-line infilling [arxiv:2308.12950 §3.2] [app. E]. Only 7B and 13B are released with infilling [Code Llama card, Paper vs code]. |
| Gap cut at arbitrary points | Infilling model in PSM layout | SPM is worse on random-span infilling [arxiv:2308.12950 §3.2]. SPM needs token healing when the prefix may end mid-token, and the paper's evaluation does not use it [app. E]. |
| Best HumanEval score at 34B, no chat | Base 34B, not Instruct 34B | 48.8 against 41.5 [arxiv:2308.12950 tab. 2]. The paper attributes part of the gap to safety tuning [§4]. |
| Python-only code, no infilling needed | Python variant | Higher HumanEval pass@1 at every size [arxiv:2308.12950 tab. 2]. It has no infilling at 7B, 13B or 34B [§2.1]. |
| Python function completion with a small model, and time to build a curated dataset | phi-1 recipe: filtering and generated textbooks and exercises | 50.6% HumanEval pass@1 at 1.3B, author-reported [arxiv:2306.11644 tab. 1]. The ablation at 350M reaches 20.12% with filtered and textbook data, against 12.19% unfiltered, though the two runs use different step budgets [§2.1]. The decoding and prompts are not stated, so the score cannot be reproduced exactly [§2.2] [tab. 1]. |
| Completion or generation in C++, Java, JavaScript or Go, for non-commercial research | CodeGeeX | HumanEval-X average pass@1 18.40% against 16.73% for CodeGen-Multi-16B [arxiv:2303.17568 tab. 5]. The weights are non-commercial only [code: MODEL_LICENSE §2-§3]. |
| Inference of a 13B model under a memory limit | INT8 weights, absolute-maximum scaling | 14.7 GB against 26.9 GB for FP16 on an A100 at 128 tokens [arxiv:2303.17568 tab. 4]. The paper scales per layer [eq. 4], but the code scales per row [code: codegeex/quantization/quantize.py#L55-L56], so check which one a reproduction needs. |
| Plain-language requests | Instruct variant | Trained for them [arxiv:2308.12950 §2.5]. HumanEval pass@1 is 67.8 at 70B against 53.0 for the base model [tab. 2]. |
| Fill-in-the-middle with a 15.5B model, where the training data's licence matters | StarCoder | Trained on permissively licensed code with 8K context and FIM [arxiv:2305.06161 §3] [§5.1]. HumanEval pass@1 is 33.6 in the authors' setup [arxiv:2305.06161 tab. 12]. The weights carry a separate OpenRAIL-M-type licence [§1], so check it before use. |
| Inputs beyond 16K tokens | Not settled by the sources | Retrieval is weaker beyond 16K [arxiv:2308.12950 §3.3], and the LCFT scope is inconsistent. |
| Whether suggestions help learners | Human study | The Code Llama numbers are benchmark pass rates, not learner outcomes [arxiv:2308.12950 tab. 2]. The student study reports associations in 21 students and makes no causal claim [arxiv:2405.17739 tab. 4]. |

## Implementation sketch

The infilling objective is the only part of this topic with an algorithm in
the sources. The training transform splits a document and reorders the parts:

```pseudocode
# Training: split one document and reorder it  [arxiv:2308.12950 §2.3]
# Input: document (str); output: token list in PSM or SPM order

def infilling_example(document, rng):
    # About 10% of documents stay whole  [§2.3]
    if rng.uniform() >= 0.9:
        return encode(document)
    # Cut the text into prefix, middle and suffix  [§2.3]
    prefix, middle, suffix = split_at_two_points(document, rng)
    # Half the examples use PSM and half SPM  [§2.3]
    if rng.uniform() < 0.5:
        return psm(prefix, middle, suffix)
    return spm(prefix, middle, suffix)

# Inference: the editor supplies the text around the cursor
def infilling_prompt(prefix, suffix):
    # PSM layout: prefix first, then suffix, then the middle marker  [§2.3]
    # The suffix is encoded without its implicit leading space
    # [Code Llama card, Paper vs code]
    return ([BOS, PRE] + encode(prefix) + [SUF]
            + encode_no_lead(suffix) + [MID])
    # Generation runs until a fourth special token ends the span  [§2.3]
```

Variants differ in the infilling layout and in how training text is formatted.
StarCoder uses character-level FIM at rate 0.5, with PSM or SPMv2 layouts
[arxiv:2305.06161 §5.1]. The paper names SPMv2 without defining it, so a
reproduction must take the layout from the FIM paper the authors cite
[StarCoder card, Pitfalls].

Decoding must match the metric. pass@1 is greedy. pass@10 and pass@100 use
nucleus sampling with p = 0.95 and temperature 0.8, and APPS uses temperature
0.6 [arxiv:2308.12950 tab. 2] [tab. 3]. Do not compare greedy pass@1 with
sampled pass@k.

## Contradictions

- **Infilling at 70B.** The abstract and §2.1 say the 70B base and Instruct
  models support infilling. The repository README lists infilling for 7B and
  13B only [arxiv:2308.12950 §2.1] [Code Llama card, Paper vs code]. The paper
  does not say whether 70B Python has infilling [arxiv:2308.12950 §2.1].
- **Which sizes got long-context tuning.** The card's Method and the summary of
  §2.1 say only the base 70B model received LCFT [arxiv:2308.12950 §2.1]. The
  card's hyperparameter and results sections give LCFT settings and LCC results
  for 7B, 13B and 34B [arxiv:2308.12950 §2.4] [§3.3]. The README says all
  models except 70B Python and Instruct support up to 100,000 tokens. The card
  says this agrees with §2.1, but on its face it does not
  [Code Llama card, Paper vs code]. This page cannot settle it.
- **RoPE base.** The paper gives 10^6 for LCFT [arxiv:2308.12950 §2.4]. The
  repository default is 10000, the Llama 2 value. The 10^6 setting is not in
  the repository, so it can be checked only in the checkpoint parameters
  [Code Llama card, Paper vs code].
- **Sampling defaults.** The repository's `generate` defaults to temperature
  0.6 and top-p 0.9. The paper's pass@10 and pass@100 setting is p = 0.95 and
  temperature 0.8 [arxiv:2308.12950 tab. 2] [Code Llama card, Paper vs code].
- **StarCoderBase against code-cushman-001.** The paper's abstract says
  StarCoderBase matches or beats code-cushman-001. On HumanEval it scores 30.4
  against 33.5, while on MBPP it scores 49.0 against 45.9 [arxiv:2305.06161
  tab. 12]. The claim holds for StarCoder (33.6 on HumanEval) but not for
  StarCoderBase on HumanEval [StarCoder card, Pitfalls].
- **Across papers.** No direct conflict. Code Llama, CodeGeeX, StarCoder and
  phi-1 all report HumanEval pass@1, but under different setups. Code Llama's
  HumanEval number is greedy and zero-shot [arxiv:2308.12950 tab. 2]. CodeGeeX
  samples at temperature 0.2 [arxiv:2303.17568 §4.1]. StarCoder samples at
  temperature 0.2 with 200 samples for open models [arxiv:2305.06161 §6.1].
  phi-1's decoding is not stated
  [arxiv:2306.11644 tab. 1]. CodeGeeX's HumanEval number is Python-only, and the
  paper's main benchmark is HumanEval-X [arxiv:2303.17568 tab. 10] [tab. 5]. The Widening Gap study measures student
  behaviour, so its numbers are not comparable with either. Treat each number as
  the authors' claim, not as a ranking.
- **Within the Widening Gap card.** In six of the seven struggling students
  whose narrative labels differ from Table 3, the two sources swap Progression
  and Interruption [arxiv:2405.17739 §4.1] [tab. 3]. Acceptance means computed
  from the Table 3 rows, 32.5% and 25.1%, do not match Table 4's 34.1% and
  24.5% [arxiv:2405.17739 tab. 3] [tab. 4]. Use the table values, and check
  the per-student labels before relying on them.

## Papers

- [Textbooks Are All You Need](/papers/2306.11644) (2023, cs.CL, cs.AI, cs.LG):
  phi-1, a 1.3B Python model trained on filtered web code, generated textbooks
  and generated exercises. Its HumanEval and MBPP results are author-reported.
- [Code Llama](/papers/2308.12950) (2023, cs.CL): Llama 2 further trained on
  code, with infilling, long-context tuning and instruction tuning, at 7B to
  70B. The paper is CC BY 4.0; the card records the repository under the Llama
  2 Community License, which must be read before reuse.
- [The Widening Gap](/papers/2405.17739) (2024, cs.AI and cs.HC): a lab study
  of 21 novices using Copilot and ChatGPT on one C++ problem, coded for
  metacognitive difficulties.
- [CodeGeeX](/papers/2303.17568) (2023, cs.LG, cs.AI, cs.SE): a 13B
  left-to-right code model in 23 languages, with the HumanEval-X benchmark and
  editor plug-ins. The weights are non-commercial; the repository code is
  Apache-2.0.
- [StarCoder](/papers/2305.06161) (2023, cs.CL, cs.AI, cs.PL, cs.SE): a 15.5B
  code model trained on permissively licensed code, with fill-in-the-middle
  and multi-query attention. Its benchmark numbers are author-reported.