Fill-in-the-middle (infilling)

Overview

Fill-in-the-middle (FIM) lets a model write a span that sits between text it can already see on both sides: code before the cursor and code after it. A plain left-to-right model only sees the text before the cursor. The approach described in the StarCoder card does not change the network. It rewrites part of the training data so that the model learns to produce the missing middle after a prefix and a suffix 2305.06161 §5.1 2305.06161 §5.4.

The idea is shared across variants: a training document is cut into three parts, prefix, middle and suffix, and the parts are reordered and tagged with sentinel tokens. The model is trained on the reordered text with the usual next-token loss.

Only one paper card in this wiki covers the technique. Other models that the StarCoder paper names as FIM models have no card here, so this page does not compare them. The code completion page gives the wider picture of editor-based code help.

Variants

Character-level FIM with PSM or SPMv2 layout (StarCoder)

StarCoder applies FIM to source code files only, at character level, with a FIM rate of 0.5. When a file is transformed, the layout is PSM with probability 0.5 and SPMv2 with probability 0.5 2305.06161 §5.1. The paper's sentinel tokens for this are <fim_prefix>, <fim_middle>, <fim_suffix> and <fim_pad> 2305.06161 tab. 10. Code files also carry repository, file and star metadata prefixes before the FIM step 2305.06161 §5.1.

The paper does not define the SPMv2 layout. It names the layout and cites the original FIM paper, Bavarian et al. (2022), for the FIM method [arxiv:2305.06161 §5.1]. That paper has no card in this wiki. An implementation that needs SPMv2 must take its definition from the original FIM paper.

Card: StarCoder: may the source be with you!.

Sentinel-tagged templates for other sources

The same paper uses sentinel tokens for three other data sources: GitHub issues, Jupyter notebooks and Git commits 2305.06161 §5.1. These are templates, not infilling. Commits show the code before a change, the commit message and the code after it, separated by sentinels. They are used to prompt the model to edit code from an instruction 2305.06161 §5.1. This variant is outside the FIM transform itself, so the rest of this page does not cover it.

When to use which

The infilling result is single-line, on Python, Java and JavaScript benchmarks built from HumanEval and MultiPL-E style prompts 2305.06161 §6.2. The card does not report multi-line or repository-level infilling, so do not extend the claim to those.

Implementation sketch

The transform applied to a code file, written in our own words. Only the PSM layout is shown, because SPMv2 is not defined in the paper.

# Apply the fill-in-the-middle transform to one code file  [§5.1]
# Input:  code (str); rate (float), 0.5 in StarCoder
# Output: training string (str)
def fim_transform(code, rate=0.5):
    # Leave the file unchanged with probability 1 - rate  [§5.1]
    if random() >= rate:
        return code
    # Cut the file at two character positions. The paper does not
    # say how the cut points are sampled  [§5.1]
    i, j = sorted(sample_two_cut_points(len(code)))
    prefix, middle, suffix = code[:i], code[i:j], code[j:]
    # PSM layout: prefix, then suffix, then the middle  [§5.1]
    # The order follows from the layout's name
    return ("<fim_prefix>" + prefix
            + "<fim_suffix>" + suffix
            + "<fim_middle>" + middle)

At inference, the prompt is the prefix and suffix in the same layout, ending at the <fim_middle> token, and the model generates the middle. The paper's text does not show this prompt. It follows from the training format and is an inference.

Contradictions

None known in the cards in this wiki. Only one card covers this topic.

Papers

• StarCoder: may the source be with you!: applies character-level FIM at rate 0.5 in PSM or SPMv2 layout during pretraining, with sentinel tokens, and reports single-line infilling results.