Program repair (LLM self-debugging)

Overview

Given a problem and a program that may be wrong, the model gets feedback about that program and writes a fixed version. The feedback comes from running the program against tests or sample inputs, from the model explaining the code, or from both 2304.05128 §3. The loop stops when the feedback says the program is correct or after 10 turns 2304.05128 §3 2304.05128 §5. A simpler form, one check with no loop, appears in text-to-SQL 2304.11015 §4.4.

The invariant all variants share is that no weights change. Generation, explanation, judgement and repair are all few-shot or zero-shot prompts to the same model 2304.05128 §1 2304.05128 §3 2304.11015 §4. What differs between variants is the feedback message and whether there is a loop at all. Test results are the strongest signal in the Self-Debugging paper. Without tests, the model judges its own explanation, and that explanation step is where the gain comes from 2304.05128 §5.1.

The corpus has two papers for this concept so far. The Self-Debugging paper gives the loop, the feedback formats and the test-based results. DIN-SQL gives a one-pass variant for text-to-SQL. Each claim below cites the paper that reports it. For the same draft-critique-rewrite loop on text tasks, see self-refinement.

Variants

Each variant takes its feedback from one of the two papers 2304.05128 §3 2304.11015 §4.4. Cards: Teaching Large Language Models to Self-Debug and DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction.

Simple verdict

The model says only whether the program is correct or wrong. There is no explanation step 2304.05128 §3. Results are mixed: on Spider with Codex it does not move the score (81.3 to 81.3), while on TransCoder it rises to 89.3 2304.05128 tab. 2. Without tests it can hurt; on MBPP with Codex it fell to 57.6 against a 61.4 baseline 2304.05128 tab. 3.

Unit-test feedback (UT)

The message adds the results of the failing unit tests. No explanation call is made 2304.05128 §3. Author-reported: TransCoder with Codex 80.4 to 91.6 and MBPP with GPT-4 72.8 to 80.6 2304.05128 tab. 2. Requires tests that can be run.

Code explanation (Expl)

Before judging, the model explains the program line by line [arxiv:2304.05128 §3]. This is the variant the paper credits for gains without tests: on Spider with Codex, 81.3 to 84.1 2304.05128 tab. 2. With tests, it adds a little on top of UT: TransCoder with Codex reaches 92.5, and on MBPP with GPT-4 it is 80.4 against 80.6 for UT alone 2304.05128 tab. 2. Each turn then costs one more model call 2304.05128 §3.

Execution trace

The model writes a line-by-line trace for a sample input. The paper notes that the trace is generated text, not real intermediate values, so it needs no more access than the explanation variant 2304.05128 §3. With tests it trails UT and Expl on TransCoder (87.9 for Codex) 2304.05128 tab. 2. Without tests it is the best of the no-test formats on MBPP with Codex (66.2) 2304.05128 tab. 3.

Text-to-SQL without tests (Spider)

There are no unit tests, so the loop works in three parts. The model restates the question and infers the column count. The SQL is executed, and the first two returned rows go into the explanation prompt. The model then checks its explanation against the question and rewrites the query if needed 2304.05128 §4.1 2304.05128 app. E.3. The loop is therefore not purely self-contained: execution feeds the explanation 2304.05128 §4.1.

Multi-sample start

Draw several initial programs. With tests, drop the ones that fail; then take the most common execution output among the rest, and debug one program from that group 2304.05128 §2. Author-reported: MBPP with Codex, 10 initial samples and UT+Expl, 72.2 to 75.6 2304.05128 tab. 1.

Single self-correction pass (DIN-SQL)

There is no loop and no run step. After decomposed generation, one zero-shot prompt shows the model the schema and the generated SQL and asks it to find and fix errors 2304.11015 §4.4. The paper has two wordings: a generic one that labels the SQL as buggy, and a gentle one that asks the model to check for possible issues. The default is gentle for GPT-4 and generic for CodeX 2304.11015 §4.4.

On Spider dev execution accuracy, the ablation gives these numbers 2304.11015 tab. 5:

• CodeX Davinci: 67.3 without self-correction, 69.9 generic, 68.7 gentle.
• GPT-4: 73.3 without self-correction, 74.2 gentle, 70.0 generic.

So the gain is 0.9 to 2.6 points, and the generic wording lowers GPT-4 by 3.3 points. Most development used Spider dev 2304.11015 §5.5.2, and each cell in the table is one number with no spread reported.

When to use which

Token cost and latency per turn are not stated in the Self-Debugging paper 2304.05128 §5. The call count per turn above is the only cost figure the paper gives.

Implementation sketch

Language-agnostic pseudocode for the common loop. The feedback choice is the only part that varies between variants.

# Self-debugging loop: frozen model, few-shot prompts, no weight updates
# Input:  problem, optional tests, llm, feedback in {ut, expl, trace}
# Output: last program written

def self_debug(problem, tests, llm, feedback="expl"):
    # Greedy first program when only one is drawn  [arxiv:2304.05128 §5]
    program = llm.generate(problem)
    for _ in range(10):  # 10 turns is the paper's limit  [§5]
        # Run on visible tests, or on a sample input  [§3] [§4.1]
        result = run(program, tests, problem)
        if feedback == "ut" and tests:
            # Verdict and failing-test list, no explanation call  [§3]
            verdict = llm.verdict_with_failures(problem, program, result)
        else:
            # Explain (or trace) the program, then judge  [§3]
            note = llm.explain(problem, program, result, mode=feedback)
            verdict = llm.judge(problem, program, note)
        if verdict.correct:
            break
        # Rewrite using the verdict and the failing details  [§3]
        program = llm.fix(problem, program, verdict)
    return program

The DIN-SQL self-correction pass is this loop with one turn and no run step. It is one prompt that gets the schema and the SQL and returns a corrected query 2304.11015 §4.4. The paper's wording is not transcribed here. The released code also returns the SQL unchanged when the model finds no problem; see the DIN-SQL card's Paper vs code section.

Contradictions

Between the papers, the same kind of bare self-check helps in one setting and hurts in another. Without tests on MBPP, the Simple verdict raised GPT-4 from 72.8 to 76.0 but lowered Codex from 61.4 to 57.6 2304.05128 tab. 3. On Spider dev, DIN-SQL's generic self-correction lowered GPT-4 from 73.3 to 70.0, while its gentle wording raised it to 74.2 2304.11015 tab. 5. The papers do not say which model, task or wording drives the difference.

The Spider Codex baselines also differ: Self-Debugging reports 81.3 2304.05128 tab. 2, and DIN-SQL reports 61.5 for few-shot CodeX Davinci 2304.11015 tab. 4a. The setups differ, so do not compare these numbers across papers.

Within the Self-Debugging paper:

• Some MBPP exemplar feedback says the code passes the assertion and then concludes that it is wrong 2304.05128 app. G.3 2304.05128 app. G.4. The paper does not say whether this is intended. Use consistent wording in your own prompts.
• Temperature is 0.7 for multi-sample runs 2304.05128 §5, but the TransCoder baseline prompt uses 0.8 2304.05128 app. F.1. The paper does not say which run uses which value.
• The MBPP baseline for Codex is 72.2 in one table 2304.05128 tab. 1 and 61.4 in another 2304.05128 tab. 2. The paper does not state the setup behind the first figure, so compare numbers only within one table.

Papers

• Teaching Large Language Models to Self-Debug: the prompt-only debugging loop, with four feedback formats and results on Spider, TransCoder and MBPP.
• DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction: a text-to-SQL pipeline whose last step is one zero-shot self-correction pass, with an ablation on Spider dev and results on Spider and BIRD.