--- kind: concept slug: program-repair title: Program repair (LLM self-debugging) summary: >- A frozen language model fixes its own program by running it, explaining it or both, then rewriting it, for up to 10 turns. No training. Gains are largest when tests exist (TransCoder with Codex 80.4 to 92.5). Without tests, explaining helps. For text-to-SQL, one zero-shot self-check adds 0.9 to 2.6 points on Spider dev. Needs code you can execute or check. tags: [code-generation, prompting, self-correction, text-to-sql] papers: ["2304.05128", "2304.11015"] related: [self-refinement, llm-self-evaluation, in-context-learning, self-consistency, task-decomposition, chain-of-thought-prompting] status: draft updated_at: 2026-10-19 --- ## Overview Given a problem and a program that may be wrong, the model gets feedback about that program and writes a fixed version. The feedback comes from running the program against tests or sample inputs, from the model explaining the code, or from both [arxiv:2304.05128 §3]. The loop stops when the feedback says the program is correct or after 10 turns [arxiv:2304.05128 §3] [arxiv:2304.05128 §5]. A simpler form, one check with no loop, appears in text-to-SQL [arxiv:2304.11015 §4.4]. The invariant all variants share is that no weights change. Generation, explanation, judgement and repair are all few-shot or zero-shot prompts to the same model [arxiv:2304.05128 §1] [arxiv:2304.05128 §3] [arxiv:2304.11015 §4]. What differs between variants is the feedback message and whether there is a loop at all. Test results are the strongest signal in the Self-Debugging paper. Without tests, the model judges its own explanation, and that explanation step is where the gain comes from [arxiv:2304.05128 §5.1]. The corpus has two papers for this concept so far. The Self-Debugging paper gives the loop, the feedback formats and the test-based results. DIN-SQL gives a one-pass variant for text-to-SQL. Each claim below cites the paper that reports it. For the same draft-critique-rewrite loop on text tasks, see [self-refinement](/concepts/self-refinement). ## Variants Each variant takes its feedback from one of the two papers [arxiv:2304.05128 §3] [arxiv:2304.11015 §4.4]. Cards: [Teaching Large Language Models to Self-Debug](/papers/2304.05128) and [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](/papers/2304.11015). ### Simple verdict The model says only whether the program is correct or wrong. There is no explanation step [arxiv:2304.05128 §3]. Results are mixed: on Spider with Codex it does not move the score (81.3 to 81.3), while on TransCoder it rises to 89.3 [arxiv:2304.05128 tab. 2]. Without tests it can hurt; on MBPP with Codex it fell to 57.6 against a 61.4 baseline [arxiv:2304.05128 tab. 3]. ### Unit-test feedback (UT) The message adds the results of the failing unit tests. No explanation call is made [arxiv:2304.05128 §3]. Author-reported: TransCoder with Codex 80.4 to 91.6 and MBPP with GPT-4 72.8 to 80.6 [arxiv:2304.05128 tab. 2]. Requires tests that can be run. ### Code explanation (Expl) Before judging, the model explains the program line by line [arxiv:2304.05128 §3]. This is the variant the paper credits for gains without tests: on Spider with Codex, 81.3 to 84.1 [arxiv:2304.05128 tab. 2]. With tests, it adds a little on top of UT: TransCoder with Codex reaches 92.5, and on MBPP with GPT-4 it is 80.4 against 80.6 for UT alone [arxiv:2304.05128 tab. 2]. Each turn then costs one more model call [arxiv:2304.05128 §3]. ### Execution trace The model writes a line-by-line trace for a sample input. The paper notes that the trace is generated text, not real intermediate values, so it needs no more access than the explanation variant [arxiv:2304.05128 §3]. With tests it trails UT and Expl on TransCoder (87.9 for Codex) [arxiv:2304.05128 tab. 2]. Without tests it is the best of the no-test formats on MBPP with Codex (66.2) [arxiv:2304.05128 tab. 3]. ### Text-to-SQL without tests (Spider) There are no unit tests, so the loop works in three parts. The model restates the question and infers the column count. The SQL is executed, and the first two returned rows go into the explanation prompt. The model then checks its explanation against the question and rewrites the query if needed [arxiv:2304.05128 §4.1] [arxiv:2304.05128 app. E.3]. The loop is therefore not purely self-contained: execution feeds the explanation [arxiv:2304.05128 §4.1]. ### Multi-sample start Draw several initial programs. With tests, drop the ones that fail; then take the most common execution output among the rest, and debug one program from that group [arxiv:2304.05128 §2]. Author-reported: MBPP with Codex, 10 initial samples and UT+Expl, 72.2 to 75.6 [arxiv:2304.05128 tab. 1]. ### Single self-correction pass (DIN-SQL) There is no loop and no run step. After decomposed generation, one zero-shot prompt shows the model the schema and the generated SQL and asks it to find and fix errors [arxiv:2304.11015 §4.4]. The paper has two wordings: a generic one that labels the SQL as buggy, and a gentle one that asks the model to check for possible issues. The default is gentle for GPT-4 and generic for CodeX [arxiv:2304.11015 §4.4]. On Spider dev execution accuracy, the ablation gives these numbers [arxiv:2304.11015 tab. 5]: - CodeX Davinci: 67.3 without self-correction, 69.9 generic, 68.7 gentle. - GPT-4: 73.3 without self-correction, 74.2 gentle, 70.0 generic. So the gain is 0.9 to 2.6 points, and the generic wording lowers GPT-4 by 3.3 points. Most development used Spider dev [arxiv:2304.11015 §5.5.2], and each cell in the table is one number with no spread reported. ## When to use which | Situation | Use | Why | |---|---|---| | Programs can be run against complete example tests | UT feedback first; add Expl if an extra call per turn is affordable | TransCoder with Codex 80.4 to 91.6 (UT) and 92.5 (+Expl); MBPP with GPT-4 72.8 to 80.6 (UT) and 80.4 (+Expl) [arxiv:2304.05128 tab. 2]. On these rows Expl adds 0.9 and -0.2 points over UT | | Tests exist but only some are shown (MBPP: 1 of 3) | UT or UT+Expl; do not treat a visible pass as final | The model must judge the program even when the visible test passes; the feedback reports only the visible result [arxiv:2304.05128 §4.3] | | No tests, but you can execute the code and show the output (text-to-SQL) | Expl, with the first rows of the result | Spider with Codex: Simple 81.3 (no change), +Expl 84.1 [arxiv:2304.05128 tab. 2] | | No tests and no execution | Expl or Trace; avoid a bare verdict | Simple fell below baseline on MBPP with Codex (57.6 vs 61.4) but rose on TransCoder with Codex (83.4 vs 80.4); Expl 64.4 and Trace 66.2 on MBPP with Codex [arxiv:2304.05128 tab. 3]. DIN-SQL's generic self-correction lowered GPT-4 from 73.3 to 70.0 [arxiv:2304.11015 tab. 5] | | Decomposed pipeline, no tests, no execution, and a call budget that allows one more call | One zero-shot self-correction pass, using the gentle wording for GPT-4 and the generic wording for CodeX, then measure on your own data | Gains were 0.9 points (GPT-4, gentle) and 2.6 points (CodeX, generic) on Spider dev [arxiv:2304.11015 tab. 5]. The cost of that extra call is not stated; the whole pipeline is about $0.5 and 60 seconds per question with GPT-4 [arxiv:2304.11015 §7] | | Latency or call budget is tight | One greedy program, UT only, stop on the first pass | Each turn adds an explanation call and a feedback call on top of the fix [arxiv:2304.05128 §3]. Most of the gain comes in the first turn [arxiv:2304.05128 §5.2.1]; successful runs mostly end within 3 turns [arxiv:2304.05128 §5] | | You can afford several samples | Multi-sample start, then debug one program | On Spider with Codex, starting from 8 samples beats the 32-sample baseline [arxiv:2304.05128 §5.2.1] [arxiv:2304.05128 fig. 6] | | A small open model such as StarCoder (15.5B) | Expect smaller gains; measure on your own tasks | StarCoder on TransCoder 70.0 to 76.6 with +Expl, against Codex 80.4 to 92.5; on Spider 64.7 to 64.9 [arxiv:2304.05128 tab. 2] | Token cost and latency per turn are not stated in the Self-Debugging paper [arxiv:2304.05128 §5]. The call count per turn above is the only cost figure the paper gives. ## Implementation sketch Language-agnostic pseudocode for the common loop. The feedback choice is the only part that varies between variants. ```pseudocode # Self-debugging loop: frozen model, few-shot prompts, no weight updates # Input: problem, optional tests, llm, feedback in {ut, expl, trace} # Output: last program written def self_debug(problem, tests, llm, feedback="expl"): # Greedy first program when only one is drawn [arxiv:2304.05128 §5] program = llm.generate(problem) for _ in range(10): # 10 turns is the paper's limit [§5] # Run on visible tests, or on a sample input [§3] [§4.1] result = run(program, tests, problem) if feedback == "ut" and tests: # Verdict and failing-test list, no explanation call [§3] verdict = llm.verdict_with_failures(problem, program, result) else: # Explain (or trace) the program, then judge [§3] note = llm.explain(problem, program, result, mode=feedback) verdict = llm.judge(problem, program, note) if verdict.correct: break # Rewrite using the verdict and the failing details [§3] program = llm.fix(problem, program, verdict) return program ``` The DIN-SQL self-correction pass is this loop with one turn and no run step. It is one prompt that gets the schema and the SQL and returns a corrected query [arxiv:2304.11015 §4.4]. The paper's wording is not transcribed here. The released code also returns the SQL unchanged when the model finds no problem; see the DIN-SQL card's Paper vs code section. ## Contradictions Between the papers, the same kind of bare self-check helps in one setting and hurts in another. Without tests on MBPP, the Simple verdict raised GPT-4 from 72.8 to 76.0 but lowered Codex from 61.4 to 57.6 [arxiv:2304.05128 tab. 3]. On Spider dev, DIN-SQL's generic self-correction lowered GPT-4 from 73.3 to 70.0, while its gentle wording raised it to 74.2 [arxiv:2304.11015 tab. 5]. The papers do not say which model, task or wording drives the difference. The Spider Codex baselines also differ: Self-Debugging reports 81.3 [arxiv:2304.05128 tab. 2], and DIN-SQL reports 61.5 for few-shot CodeX Davinci [arxiv:2304.11015 tab. 4a]. The setups differ, so do not compare these numbers across papers. Within the Self-Debugging paper: - Some MBPP exemplar feedback says the code passes the assertion and then concludes that it is wrong [arxiv:2304.05128 app. G.3] [arxiv:2304.05128 app. G.4]. The paper does not say whether this is intended. Use consistent wording in your own prompts. - Temperature is 0.7 for multi-sample runs [arxiv:2304.05128 §5], but the TransCoder baseline prompt uses 0.8 [arxiv:2304.05128 app. F.1]. The paper does not say which run uses which value. - The MBPP baseline for Codex is 72.2 in one table [arxiv:2304.05128 tab. 1] and 61.4 in another [arxiv:2304.05128 tab. 2]. The paper does not state the setup behind the first figure, so compare numbers only within one table. ## Papers - [Teaching Large Language Models to Self-Debug](/papers/2304.05128): the prompt-only debugging loop, with four feedback formats and results on Spider, TransCoder and MBPP. - [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](/papers/2304.11015): a text-to-SQL pipeline whose last step is one zero-shot self-correction pass, with an ablation on Spider dev and results on Spider and BIRD.