Self-refinement (draft, critique, rewrite loop)

Overview

Self-refinement is a test-time loop around a model M: produce an output, ask M for feedback on that output, ask M to rewrite the output using the feedback, and stop on a fixed count or a signal in the feedback 2303.17651 §2 2303.17651 alg. 1. The variants below differ in where the feedback comes from. It can be the model's own judgement, a run of the output, a test suite, a solver's error message, a ranking among candidates, or a gate on stored state. The invariant most variants share is that no weights change at test time: the only things that change are the prompt, the output, or stored notes 2303.17651 §1 2308.10144 §4.3 2502.18864 p. 31. A 2023 survey files this loop under model feedback in planning, next to Reflexion's verbal feedback 2308.11432 §2.1.3. A 2024 survey of prompt-based reasoning puts Self-Refine in its step-control column, next to Reflexion and Tree-of-Thoughts, and labels that column reinforcement learning. That label is the survey's own classification; the loop itself trains nothing 2407.11511 §3.3.3 2407.11511 tab. 2. One variant breaks the invariant: Constitutional AI runs the same critique-and-revise steps offline, to build finetuning data 2212.08073 §3.1.

The feedback is meant to be actionable (it names a concrete change) and specific (it points at the phrase or code span to change) 2303.17651 §2. Where the rewrite sees the whole history of earlier drafts and critiques, the loop is a history-conditioned refinement 2303.17651 §2 (eq. 4).

Variants

The corpus has seventeen papers that use or describe this loop, each for a different purpose. The first four variants come from the Self-Refine paper, which defines the loop with a single model and few-shot prompts. The rest come from papers that use a run, a memory, a tournament or a versioned state as the feedback source, or that apply the critique loop inside a larger pipeline. Cards are linked in each entry.

Self-feedback with few-shot prompts (the default)

The same model, with three few-shot prompts (generate, feedback, refine). Feedback can carry numeric aspect scores; the output with the best total score across rounds is returned instead of the last one 2303.17651 app. M 2303.17651 app. H. This is the form used for dialogue and acronym tasks. Card: Self-Refine: Iterative Refinement with Self-Feedback.

Stop on a feedback signal

The loop ends when the feedback contains a task-specific phrase, such as a stop score or a "correct" marker, rather than after a fixed count 2303.17651 §2. The paper leaves the signal to each task; its concrete phrases are listed in the card's Paper vs code section.

Oracle feedback (external correctness signal)

Refine only when an external check says the answer is wrong. On GSM8K this lifts GPT-3.5 from 64.1 to 68.9 and ChatGPT from 74.8 to 76.2, where self-feedback alone gave 64.1 and 75.0 2303.17651 app. H. This variant uses ground truth, so it is not self-contained. The code-focused variant in the next section is the same idea with unit tests as the check.

Retrieved evidence with template or self-critique feedback (LLM-Augmenter)

A frozen, black-box LLM answers from evidence that a retriever and linker gathered first. A utility step then scores each candidate against that evidence and returns feedback, and the model is asked again until the answer passes. The paper states no maximum number of rounds and no threshold value 2302.12813 §3.2 2302.12813 §1. Feedback comes either from a template that fires on a low score, or from ChatGPT asked to criticise its own answer 2302.12813 §3.2. The score is knowledge F1, the token overlap between answer and evidence, so the loop checks grounding in the evidence, not correctness 2302.12813 §3.2. On customer-service dialog with BM25 retrieval, KF1 goes from 34.07 without feedback to 37.41 with the template and to 37.10 with self-criticism 2302.12813 tab. 4. ChatGPT alone reaches 31.33 on the same task 2302.12813 tab. 2. On OTT-QA, the paper reports Wiki QA F1 rising from 0.59 for ChatGPT alone to 11.80 with CORE evidence and feedback 2302.12813 tab. 5. Card: Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback.

Latest-output-only refinement

Refine sees only the latest output and its feedback, not the history (eq. 3 instead of eq. 4) 2303.17651 §2. The paper's code does this for code optimization; see the card's Paper vs code section for the details.

Execution feedback from tests or a run

The program is executed, and the result (failed unit tests, a runtime error, an output mismatch) goes into the feedback. Self-Debugging generates a program, runs it, has the model explain it line by line, judges it, and asks for a fix. It stops on a correct verdict or after 10 turns 2304.05128 §3 2304.05128 §5. Feedback can be a bare verdict, a verdict with the failed tests, a model-written explanation, or a model-written trace. A trace is generated text, not real intermediate state 2304.05128 §3. On TransCoder, where all 10 unit tests are in the prompt, Codex goes from 80.4 to 91.6 with test feedback and to 92.5 with test feedback plus explanation 2304.05128 tab. 2. On MBPP, where two of three tests are hidden, GPT-4 goes from 72.8 to 80.6 with test feedback 2304.05128 tab. 2 2304.05128 §4.3. Card: Teaching Large Language Models to Self-Debug.

MetaGPT uses the same pattern inside a multi-agent software team. The engineer writes unit tests, runs them, and debugs on failure, with a retry limit of three in the text 2308.00352 §3.3. The authors report +4.2 Pass@1 points on HumanEval and +5.4 on MBPP from executable feedback 2308.00352 §4.4. Card: MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.

Explanation feedback when no tests exist

When the output cannot be tested, the check is the model's own reading of a run. For text-to-SQL, the loop executes the query, passes the first two rows of the result to the model, and asks it to explain each clause and judge the query against the question 2304.05128 §4.1. This lifts Codex on Spider from 81.3 to 84.1. Simple feedback without an explanation left Codex at its baseline 2304.05128 tab. 2. On MBPP without tests, simple feedback lowered Codex from 61.4 to 57.6, below its baseline, while explanation feedback reached 64.4 2304.05128 tab. 3.

Solver error message in a formal program (Logic-LM)

The model translates a logic problem into a formal program, and a deterministic solver runs it. Only a program that fails to execute goes back to the model, with the solver's error message and a few demonstrations of common errors. The model then rewrites the program 2305.12295 §3.3 2305.12295 fig. 3. The paper gives no revision limit in its text; the figure shows zero to three rounds 2305.12295 §4.4. The paper names this module Self-Refiner and builds it on self-debugging and self-refine 2305.12295 §3.3. Refinement raises the share of programs that execute: the paper reports a mean gain of 5.01 points 2305.12295 §4.4. Accuracy on the programs that execute moves in both directions. For GPT-3.5 on FOLIO it rises from 61.8 to 64.3, while for GPT-4 on FOLIO it falls from 80.4 to 79.9, even though that model's execution rate rises from 79.9 to 85.8 percent 2305.12295 tab. 3. The main results table reports Logic-LM without refinement, so the end-to-end effect of this loop on accuracy is not shown there 2305.12295 §4 2305.12295 tab. 2. Card: Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning.

Verbal reflection in a bounded memory (Reflexion)

After each failed trial on the same task, the model writes a verbal lesson. The lesson goes into a memory that is read before the next trial, and only the last one to three lessons are kept 2303.11366 §3 2303.11366 §4.1. The evaluator can be exact-match grading, a heuristic, an LLM classifier, or unit tests that the LLM writes itself 2303.11366 §3 2303.11366 §4.3. On HumanEval (Python) the paper reports 91.0 pass@1 against 80.1 for GPT-4 2303.11366 tab. 2. It reports +22% absolute over ReAct on ALFWorld and +20% on HotPotQA 2303.11366 §1. On MBPP (Python) it reports 77.1, below the 80.1 baseline 2303.11366 tab. 2. Its numbers are author-reported. Card: Reflexion: Language Agents with Verbal Reinforcement Learning.

Retries with cross-task insight memory (ExpeL)

Each training task gets up to three reflection retries, with every trial stored 2308.10144 §4.1 2308.10144 tab. 4. The model then compares successful and failed attempts on the same task, and summarises successes across tasks, into a list of written insights 2308.10144 §4.2.2. Each insight carries a count that goes up or down as later comparisons agree or disagree. At test time the full insight list and the most similar past successes go into the prompt 2308.10144 §4.3. With gpt-3.5-turbo as the acting model, the authors report success rates of 28.0 to 39.0 on HotpotQA and 40.0 to 59.0 on ALFWorld, against plain ReAct 2308.10144 tab. 3. Tokens per HotpotQA trajectory rise from about 1,320 to 4,310 2308.10144 tab. 6. In the authors' ablation, feeding reflections into insight extraction lowered HotpotQA to 29.0, which they attribute partly to hallucinated reflections 2308.10144 tab. 3 2308.10144 §5.6. Card: ExpeL: LLM Agents Are Experiential Learners.

Reviewer agent in a group chat (coder and reviewer)

A second agent reviews the code for correctness, task fit and safety. It either sends the code back to the coder with feedback, or hands it to an executor 2411.16723 §3.1. The paper's text gives no round limit and no stop rule. The repository uses a maximum of 12 rounds and ends the chat when the reviewer writes the word "terminate"; the card's Paper vs code section covers this and other gaps. On a Boston Dynamics Spot, with 7 spoken prompts and 3 runs, the coder-and-reviewer setup had the fewest first-attempt code errors: 9 points below a single agent and 14 points below a planner-coder-reviewer team 2411.16723 §4 2411.16723 fig. 6. Observer scores on task success, safety and sociability did not differ significantly across the three configurations 2411.16723 §4 2411.16723 fig. 8. The paper reports no statistical test. Card: Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction.

Tournament critique and evolution (Co-Scientist)

A supervisor coordinates generation, review, ranking, proximity, evolution and meta-review agents over a task queue 2502.18864 p. 29–31. Hypotheses start at an Elo rating of 1200 and are compared pairwise. Top-ranked pairs get a multi-turn debate, and others a single comparison 2502.18864 p. 34. The meta-review turns the reviews and debates into a critique, which is added to every agent's prompt in the next iteration, with no fine-tuning 2502.18864 p. 31 2502.18864 p. 35–36. Across 203 research goals, the authors report the best hypothesis's Elo rising from about 1,350 to about 1,610 over ten time buckets 2502.18864 p. 7 2502.18864 fig. 2a; these are values read from a figure. The ratings come from the system's own LLM judgements, not from ground truth 2502.18864 p. 10. No public code is available 2502.18864 p. 40. Card: Accelerating scientific discovery with Co-Scientist.

Critique-and-revise inside a hypothesis generator (MOOSE-Chem)

Each candidate hypothesis is drafted, then critiqued on validness, novelty, clarity and significance, then revised. The paper builds on Self-Refine for this critique loop 2410.07076 §6.1. Each mutation line drafts once and revises twice, and only the final draft is scored; the mean of the four self-scores sets the beam and the final order 2410.07076 §4.2.3 2410.07076 §4.2.4. The loop sits inside a larger pipeline of literature screening and recombination, so the paper does not isolate its effect. Its ablations remove the multi-step and evolutionary parts, not the critique: the top Matched Score falls from 4.020 to 3.765 without multi-step and to 2.863 without multi-step and the evolutionary unit 2410.07076 tab. 10. The Matched Score is judged by GPT-4o, not by experts 2410.07076 app. A.14. Card: MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses.

Fixed reflection passes in a research pipeline (AI Scientist)

The pipeline has no external checker, so each stage uses a fixed number of self-reflection rounds. Idea generation runs three rounds on each idea and stops early when the model signals it is finished 2408.06292 §3 2408.06292 tab. 6. The write-up gives each section one reflection pass, then a final pass that removes repetition 2408.06292 §3. The reviewer averages several independent reviews, merges them, and runs five reflection rounds on the merged review 2408.06292 §4 2408.06292 tab. 6. The paper does not remove these passes in an ablation, so their effect on output quality is not measured. The authors report about $10 to $15 of API cost per generated paper 2408.06292 §6. The reviewer and the generator are both language models, so the score is a model judgement, not a reference. Card: The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.

Gated self-modification of versioned resources (Autogenesis)

Prompts, agents, tools, environments and memory are registered as versioned resources. A reflection loop proposes an edit, assesses the candidate, and commits it only if the assessment is accepted. A rejected candidate leaves the active version as it was 2604.15034 §3.2 2604.15034 app. G. In one complete run on GPQA-Diamond with gpt-4o, a prompt edit moved accuracy from 47.98 to 53.54 percent 2604.15034 tab. 1. In three repeated gpt-4o runs, a simple reflection baseline scored 54.04 on GPQA-Diamond against 47.98 for the vanilla model, and 10.00 against 13.33 on AIME24 2604.15034 app. B. The acceptance test is the same model's judgement, not a reference answer 2604.15034 §3.2. The paper does not show recovery after a faulty version goes live 2604.15034 §6. Joint edits used about 52 to 61 percent more output tokens than matched controls 2604.15034 app. B. The card records that the paper's linked repository, at the commit checked, did not contain the reflection loop under the paper's names, so these numbers are not reproduced here. Card: Autogenesis: A Self-Evolving Agent Protocol.

Survey-listed self-checks for hallucination (nine works)

A 2024 survey of hallucination mitigation files nine works under self-refinement: prompting GPT-3 for reliability, ChatProtect, Mind's Mirror, a self-reflection method for medical QA, structured comparative reasoning, DRESS for vision-language models, MixAlign, Chain-of-Verification (CoVe) and CoNLI 2401.01313 tab. 1 2401.01313 §2.1.2. CoVe drafts an answer, plans checks, answers them independently and rewrites the answer. CoNLI judges sentences and entities with a chain of natural language inference steps, then post-edits the text 2401.01313 §2.1.2. The table gives tasks and models for each work but no result values, and it marks all nine as covering detection, a term the survey does not define 2401.01313 tab. 1. The survey describes CoNLI as detection followed by post-editing, yet files it under self-refinement 2401.01313 §2.1.2, so the family label does not tell you the mechanism. Card: A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models.

Offline critique-revision for training data (Constitutional AI)

This variant runs the loop before deployment, not at test time. For each red-team prompt, the helpful model samples a response at temperature 1. Then, for four rounds, it draws one of 16 written principles, writes a critique, and writes a revision 2212.08073 §3.1 2212.08073 §3.2. The training pair is the prompt and the final revision; the critique text is not kept 2212.08073 §3.1. The paper uses 182,831 red-team prompts and finetunes a pretrained model for one epoch on all revisions and the helpfulness samples 2212.08073 §3.2. Against helpful RLHF, the result is more harmless and less helpful 2212.08073 §3.3 2212.08073 fig. 3. Harmlessness scores rise with each revision and helpfulness falls 2212.08073 fig. 5. Critiqued revisions beat direct revisions for smaller models, and the two are similar at 52B 2212.08073 §3.5 2212.08073 fig. 7. These results are author-reported and read from plots. The RL stage that follows uses AI preference labels, not this loop; the RLAIF concept page covers that stage. Card: Constitutional AI: Harmlessness from AI Feedback.

When to use which

Practical default: start with one or two rounds, a parseable feedback format and a stop rule that is a plain string match. If the task has a checker, make the checker the stop rule rather than the model's verdict 2304.05128 §3. If the tests are the model's own, treat a pass as weak evidence: the Reflexion paper measures false passes at 16% on MBPP Python 2303.11366 tab. 3. Measure against the single-pass output on your own tasks before adding rounds. Gains in the Self-Refine paper grow with the number of rounds, but the curve is flattening 2303.17651 fig. 4.

Implementation sketch

# Self-refinement loop  [arxiv:2303.17651 alg. 1]
# Input:  input x, model M (text in, text out), few-shot prompts
#         p_gen, p_fb, p_refine, stop(feedback, t) -> bool
# Output: rewritten output

def self_refine(x, M, p_gen, p_fb, p_refine, stop, max_rounds):
    # First draft from input-output examples  [eq. 1]
    output = M(p_gen + x)
    history = []
    for t in range(max_rounds):
        # Same model critiques its latest output  [eq. 2]
        feedback = M(p_fb + x + output)
        # Stop on a fixed count or a signal parsed from feedback  [§2]
        if stop(feedback, t):
            break
        history.append((output, feedback))
        # Rewrite using all earlier outputs and feedback  [eq. 4]
        prompt = p_refine + x
        for prev_output, prev_feedback in history:
            prompt = prompt + prev_output + prev_feedback
        output = M(prompt)
    # Return the last rewrite, or the best-scored output  [app. H]
    return output

Where variants differ: the source of the feedback (the model's own reading, a run of the output, a tournament, or a judge that is not the same model), the stop test (fixed count, parsed score, string match, or a test pass), whether the rewrite gets the full history (eq. 4), only the last pair (eq. 3), or a window of the last one to three reflections 2303.11366 §3, whether only an execution error triggers a rewrite, with the solver's message as feedback 2305.12295 §3.3, whether the feedback is checked against retrieved evidence 2302.12813 §2.4, whether the loop runs at inference or offline, where only the final revision is kept as a training pair and the critique is dropped 2212.08073 §3.1, whether a candidate must pass a gate before it replaces the active output 2604.15034 §3.2, and the selection rule at the end. Also keep the feedback format strict: a parse failure should be counted and handled, not silently dropped.

Contradictions

Within arxiv:2303.17651 the reported numbers do not agree everywhere. The GSM8K self-refine result is 64.1, 75.0 and 93.1 in the main table 2303.17651 tab. 1 but 62.4, 75.1 and 94.5 in the comparison table 2303.17651 app. F. The stated decoding temperature is also self-contradictory 2303.17651 §3.1. Treat small differences as noise and check the card's Pitfalls section before reproducing a number.

Across papers, the evidence on unaided self-judgement points in different directions. We do not resolve these; the tasks differ:

• Self-judgement without a check. On MBPP without tests, simple feedback lowered Codex from 61.4 to 57.6 2304.05128 tab. 3. The Self-Refine paper reports gains from self-feedback on its own tasks 2303.17651 tab. 1. MOOSE-Chem uses the same model's self-scores to rank hypotheses and reports a rank-ratio gain for drafts with matched inspirations, but its check is weak and its judge is a language model 2410.07076 tab. 8 2410.07076 app. A.14. LLM-Augmenter finds self-criticism close to its template feedback on KF1 (37.10 against 37.41) 2302.12813 tab. 4. KF1 measures overlap with retrieved evidence, not correctness, so this does not show that the answers are right.
• Critiques that are often wrong. Constitutional AI reports that its critiques were frequently inaccurate or overstated, yet the revisions still scored as more harmless 2212.08073 §3.5 2212.08073 app. A. The score comes from a preference model trained on AI labels, not from ground truth, so the revisions are not shown to be correct.
• Math and multiple-choice reasoning. Self-Refine reports almost no change on GSM8K 2303.17651 tab. 1. The Autogenesis paper reports a rise on GPQA-Diamond, a multiple-choice science set, from 47.98 to 53.54 for gpt-4o and from 65.15 to 68.69 for gpt-4.1, with a single run each 2604.15034 tab. 1. Its repeated runs show a mixed picture on AIME24 2604.15034 app. B. GPQA is not grade-school arithmetic, so the gap may reflect the task rather than a contradiction.
• Where the gain comes from on code. On the 50 hardest HumanEval problems in Rust with GPT-4, Reflexion's ablation scores 0.60 without the reflection text, 0.52 without generated tests, and 0.68 with both 2303.11366 tab. 4. Read that as the reflection text carrying the gain. Self-Debugging's TransCoder result points the other way: test feedback carries most of the gain, and the explanation adds 0.9 points (80.4 to 91.6, then 92.5) 2304.05128 tab. 2. The tasks, models and benchmarks differ, so we do not resolve this.
• Execution rate and accuracy. In the Logic-LM paper, refinement raises how many programs run, but accuracy on the programs that run moves both ways: up for GPT-3.5 on FOLIO, down for GPT-4 on FOLIO and AR-LSAT (60.0 to 58.8) 2305.12295 tab. 3. The released code refines only FOLIO and AR-LSAT, so the refinement rows for the other three datasets in that table cannot come from it; see the card's Paper vs code section 2305.12295 tab. 3.
• Two agents versus three. The paper's Discussion concludes that the two-agent setup performs better than the three-agent one 2411.16723 §5, and calls the effect significant. The Results text says observer scores do not differ significantly across configurations 2411.16723 §4. The paper reports no test, and each configuration was scored by one observer per run. Treat the conclusion as tentative.
• Typical size of the gain. The 2024 survey gives a typical Self-Refine gain of about 30 points 2407.11511 §3.3.3 2407.11511 tab. 2. The Self-Refine table cited above does not support a single typical figure: the GPT-4 gain is 0.2 points on GSM8K 2303.17651 tab. 1 and 49 points on dialogue preference 2303.17651 tab. 1. The survey does not say which tasks its figure averages over, so treat it as a loose summary.
• Size of the rounds. Self-Debugging reports that most successful runs end within three turns 2304.05128 §5, while the Self-Refine paper reports gains that grow with more rounds 2303.17651 fig. 4.

Papers

• Self-Refine: Iterative Refinement with Self-Feedback: defines the draft, critique, rewrite loop with a single model and few-shot prompts, and tests it on seven tasks.
• Reflexion: Language Agents with Verbal Reinforcement Learning: writes a verbal lesson after each failed trial and keeps the last one to three in memory; the evaluator is exact match, a heuristic, a classifier or self-written tests.
• A Survey on Large Language Model based Autonomous Agents: a 2023 survey whose taxonomy places Self-Refine and Reflexion under model feedback in planning; it reports no experiments of its own.
• Tree of Thoughts: describes self-refinement as a special case of its tree search 2305.10601 §3, and reports an iterative refine baseline on Game of 24 that uses ground-truth feedback 2305.10601 §4.1.
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback: wraps a frozen ChatGPT with evidence consolidation and a utility step whose feedback re-prompts the model until the answer passes.
• Teaching Large Language Models to Self-Debug: runs generated code or queries, has the model explain and judge them, and fixes them for up to 10 turns; gains depend on tests or on the run result.
• ExpeL: LLM Agents Are Experiential Learners: per-task reflection retries, then written cross-task insights that go into the prompt of a frozen agent.
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework: a software-team agent system whose engineer tests and debugs its code within a retry limit.
• Accelerating scientific discovery with Co-Scientist: a multi-agent hypothesis generator where a meta-review feeds critique back to every agent and an Elo tournament ranks the output.
• MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses: a three-step chemistry hypothesis pipeline whose candidates are critiqued and revised, then ranked by self-scores.
• Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning: an LLM writes a formal program, a solver runs it, and the program is rewritten only after an execution error, with the error message as feedback.
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery: an LLM pipeline that proposes ideas, runs experiments with a coding agent, writes a paper, and has it reviewed; each stage uses a fixed number of reflection passes.
• Autogenesis: A Self-Evolving Agent Protocol: versioned prompts, agents, tools and memory that change through a propose, assess, commit cycle.
• Constitutional AI: Harmlessness from AI Feedback: runs critique and revision offline on red-team prompts under 16 written principles, then finetunes on the revisions; its RL stage uses AI preference labels.
• Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction: a coder and a reviewer agent write robot code in a group chat, compared with one agent and with a planner-coder-reviewer team.
• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models: a survey that files nine self-refinement works under prompt engineering; it runs no experiments and its table gives no result values.
• Multi-Step Reasoning with Large Language Models, a Survey: a taxonomy of prompt-based reasoning that files Self-Refine under step control and reports the cited papers' own numbers; it runs no experiments.