Self-Refine: Iterative Refinement with Self-Feedback

In plain words

People rarely get a piece of writing or code right on the first try. They write a draft, reread it, notice what is off ("this sounds rude", "this loop is too slow") and fix it. Language models normally skip that second look: they answer once and stop.

Self-Refine gives the model that second look without any training. The same model plays three roles in turn, each set up with a handful of examples in its prompt. First it writes an answer. Then it acts as a critic and writes feedback on its own answer, pointing at a concrete part and saying what to change. Then it acts as an editor and rewrites the answer using that feedback, while seeing all earlier drafts and critiques. The critique-and-rewrite cycle repeats a few times or until the critic says the answer is good enough.

The authors tried this on seven tasks, from rewriting reviews and chatting to speeding up programs and solving math word problems, with GPT-3.5, ChatGPT and GPT-4. The refined answers were preferred or scored higher on every task, but math barely moved, because the model usually could not spot its own mistakes. A smaller open model (Vicuna-13B) could not follow the critique and rewrite prompts at all, and every round costs two more model calls.

TL;DR

Self-Refine is a test-time loop around one LLM: generate y0, ask the same model for feedback fb_t on y_t, ask it again to produce y_{t+1} from the input plus all previous outputs and feedback, and stop on a feedback-derived signal or after at most 4 iterations §2 §3.1 alg. 1. All three steps are few-shot prompts; feedback is meant to be actionable (names a fix) and specific (points at a span) §2. Author-reported gains over the same model's single-pass output range from 0 (GSM8K with GPT-3.5) to +49.2 points (GPT-4 dialogue preference) tab. 1. It needs a strong instruction-following model; it failed with Vicuna-13B §4 app. G.

Problem

Single-pass LLM outputs often miss multi-faceted or hard-to-specify goals (an engaging dialogue reply, readable code) §1. Prior refinement methods train a dedicated refiner or reward model on domain data or human annotation, which is expensive and task-specific §1. Self-Refine asks whether one off-the-shelf model can critique and improve its own output using only a few in-context examples, with no training, no extra data and no RL §1 §2.

Method

One model M, three task-specific few-shot prompts and a stop condition §2 alg. 1:

1. Initial generation. p_gen holds input-output pairs; the model writes y0 eq. 1.
2. Feedback. p_fb holds input-output-feedback triples; the model writes feedback on its latest output eq. 2. The in-context feedback is written to be actionable (contains a concrete change) and specific (names the phrase or code to change) §2. In several tasks the feedback also scores aspects numerically (10 aspects out of 3 each for dialogue, 5 aspects out of 5 each for acronyms) app. M app. Q.
3. Refine. p_refine holds input-output-feedback-refined quadruples; the model rewrites the output eq. 3. In practice the prompt carries the whole history of outputs and feedback, so eq. 3 becomes eq. 4 §2.
4. Stop. stop(fb_t, t) fires at a fixed iteration count or on an indicator extracted from the feedback (e.g. a stop score), defined per task §2. Experiments cap the loop at 4 iterations §3.1.
5. Output. Algorithm 1 returns the last refinement y_t §2 alg. 1. For multi-aspect tasks (dialogue, acronyms), the output with the highest feedback score across iterations is selected instead app. H app. M.

y_0 = \mathcal{M}\left(p_{gen} \,\|\, x\right) eq. 1

fb_t = \mathcal{M}\left(p_{fb} \,\|\, x \,\|\, y_t\right) eq. 2

y_{t+1} = \mathcal{M}\left(p_{refine} \,\|\, x \,\|\, y_t \,\|\, fb_t\right) eq. 3

y_{t+1} = \mathcal{M}\left(p_{refine} \,\|\, x \,\|\, y_0 \,\|\, fb_0 \,\|\, \dots \,\|\, y_t \,\|\, fb_t\right) eq. 4

|| is string concatenation §2.

# Self-Refine  [alg. 1]
# Input:  x (task input string), model M (text in, text out),
#         prompts p_gen, p_fb, p_refine (few-shot strings),
#         stop(feedback, t) -> bool, max_iterations (4 in the paper)
# Output: refined output string

def self_refine(x, M, p_gen, p_fb, p_refine, stop, max_iterations):
    # Initial draft from input-output demonstrations  [eq. 1]
    output = M(p_gen + x)
    history = []
    for t in range(max_iterations):
        # Same model critiques its latest output  [eq. 2]
        feedback = M(p_fb + x + output)
        # Stop at a fixed t or on a signal parsed from the feedback  [§2]
        if stop(feedback, t):
            break
        history.append((output, feedback))
        # Refine conditioned on all earlier outputs and feedback  [eq. 4]
        refine_prompt = p_refine + x
        for previous_output, previous_feedback in history:
            refine_prompt = refine_prompt + previous_output + previous_feedback
        output = M(refine_prompt)
    # Return the last refinement  [alg. 1]
    return output

Per-task variants that matter for implementation:

• Score-based selection (dialogue, acronyms): keep every (output, feedback score) and return the argmax of the total score; for dialogue the initial response is excluded from the argmax app. M app. H.
• Code readability has no initial generation: the input code is y0, then critique and edit alternate for N rounds app. L.
• Math (GSM8K): initial solutions are Python programs in PaL style app. O app. S. The appendix says the correct label decides when to move to the next step, following Self-Correct app. O; "oracle" results with this signal are reported separately app. H.

Hyperparameters

Results

Author-reported. Base = the same model with one-pass few-shot generation; metrics: solve rate (GSM8K), % programs optimized (code optimization), concept coverage % (constrained generation), GPT-4 preference % (sentiment, dialogue, acronym), GPT-4-judged fraction of well-named variables (code readability) §3.2 tab. 1.

With Wilson intervals, gains are significant for nearly all GPT-4 tasks, 4 of 7 ChatGPT tasks and 3 of 7 GPT-3.5 tasks app. J. GPT-4 preference agreed with human preference 82% (sentiment), 68% (acronym) and 71% (dialogue) of the time §3.2.

Feedback ablation (code optimization and sentiment with ChatGPT, acronym with GPT-3.5) tab. 2:

Score by iteration, averaged over GPT-3.5, ChatGPT and GPT-4 fig. 4:

GSM8K with oracle feedback (refine only when the answer is wrong) app. H:

Other findings:

• Blind A/B by the authors, 150 examples per task, Self-Refine / direct / either: sentiment 75.00 / 21.43 / 3.57, acronym 44.59 / 12.16 / 43.24, dialogue 47.58 / 19.66 / 32.76 app. C.
• Against k = 4 ChatGPT samples without refinement (1 vs. k comparison), humans still preferred Self-Refine outputs §4 app. H.
• Vicuna-13B could not produce feedback in the required format or follow refine prompts. Using Vicuna for the draft and ChatGPT for feedback and refinement raised GSM8K from 24.18% to 40.5% §4 app. G.
• Manual analysis of 70 samples (code optimization and GSM8K): of failures, 33% came from feedback that located the error wrongly, 61% from feedback proposing a wrong fix, 6% from the refiner misapplying good feedback; of successes, 33% happened despite partly wrong feedback §4.
• ChatGPT's feedback on GSM8K said "everything looks good" for 94% of instances §3.3.
• Code optimization vs. other methods: Self-Refine with GPT-4 reaches 36.0% optimized with at most 4 samples; PIE few-shot with best-of-32 reaches 38.3% app. F.

Paper vs code

Checked src/*/run.py, the task_init, feedback and task_iterate modules, and src/utils.py.

• Temperature. The paper uses 0.7 everywhere §3.1. The code defaults to 0.0 for code optimization (init, feedback and refine share the CLI temperature) run.py and GSM8K init run.py, with GSM8K feedback hardcoded to 0.7 run.py; sentiment refine uses 0.0 task_iterate.py; dialogue, acronym and constrained generation use 0.7 task_iterate.py.
• GSM8K merges feedback and refine. One call returns the feedback and a corrected program, split on def solution(): feedback.py. The loop stops when the feedback contains "it is correct" run.py. History is kept by appending each (solution, feedback, fix) to the feedback prompt as a new few-shot example feedback.py. The default engine is Codex run.py, while tab. 1 reports GPT-3.5, ChatGPT and GPT-4.
• Code optimization has no history. Refine sees only the latest program and its feedback, i.e. eq. 3, not eq. 4 run.py vs §2. The stop signal is the phrase "this code is not slow" run.py. --max_attempts defaults to 3 run.py; the README command passes 4.
• Sentiment reversal never stops early. The loop runs max_attempts + 1 generations (<=) and has no break, although the paper says iterations continue until the target sentiment is reached run.py vs app. P. The code appends "Try again to make it {target}!" to feedback that lacks it run.py and filters the input file to positive targets only run.py.
• Dialogue truncation. Context is cut to the last 8 lines run.py. When a call exceeds 3000 (and again 3500) total tokens, the oldest in-context examples of the refine prompt are dropped run.py task_iterate.py. There is no early stop; all scored responses are saved and selection happens outside run.py run.py.
• Acronym. The "only iterate over improving" filter is total_score >= 0, which is always true, so every output stays in the history run.py. The refine step may also return a rewritten title, which replaces the input title for later feedback run.py. Default max_attempts for a single title is 5 run.py.
• Constrained generation stops when both concept and commonsense feedback are "none" run.py, with a batch default of 4 attempts run.py.
• Silent failures. Parse errors restart the whole loop up to 3 times, then return None utils.py; the code optimization and GSM8K drivers swallow exceptions and drop those examples from the output file run.py run.py.

Pitfalls

• Algorithm 1 vs. best-of selection. Alg. 1 returns the last y_t alg. 1, but the appendix says the best-scoring output across iterations is selected and attributes this to Alg. 1 app. H. Decide per task: last output for monotone tasks, argmax of feedback score for multi-aspect tasks where quality fluctuates (the acronym example goes 11, 17, 12, 17 out of 25) app. H.
• Self-feedback rarely catches reasoning errors. GSM8K gains are 0 to 0.2 points tab. 1; the gain comes back only with an external correctness signal app. H. The appendix text describing GSM8K mentions using the gold label to advance the loop app. O, which is oracle information; check which variant you are reproducing.
• Numbers disagree across the paper. GSM8K Self-Refine is 64.1 / 75.0 / 93.1 in tab. 1 but 62.4 / 75.1 / 94.5 in the comparison table app. F; ChatGPT code optimization is 23.9 to 27.5 in tab. 1 but 22.2 and 26.7 in app. F. The confidence-interval appendix says 99% in the text and 95% in the caption app. J. Dialogue test size is 342 app. M vs. 372 conversations app. A.
• "Greedy decoding with a temperature of 0.7" is contradictory §3.1, and the code uses both 0.0 and 0.7 depending on task and module (see Paper vs code).
• Judge bias risk. Four tasks are scored by GPT-4 preference, and GPT-4 is also one of the refined models §3.2; human A/B tests were run by the authors themselves app. C.
• Cost. Each iteration adds a feedback call and a refine call, and the refine prompt grows with every past output and feedback eq. 4; the dialogue code has to drop prompt examples once calls pass 3000 tokens run.py.
• Needs a strong instruction follower. Vicuna-13B repeated its output or hallucinated a conversation instead of refining, even with hard-coded feedback §4 app. G; the feedback format must be parseable (scores, stop phrases), and parse failures are silently retried or dropped in the code utils.py.
• Feedback quality is the bottleneck. 94% of failures in the manual analysis came from wrong feedback, not from the refiner §4. Generic feedback ("improve efficiency") loses most of the gain on sentiment reversal (43.2 to 31.2) tab. 2.
• Stop signals are task-specific string matches (e.g. "it is correct", "this code is not slow", "none") run.py run.py; the paper only says the indicator is defined per task §2.
• Closed models. All main results use OpenAI models whose details are not public; experiments are English-only §6.

Lineage

• Builds on: few-shot prompting §2; Self-Correct (Welleck et al., arXiv:2211.00053), which trains a separate corrector, while Self-Refine uses the same model without training §1 app. F; PaL prompts for GSM8K app. S; PIE (arXiv:2302.07867) data and prompts for code optimization app. S.
• Compared against: same-model single-pass generation tab. 1; k-sample generation without refinement §4; PaL, Self-Correct, PIE fine-tuned and few-shot best-of-k, Scalene, CodeGen app. F.
• Related in this wiki: Tree of Thoughts also has the model evaluate its own intermediate outputs, but searches over steps instead of rewriting a full draft.