LLM self-evaluation

Overview

Self-evaluation means the model that produced an answer is also the one that scores it, critiques it or compares it with alternatives. The judgement is then used inside the same program: to rewrite a draft, to re-run and fix a program, to decide which partial solution to extend, or to rank candidate hypotheses 2303.17651 §2 2305.10601 §3 2304.05128 §3 2502.18864 p. 34 2410.07076 §4.2.4. Reflexion writes a verbal lesson after each failed attempt and keeps it for the next one 2303.11366 §3. Kadavath et al. ask a model whether its own sampled answer is True 2207.05221 §4.1. Most of these methods train nothing. The exception is the P(IK) value head, which is trained on labelled samples 2207.05221 §5. Self-Refine drives every step with few-shot prompts 2303.17651 §2, and the tree search likewise uses prompts only 2305.10601 §3. Self-Debugging is also prompt-only 2304.05128 §3.

The problem these papers address is that a single left-to-right pass can miss goals that are hard to specify or that depend on an early choice. Self-Refine asks whether one off-the-shelf model can critique and improve its own output without extra data 2303.17651 §1. Tree of Thoughts asks whether the same model can serve as the heuristic for a search over intermediate steps 2305.10601 §1 2305.10601 §3. Self-Debugging asks whether a frozen model can find its own bugs from its own explanation or from running its code 2304.05128 §1 §3. Reflexion asks whether an agent can improve across retries from words alone, without changing weights 2303.11366 §1. Kadavath et al. ask whether models can tell when they are right, and whether that can be trained 2207.05221 §1.

The invariant across the decision variants: the judge sees a candidate, produces a signal (feedback text, a score, a class, a vote or a rating), and that signal decides the next step. The signal is the weak point, though the papers locate it differently. Self-Refine traces most failures to bad feedback 2303.17651 §4, while the tree-search paper finds that generation is the bottleneck when models are mixed 2305.10601 app. B.2. Where the candidate can be run, the run gives the judge a signal it does not have to invent 2304.05128 §3. Reflexion's coding gains rest on tests the model writes itself, which can be wrong 2303.11366 tab. 3. The confidence variants are different: the rating is the output, and nothing is re-run.

This page covers the eight papers on wikillm under a reuse license that study self-evaluation. Other self-evaluation work, such as Self-Correct (Welleck et al., arXiv:2211.00053), which trains a separate corrector, has no card here and is not summarized.

Variants

Iterative self-feedback and rewrite

The same model writes a draft, then writes feedback on its own draft, then rewrites the draft using that feedback and all earlier drafts and feedback. The feedback is meant to be actionable (it names a change) and specific (it names the phrase or code to change) 2303.17651 §2. The loop stops after at most 4 rounds or when a task-specific signal appears in the feedback 2303.17651 §3.1 2303.17651 alg. 1. Author-reported GPT-4 results over one-pass output include dialogue responses from 25.4% to 74.6% and constrained generation from 15.0% to 45.0%, while GSM8K moved from 92.9% to 93.1% 2303.17651 tab. 1. Card: Self-Refine.

The feedback carries most of the effect. On sentiment reversal with ChatGPT, feedback written in the Self-Refine style scored 43.2%, generic feedback such as "improve it" scored 31.2%, and no feedback scored 0 2303.17651 tab. 2. In the authors' manual analysis of 70 samples, 33% of failures came from feedback that pointed at the wrong location and 61% from feedback that proposed a wrong fix 2303.17651 §4.

Output selection is a per-task choice. The main algorithm returns the last rewrite, but for multi-aspect tasks the appendix keeps every output with its feedback score and returns the highest-scoring one 2303.17651 alg. 1 2303.17651 app. H 2303.17651 app. M.

Execution-grounded self-debugging

The model writes a program, the program is run on unit tests or on sample inputs, and the model explains the code, judges it from the run or from its explanation, and asks for a fixed program. The loop runs for up to 10 turns, and the paper's successful runs mostly end within 3 2304.05128 §3 §5. The feedback comes in four forms: a bare verdict, a verdict with the failed unit-test results, a line-by-line explanation, and a generated execution trace 2304.05128 §3. With unit-test feedback plus explanation, Codex on TransCoder rose from 80.4 to 92.5, and GPT-4 on MBPP rose from 72.8 to 80.6 with unit-test feedback 2304.05128 tab. 2. Card: Teaching Large Language Models to Self-Debug.

Where the tests are visible, most of the gain comes from running the program 2304.05128 §5.1 §5.2.2. On text-to-SQL, which has no unit tests, the explanation step is what helps. Codex rose from 81.3 to 84.1 with an explanation, while a bare verdict left it at 81.3 2304.05128 tab. 2 §5.1. The SQL is still executed, and its first two result rows go into the explanation prompt 2304.05128 §4.1.

Without tests, the effect depends on the model. On MBPP, a bare verdict lowered Codex from 61.4 to 57.6, while GPT-4 rose from 72.8 to 76.0 2304.05128 tab. 3.

Verbal self-reflection across retries

The agent runs a trial, an evaluator scores it, and on failure the model writes a short reflection on what went wrong. The reflection goes into a bounded memory that is read before the next trial. The evaluator can be an environment or exact-match grader, a heuristic or LLM classifier, or unit tests the model wrote itself 2303.11366 §3 §4.1 §4.2 §4.3. The memory window is 1 to 3 reflections 2303.11366 §3 §4.1. Nothing is trained. Author-reported GPT-4 pass@1 on HumanEval Python rose from 80.1 to 91.0 2303.11366 tab. 2. On ALFWorld the authors report +22% absolute over ReAct within 12 trials 2303.11366 §1 §4.1. Each trial is a full episode and each failure adds a reflection call, so cost grows with trials 2303.11366 §4.1.

The gain depends on the evaluator. In the 50-hardest-Rust ablation, removing the self-written tests drops the score from 0.68 to 0.52, and removing the reflection text leaves it at 0.60, the base level 2303.11366 tab. 4. A weak test suite passes wrong code: the false-positive rate is 0.16 on MBPP Python 2303.11366 tab. 3. Card: Reflexion.

Value scoring inside a tree search

Here the model is not rewriting a finished answer. It proposes next steps for a partial solution, then scores each new partial solution on its own. The score is a class such as sure, maybe or impossible, mapped to a number, and it is the heuristic for either breadth-first search (keep the top b states) or depth-first search (prune states below a threshold and backtrack) 2305.10601 §3 2305.10601 alg. 1 2305.10601 alg. 2. On Game of 24 with GPT-4, the breadth-first version with b = 5 reaches 74% against 4.0% for chain-of-thought and 9.0% for 100-sample chain-of-thought voting 2305.10601 tab. 2. On Mini Crosswords, depth-first search with a prune rule that rejects a state when any remaining clue looks impossible raises the word-level success from 15.6% (chain-of-thought) to 60% 2305.10601 tab. 3. Card: Tree of Thoughts.

The value scorer is also the weak point. On crosswords, the model sometimes marks valid but rare words as impossible and prunes a correct branch 2305.10601 §4.3. The output heuristic is weak too: an oracle that picks the best final state reaches 67.5% word-level success, against 60% for the model's own choice 2305.10601 tab. 3.

Comparative vote over candidates

When an absolute score is hard to pin down, the model can instead be shown several candidate states and asked which one is best, with the vote sampled several times. The authors use this for Creative Writing, where they sample 5 plans, vote on them 5 times with a zero-shot prompt and keep one plan 2305.10601 §4.2. Author-reported GPT-4 coherency scores were 6.19 for one-pass output, 6.93 for chain-of-thought and 7.56 for the tree search 2305.10601 §4.2. The authors chose the vote for properties such as coherence that are hard to score directly 2305.10601 §3. Card: Tree of Thoughts.

Pairwise tournament with debate

Co-Scientist does not score each hypothesis on an absolute scale. New hypotheses enter a rating tournament at 1200 2502.18864 p. 34. Pairs are chosen by similarity to the goal and by rank, and newer or top-ranked hypotheses come up more often 2502.18864 p. 35. Top-ranked pairs get a multi-turn scientific debate, and lower-ranked pairs a single comparison turn 2502.18864 p. 34. The Elo update rule and K-factor are not stated 2502.18864 p. 34. The ratings feed back into the next round: a meta-review writes a critique that is appended to every agent's prompt, so the system improves without fine-tuning 2502.18864 p. 31 p. 35–36. Author-reported, over 203 research goals, the Elo of the best hypothesis rose from about 1,350 to about 1,610 across ten time buckets 2502.18864 p. 7 fig. 2a. Card: Accelerating scientific discovery with Co-Scientist.

Multi-aspect rubric scores for a final ranking

Here the model scores a finished set of candidates on several named aspects, then ranks by the average. MOOSE-Chem scores each final hypothesis on validness, novelty, significance and potential, and the rank is the mean of the four 2410.07076 §4.2.4 eq. 3. The same aspects, minus potential and with clarity added, also drive the critique inside each refinement line 2410.07076 §4.2.3. Unlike the pairwise tournament, every candidate gets an absolute score, so the ranking needs no opponent pool.

Author-reported, on a 51-paper benchmark with GPT-4o as generator and judge: the top Matched Score is 4.020, against 2.882 for the MOOSE baseline. The average is 2.564, against 2.464 2410.07076 tab. 10. Removing the multi-step search drops the top score to 3.765 2410.07076 tab. 10. That ablation removes the search as a whole, so it does not isolate the scoring. The Matched Score is itself judged by GPT-4o. The authors report that experts usually rate one to two points lower 2410.07076 app. A.14, and expert-only counts are in 2410.07076 tab. 7.

The ranking itself gets only a modest check. Drafts with two matched inspirations have an average rank ratio of 0.411, against 0.521 for drafts with none, where lower is better 2410.07076 tab. 8. The GPT-4o Matched Score trend across scores is not monotone, and it rests on small groups 2410.07076 tab. 9. Card: MOOSE-Chem.

Probability that one sampled answer is correct

The model samples an answer at temperature 1, then is asked whether that answer is "(A) True" or "(B) False". The probability on True is the score 2207.05221 §3.2 §4.1. Zero-shot, the score is poorly calibrated and sits near 50% for typical samples 2207.05221 §4.1 app. B. Showing five samples and judging one of them, with 20-shot prompting, is the best setting the paper reports 2207.05221 §4.2 fig. 11. Its numbers appear only in figures. Accuracy on the answers scored above 0.5 is higher than overall accuracy, and the gap grows with model size 2207.05221 §4.2 fig. 11. Card: Language Models (Mostly) Know What They Know.

A second variant trains a value head to predict P(IK), the chance that the model knows the answer. The head is trained on hard labels: 30 samples per question at temperature 1, each marked right or wrong 2207.05221 §5 app. A.6. On TriviaQA with a 52B model, AUROC is 0.864 2207.05221 tab. 1. Trained on TriviaQA only, it transfers partly: AUROC is 0.606 on LAMBADA and 0.624 on GSM8k 2207.05221 tab. 1. With the other tasks added to training, AUROC is 0.853 on LAMBADA, and 0.752 on GSM8k, which is held out 2207.05221 tab. 1. Calibration on those other tasks is poor, and models are often underconfident there 2207.05221 fig. 13. Prepending a relevant Wikipedia article raised P(IK) on an obscure question from 18% to 78% at 52B 2207.05221 §5.3. The models are the authors' own, and no code or weights are released 2207.05221 §1.2 §7.

Verbal confidence on request

Some models, asked how sure they are, answer with a class such as "Likely" or with a number. Here the rating is the output: nothing is rewritten or searched. The paper asks where that rating comes from. On Gemma 3 27B, on TriviaQA with no chain of thought, the authors find that the model forms a judgement of its answer while writing the answer, keeps it at the newline after the answer, and reads it out at the confidence token 2603.17839 §1 §2.2 §2.3. Probes at that newline explain variance in the rating beyond six summaries of answer log-probabilities 2603.17839 §2.6 app. D.3. Author-reported behaviour on the same setup: expected calibration error 0.12, AUROC 0.71, accuracy 77.4% 2603.17839 §2 fig. 9. Card: How do LLMs Compute Verbal Confidence.

This is an interpretability study. It does not propose a better way to get a confidence score, and its main evidence is a single open model.

When to use which

Three cautions apply to the judging variants. First, when the judge and the generator are the same model family, the judge may favour its own style. Self-Refine's preference scores come from GPT-4, which is also one of the refined models 2303.17651 §3.2. Co-Scientist runs its generation and judging on Gemini 2.0 variants 2502.18864 p. 29. MOOSE-Chem uses GPT-4o for every step, including drafting, critique, scoring and the judge 2410.07076 §5 §6. Second, the generator matters more than the judge in the tree search. Mixing models showed that GPT-4 generation with GPT-3.5 evaluation reached 64% on Game of 24, while GPT-3.5 generation with GPT-4 evaluation reached 31% 2305.10601 app. B.2. A stronger model is better spent on generation first. Third, a self-judgement with no outside grounding can flatter the candidate. In Co-Scientist, removing the search step from Reflection let a known non-novel idea score 6.14 out of 10 for novelty, against 2.38 with search 2502.18864 p. 38. Reflexion's self-written tests carry the same risk in code, where they pass wrong solutions 2303.11366 tab. 3. The Elo ratings come from the system's own judgements, not from ground truth 2502.18864 p. 10.

Implementation sketch

Both decision variants share a skeleton: produce candidates with the model, let the model judge them, then keep, prune or rewrite. The first block is the Self-Refine loop, the second is one breadth-first step of the tree search. Self-Debugging is close to the first block, with a run of the program inserted before the feedback call 2304.05128 §3. Reflexion's retry loop 2303.11366 alg. 1 has the same shape, but it keeps a window of past reflections instead of the full history. Pseudocode is a sketch in the style of the cards, not a copy of either paper's code.

# Self-feedback loop  [arxiv:2303.17651 alg. 1]
# Input:  task input x, model M (text in, text out), few-shot prompts,
#         stop(feedback, t) -> bool
# Output: rewritten output

def refine_loop(x, M, p_gen, p_fb, p_refine, stop, max_iters=4):
    # Draft from input-output examples  [eq. 1]
    output = M(p_gen + x)
    history = []
    for t in range(max_iters):
        # Same model critiques its latest output  [eq. 2]
        feedback = M(p_fb + x + output)
        # Stop on a fixed count or a signal in the feedback  [§3.1]
        if stop(feedback, t):
            break
        history.append((output, feedback))
        # Rewrite using every earlier output and feedback  [eq. 4]
        refine_prompt = p_refine + x
        for old_output, old_feedback in history:
            refine_prompt += old_output + old_feedback
        output = M(refine_prompt)
    return output

# One breadth-first step of tree search  [arxiv:2305.10601 alg. 1] [§3]
# Input:  kept states (lists of thoughts), model M, proposal count k,
#         value function value(M, state) -> number, breadth b
# Output: the b states to keep for the next step

def bfs_step(states, M, propose, value, k, b):
    # Propose k candidate next thoughts for each kept state  [§3]
    candidates = []
    for state in states:
        for thought in propose(M, state, k):
            candidates.append(state + [thought])
    # Score every candidate; value() may sample several times  [§4.1]
    scored = [(value(M, c), c) for c in candidates]
    scored.sort(key=lambda pair: pair[0], reverse=True)
    # Keep the b best states  [alg. 1]
    return [c for _, c in scored[:b]]

Contradictions

Self-Refine and Tree of Thoughts report self-evaluation results on math, and the effects are small and not directly comparable. Self-Refine moves GPT-4 GSM8K from 92.9% to 93.1% and ChatGPT from 74.8% to 75.0% 2303.17651 tab. 1. The tree-search paper reports zero-shot tree search on GSM8K at 90, against 86 for chain-of-thought, on 100 random test questions 2305.10601 tab. 4. The baselines differ: Self-Refine's math starts from program-style prompts 2303.17651 app. S, while the tree-search baseline is a text chain-of-thought. The papers do not resolve whether the gap is the method or the baseline.

Whether self-ranking tracks quality is also reported differently. Co-Scientist reports that the Elo of its best hypothesis rose over a run 2502.18864 fig. 2a. MOOSE-Chem's own check finds that its GPT-4o Matched Score trend is not monotone 2410.07076 tab. 9. The two measures differ, a tournament rating and an absolute score, and the two papers use different benchmarks, so the results do not settle whether self-ranking tracks quality.

Whether self-judgement adds anything without an outside signal is also reported differently. Self-Refine finds that its feedback adds little on GSM8K, and that the gain comes back only with an external correctness signal 2303.17651 app. H. Self-Debugging reports gains from explanation alone on text-to-SQL, where the model judges its own explanation of the executed query, with Codex at +2.8 points 2304.05128 tab. 2 §5.1. On TransCoder without unit-test execution, Codex rose from 80.4 to 83.9 with an explanation 2304.05128 tab. 3. These are code and SQL tasks, not math, and the papers do not compare them.

Whether self-judgement works on smaller models is also reported differently. Kadavath et al. report that the accuracy gap on answers scored above 0.5 grows with model size 2207.05221 §4.2 fig. 11. Reflexion reports no change for Starchat-beta on HumanEval Python, at 0.26 for both baseline and Reflexion, and concludes that self-correction is an ability of larger models 2303.11366 app. A. These use different tasks and models, and neither paper tests the other's setup.

Where the bottleneck lies is also reported differently. Self-Refine attributes most failures to feedback 2303.17651 §4, whereas the tree-search model-mixing experiment points at generation 2305.10601 app. B.2. These are different tasks and setups, and both are author-reported.

Papers

• Self-Refine: Iterative Refinement with Self-Feedback: one model drafts, critiques and rewrites in a loop, with no training.
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models: the same model proposes steps, scores or votes on partial solutions, and a classic search keeps the promising ones.
• Teaching Large Language Models to Self-Debug: a frozen model runs its code, explains it and fixes it, for up to 10 turns.
• Accelerating scientific discovery with Co-Scientist: a Gemini 2.0 multi-agent system that generates, reviews and ranks hypotheses in an Elo tournament.
• MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses: a GPT-4o pipeline that screens inspirations, evolves drafts, and ranks them by four self-scores.
• How do LLMs Compute Verbal Confidence: an interpretability study of where a verbal confidence rating is computed in one open model.
• Language Models (Mostly) Know What They Know: a model scores its own sampled answers with P(True), and a trained head predicts whether it knows the answer.
• Reflexion: Language Agents with Verbal Reinforcement Learning: an agent writes a verbal reflection after each failed attempt and keeps a window of them for the next try.