Reflexion: Language Agents with Verbal Reinforcement Learning

In plain words

Teaching a language-model agent usually means changing its weights, which is expensive. Reflexion skips that. When the agent fails a task, the same model writes a short note about what went wrong and what to try differently. The agent keeps these notes and reads them before its next attempt, so the next try can avoid the earlier mistake.

The idea works for several kinds of tasks: step-by-step household tasks, question answering over documents, and writing code. The feedback can be a plain pass or fail, a score, or the error messages from running code. Where no outside judge exists, the agent can write its own tests or judge its own answer. The paper reports that this improves results on each task type, with the usual caveat that its numbers come from the authors' own runs.

TL;DR

Reflexion keeps the LLM frozen and adds a loop: act, score the attempt, write a verbal reflection on any failure, and prepend recent reflections to the next attempt §3. It reports 91.0% pass@1 on HumanEval Python against a GPT-4 figure of 80.1% tab. 2, and +22% absolute on ALFWorld over ReAct §1. Its gains depend on how good the evaluator is; the paper's own failure analysis shows false positives from weak self-written tests tab. 3.

Problem

Language agents that act in environments (games, compilers, APIs) learn from trial and error only through in-context examples. Reinforcement learning with gradient updates would need many samples and fine-tuning of a large model, which the authors consider too costly §1. Scalar rewards are also hard to turn into useful credit assignment for long trajectories §1. Reflexion targets this gap by converting the feedback into text that the model can use directly §1, §3. The paper concedes that the approach relies on the LLM's self-evaluation or on hand-written heuristics, and that it has no formal guarantee of success §1.

Method

Reflexion uses three roles, all implemented by LLM calls in the experiments §3:

1. Actor M_a: generates text or actions from the current observation. The paper tries Chain-of-Thought and ReAct as actors §3, Actor.
2. Evaluator M_e: maps a trajectory to a scalar reward. Reasoning tasks use exact-match grading; decision-making tasks use hand-written heuristics or an LLM classifier; coding tasks use self-generated unit tests [§3, Evaluator] §4.3.
3. Self-reflection M_sr: receives the sparse reward, the trajectory and the existing memory, and writes a verbal lesson, typically naming the action that should have been different §3, Self-reflection.

Memory has two parts. The short-term memory is the current trajectory. The long-term memory is a list of past reflections, of which the last few are added to the actor's prompt. The paper bounds this list to a window Ω, "usually set to 1-3" §3, Memory and The Reflexion process.

The outer loop is given as Algorithm 1 alg. 1. In words: run a first trial and score it; reflect and store the reflection; then repeat trials until the evaluator passes or a trial limit is reached, reflecting after each trial alg. 1 §3, The Reflexion process. Figure 2(a) shows the same loop as a diagram fig. 2.

# Reflexion outer loop  [alg. 1] [§3]
# Input:  task, actor, evaluator, reflector, memory window, trial limit
# Output: last trajectory and the stored reflections

def reflexion(task, actor, evaluator, reflector, window, max_trials):
    # First trial: no reflections exist yet  [alg. 1]
    trajectory = actor.run(task, memory=[])
    reward = evaluator.score(trajectory)
    # Turn the failed attempt into a stored lesson  [§3, Self-reflection]
    memory = [reflector.reflect(trajectory, reward, memory=[])]
    trial = 0

    # Retry until the evaluator passes or the limit is reached  [alg. 1]
    while not evaluator.passes(reward) and trial < max_trials:
        # Only the most recent reflections go into the prompt  [§3, Memory]
        trajectory = actor.run(task, memory=memory[-window:])
        reward = evaluator.score(trajectory)
        # Reflect after every trial and append the lesson  [alg. 1]
        lesson = reflector.reflect(trajectory, reward, memory[-window:])
        memory.append(lesson)
        trial += 1

    return trajectory, memory

Decision-making (ALFWorld) runs ReAct as the actor, with two few-shot trajectories copied from the ReAct paper, and self-evaluates with either an LLM classifier or a heuristic §4.1. The heuristic triggers a reflection when the agent repeats an action and gets the same response for more than three cycles, or when it takes more than 30 actions §4.1. The memory keeps the last three reflections §4.1.

For reasoning (HotPotQA), the reward is exact-match grading against the gold answer, computed by the environment §4.2. Two agents are tested: Chain-of-Thought given the gold context (CoT (GT)) and ReAct with a Wikipedia search tool §4.2.

For programming, the evaluator is a set of unit tests that the LLM writes itself. Candidate tests are kept only if they parse as valid syntax §4.3. The paper samples up to 6 tests per problem §4.3. A failing candidate is run against these tests; the reflection is then conditioned on the test output, and the next attempt keeps one reflection of memory §4.3. The paper also adds LeetcodeHardGym, 40 hard LeetCode problems released after 8 October 2022, in 19 programming languages §1, §4.3.

# Programming variant: LLM-written tests act as the evaluator  [§4.3]
# Input:  function signature, model, trial limit
# Output: implementation

def solve_with_tests(signature, model, max_trials):
    # Generate tests, then keep only the syntactically valid ones  [§4.3]
    tests = valid_tests(model.write_tests(signature), max_count=6)
    # First attempt, without memory  [§4.3]
    code = model.write_code(signature, memory=[])
    feedback = run(code, tests)
    trial = 0

    # Repeat until all self-written tests pass or the limit is reached
    while not all_passed(feedback) and trial < max_trials:
        # Reflect on the failure, then keep only that reflection  [§4.3]
        lesson = model.reflect(code, feedback)
        code = model.write_code(signature, memory=[lesson])
        feedback = run(code, tests)
        trial += 1

    return code

Hyperparameters

Results

Author-reported. The base models and baselines differ by task, so the numbers are not directly comparable across tasks §4.

Programming pass@1 by failure type, from the same table set:

Ablation on the 50 hardest Rust problems with GPT-4: the base model scores 0.60. Removing test generation drops it to 0.52. Removing the self-reflection text leaves it at 0.60. The full method reaches 0.68 tab. 4.

Other models: on HotPotQA with 100 questions, Reflexion improves every model tested, from text-davinci-003 to GPT-4 app. A. Starchat-beta on HumanEval Python shows no change: 0.26 for both baseline and Reflexion app. A. The paper's conclusion is that self-correction is an ability of larger models app. A.

WebShop: a two-shot ReAct + Reflexion agent did not improve over four trials, and the authors stopped the runs there app. B.

Paper vs code

The official repository is noahshinn/reflexion (MIT, owner noahshinn, which matches the first author). The paper names noahshinn024/reflexion; that URL returns 404 now, so we use the current address. Cloned at 218cf0ef1df84b05ce379dd4a8e47f17766733a0.

• Unit tests per attempt. Paper: up to 6 generated tests per problem §4.3. Code: the Reflexion loop asks for 1 test per attempt [code: programming_runs/reflexion.py#L37]. The generator's default is 5 [code: programming_runs/generators/py_generate.py#L288].
• ALFWorld stop rule. Paper: reflect if the same action gets the same response for more than 3 cycles, or after more than 30 actions §4.1. Code: the trial stops only when the new action equals the previous action [code: alfworld_runs/env_history.py#L17-L19]. The code has no 30-action limit and no check on the response. Each trial is capped at 49 environment steps alfworld_trial.py.
• ALFWorld trial count. Paper: 12 trials §1 §4.1. Code: the Reflexion script runs --num_trials 10 run_reflexion.sh.
• HotPotQA temperature. Paper: the baselines were run at temperature 0.7 §4.2. Code: the agents set temperature=0 for both the action and the reflection model agents.py, [code: hotpotqa_runs/agents.py#L52], agents.py.
• HotPotQA memory. Paper: the window is 3 experiences §4.2. Code: the reflection list grows without a cap where reflections are appended [code: hotpotqa_runs/agents.py#L113], agents.py. The scratchpad is trimmed to 1600 tokens react.py.
• Programming memory. Paper: memory of 1 experience §4.3. Code: the full list of reflections is recorded, but only the latest one is passed to the next attempt reflexion.py [code: programming_runs/reflexion.py#L64-L70].
• Programming trial limit and test checks. The paper does not state a trial limit for programming. The code uses max_iters from the command line run_reflexion.sh. On the final iteration the code runs the held-out tests and stops regardless [code: programming_runs/reflexion.py#L81-L87].
• ALFWorld retries. The code retries a model call up to six times with rising temperature when the output is shorter than five characters [code: alfworld_runs/alfworld_trial.py#L25-L33]. The paper does not mention this.
• Reflection generation in ALFWorld. The reflection call uses the default completion function with temperature 0 and 256 max tokens [code: alfworld_runs/generate_reflections.py#L44] [code: alfworld_runs/utils.py#L22].

No discrepancies were found for the ALFWorld memory window (3 in both) [code: alfworld_runs/alfworld_trial.py#L47-L48].

Pitfalls

• The evaluator sets the ceiling. Reflexion can only fix failures it can see. For coding, a weak generated test suite causes false positives, which the paper measures at 16% on MBPP Python tab. 3 and which make the agent stop too early. A false negative is less harmful: the agent may keep a correct solution §4.3.
• Self-written tests need budget. The programming result depends on generating and running a test suite on every attempt. Removing test generation lowers accuracy (0.60 to 0.52 in the Rust ablation) tab. 4.
• Cost scales with trials. Each trial is a full episode, and each failure adds a reflection call. ALFWorld uses up to 12 trials per task §4.1. Reflection prompts also grow with the memory window; the code trims the memory to the last 3 reflections in ALFWorld [code: alfworld_runs/alfworld_trial.py#L47-L48].
• Prompt length limits. The paper bounds memory to avoid exceeding the context window §3. The HotPotQA code also trims the scratchpad and halts near 3896 tokens react.py [code: hotpotqa_runs/agents.py#L253].
• Two different stop rules. The paper's heuristic counts repeats over several cycles and also caps actions at 30 §4.1. The code flags only an immediate repeat env_history.py. The two rules flag different trajectories, so the 130 of 134 result §4.1 may not follow from the code's rule.
• Exploration. The authors report that Reflexion cannot escape local minima that need creative behaviour, such as WebShop app. B. The limitations section adds that the policy may reach non-optimal local minima and that long-term memory is a sliding window §5.
• Execution safety. The programming experiments run generated code. The authors advise isolated environments §8.
• Mixed evidence. The paper reports gains on the tasks it chose and no gain on Starchat app. A. MBPP Python is worse than the GPT-4 baseline tab. 2.
• Unstated values. The paper gives no trial limit for programming, no step cap for ALFWorld, and no max-token budget. Use the code's values only if you accept them as the authors' runs §4 alfworld_trial.py.

Lineage

• Builds on: ReAct as the action generator in decision-making and HotPotQA §4.1 §4.2; Chain-of-Thought for reasoning and test generation §3 §4.2 §4.3.
• Compared against: Self-Refine, CodeT, Self-Debugging, CodeRL and AlphaCode as related work §2 tab. 1.
• Superseded by: not determined from this wiki.