ReAct: Synergizing Reasoning and Acting in Language Models

In plain words

A language model can reason in writing, or it can act: call a tool, pick a move in a game, click a button on a website. Reasoning alone tends to invent facts, because nothing checks the model's notes against the world. Acting alone tends to lose the plan, because no notes explain why each action is taken.

ReAct asks the model to do both in one stream of text. Each step is a short thought, then an action, then the result of that action. The thoughts can plan, track progress, fix a wrong assumption, or decide what to look up next. The results from the outside world then feed back into the next thought. The method needs no training: a few worked examples in the prompt are enough.

It was tried on question answering with a Wikipedia lookup tool, fact checking, a text-based household game, and an online shopping site. Its main selling point is that the written trace is easy for a person to read and correct.

TL;DR

ReAct is a prompting pattern, not a new model. The model alternates "Thought", "Action" and "Observation" lines, and the environment returns the observations. The authors ran it on a frozen PaLM-540B with 1 to 6 examples per task. It is strongest when combined with chain-of-thought self-consistency for question answering and clearly ahead of action-only prompting on the two interactive tasks. Results are author-reported, and the released code targets GPT-3, not PaLM.

Problem

Chain-of-thought prompting reasons over the model's own internal knowledge only. That gives hallucinated facts and errors that compound over the chain §1 fig. 1. Earlier language-model agents picked actions with little or no explicit reasoning, so they had no working plan and no way to explain a choice §1. The paper asks whether reasoning and acting can be combined systematically, and whether that beats either one alone §1.

Method

Action space. The agent's action space is extended with a language space. A "thought" is an action in that language space. It does not change the environment and returns no observation. It is written into the context so that later steps can use it §2.

Policy and context. At step t the agent sees a context made of earlier observations, actions and thoughts, and produces the next action, which can be a thought or an environment action §2.

Two densities of thoughts. For knowledge tasks, thoughts and actions alternate at every step ("dense" thoughts). For decision-making tasks with many actions, thoughts appear only where the model judges them useful ("sparse" thoughts). The model decides when to think §2 §3.2 §4.

Prompt construction. The prompt holds a few human-written trajectories, each a sequence of thoughts, actions and observations for one solved task instance, followed by the new question or environment state §2 app. C. No special example selection or thought format is used §2.

Knowledge tasks (HotpotQA, FEVER). The action space is a Wikipedia API with three actions §3.1:

• search[entity]: returns the first 5 sentences of the page for entity if the page exists; otherwise it returns up to 5 similar entity names from the Wikipedia search engine §3.1.
• lookup[string]: returns the next sentence of the current page that contains string, like Ctrl+F §3.1.
• finish[answer]: ends the episode with answer §3.1.

Tasks are run question-only: the model gets no supporting passage §3.1.

Combining with chain-of-thought self-consistency (CoT-SC). Two switching rules are proposed §3.2:

• ReAct → CoT-SC: if ReAct has not returned an answer within 7 steps (HotpotQA) or 5 steps (FEVER), fall back to CoT-SC.
• CoT-SC → ReAct: if the majority answer among the n CoT-SC samples appears fewer than n/2 times, fall back to ReAct.

Pseudocode for one ReAct episode, following the authors' prompting loop and the released HotpotQA notebook:

# One ReAct episode  [§2] [code: hotpotqa.ipynb#L85-L111]
# Input:  question, examples (few-shot prompt text), env, max_steps
# Output: answer string (empty if the step budget runs out)

def react_episode(question, examples, env, max_steps):
    # Prompt = instructions + exemplar trajectories + new question  [§3.2]
    prompt = examples + "Question: " + question + "\n"
    for step in range(1, max_steps + 1):
        # Sample a thought; stop before the model writes an observation  [§2]
        output = llm(prompt + f"Thought {step}:", stop=f"\nObservation {step}:")
        # Split into thought and action  [code: hotpotqa.ipynb#L94-L101]
        thought, action = split_on(output, f"\nAction {step}: ")
        if the split failed:
            # Keep the first line as the thought and request the action again
            thought = first_line(output)
            action = llm(prompt + f"Thought {step}: {thought}\nAction {step}:")
        # Execute the action in the environment  [§3.1]
        observation, done = env.step(action)
        # Append the whole step so later thoughts see it  [§2]
        prompt += (
            f"Thought {step}: {thought}\n"
            f"Action {step}: {action}\n"
            f"Observation {step}: {observation}\n"
        )
        if done:
            return env.answer()
    # Budget used up: finish with an empty answer  [code: hotpotqa.ipynb#L111]
    return ""

Combined method, ReAct first with CoT-SC as back-off §3.2:

# ReAct -> CoT-SC  [§3.2]
# Input:  question, react_examples, cot_examples, env
# Output: answer string

def react_then_cot_sc(question, react_examples, cot_examples, env):
    # Stop ReAct at 7 steps for HotpotQA (5 for FEVER)  [§3.2]
    answer = react_episode(question, react_examples, env, max_steps=7)
    if answer is not None and answer != "":
        return answer
    # Back off: majority vote over 21 CoT samples at temperature 0.7  [§3.2]
    samples = [sample_cot(question, cot_examples, temperature=0.7)
               for _ in range(21)]
    return majority_answer(samples)

The reverse order (CoT-SC → ReAct) samples the n CoT chains first and calls ReAct only when no answer has at least n/2 votes §3.2. The paper does not say how a tie or a missing answer is handled in the vote.

Decision-making tasks (ALFWorld, WebShop). ALFWorld is a text-based household game with six task types §4. WebShop is an online shopping site with 1.18M products and 12k instructions §4. Thoughts are sparse §4. For ALFWorld, the annotated thoughts (1) decompose the goal, (2) track subgoal completion, (3) pick the next subgoal and (4) apply commonsense about where an object is likely to be §4. For WebShop, the thoughts decide what to explore, when to buy, and which options match the instruction §4.

Finetuning. The paper bootstraps training data: 3,000 correct trajectories generated by ReAct (and by the other baselines) are used to finetune PaLM-8B and PaLM-62B to produce whole trajectories, with thoughts, actions and observations, conditioned on the question §3.2. Batch size 64 app. B.1.

Baselines. Standard prompting removes all thoughts, actions and observations. CoT removes actions and observations. Act removes thoughts. CoT-SC samples 21 CoT chains at temperature 0.7 and takes the majority answer §3.2. On ALFWorld, Act uses the same trajectories with thoughts removed, which the paper calls a fair controlled comparison §4. The IM-style variant (ReAct-IM) re-annotates the same trajectories with dense feedback thoughts that only restate the goal and current subgoal §4 app. B.2.

Hyperparameters

Results

Author-reported throughout. Setups are as stated in §3.1 and §4.

Knowledge tasks, PaLM-540B prompting (HotpotQA exact match, FEVER accuracy) tab. 1:

Note: the paper's footnote gives HotpotQA EM of 27.1, 28.9 and 33.8 for Standard, CoT and CoT-SC as reported by Wang et al. tab. 1.

The paper states that ReAct beats Act on both tasks, that ReAct beats CoT on FEVER, and that CoT beats ReAct on HotpotQA §3.3. CoT-SC plus back-off is the best prompting method on both tasks §3.3. The paper also reports that a few CoT-SC samples (3 to 5) reach CoT-SC's 21-sample performance when combined with ReAct, shown in fig. 2.

Finetuning (HotpotQA): the paper reports that, after finetuning, ReAct is the best of four methods. Finetuned PaLM-8B ReAct beats prompted PaLM-62B methods, and finetuned PaLM-62B ReAct beats prompted 540B methods §3.3 fig. 3. The figure's values are not given in the text.

Failure analysis (HotpotQA, 200 examples judged by hand) tab. 2:

ALFWorld, success rate by task type (134 unseen games) tab. 3:

The paper says ReAct's worst trial (48%) beats the best trial of Act (45%) and of BUTLER (37%) §4. It also says the ReAct advantage over Act ranges from 33% to 90% across the six controlled trials, averaging 62% §4. Those two figures are not in the table and are not recomputed here.

WebShop (500 test instructions) tab. 4:

GPT-3 check (appendix, text-davinci-002, greedy) tab. 5:

The paper notes that ReAct prompting also works on GPT-3 app. A.1.

Paper vs code

Official repository: ysymyth/ReAct (MIT, Copyright Shunyu Yao), linked from the authors' project page given in the paper's abstract and footnote §1. Commit 6bdb3a1fd38b8188fc7ba4102969fe483df8fdc9 was read.

• Model. The paper's main results use PaLM-540B §2. The released notebooks call OpenAI text-davinci-002 (GPT-3) with temperature 0, top_p 1 and max_tokens 100 hotpotqa.ipynb. Only the GPT-3 setting is in the repository.
• CoT-SC and the switch. The 21-sample CoT-SC baseline and both ReAct/CoT-SC switching rules §3.2 are not in the released notebooks. A search for majority voting or a CoT notebook finds none. Reproducing the combined method means implementing these rules from the paper text.
• Step budget. The paper gives 7 steps as the back-off threshold for HotpotQA and 5 for FEVER §3.2. The released HotpotQA and FEVER notebooks both loop 7 times hotpotqa.ipynb FEVER.ipynb. For HotpotQA this equals the paper's threshold, but for FEVER the code uses 7 where the paper says 5. When the budget runs out, the code calls finish[] with an empty answer hotpotqa.ipynb.
• Thought observations. The paper says a thought gets no observation feedback §2. The code keeps the trace format but returns the literal text OK. as the observation for a thought in ALFWorld and WebShop alfworld.ipynb WebShop.ipynb. A thought therefore appears in the context with an Observation: OK. line.
• Parsing and retries. If the model output lacks the Action i: line, the code keeps the first line as the thought and asks for the action again, and counts these as bad calls hotpotqa.ipynb. The paper does not mention this.
• Search. The code queries the live English Wikipedia search page over HTTP and parses its HTML wikienv.py. If the result is a disambiguation page ("may refer to:"), it searches again with the name in square brackets wikienv.py. It returns the first 5 sentences of the page, joined wikienv.py.
• Lookup. The code splits the page into sentences on ". ", matches case-insensitively as a substring, and returns the next match with a (Result k / N) prefix. It returns No more results. at the end wikienv.py wikienv.py. The paper says only "the next sentence containing the string" §3.1.
• Answer matching. The paper says "exact match" but does not define normalization. The code lowercases, removes punctuation and the articles "a", "an" and "the", and collapses whitespace before comparing wrappers.py.
• Evaluation subset. The released notebooks sample 500 random dev items with a fixed seed of 233 hotpotqa.ipynb FEVER.ipynb. The paper does not state the subset for the PaLM table §3.1.
• Context limit (WebShop). The code truncates the prompt to the last 6400 characters before each call WebShop.ipynb. The paper does not mention a limit.
• README numbers. The repository README labels a row "PaLM-540B (paper)" with HotpotQA 29.4, FEVER 62.2, ALFWorld 70.9 and WebShop 40. In the paper, 29.4 is CoT's HotpotQA score, not ReAct's (27.4) tab. 1. The FEVER value 62.2 does not appear in the paper; ReAct's FEVER score is 60.9 tab. 1. The ALFWorld and WebShop values match tab. 3 tab. 4 tab. 5. Treat the README row as informal.
• Licence. MIT. The card quotes no code.

Pitfalls

• Live Wikipedia breaks reproduction. Search and lookup hit today's Wikipedia, so pages, search suggestions and the HTML parsed by the code all change over time. The paper itself notes that some HotpotQA labels are out of date and that live search can return newer facts app. A.2 fig. 4. Freeze a snapshot if you need repeatable numbers.
• Thought-only is not always worse. On HotpotQA, ReAct scores below CoT (27.4 vs 29.4) tab. 1. Its advantage is on FEVER and on the interactive tasks. The paper's own answer is to combine the two §3.2 §3.3.
• Repetition loops. The paper's largest ReAct failure is reasoning error, including repeated thoughts and actions that never recover tab. 2 §3.3. It suggests greedy decoding as a possible cause but did not test beam search or other decoding §3.3, footnote.
• Prompt choice moves ALFWorld a lot. Six prompt sets from pairs of three annotated trajectories give a range from 48% to 71% success tab. 3 §4. "Best of 6" is an upper bound over prompt sets, not a single run. The paper does not say whether the best set was chosen on the test games.
• Step budgets are not all stated. The paper does not give a step limit for ReAct on ALFWorld or WebShop. The released code uses 49 and 15 alfworld.ipynb WebShop.ipynb. Results depend on these budgets.
• The "10%" WebShop claim does not match a single row. The text says ReAct improves success by an absolute 10% over the previous best §4. In tab. 4, 40.0 minus Act's 30.1 is 9.9, and 40.0 minus IL's 29.1 is 10.9. The paper does not say which baseline it means.
• Ambiguous tie and fallback rules. The CoT-SC switching rule uses "fewer than n/2" §3.2. It does not say how a tie, an unparseable answer or an empty answer is counted in the vote.
• Decoding for the main results is not stated. The paper's PaLM results do not give a temperature or top-p for ReAct or Act. Only the GPT-3 appendix says greedy app. A.1, and the code uses temperature 0 [code: hotpotqa.ipynb#L25].
• Unequal comparison data. ALFWorld's BUTLER baseline is trained on 10^5 expert trajectories for each task type §4, and WebShop's IL baselines use 1,012 human trajectories §4. ReAct uses one to three examples per task. Compare them as different data regimes, not as equal conditions.
• Small human study. The error analysis covers 200 HotpotQA examples, and the paper calls them randomly sampled §3.3 tab. 2.
• Figure and table numbers. Numbers above follow the order of the LaTeX source, and the compiled PDF was not checked. The ALFWorld and WebShop results share one float in the source, so tab. 3 and tab. 4 may be numbered differently in the compiled PDF.
• Not reproducible for the public. The main results use PaLM-540B, which the reproducibility statement describes as not openly accessible. Only the GPT-3 check is reproducible with public APIs app. A.1.

Lineage

• Builds on: chain-of-thought prompting, Wei et al. §1; self-consistency sampling, Wang et al. §3.2 (card); the bootstrap finetuning idea of STaR §3.2.
• Compared against: Act-only prompting, similar in spirit to WebGPT §3.2; Inner Monologue for embodied tasks §4; BUTLER and IL/IL+RL on ALFWorld and WebShop §4.
• Superseded by: not stated on this card.