ExpeL: LLM Agents Are Experiential Learners

In plain words

A language model agent that works through a series of tasks normally starts each one from scratch. It can retry a task after reflecting on what went wrong, but that lesson is usually lost once the next task begins. ExpeL adds a memory step. It works like a student who practises on old exam questions: it first attempts training tasks, keeps both the attempts that worked and the ones that failed, and then writes down general rules by comparing them.

At test time the model's weights stay the same. The written rules go into the prompt, together with a few past successful attempts at tasks that look most like the new one. Because only the prompt changes, the method can be used with models that are available only through an API.

On three interactive text benchmarks (question answering, household tasks, online shopping) this beat a plain ReAct agent. In a separate transfer test, rules learned on question answering also helped on a fact-checking task. The cost is extra model calls during the training phase and longer prompts at test time.

TL;DR

ExpeL wraps a frozen LLM agent in three stages. In training, it runs ReAct with Reflexion-style retries over training tasks and stores every trajectory §4.1 alg. 1. It then has an LLM compare failed and successful attempts on the same task, and summarise successes across tasks, into a list of natural-language insights §4.2.2 alg. 2. At test time the policy prompt gets the full insight list plus the k most similar past successes as examples §4.3 alg. 3. Author-reported, with gpt-3.5-turbo as the policy: HotpotQA 39% vs 28% for ReAct, ALFWorld 59% vs 40%, WebShop reward 0.701 vs 0.665 tab. 3 tab. 5. Relevant if you want an agent to improve across tasks without fine-tuning and can afford an extra training phase and longer prompts.

Problem

Fine-tuning an LLM for a task needs its weights and a lot of compute, and it can reduce generalisation §1. Prompting planners such as ReAct avoid that, but the context window limits how much past experience they can use, so they learn nothing across tasks §1. Reflexion retries a task after self-reflection, but its memory is per task and it has no cross-task learning §1 app. A.3.2. ExpeL targets learning across tasks from the agent's own experience, with no parameter updates §1 §4.

ExpeL needs no access to model parameters, so the paper argues it suits API-only models such as GPT-4 and Claude §1. Its experiments use gpt-3.5-turbo and gpt-4 §5.1 §4.2.2.

Assumptions: the environments are deterministic §3; each task yields a success signal §5.1; observations are text only §6.

Method

ExpeL has three stages fig. 1: gather experiences on training tasks, extract insights from them, and use both insights and recalled trajectories at evaluation. The base planner is ReAct §4.1 app. A.3.1.

Stage 1: gathering experiences §4.1 alg. 1. Each training task gets up to Z+1 trials, where Z is the maximum number of reflection retries (3) tab. 4. Every trial runs ReAct with the manual few-shot examples and the reflections so far. Every trajectory is added to the experience pool B, including failures. A success ends the task. A failure triggers a self-reflection, which is appended to the reflections for the next trial. The pool starts out holding the manual few-shot trajectories alg. 1.

Stage 2: insight extraction §4.2.2 alg. 2 fig. 2. The insight list starts empty. Two kinds of input are passed to an insight LLM together with the current list:

• Comparisons: a successful and a failed trajectory for the same task §4.2.2 alg. 2.
• Success batches: chunks of L successful trajectories from different tasks, sampled without replacement §4.2.2 alg. 2.

The insight LLM answers with operations on the list: ADD a new insight, EDIT an existing one, UPVOTE to agree with one, or DOWNVOTE to disagree §4.2.2. Each insight carries an importance count. A new insight starts at 2. UPVOTE and EDIT raise the count by one. DOWNVOTE lowers it by one. An insight whose count reaches zero is removed. The paper says this counting makes the process robust, because even successful trajectories can be suboptimal §4.2.2. The default insight LLM is gpt-4-0613 §4.2.2 tab. 4.

Stage 3: task inference §4.3 alg. 3 fig. 3. For each evaluation task, the policy prompt is the base ReAct prompt, plus the full concatenated insight list, plus the top-k past successful trajectories ranked by task similarity. Those examples are retrieved from a vector store built from the successes in the pool. The policy acts in a single attempt, with temperature 0 and greedy decoding §5.1 app. E.

Experience recall §4.2.1. Task descriptions are embedded with all-mpnet-base-v2. The vector store is Faiss with kNN retrieval. The top-k successful trajectories with the largest inner-product task similarity to the evaluation task are returned §4.2.1 tab. 4. The paper's reason for using task similarity: if a task repeats or resembles a stored success, the agent can mostly copy that success §4.2.1.

Transfer §4.4 fig. 4. Insights from a source task family are rewritten for a target family by an LLM (gpt-4-0613 in the paper), which is shown a few target few-shot examples. The target set has no experience pool, so the fixed few-shot examples serve as in-context examples instead §5.4. The paper evaluates this on HotpotQA-to-FEVER §5.4.

# Stage 1: experience gathering  [alg. 1]
# Input:  training tasks, policy, reflector, manual fewshots F
# Output: experience pool (all trajectories, successful or not)

def gather_experiences(tasks, policy, reflect, fewshots, max_retries):
    # The pool starts with the manual fewshot trajectories  [alg. 1]
    pool = list(fewshots)
    for task in tasks:
        # Reflections accumulate across the retries of one task  [§4.1]
        reflections = ""
        for trial in range(max_retries + 1):
            trajectory = react(policy, task, fewshots, reflections)
            # Every trial is stored, including failures  [alg. 1]
            pool.append(trajectory)
            if trajectory.succeeded:
                break
            # Reflect only if a retry is still left  [§4.1] [alg. 1]
            if trial < max_retries:
                reflections += reflect(trajectory)
    return pool

# Stage 2: insight extraction  [alg. 2] [§4.2.2]
# Input:  experience pool, insight LLM, chunk size L
# Output: list of insights, each with an importance count

def extract_insights(pool, insight_llm, L):
    # Start with an empty insight list  [alg. 2]
    insights = []
    # Compare each success with each failure of the same task  [§4.2.2]
    for success, failure in same_task_pairs(pool):
        operations = insight_llm(insights, success, failure)
        insights = apply(insights, operations)
    # Then summarise random chunks of L successes from different tasks  [§4.2.2]
    for chunk in random_chunks_without_replacement(successes(pool), L):
        operations = insight_llm(insights, chunk)
        insights = apply(insights, operations)
    return insights

# Stage 3: task inference  [alg. 3] [§4.3]
# Input:  evaluation task, insights, experience pool, embedder, k
# Output: one trajectory for the evaluation task

def run_eval_task(task, insights, pool, embedder, k):
    # Retrieve the k successes whose tasks are most similar  [§4.2.1]
    examples = top_k_similar_successes(task, pool, embedder, k)
    # Prompt with all insights and the examples, then act once  [§4.3]
    return react(policy, task, insights, examples)

Hyperparameters

Values are from the paper unless marked as from the code. "Not stated" means neither the paper nor the code states it in the places checked.

Results

Author-reported. Setup: four text-based environments (HotpotQA, ALFWorld, WebShop, FEVER), gpt-3.5-turbo-0613 for acting §5.1 app. D.5, 100 HotpotQA dev tasks, 134 solvable ALFWorld tasks, 100 WebShop tasks app. D.1. Success rate (SR) is the metric unless noted §5.1.

Ablations tab. 3, all author-reported:

Token cost per trajectory, HotpotQA tab. 6: ReAct 1,319.75 tokens; ExpeL 4,310.06 tokens. ALFWorld: ReAct 2,051.49; ExpeL 2,856.7. WebShop: ReAct 2,575.41; ExpeL 3,291.31. Training-phase cost is not reported.

Paper vs code

The code is the official repository linked in the abstract footnote, at commit e41ec9a (Apache-2.0). We searched the paper's links first (abstract footnote), so no other search was needed.

• Insight LLM default. Paper: gpt-4-0613 §4.2.2. Code default: agent.llm = gpt-3.5-turbo, which the insight script uses for its critiques expel.yaml insight_extraction.py. The README's paper commands pass agent.llm=gpt-4 README.md.
• Model snapshot. Paper: gpt-3.5-turbo-0613 §5.1 tab. 4. Code: the unpinned name gpt-3.5-turbo expel.yaml llm.py. Results may change when OpenAI updates that alias.
• Target number of insights. Paper: not stated. Code: max_num_rules default 20 expel.yaml; README sets 10 or 8 per benchmark README.md. In the code this is a soft target: rules are never truncated to this number, and it only changes how strongly REMOVE operations count expel.py expel.py.
• Operation semantics. Paper: ADD, EDIT, UPVOTE, DOWNVOTE; counts start at 2, UPVOTE and EDIT add one, DOWNVOTE subtracts one §4.2.2. Code: ADD, EDIT, AGREE, REMOVE; ADD gives 2, AGREE and EDIT add one, REMOVE subtracts one, or three when the list is full expel.py. Rules with count zero or less are dropped expel.py. The code also drops an ADD whose text already exists, turns an EDIT that matches an existing rule into an AGREE, and discards any operation whose text does not end in a period expel.py.
• Retrieval metric. Paper: maximum inner-product task similarity §4.2.1. Code: embeddings are not normalised [code: memory/init.py#L24] and the FAISS call sets no distance metric expel.py. The metric is therefore whatever the pinned langchain FAISS wrapper uses by default. We did not verify that version.
• What is retrieved. Paper: top-k successful trajectories §4.2.1. Code: one trajectory per task, the shortest successful one expel.py. The manual few-shot examples are also added to the candidate pool expel.py. Candidates are filtered by token count and by duplicate task expel.py.
• Per-step retrieval. Paper: retrieval is described per task alg. 3. Code: update_dynamic_prompt_components rebuilds the FAISS store on every step, with the task as query under the default strategy expel.py.
• Success batches. Paper: chunks of L successes §4.2.2. Code: each task contributes only its first success expel.py.
• FEVER fewshots. Paper: k = 3 tab. 4. Code: num_fewshots: 5 fever.yaml.
• Folds. Paper text: four-fold validation §5.1. Appendix: train on one half, evaluate on the other, and reverse app. D.1. Code: k_folds: 2 hotpotqa.yaml.
• WebShop changes. Paper: prices fixed to averages rather than sampled, and 10 items per page app. D.3. These changes are in the external WebShop server. The repository only points to a local URL webshop.py. We could not verify them here.
• Unused code. task_critique and failure_critique are defined but not called from the training or evaluation paths expel.py expel.py. The paper does not describe them.

No discrepancy was found in the benchmark step budgets H (7, 20, 15, 7) or in the reflection depth Z = 3 tab. 4 .

Pitfalls

• Insight count is a soft cap. The code does not truncate the insight list, so the number of insights can exceed the target and grow the prompt. The paper argues that the list fits the context window, but notes that retrieval may be needed for long-running agents §6.
• Prompt growth. The full insight list is concatenated into every evaluation prompt §4.3. Token use per trajectory roughly doubles to triples on HotpotQA tab. 6.
• Reflections and insights. The paper found that feeding reflections into insight extraction hurt performance, and it suspects hallucinated reflections §5.6 tab. 3. The default pipeline does not pass them.
• Insight quality depends on the LLM. The paper says gpt-4 hallucinated less than gpt-3.5-turbo at following the operator format, and that a better model gave better insights §4.2.2 §5.6. A weaker insight LLM is a likely cause of the gap seen in the ablation.
• Operator parsing is fragile. Malformed operations, unmatched rule text and text not ending in a period are dropped silently expel.py. Check how many operations survive before you trust a run.
• Fold statistics. If the two-fold setup in the appendix and code is the one used, the reported standard error is taken over two values app. D.1 and is a weak spread estimate.
• Figure-only prompts. The insight, policy and transfer prompt templates appear only as images in the paper fig. 2 fig. 3 fig. 4 app. F. This card does not reproduce them. The code contains the prompt text under prompts/ and prompts/templates/ (not cited line by line here).
• Determinism. Decoding is greedy at temperature 0 §5.1. The paper assumes deterministic environments §3. Its WebShop change (fixed average prices instead of sampled ones) was made so that each run is deterministic app. D.3.
• Failed training tasks. Training tasks that never succeed contribute only failure trajectories. Their comparisons with successes exist only for tasks with at least one success alg. 2 expel.py.
• Compute. Each training task can trigger up to four ReAct trials plus reflections, and insight extraction makes one LLM call per comparison or chunk alg. 1 alg. 2. The paper does not report training-phase cost.

Lineage

• Builds on: ReAct (Yao et al., 2022) as the base planner §4.1 app. A.3.1; Reflexion (Shinn et al., 2023) for reflection-based retries §4.1 app. A.3.2. Neither has an arXiv id in this paper's bibliography, so this card does not link them.
• Compared against: ReAct, Act (ReAct without reasoning steps), Reflexion, and imitation-learning results taken from the ReAct paper §5.1 §5.2.2 tab. 5.
• Cites experience replay and off-policy learning as conceptual background app. A.4.
• Published at AAAI 2024 according to the repository README README.md.
• Superseded by: not stated.