Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions

In plain words

Large language models can be wrapped in a loop that lets them choose an action, use a tool, read the result and choose again. This paper tests that setup on two simulated decision tasks: buying a product in a copy of an online store, and carrying out household chores in a text-based home. It compares several chatbot models inside the same loop.

The authors also add a second opinion. A separately trained model proposes an action, and that proposal is written into the language model's prompt. The language model still makes its own decision. Nothing is retrained. The strongest model, GPT-4, does best with the extra opinion, and the other models do less well in this setup.

TL;DR

• The paper benchmarks Auto-GPT-style agents built on GPT-4, GPT-3.5, Claude and Vicuna on WebShop and ALFWorld §1 §3.1.
• GPT-4 has the highest success rate on both tasks, and Claude and GPT-3.5 stay below the imitation baseline tab. 1 tab. 2.
• Additional Opinions adds a supervised expert's top action(s) to the prompt; it raises GPT-4's success on both tasks §2.3 tab. 1 tab. 2.
• Test sets are small (50 WebShop instructions, 134 ALFWorld games) and the runs are mostly single; treat gaps of a few tasks as noise §3.1.1 §3.1.2.
• No runtime or cost figures are reported. The expert adds a model pass at each step where it runs §2.3 autogpt.py.

Problem

The two environments are WebShop, a simulated shopping site built from scraped product listings, and ALFWorld, household tasks in simulated rooms §2.1.1 §2.1.2. LLM agents need long-term memory, a context window and deterministic control, and these are weak points §1. Auto-GPT-style agents were not benchmarked on online decision tasks that respond to an unknown environment, and their effectiveness there was unknown §1 [abstract]. Reasoning techniques such as chain-of-thought and self-consistency were mostly evaluated on offline tasks that can be planned ahead §4.

The authors' premise is that a supervised learner's output can help a foundation model without fine-tuning the model, which they call "additional opinions" §1 §2.3.

Method

Agent loop

1. Goal and system prompt. The task text is the goal. The system prompt says to use a finish command when done, then lists the goals, constraints, commands and a JSON response format §2.2 prompt.py prompt_generator.py. The paper says the goal is the task requirement, typed directly §2.2.
2. Tools. Each available action is a tool with a description. The authors found that instructions without examples performed much worse, so they add one to three few-shot demonstrations per tool §2.2. WebShop exposes search and click webshop_gpt_agent.ipynb. ALFWorld exposes one tool, alfworld_action, whose argument is an admissible action app. 3.
3. Step. At each step the model returns a JSON object with thoughts (text, reasoning, plan, criticism) and a command with a name and arguments §1 prompt_generator.py. The named tool runs and its observation is appended to the history and to a memory store autogpt.py.
4. Stop. The loop ends on finish, or when the step limit is reached autogpt.py autogpt.py.

Additional opinions

5. Expert call. At a step where the last tool call returned both an observation and an info dictionary, the expert proposes action(s) for that observation §2.3 autogpt.py.
6. Suggestion text. The proposal is wrapped in a suggestion sentence that asks the model to treat it as a reference and make its own judgement; the default request follows it §2.3 alg. 1 autogpt.py. The paper's template names the action with its parameters and asks the model to weigh it as one reference, for a single action or a few §2.3.
7. Persistence. The suggestion is stored in the message history like any user message, so it stays in context for later steps until it leaves the window autogpt.py.

# Auto-GPT loop with Additional Opinions  [alg. 1] [code: autogpt.py#L95-L216]
# Input:  goals, tools, memory, llm, expert (may be None), top_k
# Output: message history, actions taken

def run_episode(goals, tools, memory, llm, expert, top_k):
    # Start from the reset observation and info  [code: autogpt.py#L101-L102]
    observation, info = reset_observation()
    history = []
    actions = []
    for step in range(loop_limit):
        # Default request to the model  [code: autogpt.py#L104-L107]
        request = DEFAULT_REQUEST
        # Expert only if the last tool returned obs and info  [§2.3] [alg. 1]
        if observation and info and expert is not None:
            suggestions = expert.predict(observation, info, top_k=top_k)
            # Suggestion goes before the request  [code: autogpt.py#L137-L146]
            request = wrap_suggestions(suggestions) + DEFAULT_REQUEST
        # Ask the model  [code: autogpt.py#L151-L161]
        reply = llm(goals, history, memory, request)
        history.append(request)
        history.append(reply)
        # Parse the command  [code: output_parser.py#L34-L55]
        action = parse_command(reply)
        actions.append(action)
        if action.name == "finish":
            break
        # Run the tool  [code: autogpt.py#L175-L195]
        observation, info, result = run_tool(action, tools)
        memory.add(reply + result)  # [code: autogpt.py#L204-L214]
        history.append(result)
    return history, actions

WebShop expert

8. Two learned models. A BART model generates search queries from the instruction, and a BERT model with image features scores the clickable options; the imitation baseline is the same pair §2.1.1 §3.1.1.
9. Search. For search actions, the expert takes beams from BART over the goal text. The code keeps the top beam when top_k=1, and the first top_k beams otherwise test.py.
10. Clicks. For other actions, the expert scores every valid action with the BERT model and samples from the softmax of the scores when softmax is set; the notebooks set it test.py webshop_gpt_agent.ipynb.
11. Random control. A "random" flag replaces the choice by a uniform draw over valid actions. It is the ablation with a random opinion given to GPT-3.5 test.py §3.4.1.

# WebShop expert  [§2.1.1] [code: webshop/baseline_models/test.py#L28-L89]
# Input:  observation text, info (valid actions, image features), top_k
# Output: one action string, or a list of top_k action strings

def expert_opinion(observation, info, top_k, random_pick):
    valid_actions = info["valid"]
    if valid_actions[0].startswith("search["):
        # Beam search on the goal; keep top beams  [code: test.py#L34-L42]
        queries = bart_beams(observation, num_beams=5)
        return first_k_as_search_actions(queries, top_k)
    if random_pick:
        # Uniform draw: the random ablation  [§3.4.1] [code: test.py#L69-L70]
        return valid_actions[random_index(len(valid_actions))]
    # Score each valid action with BERT and images  [code: test.py#L52-L67]
    scores = bert_scores(observation, valid_actions, info["image_feat"])
    # Softmax sampling; with replacement when top_k > 1  [code: test.py#L68-L82]
    return sample_softmax(scores, top_k)

ALFWorld expert

12. Expert. The imitation model is the text agent of BUTLER, trained with DAgger on expert demonstrations; the vision agent is excluded §2.1.2 §3.1.2. The notebook wraps the agent's greedy action generation as the single suggestion for each step alfworld_gpt_agent.ipynb.
13. Failure handling. The paper says beam search generates alternative action sentences when the agent fails, and that it is greedy in practice §2.1.2. The repository's config has this switch set to False, with a comment saying to enable it during evaluation eval_config.yaml.

Compared systems

14. Baselines. WebShop: Rule, and IL with and without image input. ALFWorld: the DAgger IL agent with and without beam search. Each Auto-GPT row is compared with these §3.1.1 §3.1.2 tab. 1 tab. 2.
15. LLM settings. Each model runs with temperature 0.01 according to the paper, under a fixed step limit §3.1.1 §3.1.2.

Hyperparameters

Results

Author-reported. Numbers come from single runs or averages of two or three runs, on the test sets above §3.1.1 §3.1.2. Model sizes are not stated.

WebShop, 50 instructions tab. 1:

ALFWorld, unseen set of 134 games tab. 2; the IL rows average three runs:

Headline comparisons:

• WebShop: GPT-4 + IL (top 5) reaches 0.320 success against 0.227 for IL tab. 1. The paper notes that reward is not the best end measure, because it counts purchases that miss requirements §3.2.1.
• ALFWorld: GPT-4 + IL reaches 0.515 success against 0.306 for IL tab. 2.
• The paper reports that the LLM-to-opinion agreement rate rises from 0.549 to 0.602 for GPT-4 when going from one to five WebShop opinions §3.4.1 fig. 2.
• Disagreement with the ALFWorld opinion is 0.814 for Claude, 0.769 for GPT-3.5 and 0.854 for GPT-4 §3.4.2 fig. 3.
• The paper makes a speed claim, Claude faster than GPT-3.5 and GPT-4. No timings are reported §3.3.1.

Paper vs code

The paper links the repository in its conclusion §5. The repository belongs to the authors (owner younghuman, and the paper links it). Its top-level code is Apache-2.0. The webshop/ subfolder bundles WebShop under its own MIT license LICENSE.md; this card quotes no text from it, only pseudocode and citations.

Discrepancies found:

• Vicuna temperature. The paper says 0.01 for all models §3.1.1 §3.1.2; the notebooks set Vicuna to temperature 1 webshop_gpt_agent.ipynb alfworld_gpt_agent.ipynb.
• Top-k opinions. The paper says the top k opinions are sampled from the expert §2.3. The code uses torch.multinomial with replacement=True over the softmax test.py. Repeats are possible, so five opinions can contain duplicates. The non-softmax top-k branch test.py is not used by the notebooks.
• Top-1 opinion. The notebooks sample one action from the softmax rather than taking the argmax test.py. The paper does not say which.
• Search query source. The code comment says the paper samples from BART's top-5 queries, and that the code uses the top-1 instead because it does better test.py. The paper describes BART only as producing "the most effective search query" §2.1.1.
• Rule baseline. The paper says the rule model buys the first item after search §3.1.1. The code clicks the first item action at any point when one is available, and buys otherwise test.py.
• ALFWorld beam search. The paper says beam search is used for failed actions §2.1.2. The config switch is False eval_config.yaml, and no notebook assigns True (the printed config shows False alfworld_gpt_agent.ipynb). The repository does not show which setting produced the reported IL row.
• ALFWorld numbers in text. Section 3.3.2 gives Claude 0.075 and GPT-3.5 0.082 §3.3.2; table 2 gives the reverse (Claude 0.082, GPT-3.5 0.075) tab. 2. This card uses the table.
• Suggestion wording. For more than one opinion the code says "Here's a few suggestions" autogpt.py; the paper's template only shows the single-opinion form §2.3.
• Reward finish. When a WebShop reward message appears, the code overrides the model's action with finish and a fixed sentence about a hair mask, which is not the product in the instruction autogpt.py. The paper does not mention this override.
• Reward scale. The test script states that the environment score is 0 to 10 and the paper uses 0 to 100 test.py.

Pitfalls

• Small test sets. A WebShop success step is 0.02 (50 tasks), an ALFWorld step is about 0.0075 (134 games). The gap between GPT-4 + IL (0.300) and GPT-4 + IL (top 5) (0.320) is one task.
• Single runs. The Auto-GPT rows are single runs at low temperature, and the + IL rows are averages of two. No variance is reported for the LLM rows §3.1.1.
• Cost per step. Each step can call the LLM once, plus the expert models; the BERT scorer moves its inputs to CUDA test.py. The paper reports no timing or token cost; the code also sends a memory retrieval of up to 2500 tokens with every call prompt.py.
• Prompt construction. The history passed to the model is the last 10 messages and is trimmed to a token budget, so earlier opinions drop out silently prompt.py prompt.py.
• Token counting. WebShop counts tokens with the GPT-3.5 tokenizer for all models webshop_gpt_agent.ipynb. Budgets are approximate for Claude and GPT-4.
• Parsing. The expert's string is split on ] and [ to build the suggestion, so a search query that contains either character breaks the parse autogpt.py autogpt.py.
• Observation split. Tool results are split on the literal %%%%, and the expert runs only when both parts exist autogpt.py autogpt.py.
• Reward parsing. ALFWorld reward is read from a fixed message position before finish, and is 0 if the pattern is missing alfworld_gpt_agent.ipynb. Failures there count as zero reward, not as missing data.
• Vicuna zero. The paper reports Vicuna at 0 because it could not produce the required format §3.3.1. Format compliance is a hidden requirement; the parser returns ERROR for any malformed reply output_parser.py.
• Imitation strength. The paper's expert gains come from a model trained on demonstrations for these very environments §2.1.1 §2.1.2. The approach needs such a learner and data for each new task; the paper says it "does not fully encapsulate" real scenarios §3.5.
• Precision and purchase rate are used without a formal definition in the paper §3.2.1.

Lineage

• Builds on: Auto-GPT, the GPT-based autonomous agent that the paper names with a link to its repository §1. Chain-of-thought and Reflexion-style self-monologue, the thought format of the agent §1 [arXiv:2201.11903] [arXiv:2303.11366].
• Compared against: the WebShop imitation baseline [arXiv:2207.01206] (card) and the ALFWorld DAgger agent [arXiv:2010.03768] §2.1.2.
• Related: self-consistency sampling (card) and group voting, which the paper contrasts with its approach §4.
• Superseded by: not stated.