WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

In plain words

Many language-understanding benchmarks test a model on a fixed text, not on a task it has to carry out step by step. WebShop builds a fake online shop with about a million real product listings scraped from Amazon. Each episode gives the agent a shopping request written by a person, such as a pair of shoes with certain features and a price limit. The agent must type a search, click through results and product pages, choose options such as size or colour, and press buy.

The score is partial credit: the bought product is compared with the request on its attributes, its options, its price and its product type. The authors trained two learned parts, one that writes search queries and one that picks the next click, from a set of recorded human shopping sessions, then added reinforcement learning on top. The best learned agent does clearly better than a simple rule, but still well below human experts, so the benchmark leaves a lot of room for improvement. The authors also let the agent run briefly on the real Amazon and eBay sites, with mixed but encouraging results.

TL;DR

• WebShop is an interactive, text-based shopping environment with a reward you can compute automatically, so no human rater is needed in the loop §3.
• The reward is a partial-credit match on attributes, options, price and product type eq. 1 app. A.5.
• Best learned agent (IL+RL) reaches score 62.4 and success rate 28.7%; expert humans reach score 82.1 and success rate 59.6%; the rule baseline reaches 45.6 and about 10% §5.2 tab. 2.
• All numbers are author-reported on a 500-instruction test split §5.1.
• Use it for an agent benchmark or as a testbed for search-query generation and option choice. The code is MIT-licensed; the scraped product data is not covered by this card's licence.

Problem

Existing grounding benchmarks either use little real-world language or need a lot of human feedback to score each episode. A web shop has real text and multi-step navigation, and the reward can be computed by a program, so it offers a path to scaling. The paper argues that real web sites (rather than pixel-level browser control) need a semantic action space §1 §3.1.

Method

Environment (WebShop)

• State and pages. There are four page types: search, results, item and item-detail §3.1.
• Actions. search[query] is allowed only on the search page. On all other pages the agent can only choose[button]. A buy click ends the episode tab. 1 §3.1.
• Observation. Two modes: a full HTML mode for humans and a simplified text mode (simple) for the models. All models are trained and evaluated in the simple mode §3.1.
• Search engine. BM25 via Pyserini over title, description, overview and options. The engine is deterministic. Each search returns the top 50 items, shown as five pages of 10 §3.2 app. A.3.
• Attributes. 670 attributes were mined with TF-IDF over bigrams and then filtered by hand, so each product has hidden attributes §3.2 app. A.2.
• Instructions. 12,087 instructions were written by crowd workers from a sampled target product, its attributes and options §3.2 app. A.4.

Reward

The episode reward is given only when the agent buys §3.1:

r = r_{\text{type}} \cdot \frac{|U_{\text{att}} \cap Y_{\text{att}}| + |U_{\text{opt}} \cap Y_{\text{opt}}| + \mathbf{1}[y_{\text{price}} \le u_{\text{price}}]}{|U_{\text{att}}| + |U_{\text{opt}}| + 1} eq. 1

Here U_{\text{att}}, U_{\text{opt}} and u_{\text{price}} are the goal's attributes, options and price limit, and Y_{\text{att}}, Y_{\text{opt}} and y_{\text{price}} those of the bought product.

The type reward r_{\text{type}} is a text-match score on the product title against the goal title, with noun-tagged words only (SpaCy). It is zero for no overlap, 0.1 for weak overlap, 0.5 when the overlap is above 0.2 and both the coarse and fine categories match, and 1 otherwise app. A.5.

# Episode reward  [eq. 1]
# product: bought item; goal: attributes, options, price limit, title

def episode_reward(product, goal):
    # Count goal attributes and goal options the product satisfies  [eq. 1]
    matched_attrs = count_matches(goal.attributes, product.attributes)
    matched_opts = count_matches(goal.options, product.options)
    # Price is a 0/1 term: 1 if price is at most the goal's upper bound  [eq. 1]
    price_ok = 1 if product.price <= goal.price_upper else 0
    # Normalise by goal terms plus one for the price term  [eq. 1]
    partial = (matched_attrs + matched_opts + price_ok) / (
        len(goal.attributes) + len(goal.options) + 1)
    # Scale by the type reward; wrong product types are penalised  [app. A.5]
    return type_reward(product, goal) * partial

# Type reward  [app. A.5]
# text_match: overlap of noun-tagged title words, in [0, 1]
# coarse_match, fine_match: category agreement flags

def type_reward(product, goal):
    # No text overlap at all  [app. A.5]
    if text_match == 0:
        return 0.0
    # Very weak overlap  [app. A.5]
    if text_match < 0.1:
        return 0.1
    # Overlap is above 0.2 and both category levels agree  [app. A.5]
    if text_match > 0.2 and coarse_match and fine_match:
        return 0.5
    return 1.0

Imitation learning (IL)

Two pre-trained language models are trained separately on human demonstrations §4.2.

• Search model. A BART model maps the instruction to one search string, trained by maximum likelihood on 1,421 instruction–search pairs from 1,012 training demonstrations §4.2:

\mathcal{L}_{\text{search}} = \mathbb{E}_{u, a \sim \mathcal{D}}\left[-\log \pi_{\phi}(a\mid u)\right] eq. 2

• Choice model. A 12-layer BERT encodes the observation o and each candidate action. A cross-attention layer fuses them, and a mean-pooled vector is projected to a scalar score S(o,a). The policy is a softmax over the scores of the available buttons. The training set has 9,558 samples §4.2:

\mathcal{L}_{\text{choose}} = \mathbb{E}_{o, \mathcal{A}(o), a^*}\left[-\log \pi_\theta(a^* \mid o, \mathcal{A}(o))\right] eq. 3

\pi_\theta(a \mid o, \mathcal{A}(o)) \sim \exp\left(W^\top \text{mean}\left[\text{cross-attn}\left(\text{BERT}(o;\theta), \text{BERT}(a;\theta)\right)\right]\right) eq. 4

• Image features. A pre-trained ResNet-50 maps each product image to 512 dimensions. A learned linear layer maps that to 768 and it is concatenated to the observation representation §4.2.

# Choice IL step  [§4.2] [eq. 3] [eq. 4]
# obs: simple-mode text; buttons: valid click strings; target: human click

def choice_il_loss(obs, buttons, target, image):
    # Encode observation text with BERT  [§4.2]
    obs_tokens = bert(obs)
    # Project the product image and prepend it to the observation  [§4.2]
    obs_tokens = concat(linear_512_768(image), obs_tokens)
    # Score every button against the observation  [§4.2] [app. B.1]
    scores = [score(obs_tokens, bert(button)) for button in buttons]
    # Negative log-likelihood of the human's click  [eq. 3]
    return -log_softmax(scores)[index_of(target, buttons)]

Full pipeline at test time. On a search page the BART model proposes search strings and one is chosen at random from the top 5 (beam search). On every other page the choice model samples one button from its softmax policy. The authors state that sampling avoids being stuck and gives more diverse candidates §4.2.

Reinforcement learning (RL)

The authors fine-tune the choice model with on-line policy gradient, starting from IL weights. The BART model is frozen, so its top-10 search strings become a fixed set of extra actions. This avoids language drift in the generator §4.3.

• Return: R_t = r_t + \gamma R_{t+1}, with a learned value baseline V(o) computed on BERT.
• Loss: policy gradient, value L2 loss and an entropy term, summed with equal weight §4.3.

\mathcal{L}_{\text{PG}} = \mathbb{E}_{\pi}\left[-(R_t - V(o_t))\log\pi(a_t \mid o_t, \mathcal{A}(o_t))\right] eq. 5

# One RL update over a rollout of length bptt  [§4.3] [app. C.2]
# rollout: list of (state, actions, action_taken, reward, value, done)

def rl_update(rollout, bootstrap_value):
    # Discounted returns, bootstrapped from the last state  [§4.3]
    returns = discounted_returns(rollout, bootstrap_value, gamma)
    for step in rollout:
        # Advantage uses the value baseline  [§4.3]
        advantage = returns[step] - step.value
        # Policy gradient term  [eq. 5]
        loss = -advantage * log_prob(step.action)
        # Value regression term  [§4.3]
        loss = loss + advantage ** 2
        # Entropy term, as written in the paper  [§4.3]
        loss = loss + sum_p_log_p(step.action_probs)
        # Accumulate gradients over the rollout  [§4.3]
        backward(loss)
    adam_step()

Hyperparameters

Results

Author-reported. Test split of 500 instructions. Learned models averaged over 3 trials fig. 4. Human scores use the HTML mode of the environment §3.2.

Breakdown (score sub-parts and trajectory length). tab. 2

Choice oracle (search queries held fixed, choice made by an oracle). tab. 4

Zero-shot sim-to-real, 100 instructions, manual scoring. tab. 5

Decoding and image ablations (appendix tables, 500 test cases, 3 seeds).

The parenthesised values are not defined in the source.

Ablations in the main results figure fig. 4, values only in the figure: removing language pre-training from the choice model (IL w/o LP Choice) cuts success by nearly two-thirds. Replacing the BART search with the rule query (IL w/o LP Search) costs about 3 points on both metrics. Adding one past observation and the last five actions lowers the IL score from 59.9 to 57.3 §5.2.

Paper vs code

Official repository: princeton-nlp/WebShop, MIT licence, commit 64fa2a5c15c7daa698b9ac93f5bb5437b634c9bd. It is linked from the paper's footnote and owned by the authors' lab. Repository searched via the project page and GitHub.

• Search model size. Paper: "a BART model" §4.2. Code trains facebook/bart-base for the model, while the tokenizer is facebook/bart-large train_search_il.py train_search_il.py.
• Search training hyperparameters. Paper gives none. Code: 10 epochs, batch 4, 50 warm-up steps, weight decay 0.01, no learning rate set train_search_il.py.
• Search query at test. Paper: a random pick from the top 5 beam results §4.2. Code: top-1 by default, with a comment that top-1 "will lead to better results than the paper" test.py. Appendix C.3 reports top-1 as slightly better app. C.3.
• Choice training data split. Paper: 10,587 / 1,000 / 500 §5.1. Code uses human-goal indices 1500+ for training, 500–1500 for evaluation, and 0–500 for testing train_choice_il.py. The search model uses 500+ / 500–1500 / 0–500 train_search_il.py.
• Choice IL: long action lists. Not in the paper. If a state offers more than 20 actions, the code keeps the first 6 plus 10 random others, and always keeps the human's action train_choice_il.py.
• Choice IL: optimiser and scheduler. Not in paper. Code: AdamW, linear decay, no warm-up, weight decay 0 by default train_choice_il.py train_choice_il.py.
• Choice IL: sequence lengths. Not in paper. Code: state up to 512 tokens, action up to 128 train_choice_il.py.
• Choice head. Paper formula eq. 4 with a leaky ReLU inside the cross-attention app. B.1. Code: the attention output (a 4 × 768 concatenation) goes through a 3072→768 linear layer, a ReLU, mean pooling and a 768→1 linear layer bert.py bert.py modules.py.
• Value head. Paper: a linear layer W_v on BERT(o) §4.3. Code: a two-layer MLP (768→128→1, LeakyReLU) on the first token of the state's BERT output bert.py bert.py.
• Reward scale in RL. Paper: reward in [0, 1] fig. 1 caption. Code's RL environment returns the reward multiplied by 10, so returns, the value loss and gradient clipping all work on a 0–10 scale env.py env.py.
• Discount. Paper: \gamma symbolic, value not given. Code default \gamma = 0.9 train_rl.py.
• RL length. Paper: 100,000 steps app. C.2. Code default: 300,000 train_rl.py. The run that produced the paper's numbers is not recorded in the repository.
• RL episode limit. Not in paper. Code: 100 steps per episode train_rl.py.
• RL action pool. Paper: top-10 BART queries, frozen §4.3. Code: the 10 beam queries are precomputed (num_beams=10) and stored, and the raw instruction is appended as one more query generate_search.py env.py.
• Advantage baseline. Paper: V(o) on the current observation §4.3. Code: values are computed when the action is sampled and stored in the rollout, then reused for the advantage in the update agent.py bert.py.
• Entropy term. Paper: sum over actions for each state §4.3. Code: mean over every candidate-action entry in the transition agent.py.
• Loss normalisation and clipping. Code divides each transition's loss by the rollout length, takes a backward pass, and clips gradients at norm 10 after each transition. One optimiser step is taken per rollout agent.py train_rl.py.
• Reward function, type term. Paper: type reward zero, 0.1, 0.5 or 1 by the cases in app. A.5. Code: starts at 1; falls to 0.5 when the query does not match, the coarse category does not match (with at least two shared category labels), and the title score is at most 0.2; 0.1 for title score below 0.1; 0 for title score exactly 0 goal.py. The code's 0.5 case uses query match as well, and the category test is not the paper's coarse-and-fine test.
• Reward, attribute and option matching. Paper: exact set intersection in eq. 1 and app. A.7 ("exact matching"). Code: fuzzy token-set ratio above 85, plus a substring check in title, bullet points and description for attributes goal.py.
• Reward, price term. Paper: always present in eq. 1. Code: r_price is None when the goal has no price limit, which makes the sum in get_reward fail, while the denominator still counts the +1 goal.py.
• Search depth. Matches the paper: 50 results, 10 per page engine.py.
• Licence of the code. MIT for WebShop [LICENSE.md]. The IL training script is adapted from HuggingFace's Apache-2.0 example, as the paper says app. C.1; no code is quoted here.

No verbatim code is quoted in this card.

Pitfalls

• Success-rate headline is ambiguous. The abstract says the best model reaches 29%. The introduction names IL+RL (28.7%) as the best agent. Section 5.2 says the best model's success rate is 29.1%, which is the IL model §5.2. Pick one model and state which.
• Rule baseline success rate is 9.6% in the abstract and introduction and 10% in Section 5.2 §5.2.
• Expert human success rate is 59% in the abstract and 59.6% in the text and tab. 2. The average-human figure in Section 5.2 is 50% and matches appendix A.6 app. A.6.
• Sim-to-real human score on Amazon is 88.0 in the text and 88.2 in tab. 5 §5.4.
• Product categories conflict. Section 3.2 lists fashion, makeup, electronics, furniture and food, with 113 sub-category queries. Appendix A.1 lists beauty, food, fashion, furniture and electronics, with 313 sub-categories §3.2 app. A.1. Use the appendix as the scrape spec and check the released data before reproducing.
• RL reward is 10 times the paper's reward in the official code (see Paper vs code). The gradient-clip threshold of 10 and the value-loss scale are relative to that 0–10 reward, so they do not transfer directly to a [0, 1] reward.
• Test-time search is top-1 in the code and sampled in the paper's main text. Appendix C.3 reports that top-1 is slightly better. Results depend on which you use.
• Reward is not exact. The paper's own check finds the automatic reward a lower bound on a manual score: average trajectories 74.9 automatic vs 76.3 manual; expert 81.5 vs 89.9 app. A.7. The authors note over-penalisation from exact matching.
• Reward-type cases can be read in more than one way (for example whether the 0.5 case needs both category levels). Follow the code only if you also follow its fuzzy matching.
• Choice head is under-specified. The appendix formula app. B.1 uses several weight symbols (w_4, w_5, w_6) without stating which are vectors and which are scalars. Implement the code's head (linear, ReLU, mean, linear) and say that you did.
• No memory in the choice model. Adding a past observation and the last five actions lowered the IL score from 59.9 to 57.3 §5.2. The paper leaves memory to future work.
• Human data is small. 13 workers were trained, 7 were called experts and 8 took part in the human evaluation §3.2 app. A.6. Averages carry large spread: the best expert reached 69.5% success and the weakest 10% app. A.6.
• RL is sensitive to initialisation. RL from raw BERT is worse than the rule baseline; RL with the RNN encoder is more than 10 points lower in success and more variable §5.2.
• Costs. The choice oracle needs hundreds to thousands of steps per episode §5.3. RL runs take 20–27 GPU-hours per model app. C.2. Sim-to-real episodes take under 8 seconds per episode for the models, against 815 seconds for humans §5.4.
• Live-site results are not reproducible in full. The sim-to-real numbers used a scraping-based adaptor, manual scoring of 100 episodes, and a search engine that changes over time app. D.1 §5.4. Treat them as an indication, not a benchmark result.
• Data licence. The product data was scraped from amazon.com using ScraperAPI; the authors' random sampling checks found no personal data app. A.1. The code is MIT but the product data is not covered by this card's licence; check Amazon's terms before redistributing it.
• Figure-based results. Ablation numbers (IL w/o LP Choice, IL w/o LP Search, the 57.3 history result) appear only in the figure or as a qualitative statement fig. 4 §5.2.

Lineage

• Builds on: BART (search generator) and BERT (choice model), cited in the paper as Lewis et al. (2019) and Devlin et al. (2019) §4.2; policy gradient, cited as Mnih et al. (2016) §4.3; freezing the generator, motivated by the language-drift concern of Lazaridou et al. (2020) §4.3; the cross-attention layer follows Seo et al. (2016) app. B.1.
• Compared against: rule baseline (the instruction text as the query, the first item bought) §4.1.
• Related work named in the paper: World of Bits, with low-level mouse actions §3.1; WebGPT, for web browsing with human feedback §1.