DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents

In plain words

Travel-planning assistants often work from a long hand-written prompt and a fixed sequence of steps. They break when a user asks something new or when a booking tool fails. DeepTravel instead lets the language model decide for itself which tool to call next, look at what came back, and correct its plan, and it teaches this with trial and error.

Trial and error needs a stable practice room and a fair scorer. The authors record real travel-search answers once and replay them, so the same question gets the same answers during training. The scorer checks the finished plan in two passes: first whether the overall trip is possible, then whether every claim in the plan matches what the tools returned. A plan passes only if both checks succeed. Queries the model keeps failing are saved and tried again later, so the model revisits its hardest cases. The authors say the resulting model is small enough to run in a real booking app and beats much larger general models on their travel tests.

TL;DR

• An end-to-end RL recipe for a travel-planning LLM agent that calls travel tools, reads their outputs and revises its plan §3.
• Tool responses come from a cache (a "sandbox") that is refreshed every 24 hours, so training does not depend on live APIs §3.1.
• Reward is binary: a trajectory-level check on feasibility, then a turn-level check that the plan matches each tool response §3.2.
• Failed queries go into a buffer and are replayed during RL §3.3.2.
• Author-reported: 82.58% online final pass rate for DeepTravel-32B (Qwen3-32B backbone) and better offline scores than several larger reasoning models tab. 4 tab. 5.

Problem

Existing travel-planning agents rely on hand-written prompts or fixed workflows. The authors say these are labour-intensive and do not adapt to new queries or recover from failed tool calls §1.

Two further difficulties are named. Travel data (hotels, prices, transport) changes over time, so the same query can produce different tool outputs, which makes training unstable §1. Travel planning is also open-ended: there is no single ground-truth itinerary, so the reward must be built from constraints and tool evidence rather than matching an answer key §1.

The task is formalised as a multi-turn process. At turn t the policy reads the query and the earlier thoughts, actions and observations and emits a thought and an action; the final itinerary is the action of the last turn §2 eq. 1.

Method

The pipeline has three parts: a travel sandbox, a hierarchical reward model, and a two-stage training recipe with replay fig. 2 §3.1 §3.2 §3.3.

Policy and output format. Each turn has a thought, a tool call or a final answer, and a tool response. The authors wrap these in special tags (thought, tool call, tool response, answer) §2 §3.3.1.

Sandbox (§3.1). Real travel APIs are wrapped in a single tool interface with six tools in three groups tab. 1 §3.1.1:

• Transportation: flight search, train search, and route planning (distance and time between two named places).
• Accommodation: hotel search (city, hotel name, check-in and check-out dates).
• Attraction: POI search (address of a place) and web search (a free-text travel query).

Flight, train and hotel tools reuse the authors' enterprise travel interface, route and POI tools use a map routing API, and web search wraps a third-party AI search service §3.1.1.

Responses are cached. When a query is new, the system fetches fresh data and stores it in a local database; later identical calls read the cache §3.1.2. The cache is refreshed on demand with a 24-hour cycle, so replays of the same trajectory see identical tool outputs §3.1.3.

Hierarchical reward (§3.2). The reward is decided in two stages, and the turn-level stage runs only if the trajectory stage passes §3.2.3.

• Trajectory-level verifier (§3.2.1): given the whole trajectory, it checks the final itinerary against spatiotemporal constraints: a sensible order of events, geographic plausibility, and the user's explicit requirements such as visiting a named place in a time window.
• Turn-level verifier (§3.2.2): for each earlier turn i, it checks whether the final itinerary agrees with the tool response of that turn, which targets factual errors.
• Combination (§3.2.3): reward is 0 if the first check fails. Otherwise it is 1 only if every turn passes, and 0 otherwise. The reward is therefore binary.

Both verifiers are LLM judges. The authors write human rubrics and prompt DeepSeek-R1 with them §3.2.3. The prompts themselves are not in the source used here (see Pitfalls).

# Hierarchical reward  [§3.2] [§3.2.3]
# Input:  trajectory of thoughts, tool calls and tool responses; itinerary
# Output: scalar reward in {0, 1}

def reward(trajectory, itinerary):
    # Trajectory-level verifier checks spatiotemporal feasibility first;
    # on failure the reward is 0 and no turn checks are run  [§3.2.1]
    if not trajectory_verifier(itinerary):
        return 0
    # Turn-level verifier compares the itinerary with each tool response
    # o_1 .. o_{t-1}; any mismatch gives reward 0  [§3.2.2]
    for observation in trajectory.tool_responses:
        if not turn_verifier(itinerary, observation):
            return 0
    # Every turn is consistent, so the reward is 1  [§3.2.3]
    return 1

Cold start by SFT (§3.3.1). The authors distil multi-turn trajectories from DeepSeek-R1 in the sandbox. They keep only trajectories the reward model accepts and that pass strict tag-format checks §3.3.1. Training fits the policy to the verified trajectories, given the system prompt and the query; tool-response tokens are masked out of the loss §3.3.1. The objective is written as the expected log-likelihood of the verified trajectory eq. 2; whether it is maximised or minimised is not stated.

Replay-augmented RL (§3.3.2). RL starts from the cold-start policy and uses GRPO-style group sampling §3.3.2:

• For each query, n trajectories are sampled and scored with the reward above.
• If no trajectory in the group is verified correct, the query is stored in a replay buffer B.
• The buffer is replayed after a fixed number of steps gamma; eq. 3 draws the query from the union of the training set D and B §3.3.2 eq. 3.
• A group is excluded when the standard deviation of its rewards is at most eta = 0.1, which the authors describe as removing queries that are too easy or too hard §3.3.2.
• The advantage is the group-normalised reward, and the update is a clipped ratio objective with a KL penalty to a reference policy eq. 3.
• Tool-response tokens are masked so only agent tokens get policy gradient §3.3.2.

# Replay-augmented RL step  [§3.3.2] [eq. 3]
# Input:  train set D, replay buffer B, policy, reference, group size n
# Output: updated policy

def rl_step(step, D, B, policy, reference, n):
    # Draw the query from D or from B; the text says B is replayed after
    # every gamma steps, while eq. 3 writes q ~ {D, B}  [§3.3.2] [eq. 3]
    query = pick_query(step, D, B)
    # Sample a group of n trajectories for the query  [§3.3.2]
    group = [policy.rollout(query) for _ in range(n)]
    # Score every trajectory with the hierarchical reward  [§3.2.3]
    rewards = [reward(y, y.itinerary) for y in group]
    # Keep the query for replay when no trajectory passed  [§3.3.2]
    if max(rewards) == 0:
        B.add(query)
    # Drop the group when the reward spread is at most eta = 0.1  [§3.3.2]
    if std(rewards) <= 0.1:
        return policy
    # Group-normalised advantage for each trajectory  [eq. 3]
    advantages = [(r - mean(rewards)) / std(rewards) for r in rewards]
    # Clipped ratio objective with KL penalty; tool-response tokens are
    # excluded from the gradient  [eq. 3] [§3.3.2]
    objective = clipped_kl_objective(group, advantages, policy, reference)
    return gradient_ascent_step(policy, objective)

Deployment recipe (§4). The authors first prototyped on DeepSeek-R1 (671B) and found it both less accurate for this domain and too slow. They then trained Qwen3-32B with the recipe above, producing DeepTravel-32B, which is served on two nodes of four H800 GPUs with an average latency of about 25 seconds §4. An asynchronous loop adds failure cases from production to SFT and RL data after review by product staff §4.

Hyperparameters

Values are as stated in the paper (main text and appendix). "not stated" means the paper does not give the value.

Results

Author-reported. Two evaluation settings are used §5.2.

• Online: 6,224 real production queries from the DiDi Enterprise Solutions app, collected between 1 June and 31 August 2025 tab. 2.
• Offline: 500 synthetic queries with explicit user constraints and 500 without, each labelled easy, medium or hard by product experts §5.2 tab. 2.

The metric is the Final Pass Rate: an itinerary counts as correct only if it passes all seven human dimensions §5.2.1. For full-sample scoring the authors use the same LLM reward system as a judge, and check 200 samples by hand §5.2.1. The judge agrees with humans on 82% of checked samples; after allowing for human label error, the authors estimate 89% accuracy for the judge tab. 3 §5.2.1.

Online, DeepTravel-32B tab. 4 §5.3:

Offline, all systems tab. 5 §5.4. Columns are the pass rate (%) on the offline sets and the human-evaluation column given in the table.

Ablation on the 8B model tab. 6 §5.5:

Other reported numbers. In the online setting the 8B and 32B cold-start models rise to 49.75 and 62.77 after RL, from 40.00 and 50.03 §5.4. Sandbox statistics for one RL epoch on 500 queries: 12,883 tool calls, of which web search is 3,014, route planning 3,267, POI search 2,657, hotel search 1,211, train search 1,835 and flight search 899 tab. 7. On 50 sampled online cases, the cold start mainly improves intent understanding, completeness, feasibility and clarity; personalisation is said to need RL, and the authors report factual error rates near 50% for the base and cold-start models, falling below 20% after RL fig. 5 §5.7.

Paper vs code

No official code.

Search order: the paper has no code or project link; the arXiv abstract page lists no code, data or project link; the LaTeX source (sample-sigconf.tex) has no URL besides the ACM classification tool. A GitHub repository search for "DeepTravel" returned 17 repositories, none by the authors. A search restricted to the didi GitHub organisation returned none. No code was cloned.

Pitfalls

• Conflicting counts in the dataset table. The main Table 2 gives 222, 78 and 200 for the constrained sets and 156, 45 and 299 for the unconstrained sets tab. 2. A commented-out appendix paragraph gives the reverse. The main table is used on this card, but the assignment is not confirmed.
• Headline number. The abstract rounds the online result to 82%; the table gives 82.58% tab. 4. The offline human-evaluation column reports 82.00 for DeepTravel-32B-RL, but the set it was measured on is not stated tab. 5.
• Relative gains in the text do not match the table. The text says the 8B and 32B models beat K2 by 0.1% and 28.9% on the unconstrained easy set; the table gives gaps of 0.24 and 15.33 percentage points (about 28% relative) §5.4 tab. 5.
• Online cold-start and RL numbers are not in any table. The values 40.00, 49.75, 50.03 and 62.77 appear only in the text §5.4.
• Baseline RL backbone conflict. The text says PPO, GRPO and DAPO start from DeepTravel-8B-Cold-Start §5.4, while Table 5 labels them Qwen3-8B with each method tab. 5. The number of RL steps for DeepTravel-8B-RL is not stated, only the 100 steps for the baselines.
• Advantage formula. eq. 3 is printed as r_i - avg(r) / std(r) with no brackets; the intended form is (r_i - avg(r)) / std(r). Whether std is the population or sample standard deviation is not stated eq. 3.
• Replay schedule is ambiguous. The text replays the buffer after a fixed number of steps gamma, but eq. 3 draws queries from D and B together. The mixing ratio, buffer capacity and whether replayed queries stay in the buffer are not stated §3.3.2 eq. 3.
• The eta filter. The text speaks of filtering "samples", but the rule is applied to groups. Because the reward is binary §3.2.3, a group's reward standard deviation is at most 0.1 only when all rollouts get the same reward, and any group with a single pass is kept. This is inferred from the text and not confirmed by code.
• Reward and evaluation use the same judge. Full-sample evaluation uses the reward system as judge §5.2.1. The hand check covers 200 samples and the reported 82% to 89% correction depends on an assumed 7% human error rate tab. 3. Judge bias is not measured against a fully independent metric.
• Verifier prompts and rubrics are missing. The text points to an appendix for the verifier and policy prompts, but those sections are commented out in the source. The rubrics are not released §3.2.3 §3.3.1.
• Sandbox size conflict. §5.4 says the sandbox caches about one million records; Table 7 lists 117,844 mock records tab. 7 §5.4.
• Sandbox coverage is unclear. §3.1.2 names flight, train and hotel data as the cached dynamic data; the abstract also mentions POI data. Whether route, POI and web results are cached on the same 24-hour cycle is not stated.
• Tool schema mismatch. Table 1 names arguments such as depart_city_name and arrival_city_name. The case study in the appendix calls train_search with depart_station and arrive_station, and uses hotel_search with the same argument names as Table 1 tab. 1 app. A.3. Implement from Table 1 and expect differences from the trajectories shown.
• Training time conflict. §4 gives about 30 hours per "hundreds of steps"; the appendix gives 30 hours per 100 steps §4 app. A.1.
• Costly reward. Every training trajectory is scored by a DeepSeek-R1 judge, and the turn-level check runs once per earlier turn §3.2.2 §3.2.3. The per-call cost and throughput are not stated.
• Latency claim without a baseline. The paper says the deployed model saves about 90 seconds against DeepSeek-R1 but gives no measured DeepSeek-R1 latency in the same setup §4.
• Non-replicable inputs. Training queries and the sandbox data are not released, citing ethical requirements app. A.3. The tools are proprietary to the authors' company, so only the method can be reimplemented.
• Entropy behaviour. The authors observe entropy that does not decrease during agentic RL, which they attribute to tool outputs changing the policy's input §5.6. No repeat runs or variance are reported for it.
• Table column order differs. Table 2 lists the constrained columns first, Table 5 lists the unconstrained columns first. Check the header before copying numbers.

Lineage

• Builds on: GRPO (Shao et al., 2024, DeepSeekMath) for group sampling §3.3.2; replay of earlier failed queries, cited to RLEP (Zhang et al., 2025) §3.3.2; RLVR as the setting for verifiable reward (DeepSeek-R1, Guo et al., 2025) §1.
• Compared against: TripTailor and TravelAgent with DeepSeek-R1 as backbone; PPO, GRPO and DAPO on Qwen3-8B; nine reasoning LLMs §5.2.2.
• Related travel-planning work named in the paper: TravelPlanner, TripTailor, TravelAgent, PTS, RETAIL, ChinaTravel §6.
• Superseded by: not known from content on this wiki.