Paper · 2026-09-30 · cs.LG, cs.AI
Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents
TJPO adds explicit diversity to GiGPO-style RL for LLM agents: user-defined trajectory descriptors, a set-level diversity score per sampled group, and per-trajectory credit added to the advantage. Author-reported: higher descriptor diversity on ALFWorld and Sokoban with small success changes; on an ALFWorld subset, +8.3 points compliant success under user constraints and +7.5 under disruption.
LLM agentPolicy gradientCredit assignmentQuality-diversity searchreinforcement-learningllm-agentsdiversitypolicy-optimization
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
An LLM agent that solves a multi-step task, such as a text-based household task or a box-pushing puzzle, can often finish it in several different good ways. The routes differ in which actions they use and where they go. Standard reinforcement learning rewards only success, so over time the agent tends to keep repeating the one way that works most often and drops the others.
This paper asks the trainer to care about variety too, but only in the way the person building the agent chooses. The builder writes simple measurements of each attempt, such as which kinds of actions it used or which places it visited. Within a group of attempts on the same task, the more the attempts differ on those measurements, the more credit the unusual ones get. The agent is then pushed toward a spread of successful strategies, not one favourite path.
The point is control. Because the builder decides what counts as different, the agent varies in the directions that matter for the job. The authors report that this keeps success roughly the same while the chosen kind of variety goes up, and that the extra variety helps when a user forbids a place or a route is blocked.
TL;DR
TJPO (Trajectory-guided Joint Policy Optimization) adds a diversity term to group-based RL for LLM agents, built on GiGPO §2.2. Diversity is a set-level score over trajectory descriptors chosen by the user §3.2 §3.3. The score is turned into a per-trajectory credit, standardised within each group and added to the advantage §3.4 eq. 19. Results are author-reported and come from single training runs; gains are largest on the descriptor being optimised.
Problem
Group-based RL for agents rewards success only, so the policy can concentrate on a few reliably rewarded behaviours while other valid strategies are sampled and reinforced less §1. Entropy bonuses and decoding penalties add variety but do not say which behaviour should vary, so the variety may not match the task §1 §5. The paper asks how to define and optimise task-relevant trajectory diversity during RL post-training without losing task performance §1.
The method assumes the practitioner can write descriptors that map a trajectory to numbers for the behaviours they care about §3.2. It also assumes groups of several sampled trajectories per input, as in GRPO and GiGPO §2.2.
Method
TJPO takes the GiGPO advantage and adds a diversity credit computed from a group of trajectories sampled for the same input §3.1. The steps are:
- Descriptors. Each trajectory is mapped to a vector of m scalar descriptors, the rows of a G×m matrix Z §3.2 eq. 8 eq. 9. The paper requires the descriptors to be user-specified and intent-aligned §3.2.
- Set-level diversity. A diversity functional D maps Z to one scalar per group. It must not depend on the order of trajectories §3.3 eq. 10 eq. 11. Two options are given: the average pairwise distance eq. 12, and a determinantal score, the log-determinant of an RBF kernel matrix plus a small diagonal term eq. 13 eq. 14.
- Why per-trajectory credit. GRPO centres rewards within the group, so a bonus added equally to every trajectory cancels out §3.4 eq. 15. The diversity signal must instead differ between trajectories.
- Credit. For pairwise diversity, the credit of trajectory i is its average distance to the other G−1 trajectories eq. 16. The paper shows this is a positive affine transform of the exact leave-one-out change in D, so after group standardisation both give the same values app. A. For determinantal diversity, the credit is the ratio of the full determinant to the determinant with trajectory i removed eq. 17 app. A.
- Standardise. Credits are centred and divided by the group standard deviation, with a floor on the denominator eq. 18 app. C.
- Augment the advantage. The standardised credit, scaled by λ ≥ 0, is added to the GiGPO advantage eq. 19 §3.4. The same value is shared by every valid response token of that trajectory §3.4 app. C.
The GiGPO advantage being augmented is A_i plus ω times a step-level micro advantage, where the micro term compares an action with others from the same anchor-state group eq. 4 eq. 5 §2.2. The paper sets ω to 1.0 tab. 7.
Pseudocode for the pairwise variant, steps 4 to 6. Distances are Euclidean tab. 7:
# Pairwise diversity credit for one prompt group [§3.4] [eq. 16]
# Input: descriptors (G, m) [eq. 9]
# Output: credit (G,)
def pairwise_credit(descriptors):
# Euclidean distance between every pair of trajectories [§3.4]
distances = pairwise_euclidean(descriptors) # (G, G)
# Average distance to the other G-1 trajectories [eq. 16]
group_size = len(descriptors)
credit = distances.sum(axis=1) / (group_size - 1) # (G,)
return credit# Standardise the credit inside the group [eq. 18] [app. C]
# eps is the std floor, 1e-6 in the paper [tab. 7]
def standardise(credit, eps):
# Centre by the group mean [eq. 18]
centred = credit - mean(credit)
# Divide by the group std, floored at eps [eq. 18]
spread = max(std(credit), eps)
return centred / spread# Augmented advantage used for policy optimisation [eq. 19] [§3.4]
# Input: gigpo_advantage (G, T), credit_hat (G,), lam
# Output: advantage (G, T)
def augmented_advantage(gigpo_advantage, credit_hat, lam):
# Shift every token of trajectory i by lam * credit_hat[i] [§3.4]
shift = lam * credit_hat[:, None] # (G, 1)
return gigpo_advantage + shiftDeterminantal variant, as implemented in the appendix. The paper uses a different kernel here: an RBF with bandwidth σ = 10 and jitter η = 1e-4 app. A tab. 7. The main text leaves the kernel unspecified eq. 13.
# Determinantal diversity credit [eq. 17] [app. A]
# Input: descriptors (G, m); sigma = 10; eta = 1e-4
# Output: raw credit (G,), divided by its group maximum before standardising
def determinantal_credit(descriptors, sigma, eta):
# RBF Gram matrix with jitter on the diagonal [app. A]
gram = exp(-squared_distances(descriptors) / sigma)
full = gram + eta * identity(len(descriptors))
# Full determinant is the reference [eq. 17]
credit = zeros(len(descriptors))
for i in range(len(descriptors)):
# Ratio of full determinant to the one without trajectory i [eq. 17]
reduced = remove_row_and_column(full, i)
credit[i] = det(full) / det(reduced)
return creditHyperparameters and the policy-side details are in tab. 6 and tab. 7. The environment-specific descriptors are in app. D.
Hyperparameters
Environment setup tab. 6:
| Setting | ALFWorld | Sokoban | Source |
|---|---|---|---|
| Base model | Qwen2.5-7B-Instruct | Qwen3-VL-4B-Instruct | tab. 6 §4.1 |
| Input modality | Text | Vision-language (RGB image) | tab. 6 |
| Max environment steps | 50 | 15 | tab. 6 |
| Training data size | 16 | 32 | tab. 6; unit not stated |
| Total epochs | 150 | 200 | tab. 6 |
| Advantage normalisation | mean_std_norm | mean_norm | tab. 6 |
| Max prompt length | 2048 | 1024 | tab. 6 |
| PPO mini-batch size | 256 | 64 | tab. 6 |
| PPO micro-batch per GPU | 16 | 8 | tab. 6 |
| Validation frequency (steps) | 5 | 10 | tab. 6 |
Shared training and evaluation settings tab. 7:
| Hyperparameter | Value | Source |
|---|---|---|
| Training framework / RL backbone | verl / GiGPO | tab. 7 app. C |
| Group size G | 8 | tab. 7 |
| Max response length | 512 | tab. 7 |
| Learning rate | 1e-6 | tab. 7 |
| KL loss type / coefficient | low-variance KL / 0.01 | tab. 7 |
| Discount factor γ | 0.95 | tab. 7 |
| GiGPO step-advantage weight ω | 1.0 | tab. 7 |
| Diversity integration mode | advantage | tab. 7 |
| Diversity-credit normalisation | group mean/std | tab. 7 eq. 18 |
| Diversity-credit std floor | 1e-6 | tab. 7 app. C |
| Pairwise distance | Euclidean | tab. 7 |
| DPP kernel bandwidth σ | 10 | tab. 7 app. E |
| DPP diagonal jitter η | 1e-4 | tab. 7 app. E |
| Diversity weight λ | 0.3 for Tables 1–3; 0.5 for Table 5; sweep in Table 4 | §4.2 §4.3 §4.5 §4.6 |
| Invalid action penalty | 0.1 | tab. 7 |
| Validation temperature | 0.4 | tab. 7 app. C |
| Standard evaluation presence / frequency penalty | 0.0 / 0.0 | tab. 7 |
| Standard repetition penalty | 1.0 | tab. 7 |
| Evaluation trajectories per input N_eval | 8 | tab. 7 app. C |
| Hardware | 1 node, 8 NVIDIA H20 GPUs, 144 GB each | tab. 7 app. C |
| Tensor parallel size | 1 | tab. 7 |
| GPU memory utilisation (vLLM) | 0.6 | tab. 7 |
| Environment seed | 0 | app. C |
| Training runs per configuration | 1 | app. C |
Descriptor dimensions app. D tab. 8:
| Environment | Descriptor | Dimension | Definition |
|---|---|---|---|
| ALFWorld | action_type | 15 categories | share of each action type app. D |
| ALFWorld | location | |L(x)|, per prompt | share of go-to destinations app. D |
| Sokoban | direction_transition | 16 (4×4 pairs) | consecutive move-direction pairs app. D |
| Sokoban | spatial_coverage | 7×7 = 49 | visits on a 7×7 grid centred on the start app. D |
The paper gives no learning-rate schedule and no warm-up tab. 7. It does not state how many groups make up a batch, and it does not state the number of epochs in terms of samples tab. 6.
Results
Author-reported. Setup: ALFWorld with Qwen2.5-7B-Instruct and Sokoban with Qwen3-VL-4B-Instruct, both trained from the same base with the same budget §4.1 tab. 6. Diversity is measured on successful trajectories only, for N_eval = 8 samples per input §4.1 app. C. The main diversity tables use λ = 0.3 §4.2 §4.3.
ALFWorld, pairwise diversity tab. 1. Success is the task success rate. D_act and D_loc are the pairwise diversity of the action-type and location descriptors tab. 1:
| Method | Success ↑ | D_act^pair ↑ | D_loc^pair ↑ |
|---|---|---|---|
| GiGPO | 0.9160 | 0.0495 | 0.0858 |
| GiGPO + PP (inference only) | 0.9082 | 0.0458 | 0.0895 |
| GiGPO + RP (inference only) | 0.9229 | 0.0494 | 0.0873 |
| TJPO act-pair | 0.8682 | 0.0780 | 0.1002 |
| TJPO loc-pair | 0.9053 | 0.0601 | 0.1316 |
ALFWorld, determinantal diversity, values ×10⁻³ tab. 2:
| Method | Success ↑ | D_act^dpp ↑ | D_loc^dpp ↑ |
|---|---|---|---|
| GiGPO | 0.9160 | 1.887 | 2.500 |
| GiGPO + PP | 0.9082 | 1.858 | 2.063 |
| GiGPO + RP | 0.9229 | 1.135 | 1.544 |
| TJPO act-dpp | 0.9160 | 4.100 | 6.655 |
| TJPO loc-dpp | 0.8779 | 3.058 | 7.940 |
Sokoban tab. 3:
| Method | Success ↑ | D_dir^pair ↑ | D_spa^pair ↑ |
|---|---|---|---|
| GiGPO | 0.8281 | 0.0146 | 0.0092 |
| TJPO dir-pair | 0.8428 | 0.0675 | 0.0503 |
| TJPO spa-pair | 0.8057 | 0.0595 | 0.0518 |
λ ablation, ALFWorld, location-pairwise tab. 4:
| λ | Success ↑ | D_loc^pair ↑ | Avg. steps ↓ |
|---|---|---|---|
| 0 | 0.9160 | 0.0858 | 9.2217 |
| 0.1 | 0.9209 | 0.0942 | 9.5811 |
| 0.3 | 0.9053 | 0.1316 | 9.6947 |
| 0.5 | 0.9004 | 0.1356 | 8.1150 |
Practical benefit, ALFWorld subset with verified alternative routes, λ = 0.5, location-pairwise tab. 5 app. G. Standard is the original task; User Constraints forbids a location type; Disruption makes it unavailable §4.6:
| Method | Standard ↑ | User Constraints ↑ | Disruption ↑ |
|---|---|---|---|
| GiGPO | 0.9917 | 0.7708 | 0.7792 |
| TJPO (λ = 0.5) | 0.9750 | 0.8542 | 0.8542 |
The paper's headline gains, read from these tables: the location-pairwise variant lifts D_loc^pair from 0.0858 to 0.1316 with success 0.9053 against 0.9160 tab. 1. Sokoban direction-transition diversity lifts D_dir^pair from 0.0146 to 0.0675 with success 0.8428 against 0.8281 tab. 3. On the subset, TJPO adds 8.3 points under User Constraints and 7.5 under Disruption tab. 5 §4.6.
Case study. On one prompt ("put a spraybottle in cabinet") with eight rollouts, GiGPO follows one route in seven rollouts. TJPO spreads its eight successful rollouts over three routes, one of which goes directly through the Toilet §4.4 fig. 2 app. F fig. 4. The ladle case shows a smaller version of the same effect app. F fig. 5.
Paper vs code
No official code.
Search, as recorded on 2026-11-21:
- The paper's arXiv source contains no repository, project page or code link
[raw source,
arxiv.tex]. - GitHub repository search (via the GitHub API) for "TJPO", the title words, "TJPO GiGPO" and "explicit trajectory diversity": no matching repositories.
- Inside the
tuyoogameorganisation (an author's affiliation) and themicrosoftorganisation (another author's affiliation), searches for TJPO, diversity and GiGPO: no matching repositories. A user search for the first author's name returned no users.
A GiGPO code repository exists (langfengQ/verl-agent, Apache-2.0). It belongs to the baseline paper, not to this one, so it was not used.
Pitfalls
- The diversity term cancels unless it is per trajectory. Adding one group value to all rewards does nothing after GRPO centring §3.4 eq. 15. Use the per-trajectory credit from eq. 16 or eq. 17.
- The pairwise identity needs at least three trajectories. The derivation divides by G−2 app. A. The paper uses G = 8 tab. 7; the restriction is not stated in the main text.
- Determinantal kernel scale (inference, not stated). Descriptors are histograms app. D, so squared distances between two of them are at most 2. With σ = 10 app. C tab. 7, every kernel entry is at least about e^−0.2 ≈ 0.82. The Gram matrix is then close to all-ones, so determinants are small and η = 1e-4 matters. The paper does not discuss this. Check the outputs of the credit on real groups before trusting it.
- Undocumented normalisation in the determinantal branch. The appendix
divides raw determinant-ratio credits by their group maximum before
standardising app. A. The main text eq. 17 eq. 18 does not. The
appendix also has fallbacks app. A:
- if the full regularised determinant is at most 1e-10, use the mean of 1 − K_ij instead;
- if a leave-one-out determinant is at most 1e-10 while the full one is valid, set that trajectory's raw credit to 1;
- if a group's credit has zero variance, its normalised diversity advantage is zero.
- The standardisation floor is given only in the appendix. The main text writes max{std, ε} without a value eq. 18. The appendix uses 1e-6 app. C tab. 7.
- Location descriptors change size per prompt. The dimension is |L(x)|, the number of locations in that prompt's environment app. D. Descriptor vectors from different prompts are not comparable, so credits must stay within one prompt group.
- Evaluation counts only successful trajectories. The paper's diversity scores use the successful subset of the N_eval samples §4.1 app. C. Success counts therefore change what is measured. For the determinantal score, the n-th root of the log-determinant is used, and groups with fewer than two successes score zero app. E. The paper does not say how the pairwise score behaves with fewer than two successes.
- The diversity gains are measured on the optimised descriptor. The evaluation uses the same descriptor and metric as training §4.1 app. C. Gains on the descriptor itself are expected by construction. The evidence that the variety matters is in the practical-benefit test §4.6.
- Small effects, single seed. Each configuration comes from one training run with environment seed 0 app. C. No variance is reported. Differences of about one point in success, such as GiGPO 0.9160 against GiGPO+RP 0.9229 tab. 1, are within what a single run can show.
- Small practical-benefit sample. The subset has 15 tasks and 30 cases per intervention, with eight rollouts each app. G. Standard has 120 trajectories per model and each intervention has 240 app. G. Derived from tab. 5: 8.3 points is 20 more successes out of 240, and 7.5 points is 18 out of 240. Standard success drops from 0.9917 to 0.9750, which is 2 fewer successes out of 120.
- Different λ across experiments. The main diversity tables use λ = 0.3 §4.2 §4.3. The practical-benefit test uses λ = 0.5 §4.6 app. G. The paper does not say how λ was chosen, and the λ ablation tab. 4 is one run per value.
- Cost. Rollouts are the same as GiGPO tab. 7. The extra work is O(G²) distances or one small determinant per trajectory, which is cheap for G = 8. The real cost is the descriptor parser: ALFWorld actions are matched by prefix app. D, and Sokoban moves are parsed from action tags in the response app. D. The paper does not say what happens when parsing fails.
- Unstated details. The paper gives no learning-rate schedule, no warm-up and no batch count. Treat them as not stated.
Lineage
- Builds on: GiGPO, arXiv:2505.10978 (not yet compiled here) §2.2 §4.1. TJPO keeps its advantage and adds the diversity credit eq. 19.
- Builds on: GRPO-style group-relative advantages §2.2.
- Compared against: GiGPO, GiGPO with a presence penalty of 0.5 (PP) and with a repetition penalty of 1.05 (RP), which are inference-only baselines §4.1.
- Superseded by: not known.
Source and license
Adapted from “Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents” by Huaiyu Fu, Heng Cao, Hao Wang, Jian Ya, Tao Chen (arXiv:2609.38805), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.