Paper · 2026-10-08 · cs.CL, cs.AI, cs.LG
Recursive Semantic Divergence for LLM Agent Consistency
RSD is a per-turn consistency score for LLM agents: the expected discounted disagreement between two independent rollouts from the same history, judged by gpt-4.1 NLI and learned by a small MICo-style network with 3-step TD targets. Used as a fine-tuning reward, it cuts AUC coefficient of variation by up to 30.9% on tau2-bench and 50.5% on ToolSandbox (Qwen-2.5-3B), all author-reported.
LLM agentUncertainty estimationNatural language inferencePolicy gradientagentsconsistencytool-usereinforcement-learningbisimulationnli
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Ask an AI agent that uses tools the same question twice from the same starting point and it may take different steps: a different tool, a different answer, or a different plan. Standard tests mostly ask whether the final result was right, not whether the agent behaves the same way each time.
The idea here is to run the agent twice from the same point, let the two runs play out, and measure how much they disagree, both at the current step and in the steps that follow. A language-model judge decides whether two actions mean the same thing. A small network is trained to predict that disagreement from a single snapshot of the conversation, so the agent can be scored at every turn without finishing the task.
The same score can also be used as a training reward. The authors report that this makes agents more consistent on two tool-use benchmarks, though not in every setting, and that a consistent agent can still be consistently wrong.
TL;DR
Recursive Semantic Divergence (RSD) scores how much an agent's future behaviour varies from a given state. Training uses independent rollouts, an NLI judge (gpt-4.1) and n-step TD targets. Inference needs one embedding and one small network pass per state. In the authors' Qwen-2.5 experiments it lowered variation in 3 of 4 settings and raised task progress in all 4. Results are author-reported on 8 test samples per benchmark.
Problem
Agents calling tools can diverge across runs from an identical state: different tools, contradictory outputs or different strategies §1. Correctness metrics such as pass^k are binary and used for reporting, not for enforcing consistency §2. Token-level entropy and single-response semantic entropy do not capture how a disagreement propagates over later turns §2 §3.1. The paper treats consistency and correctness as separate properties: a consistent agent can be consistently wrong §1.
Method
The paper casts an LLM agent as a POMDP and treats the interaction history as the state §4.1 eq. 3 eq. 4. Each step of the method:
- States and rollouts. A state is the whole conversation so far, up to turn t. It covers what the user said, what the agent did (tool calls included) and what the tools returned §4.1 app. A.3.1. Two independent rollouts from the same state are paired at the same depth t §4.3.
- NLI distance. Two actions are compared with a three-class NLI judge. The distance is 0 for bidirectional entailment, 1 for contradiction in either direction, and 0.5 otherwise [def. 1] eq. 5. Gpt-4.1 is the judge app. A.3.1.
- Recursive definition. The distance between two states is the expected NLI distance now, plus a discounted expectation of the same quantity at the next states eq. 6 §4.2. The RSD of a state is this quantity for two rollouts from that state, discounted over an infinite horizon [def. 2] eq. 10.
- Learned approximation. A MICo-style distance eq. 2 is trained to match an n-step TD target: a discounted sum of NLI distances over the next n paired steps, plus a bootstrap from a soft-updated target copy at depth t+n eq. 7 §4.3. The loss is the squared Bellman residual eq. 8.
- Score. The per-state RSD is the self-distance 2α‖φ(z_s)‖², where the angle term vanishes eq. 9 alg. 2. The state is encoded once by a frozen embedding model and mapped to a 256-dimensional latent by a three-layer MLP app. A.3.1.
- Fine-tuning. The score is used as a reward in GRPO §6. The paper also tests mixing it with a per-step progress reward using a weight λ §7.2.
Algorithms 1 and 2 in the appendix give the training and inference loops alg. 1 alg. 2. The pseudocode below follows them; the code also differs in places, as listed under Paper vs code.
# RSD training: fit the self-distance to n-step TD targets [alg. 1]
# Input: per task, two or more rollouts of (state, action, next state)
# Output: trained network phi, plus its soft-updated target copy
def train_step(tasks, phi, phi_target, nli, gamma, n_steps):
# Draw a task, then two different rollouts of that task [alg. 1]
task = sample_task(tasks)
run_a, run_b = sample_two_runs(task)
# Pick a shared depth t for both rollouts [§4.3]
t = sample_common_depth(run_a, run_b)
# Sum discounted NLI distances over the paired steps [eq. 7]
target = 0.0
for k in range(n_steps):
# NLI distance between the actions at depth t + k [def. 1] [eq. 5]
target += gamma ** k * nli_distance(run_a, run_b, t + k)
# No padding and no bootstrap if either rollout ends first [app. A.3.2]
if is_last(run_a, t + k) or is_last(run_b, t + k):
break
else:
# Bootstrap from the target network at depth t + n [eq. 7]
target += gamma ** n_steps * distance(
phi_target, run_a, run_b, t + n_steps
)
# Squared Bellman residual against the online network [eq. 8]
prediction = self_pair_distance(phi, run_a, run_b, t)
loss = (prediction - target) ** 2
# Gradient step on phi, then soft-update the target copy [alg. 1]
take_gradient_step(phi, loss)
soft_update(phi_target, phi)# RSD inference: one score per state [alg. 2]
# Input: interaction history s (text), frozen encoder enc, trained phi
# Output: scalar U(s, s) >= 0; higher means less consistent
def rsd_score(history, enc, phi, alpha):
# Embed the history with the frozen encoder [alg. 2]
z_state = enc(history)
# Map to the learned 256-dimensional latent [app. A.3.1]
phi_state = phi(z_state)
# The angle term vanishes at zero angle, leaving the norm [eq. 9]
return 2.0 * alpha * sum_of_squares(phi_state)Hyperparameters
The paper gives few numeric values. Rows marked "code only" come from the official repository, not the paper, and the paper states no value for them.
| Name | Value | Where | Notes |
|---|---|---|---|
| discount γ | 0.8 | app. A.3.1 | paper and code agree |
| TD steps n | 3 | app. A.3.1 | paper and code agree |
| latent dimension m | 256 | app. A.3.1 | 3-layer MLP output; hidden widths not stated in paper |
| embedding model | Cohere embed-v4-0, frozen | app. A.3.1 | output dimension not stated in paper; code requests 1536 train.py |
| NLI judge | gpt-4.1 | app. A.3.1 §7.3 | judge temperature not stated; code uses 0.0 train.py |
| norm weight α | not stated | eq. 2 | paper says only "fixed, positive"; code 0.5 train.py (code only) |
| angle weight β | not stated | eq. 2 | paper says only "fixed, positive"; code 5.0 train.py (code only) |
| training rollouts, τ²-bench | M = 30 per task, 13 task instances | app. A.3.1 | rollout temperature 1 |
| training rollouts, ToolSandbox | M = 8 per task, 29 task instances | app. A.3.1 | rollout temperature 1 |
| evaluation trials | 8 trials × 8 test samples = 64 trajectories per benchmark | §6 | Qwen-2.5 models |
| inference temperature | 0 | §6 | |
| fine-tuning algorithm | GRPO | §6 app. A.3.3 | GRPO hyperparameters not stated; no fine-tuning code in repo |
| learning rate | not stated | app. A.3.1 | code only: 1e-4 train.py |
| optimiser | not stated | code only: AdamW, weight decay 1e-5 train.py | |
| batch size | not stated | code only: 32 train.py | |
| gradient clipping | not stated | code only: global norm 1.0 train.py | |
| target-network rate τ | "soft update", rate not stated | §4.3 | code only: 0.005 train.py |
| epochs | not stated | code only: 300 train.py | |
| train/test split | not stated | code only: 80/20 by task id, seed 42 train.py | |
| loss | squared Bellman residual | eq. 8 | code default MSE train.py |
Results
Author-reported. Setup: τ²-bench (airline customer service) and ToolSandbox, Qwen-2.5 3B and 7B; Qwen-3.5 4B and 9B in the appendix §6 app. A.4. Each benchmark uses 8 test samples, each run 8 times. The metrics are AUC and final progress, both with their coefficient of variation (CV) over the 8 trials, plus pass@1 and pass^3 §6 app. A.3.4. The text reports no confidence intervals and no repeated training seeds.
Headline claims: RSD lowers AUC CV by 30.9% on τ²-bench and 50.5% on ToolSandbox, both for Qwen-2.5-3B §7.1 tab. 1. It lowers AUC CV in 7 of 8 Qwen model–benchmark settings, and raises progress in all 8 [abstract].
Table 1 – Main results (Qwen-2.5 and fine-tuning baselines) tab. 1
| Setting | Method | Area under curve | CV of area | Final progress | CV of progress | pass@1 | pass^3 |
|---|---|---|---|---|---|---|---|
| ToolSandbox, Qwen-2.5-3B | Base | 0.271 | 0.90 | 0.293 | 0.86 | 14.1% | 20.0% |
| Token entropy | 0.302 | 0.50 (−43.8%) | 0.317 | 0.53 (−37.8%) | 15.6% | 20.0% | |
| Semantic entropy | 0.326 | 0.83 (−7.0%) | 0.354 | 0.82 (−4.0%) | 17.2% | 20.0% | |
| SFT | 0.311 | 0.82 (−8.9%) | 0.327 | 0.81 (−5.6%) | 17.2% | 20.0% | |
| RSD | 0.340 | 0.44 (−50.5%) | 0.362 | 0.43 (−49.3%) | 18.8% | 21.4% | |
| ToolSandbox, Qwen-2.5-7B | Base | 0.539 | 0.38 | 0.557 | 0.36 | 29.7% | 28.6% |
| Token entropy | 0.510 | 0.50 (+32.4%) | 0.523 | 0.50 (+40.5%) | 21.9% | 19.4% | |
| Semantic entropy | 0.504 | 0.36 (−6.2%) | 0.512 | 0.34 (−6.1%) | 23.4% | 23.2% | |
| SFT | 0.518 | 0.45 (+17.9%) | 0.530 | 0.43 (+20.6%) | 31.2% | 28.8% | |
| RSD | 0.555 | 0.28 (−26.8%) | 0.562 | 0.27 (−23.8%) | 25.0% | 28.6% | |
| τ²-bench, Qwen-2.5-3B | Base | 0.200 | 1.32 | 0.240 | 1.25 | 9.4% | 0.3% |
| Token entropy | 0.204 | 1.38 (+3.9%) | 0.247 | 1.32 (+6.1%) | 9.4% | 0.3% | |
| Semantic entropy | 0.180 | 1.11 (−15.9%) | 0.239 | 1.00 (−20.0%) | 6.2% | 1.0% | |
| SFT | 0.289 | 0.95 (−28.0%) | 0.353 | 0.90 (−27.8%) | 15.6% | 5.4% | |
| RSD | 0.274 | 0.91 (−30.9%) | 0.332 | 0.92 (−26.1%) | 15.6% | 3.6% | |
| τ²-bench, Qwen-2.5-7B | Base | 0.249 | 0.78 | 0.309 | 0.74 | 11.2% | 0.6% |
| Token entropy | 0.304 | 0.75 (−3.7%) | 0.387 | 0.70 (−4.6%) | 15.6% | 1.2% | |
| Semantic entropy | 0.242 | 0.78 (+0.5%) | 0.296 | 0.74 (±0%) | 7.8% | 0.0% | |
| SFT | 0.366 | 0.73 (−6.5%) | 0.462 | 0.72 (−1.9%) | 25.0% | 7.1% | |
| RSD | 0.278 | 0.89 (+14.2%) | 0.349 | 0.85 (+15.1%) | 12.5% | 3.0% |
Table 2 – Hybrid ablation, ToolSandbox Qwen-2.5-7B (λ weights the RSD reward; 1−λ weights the progress delta) tab. 2
| Reward | Area under curve | CV of area | Final progress | CV of progress | pass@1 | pass^3 |
|---|---|---|---|---|---|---|
| RSD (λ = 1.0) | 0.555 | 0.28 | 0.562 | 0.27 | 25.0% | 28.6% |
| Hybrid (λ = 0.8) | 0.533 | 0.36 (+29.3%) | 0.541 | 0.33 (+21.1%) | 26.6% | 28.6% |
| Hybrid (λ = 0.5) | 0.532 | 0.37 (+33.6%) | 0.539 | 0.35 (+28.2%) | 29.7% | 21.4% |
| Hybrid (λ = 0.2) | 0.518 | 0.42 (+51.0%) | 0.536 | 0.35 (+27.0%) | 23.4% | 21.4% |
Table 3 – Direct RSD evaluation of fine-tuned models (lower = more consistent; 8 runs per sample, pairwise gpt-4.1 NLI) tab. 3 §7.3
| Model | ToolSandbox 7B RSD | Change | τ²-bench 7B RSD | Change |
|---|---|---|---|---|
| Base Qwen | 0.672 | — | 0.873 | — |
| Token entropy | 0.400 | −40.5% | 0.877 | +0.5% |
| Semantic entropy | 0.440 | −34.4% | 0.860 | −1.5% |
| SFT | 0.664 | −1.1% | 0.800 | −8.3% |
| Hybrid (λ = 0.8) | 0.387 | −42.5% | 0.867 | −0.7% |
| RSD | 0.317 | −52.8% | 0.875 | +0.2% |
Table 4 – Qwen-3.5 base vs RSD fine-tuning (appendix) tab. 4 app. A.4
| Setting | Method | Area under curve | CV of area | Final progress | CV of progress |
|---|---|---|---|---|---|
| τ²-bench, Qwen-3.5-4B | Base | 0.627 | 0.444 | 0.705 | 0.454 |
| RSD | 0.686 | 0.355 (−20.0%) | 0.768 | 0.359 (−20.9%) | |
| τ²-bench, Qwen-3.5-9B | Base | 0.607 | 0.480 | 0.686 | 0.483 |
| RSD | 0.650 | 0.428 (−10.8%) | 0.720 | 0.427 (−11.6%) | |
| ToolSandbox, Qwen-3.5-4B | Base | 0.405 | 0.742 | 0.412 | 0.738 |
| RSD | 0.447 | 0.662 (−10.8%) | 0.451 | 0.662 (−10.3%) | |
| ToolSandbox, Qwen-3.5-9B | Base | 0.488 | 0.758 | 0.503 | 0.720 |
| RSD | 0.507 | 0.738 (−2.6%) | 0.515 | 0.721 (+0.1%) |
NLI validation (appendix). On 12,000 τ²-bench airline action pairs, the three-class score and a 10-class score give a Spearman rank correlation of 0.97 for RSD app. A.5.1. A manual check of 30 pairs found 96.7% agreement with gpt-4.1 app. A.5.2. Table 5 gives gpt-4.1's label distribution by tool-call category tab. 5; Table 6 gives agreement with other judges on ToolSandbox pairs tab. 6.
Table 5 – gpt-4.1 labels for tool-call pairs tab. 5
| Pair type | Count | Entailment | Neutral | Contradiction |
|---|---|---|---|---|
| Same tool and identical arguments | 1,602 | 97.6% | 0.1% | 2.3% |
| Same tool, different arguments | 343 | 6.1% | 0.0% | 93.9% |
| Different tools | 695 | 0.0% | 0.0% | 100.0% |
Table 6 – Cross-model agreement with gpt-4.1 on ToolSandbox pairs tab. 6
| Judge | Agreement |
|---|---|
| Kimi-K2.5 | 95.0% |
| GPT-5.1 | 98.0% |
| GPT-4o | 97.5% |
| GPT-4o-mini | 97.0% |
| DeBERTa-v3-large | 41.6% |
Interpretation the authors give: token entropy raised CV in several cases, and semantic entropy gave only modest ToolSandbox gains §7.1. SFT lowers CV on τ²-bench but raises it on ToolSandbox 7B §7.1. On τ²-bench 7B, RSD raises CV, and the authors trace this to one sample where mean progress fell from 0.375 to 0.125 §7.1. In the hybrid ablation, AUC and progress fall monotonically as λ falls, and the CV rises tab. 2 §7.2. These are author-reported.
Paper vs code
Official repository SAP-samples/agent-consistency-rsd at commit
01f54e53aa7aeb6a3c4f64e20d9789ef4178ce22, Apache-2.0. The repository contains
the metric training (train.py), held-out inference (test.py), an embedding
client and a distance-analysis script. It has no GRPO fine-tuning code, no code
for AUC, CV or pass^k, and no code for the Table 3 pairwise evaluation or for the
Qwen-3.5 runs.
Differences between the paper and the code:
- NLI "neutral" is much narrower in code. The paper describes neutral as partial overlap or ambiguous equivalence §4.2 app. A.5.1. The judge prompt defines neutral only for a system error, a timeout, or a transfer to a human while the other output is normal train.py. Text against a tool call is a contradiction train.py, and so are different tool names or different arguments train.py.
- Angle term. The paper gives a normalised arccos §3.2. The default code uses an atan2 form train.py. For non-zero vectors the two are equal, but the code also sets the distance to 0 when both vectors are zero and to 1/2 when exactly one is zero train.py. The paper states neither convention.
- Non-default angle modes. The README lists two clipped-arccos angle modes [README]. These give a non-zero self-angle, so the self-distance is no longer the norm term alone train.py train.py. The paper's eq. 9 has no angle term eq. 9.
- Squared norm. The paper writes ‖φ‖² without specifying a reduction. The code's default is a sum of squares train.py. A mean option divides by the latent dimension and is not in the paper train.py train.py.
- Network. The paper says only "3-layer MLP to 256 dimensions" app. A.3.1. The code uses 1536 → 512 with LayerNorm and ReLU → 256 → ReLU → 256, with Xavier initialisation and zero biases train.py.
- Loss. The default is MSE, matching eq. 8 train.py. Huber is an option train.py.
- Terminal handling. The code stops at the first rollout end, with no padding and no bootstrap train.py, matching the appendix app. A.3.2.
- Reward without NLI. If NLI is disabled, the code uses a sigmoid of cosine distance as the reward train.py. The paper does not describe this path.
- Pair sampling. The code draws a task uniformly, then two distinct runs, then a depth uniformly from the depths both runs share train.py. The paper does not specify the sampling.
- Training split. The code holds out 20% of task ids using seed 42 [code: train.py#L1224-L1254]. The paper does not state a split.
- Embedding truncation. Training embeddings set
truncatetoSTART[code: train.py#L797]. Over-length histories may therefore lose their beginning. The paper does not mention truncation.
No discrepancies were found in the TD target structure, the bidirectional entailment rule train.py, the soft target update [code: train.py#L1031-L1038] or the inference score test.py, which matches eq. 9 under the default angle mode.
Pitfalls
- Reward is mostly binary. Because the code's neutral class is almost unused, most pairs score 0 or 1. The paper's own validation table agrees: 0.0% to 0.1% neutral on tool-call pairs tab. 5. A reimplementation that follows the paper's wording may get a different reward.
- Judge overlap. The reward judge (gpt-4.1) is also the judge in the Table 3 evaluation §7.3. The gain there may partly reflect the judge rather than behaviour. Table 6 shows other large LLM judges agree with gpt-4.1 at 95–98%, but a DeBERTa judge at 41.6%. The code offers DeBERTa and RoBERTa as options [README]; the paper's results do not use them.
- Table 3 metric is undefined in the active text. The text says pairwise NLI labels each turn, but the aggregation into one RSD number is only in a commented-out line, which defines it as 1 − entailment rate §7.3. Check the aggregation before comparing numbers.
- Reward wording is ambiguous. The text says the policy is trained to "minimize RSD self-distance −Û(s,s)" §6. The likely meaning is that the reward is −Û and GRPO maximises it, but the paper does not say. It also does not say whether the reward is per turn or per trajectory, or how it is normalised inside GRPO.
- Consistency is not correctness. The paper reports a case where RSD training reduced variance but made the mean worse, so CV rose §7.1. The paper's own limitations say RSD cannot tell consistent failure from consistent success §8. The authors also note that training rollouts come from a competent base policy, so degenerate policies are not in the training distribution §7.1.
- Small evaluation. Each benchmark has 8 test samples with 8 runs each §6. Differences of a few points in CV are within what 8 samples can resolve, and no confidence intervals are reported.
- Numbers that disagree. The text gives the ToolSandbox 7B semantic-entropy change in AUC CV as −6.0% §7.1, while Table 1 shows −6.2% tab. 1.
- Learned score need not be zero at deterministic states. The paper's zero-score result holds for the exact fixed point §4.4 §5. The learned network approximates it, and its error bound scales with 1/(1−γ) = 5 for γ = 0.8 app. A.2.3. Check the score on states known to be deterministic.
- Theorem numbers. The appendix resets the theorem counter, so the printed numbers may repeat. Cite the theorems by name app. A.2.
- Cost. Each labelled pair can take up to two gpt-4.1 calls, one per direction train.py. Each distinct state and output text needs one embedding call, cached by text train.py. The code labels all aligned pairs before training train.py, so cost grows with the number of pairs.
- Epoch is not a pass over pairs. The loader iterates over transitions in batches of 32, and each batch draws 32 new pairs train.py train.py. One epoch is therefore len(transitions)/32 steps.
- Inference is not free. The paper says inference needs one forward pass with no NLI or rollouts §4.3. It still needs one embedding call per state.
Lineage
- Builds on: bisimulation metrics for reinforcement learning (Ferns et al., 2004) §3.1, extended to interaction histories with NLI as the distance, and MICo (Castro et al., 2021) §3.2, used for its sampled, diffuse distance §4.4.
- Compared against: token-level entropy (Malinin & Gales, 2020) and semantic entropy (Kuhn et al., 2023) as GRPO rewards, and SFT §6 app. A.3.3. Consistency is also measured with the pass^k metric (Yao et al., 2024; Barres et al., 2025) §2 and the progress-curve framework of Chong et al. (2026) §6.
- Benchmarks: τ²-bench (Barres et al., 2025) and ToolSandbox (Lu et al., 2024) §6.
- Superseded by: none known from this wiki.
Source and license
Adapted from “Recursive Semantic Divergence for LLM Agent Consistency” by Harshavardhan Abichandani, Penny Chong, Atin Ghosh, Daniel Dahlmeier (arXiv:2610.33012), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/SAP-samples/agent-consistency-rsd at commit 01f54e53aa7a (Apache-2.0). Any quoted code is shown under that license.