Paper · 2026-11-09 · cs.LG, cs.AI
CARE-PO: Controllable Tool-Call Efficiency for LLM Agents
GRPO for tool-using agents. The reward multiplies delivered value by an efficiency factor priced against a reference budget fixed before training. One run gives low and high operating points. Author-reported GAIA, 27B low tier: success 51.3 to 57.4, calls 10.66 to 5.20.
LLM agentTool usePolicy gradientReward designreinforcement-learningllm-agenttool-usegrpoefficiency
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
An AI agent that searches the web or runs code often makes calls it does not need. Each call costs time and money, and its result stays in the agent's memory and is re-read on every later step, so an unneeded call slows every step after it. Training an agent to finish tasks does not penalise this, so the agent keeps calling.
CARE-PO trains the agent with a reward that cares about calls. Each task gets a fixed budget, estimated once before training: roughly how many calls a good solution needs. A call is priced against that budget, so the same extra call costs more on an easy task than on a hard one. Two details stop the agent from learning to stop calling altogether. Calls that bring no evidence are charged, and every training group is shown a known successful solution as a reference point that the agent should not fall below.
The efficiency price is set at inference. The training run teaches two behaviours, one for a low-effort tag in the prompt and one for a high-effort tag, so a user can pick cheaper or more thorough answers from the same model.
TL;DR
- Agents pay for every tool call, and the usual training objective does not price calls. Some methods add an efficiency reward, but its reference point moves with the policy, and training can collapse to making no calls §1 §3.2.
- Each training task gets a reference budget and an evidence checklist from one offline pass. The reward multiplies delivered value by an efficiency factor based on calls per budget, plus a term that charges missing evidence §3.3 §3.4 eq. 2 eq. 3.
- A successful source trajectory from the training data is added to every GRPO group as a clipped floor, and the effort tag in the prompt selects the price of a call §3.4 §3.5.
- Author-reported on GAIA-165, five of six operating points raise success rate and cut calls together. At 27B the low tier goes from 51.3 to 57.4 success and 10.66 to 5.20 calls tab. 1.
- The paper's code is not public: only a supplementary archive is mentioned. Every number here is from the paper itself §7.
Problem
Tool calls compound. Each observation stays in the context, so a low-value call adds cost to all later turns §1. When 221 GAIA runs from Qwen3.5-9B and 27B were reviewed step by step, 82.8% contained at least one call that could be deleted or merged into a neighbour, and 68.0% of all calls were removable without breaking the evidence chain §3.1 app. A.
The authors name three failure modes of efficiency rewards. The anchor is often the policy's own shortest correct rollout, so it moves as the policy changes. Training can drift to a policy that issues no calls, which the authors observed at 27B for OTC-PO §1 §6.3. A single training run yields one operating point, so the efficiency-capability trade-off cannot be chosen after training §1.
Assumptions: the method needs goals that a program can grade automatically, and every training task needs a successful earlier run to start from §7. Evaluation covers only short-answer question answering §7.
Method
CARE-PO keeps GRPO as the optimiser. The reward has two factors: a delivered value, which is 1 for a correct answer and small partial credit for a failure, and an efficiency factor that falls as the cost rises eq. 1 eq. 2. An effort tier e is an input to the policy, with three values: no tag, low and high §3.2. The objective is written per tier, and one conditional policy serves all three §3.2 eq. 1.
The diagnosis in §3.1 shows where the reward should act. Repeated calls and calls that bring no new information are priced by the call count. Evidence gaps belong to no single call, so they are priced by a coverage term §3.1 app. A.
Reference budget and evidence checklist
Preprocessing never involves the policy §3.3. From MiroVerse v0.1 (about 148k trajectories), the authors keep the successful runs that used five or more calls, a pool of about 64k. A judge model reduces each source to a minimal chain of essential steps. The chain's length is the candidate reference budget. The entities that the chain needs and that appear in tool observations form the evidence checklist E(x) §3.3. A second prompt sees the chain but not the estimate. If it fails, a corrected value is returned and checked again. A task whose budget must rise to the source trajectory's call count is discarded §3.3 app. D. The chain is never shown to the policy §3.3.
Three filters apply to the output: at least two entities must need retrieval, the budget must be at least the checklist length, and the source must make at least twice the budget in calls. 1825 candidates pass; 300 are sampled, balanced across budgets 3, 4 and 5 app. D. The authors state that the estimate need only be reachable in execution, not the true minimum §3.3.
Cost
Cost is the number of budgets a trajectory spends, plus a coverage term eq. 3. A repeated call counts twice, and missing evidence adds in proportion to its share of the checklist. A trajectory that makes no call costs exactly one budget §3.4 eq. 3.
Reward
The efficiency factor is 1 + \lambda \tanh(\gamma_e (1-u)), with \lambda = 0.5. It is bounded in [1-\lambda, 1+\lambda], falls as u rises, and is steepest at u = 1. The price factor \gamma_e is t at the low tier, 1 with no tag and 1/t at the high tier, with t = 1.5. Partial credit b_e scales inversely: b_{\text{high}} = 0.30, b_{\varnothing} = 0.20, b_{\text{low}} = 0.13. The bound b_e < (1-\lambda)/(1+\lambda) = 1/3 keeps every failure below every success §3.4 app. D.
Because the factor equals 1 at u = 1 for every tier, a correct trajectory that spends exactly one budget scores the same at all tiers. Tiers separate only away from u = 1 §3.4.
# Cost u in reference budgets [eq. 3]
def cost(calls, repeats, budget, coverage):
# Repeated calls are charged twice [eq. 3]
# Training counts exact repeats only [app. A]
charged = calls + repeats
# Missing evidence adds in proportion to the checklist [eq. 3]
return charged / budget + (1 - coverage)# Reward for one rollout, with lam = 0.5 and t = 1.5 [eq. 2] [app. D]
def rollout_reward(delivered, coverage, cost, tier):
# Price factor: t at low, 1 with no tag, 1/t at high [§3.4]
gamma = price_factor(tier)
# Partial credit: 0.13 low, 0.20 none, 0.30 high [§3.4] [app. D]
partial = partial_credit(tier)
# Failures earn partial credit scaled by evidence coverage [§3.4]
value = delivered + (1 - delivered) * partial * coverage
# Efficiency factor, bounded in [1 - lam, 1 + lam] [§3.4] [eq. 2]
efficiency = 1 + lam * tanh(gamma * (1 - cost))
return value * efficiencySource trajectory as a group floor
Each group of G rollouts gets the task's source trajectory as member G{+}1, scored at the no-tag tier. It is not sampled from the policy, so its ratio is set to one at the start, and a negative advantage for it is truncated to zero. The authors describe it as a floor, not a target. It usually scores below the group mean and then adds no gradient. When most rollouts fail, it pulls the policy back toward a successful path. When no rollout gathers evidence, it raises the group mean, so call-free failures get negative advantages §3.5 app. D.
# One GRPO group for one training task [§3.4] [§3.5] [app. D]
def group_advantages(task, policy):
rollouts = []
rewards = []
# G = 8 rollouts, each with a tier drawn 0.2 / 0.4 / 0.4 [app. D] [§3.4]
for _ in range(G):
tier = sample_tier([0.2, 0.4, 0.4])
rollout = sample(policy, task, tier)
rollouts.append(rollout)
rewards.append(score(rollout, tier))
# Fixed source trajectory as member G+1, no-tag tier [§3.5]
floor = score(task.source, "none")
rewards.append(floor)
# Group-relative advantage; normalisation not stated [§3.5]
advantages = group_relative(rewards)
# The source's advantage is clipped at zero [§3.5] [app. D]
advantages[-1] = max(advantages[-1], 0.0)
return rollouts, advantagesEffort tags at inference
The low tier appends an effort block to the user turn that tells the model to prefer fewer calls and stop once the evidence is enough. The high tier appends a complementary instruction that allows extra calls when they may raise confidence or recover missing evidence. The no-tag tier appends neither. Evaluation uses the same text app. D. The paper states that all methods share one system prompt that already asks for economical tool use app. D.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Base model | Qwen3.5 at 4B, 9B, 27B | §4 | 9B used for analyses and ablations |
| Optimiser | GRPO, no KL term | app. D | token-level loss normalisation |
| Rollouts per group G | 8, plus the source trajectory | app. D §3.5 | source is member G{+}1 |
| Training steps | 500 | app. D | |
| Learning rate | 6\times10^{-6} (4B, 9B); 1\times10^{-5} (27B) | app. D | one recipe, rate per size |
| LoRA | rank 8, attention and MLP projections | app. D | alpha, dropout: not stated |
| Call cap T_{\max} | 20 | §4 app. D | exceeds the longest source trajectory |
| Lambda \lambda | 0.5 | §3.4 app. D | |
| Price factor t | 1.5 | §3.4 app. D | \gamma = t, 1, 1/t |
| Partial credit b_e | high 0.30, none 0.20, low 0.13 | §3.4 app. D | low value derived from the ratio; paper gives 0.13 |
| Effort tier sampling | none 0.2, low 0.4, high 0.4 | §3.4 | per rollout |
| MiroVerse pool | about 148k trajectories; about 64k candidates | §3.3 | at least 5 calls, successful |
| Filter checks | \|E\| \ge 2; B \ge \|E\|; S_{\text{long}} \ge 2B | app. D | 1825 pass |
| Training tasks | 300, budgets 3, 4, 5 about equal | app. D | shared by all sizes |
| Listed tasks | source calls 7–12 (mean 10.5); budgets 3–5 (mean 4.0) | app. D | checklist length 2–3 |
| Evaluation decoding | greedy | app. D | |
| Diagnosis judge | Qwen3.8-27B | app. A | not the diagnosed models |
| Offline-pass judge | "a judge model" | §3.3 | model not named |
| Coverage judge | "a large model" | app. D | model not named |
| Rollout sampling temperature | not stated | ||
| PPO-style clip \varepsilon | not stated | "clipped surrogate" only | |
| Advantage normalisation | not stated | ||
| Optimiser type, betas, schedule, warm-up, batch size | not stated | ||
| Seeds | not stated | three runs on GAIA, one on multi-hop |
Results
Author-reported. Setup: GAIA's 165-question validation split with four tools: search, fetching a page, running Python, and reading a local file. The held-out set adds 500 questions each from 2Wiki, HotpotQA and MuSiQue, plus the 125 Bamboogle questions. GAIA numbers are mean ± sample s.d. over three runs, and the multi-hop numbers are one deterministic run §4 tab. 1 tab. 2. Hypervolume is \mathrm{HV} = \mathrm{SR} \cdot (1 - \mathrm{AC}/T_{\max}) §4. Because HV fixes an exchange rate between the two, the authors judge GAIA on SR and AC individually §4 app. D.
GAIA-165, main table (tab. 1; best per size in bold in the paper):
| Model | Method | SR ↑ | AC ↓ | HV ↑ |
|---|---|---|---|---|
| Qwen3.5-9B | Vanilla ReAct | 40.2 ± 0.7 | 8.26 ± 0.14 | 0.236 ± 0.005 |
| Qwen3.5-9B | Budget-Prompt | 35.4 ± 0.3 | 8.50 ± 0.05 | 0.203 ± 0.001 |
| Qwen3.5-9B | DEPO | 34.7 ± 0.9 | 7.54 ± 0.05 | 0.217 ± 0.005 |
| Qwen3.5-9B | OTC-PO | 40.6 ± 1.2 | 5.01 ± 0.13 | 0.304 ± 0.012 |
| Qwen3.5-9B | CARE-PO (low) | 39.8 ± 2.3 | 3.42 ± 0.30 | 0.330 ± 0.015 |
| Qwen3.5-9B | CARE-PO (high) | 43.8 ± 0.3 | 4.44 ± 0.15 | 0.341 ± 0.002 |
| Qwen3.5-27B | Vanilla ReAct | 51.3 ± 0.3 | 10.66 ± 0.06 | 0.240 ± 0.002 |
| Qwen3.5-27B | OTC-PO (earlier checkpoint) | 52.1 ± 2.4 | 5.57 ± 0.21 | 0.376 ± 0.014 |
| Qwen3.5-27B | CARE-PO (low) | 57.4 ± 1.5 | 5.20 ± 0.07 | 0.425 ± 0.013 |
| Qwen3.5-27B | CARE-PO (high) | 54.7 ± 0.3 | 6.12 ± 0.22 | 0.380 ± 0.006 |
| Qwen3.5-4B | Vanilla ReAct | 27.5 ± 0.9 | 8.27 ± 0.11 | 0.161 ± 0.005 |
| Qwen3.5-4B | OTC-PO | 28.1 ± 1.3 | 4.14 ± 0.29 | 0.223 ± 0.013 |
| Qwen3.5-4B | CARE-PO (low) | 31.9 ± 0.9 | 5.01 ± 0.16 | 0.239 ± 0.008 |
| Qwen3.5-4B | CARE-PO (high) | 31.9 ± 0.9 | 7.60 ± 0.41 | 0.198 ± 0.011 |
Against Vanilla ReAct at the same size, five of six CARE-PO points raise SR by 3.4 to 6.1 points and cut AC by 8.1% to 51.2%. The exception is the 9B low tier, at −0.4 SR (not significant, p = 0.89) and −58.6% AC §5 tab. 1. The authors report p = 0.01 for the largest gain. At 9B, the two baselines Budget-Prompt and DEPO score lower than the base model on SR, while OTC-PO is level with it.
Held-out multi-hop (tab. 2; single deterministic run per cell):
| Model | Method | 2Wiki SR | AC | HV | HotpotQA SR | AC | HV | MuSiQue SR | AC | HV | Bamboogle SR | AC | HV |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 9B | Vanilla ReAct | 75.8 | 5.64 | 0.544 | 48.6 | 4.87 | 0.368 | 29.0 | 6.59 | 0.194 | 76.0 | 4.05 | 0.606 |
| 9B | OTC-PO | 70.6 | 2.24 | 0.627 | 48.0 | 1.59 | 0.442 | 25.6 | 2.23 | 0.227 | 68.8 | 1.38 | 0.640 |
| 9B | CARE-PO (low) | 63.2 | 1.94 | 0.571 | 41.4 | 1.09 | 0.391 | 21.4 | 1.55 | 0.197 | 43.2 | 0.50 | 0.421 |
| 9B | CARE-PO (high) | 70.4 | 2.19 | 0.627 | 46.2 | 1.51 | 0.427 | 25.2 | 2.10 | 0.226 | 60.8 | 1.20 | 0.572 |
| 27B | Vanilla ReAct | 80.2 | 6.78 | 0.530 | 55.8 | 5.97 | 0.391 | 34.6 | 8.07 | 0.206 | 74.4 | 2.46 | 0.653 |
| 27B | OTC-PO | 66.8 | 2.11 | 0.598 | 38.8 | 0.98 | 0.369 | 29.2 | 2.78 | 0.251 | 45.6 | 0.42 | 0.446 |
| 27B | CARE-PO (low) | 81.4 | 2.44 | 0.715 | 53.6 | 1.77 | 0.488 | 35.2 | 3.20 | 0.296 | 72.0 | 0.86 | 0.689 |
| 27B | CARE-PO (high) | 81.2 | 2.75 | 0.700 | 54.2 | 2.35 | 0.478 | 34.8 | 4.00 | 0.278 | 75.2 | 1.70 | 0.688 |
| 4B | Vanilla ReAct | 66.0 | 3.09 | 0.558 | 45.2 | 2.89 | 0.387 | 21.4 | 6.22 | 0.147 | 47.2 | 2.19 | 0.420 |
| 4B | OTC-PO | 52.0 | 1.51 | 0.481 | 30.6 | 0.77 | 0.294 | 15.6 | 1.11 | 0.147 | 33.6 | 0.47 | 0.328 |
| 4B | CARE-PO (low) | 66.8 | 1.99 | 0.601 | 44.8 | 1.25 | 0.420 | 19.8 | 2.74 | 0.171 | 48.8 | 0.89 | 0.466 |
| 4B | CARE-PO (high) | 75.4 | 2.80 | 0.649 | 48.8 | 2.78 | 0.420 | 23.2 | 5.97 | 0.163 | 69.6 | 2.12 | 0.622 |
On these benchmarks the effect on success depends on size. At 27B, the two points are not significantly different from the base model. At 4B, neither is significantly worse, and the high tier is significantly better on three of the four benchmarks. At 9B, both points score lower than the base model on every benchmark. The authors partly blame the fact that most of these questions need only two hops, which leaves little slack to remove §5 tab. 2. The text says the 9B low tier calls 23% as often as the base model, against 35% at 27B and 48% at 4B. That 23% matches the mean of the four per-benchmark ratios in tab. 2, not the pooled ratio (about 24%, a compiler calculation).
Ablation, 9B (tab. 3):
| Configuration | SR ↑ | AC ↓ | HV ↑ | Note |
|---|---|---|---|---|
| Flat price (t=1), low tag | 39.2 | 4.07 | 0.312 | one price for all tiers |
| Flat price (t=1), high tag | 40.6 | 4.91 | 0.306 | same price, other tag |
| Trained model, untagged prompt | 39.8 | 4.29 | 0.313 | untagged tier, trained |
| Base model, tag only, low | 37.0 | 7.01 | 0.240 | no training |
| Base model, tag only, high | 40.6 | 7.77 | 0.248 | no training |
| CARE-PO (low) | 39.8 | 3.42 | 0.330 | efficiency operating point |
| CARE-PO (high) | 43.8 | 4.44 | 0.341 | capability operating point |
Tags on the untrained base separate call counts but barely change HV. The trained low tier removes 4.84 calls for 0.4 SR points, while prompting the base model with the low tag removes 1.25 calls for 3.2 points §6.2 tab. 3. Dropping the coverage term from eq. 3 leaves SR at 39.6 (against 39.8) and raises AC from 3.42 to 4.00 §6.2.
Token cost, 9B (tab. 4; averages over three runs of tab. 1):
| Method | AC | Turns | Generated | Observations | Context processed |
|---|---|---|---|---|---|
| Vanilla ReAct | 8.26 | 9.1 | 1.91k | 13.1k | 104.4k |
| OTC-PO | 5.01 | 6.0 | 1.41k | 8.90k | 53.2k |
| CARE-PO (low) | 3.42 | 4.4 | 1.04k | 6.54k | 34.6k |
| CARE-PO (high) | 4.44 | 5.4 | 1.24k | 8.11k | 45.3k |
Relative to the base model, the low tier generates 45% fewer tokens and processes 67% less context. The high tier reduces these by 35% and 57% app. B tab. 4.
Source of the reduction §6.1: trajectories that exhaust the call cap fall from 15.6% to 1.4% (low) and 3.4% (high). Over half the call reduction comes from these trajectories, and almost all of them were wrong for the base model. The authors say the reduction is not abandonment: on 37 questions answered correctly in every run by both models, calls still fall by 57.5%.
Training collapse §6.3 fig. 3 app. C fig. 6: at 27B, OTC-PO's average calls drop to 0.01 per rollout once training passes step 300. CARE-PO's group-mean call count stays well above zero at 4B, 9B and 27B. The share of call-free rollouts over the last fifth of training is 2.22% (4B), 1.42% (9B) and 0.28% (27B) app. C. A 1000-step 27B run is described in the text only, with no table app. C.
Paper vs code
No official code. The paper has no code link. Its only mention of code is a supplementary archive, which is not public (Reproducibility Statement). I searched the LaTeX source for URLs (the only ones are dataset, Qwen and LoRA references). I then ran GitHub repository searches for "CARE-PO", "carepo", the expansion "Controllable Anchored Relative Efficiency", and "tool-call efficiency GRPO GAIA". The matches were unrelated, and none belong to the authors. The arXiv JSON has no comments field, so I could not check arXiv comments.
Pitfalls
- Reference budget is fixed before training and never updated. The estimate is made by an unnamed judge model and checked by a second prompt. The paper argues that estimation error changes the size of the pressure but not its direction §3.3 app. D. Implementers still need a judge and a prompt the paper does not give.
- "Repeat" is ambiguous. The training reward counts only exact matches on tool name and normalised arguments. The diagnosis uses near-duplicate thresholds (argument Jaccard at least 0.8 and return Jaccard at least 0.9). Use exact matching for the reward app. A eq. 3.
- Coverage judging is under-specified. A large model, not named, checks each checklist entity against the tool observations, allowing "an equivalent rendering" of it. A plain string match will differ app. D.
- Partial credit has a hard ceiling. Partial credit must stay below 1/3 (with \lambda = 0.5), or a failure could outscore a success. The paper's b_{\text{low}} = 0.13 is a rounded value derived from the ratio, not a separate setting §3.4 app. D.
- The tiers only separate away from u=1. At exactly one budget every tier gives factor 1. An implementation that tests tier differences on one-budget trajectories will see none §3.4.
- The source trajectory is off-policy and clipped. Its importance ratio starts at 1 using the current policy's log-probabilities as old ones. Its advantage is clipped at zero. Whether it enters the group mean used for normalisation is implied by "raises the group mean" but not stated §3.5 app. D.
- Normalisation and clipping are not fully specified. The paper does not state whether advantages are normalised by the group standard deviation, what the clip ε is, or the rollout temperature. Use the GRPO defaults of your framework and note the choice.
- Tiers are resampled per rollout. Training mixes all three tiers in every group, with probabilities 0.2, 0.4 and 0.4. The no-tag tier also contributes to the policy update §3.4.
- The OTC-PO 27B checkpoint is not identified. The paper evaluates an unnamed earlier checkpoint from before the collapse. Appendix D says checkpoint selection is described in Appendix C, but Appendix C gives no selection rule §5 app. C app. D.
- Single runs on the held-out benchmarks. Those results are one deterministic run. Search is live and non-stationary, so GAIA runs vary and the multi-hop differences have no error bars §4 app. D.
- HV depends on T_{\max}. At T_{\max} = 20, one saved call equals about 2.4 SR points. Compare SR and AC separately unless you use the same cap app. D.
- Calls are not tokens. Token savings are reported on 9B only, and the paper counts tokens with the Qwen3.5 tokenizer over stored traces app. B.
- Judge models are a hidden dependency. The diagnosis uses Qwen3.8-27B. The offline-pass and coverage judges are not named. Results could move with a different judge app. A app. D.
- The effort tags are text in the user turn. Use the same system prompt across baselines, as the paper does, or the comparison is unfair app. D.
- Shared decontamination. The paper reports exact and 8-gram matching against the evaluation sets, and no training question matched §3.3.
Lineage
- Builds on: GRPO, arXiv:2402.03300, with no KL term. The source-trajectory floor responds to advantage collapse in group relative optimisation (avspo2026, ICML 2026, no arXiv id in the source) §3.5.
- Compared against: Budget-Prompt, an inference-only call budget in the prompt (bats2025, chen2026steering, fang2026budget); DEPO, an efficiency reward added to offline preference alignment (AAAI 2025 paper, no arXiv id in the source); OTC-PO, arXiv:2504.14870, the online-RL case closest to this method §4.
- Superseded by: not known.
Source and license
Adapted from “CARE-PO: Controllable Tool-Call Efficiency for LLM Agents” by Xiaoyang Ye, Xiao Yao, Hairui Zhu, Yiying Yang, Bei Gan, Jinliang Lu, Tengjin Weng, Yuan Yao, Wenhao Jiang (arXiv:2611.12125), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.