Paper · 2026-10-28 · cs.AI
Think Before You Act: Decision Gate for LLM Agents
JevAct checks an LLM agent's write calls and hand-offs before they run. Code checks tool-state facts, a prefill-only decision model (Kev-27B) answers yes/no questions with probabilities, and a failed check returns feedback for a bounded revision. Author-reported: all eight Telecom agents improve pass^3 (average +16.9 points), BFCL gains are consistent, Airline is mixed.
LLM agentTool useLLM-as-a-judgeagentstool-useverificationdecision-modelguardrails
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
An agent that uses tools can change real records: it books a flight, cancels a reservation, or changes a setting. Most agent loops run such an action as soon as the model writes it down. This paper adds a checkpoint in between. Before a write goes through, or before the agent hands the customer off to a human, a separate check looks at the proposed action.
The check has two parts. Some facts can be read straight from the tool results, such as dates, amounts and counts, so ordinary code handles those. Other questions depend on what was said in the conversation, such as whether the customer actually asked for the change. A small decision model answers those with a probability, and the check only acts on confident answers. If the check fails, the agent gets a short note explaining what is wrong and gets one more try, within a fixed limit. The agent still decides what to do; the checker only says whether a proposed step is allowed.
The idea is useful because the checker does not depend on the agent's model or tools. The paper's results are strongest on a telecom troubleshooting benchmark and on a function-calling benchmark, and mixed on an airline one.
TL;DR
JevAct is a runtime gate for tool-using LLM agents. It runs before selected actions: write calls and hand-offs. It compiles a domain policy offline into yes/no questions, Boolean rules, Python code over tool records, and a hand-off checklist §3.3. At runtime, code computes what the records establish, and the prefill-only Kev-27B judge answers the remaining questions §3.4§3.5. Failed checks return feedback and allow a bounded revision §3.2. Author-reported gains are consistent on Telecom and BFCL and mixed on Airline §5.
Problem
Agents often run a proposed tool call immediately, so a wrong write or a premature hand-off can be hard to undo §1. Self-Refine and Reflexion show that feedback helps, but they do not check a single action against a domain policy just before it runs §1§2.1. Using a generative LLM as a judge is costly, and free-text verdicts must be parsed before a gate can use them §1. The paper asks whether a fixed-choice decision model with probability thresholds can serve as a verifier next to executable checks §1. The authors note that decision-model probabilities are not guaranteed to be calibrated, so the value has to be tested inside the agent loop §1.
Method
JevAct wraps a ReAct agent. Everything about the agent (model, base prompt, tools) is left alone. Only the step between proposal and execution changes §3. Four components work together (see fig. 2):
- Decision loop. Each step runs propose, trigger, check, then commit or revise §3.2. Only write calls and hand-offs are checked; other messages go through as in ReAct §3.2.
- Compiler (offline). For each domain, a model converts the policy into questions, rules, Python code and, for hand-offs, a checklist §3.3. A single repair pass shows the compiler logged calls that carry no outcome labels §3.3.
- Judge (online). Kev-27B gives a probability for every answer option from a single forward pass and generates no text. Questions about the same prompt reuse that pass §3.4.
- Checks. The write gate and the hand-off check, plus two generic checks for BFCL §3.5.
Decision loop §3.2
- Propose: the LLM drafts a message from the history.
- Trigger: a write call starts the write gate; a hand-off starts the hand-off check; all other messages skip checking.
- Check: the result is either pass or flag, and a flag carries feedback text.
- Commit or revise: a passing draft runs and enters the history. A flagged draft is dropped before execution and kept out of the history; the LLM then drafts again with the feedback attached to that single call §3.2.
The write gate may ask for up to two revisions and the hand-off check for one. When the budget is spent, the next proposal is committed without a check. A check therefore adds at most R extra LLM calls §3.2.
# One proposal in the decision loop [§3.2]
# Input: history, feedback (shown for this call only), budget per check
# Output: the message to commit
def decide(history, feedback, budget):
# 1. Propose: the agent LLM drafts a message [§3.2]
message = llm(history, feedback)
# 2. Trigger: pick the check for a write call or a hand-off [§3.2]
if is_write_call(message):
check = write_gate
elif is_handoff(message):
check = handoff_check
else:
return message # committed as in ReAct [§3.2]
if budget[check.name] == 0:
return message # budget spent: commit unchecked [§3.2]
# 3. Check: returns pass or a flag with feedback [§3.2] [§3.5]
flagged, new_feedback = check(message, history)
if not flagged:
return message # passed: execute and store [§3.2]
# 4. Revise: discard the draft; feedback is not kept in history [§3.2]
budget[check.name] -= 1
return decide(history, new_feedback, budget)Compiler §3.3 app. A.1
For each write tool, the compiler produces:
- questions about policy requirements, argument provenance, user intent and prerequisite steps or confirmations, each answered yes or no;
- a Boolean rule over the answers that says when the call is allowed;
- a Python function that computes policy facts from tool records (dates, amounts, counts) and returns the violations those records establish.
For hand-offs, it produces the problem types the policy covers and, for each problem, the steps to try before giving up. Every step carries a settled question that asks whether it was attempted, confirmed or excluded. The compile prompt includes a rule that a condition never discussed in the conversation is not treated as ruled out §3.3. The paper's example is "enable roaming", which otherwise compiles into a step that looks settled because "the user is not traveling" §3.3.
Two filters remove weak items §3.3 app. A.1:
- Items that fire on over 10 % of the calls logged for their tool are removed. A removed question is read as "yes" by the rule.
- A checklist step that fires on more than 5 % of hand-offs from solved training episodes is removed.
Judge §3.4
Kev is an open-source, Jev-like decision model. It is used as released, with no further training. Its base model is a fine-tune of Qwen3.8-27B §3.4. The judge gets the policy, the conversation, the tool results (dates marked, list values ranked), and the proposed call. Because the judge is a different model from the agent, the agent never grades its own draft §3.4. A check's cost stays flat as questions are added §3.4.
Checks §3.5
Write gate runs three steps:
- Code: the compiled function reads the call's arguments and earlier results and returns the violations they establish.
- Judge: Kev answers each question for that tool. An answer counts as no only when P(yes) ≤ 0.05; otherwise it counts as yes.
- Rule: flag when the code reports a violation, or when the Boolean rule evaluates to false on the answers.
The feedback names the violations and the failing questions, and tells the agent to fix the call, request missing details from the user, or say why the request cannot be met. If the agent is confident the call is right, it can resubmit it. Once the call has been flagged twice, the feedback redirects the agent to reply to the user instead §3.5.
Hand-off check applies to a transfer call and to a message in which the agent gives up. A keyword screen first flags candidate give-up messages, and the judge then confirms them §3.5. Kev then reports, from the conversation, which problem the user has, whether the user says it is resolved, and whether each checklist step is settled §3.5. A step is open when P(problem requires the step) ≥ 0.5 and P(settled) ≤ 0.5. When the problem is unresolved and some step is open (still pending), the check flags the hand-off and sends the open steps back as feedback §3.5.
# Write gate for one proposed write call [§3.5]
# Input: call (tool, args), tool_results, history
# Output: (flagged, feedback)
def write_gate(call, history, tool_results):
# 1. Code: facts that the records establish [§3.5] [app. A.1]
violations = derive[call.tool](call.args, tool_results)
# 2. Judge: one prefill pass covers every question [§3.4] [§3.5]
answers = {}
for question in questions[call.tool]:
p_yes = judge(history, tool_results, call, question)
# a "no" only when P(yes) <= 0.05, else counts as "yes" [§3.5]
answers[question.id] = p_yes > 0.05
# 3. Rule: Boolean expression over the answers [§3.5] [app. A.1]
rule_ok = eval(rules[call.tool], {"__builtins__": {}}, answers)
flagged = bool(violations) or not rule_ok
feedback = build_feedback(violations, answers)
return flagged, feedbackGeneric checks for BFCL (no written policy, fixed user turns) §3.5:
- End-of-turn check: fires when the agent ends its turn with an unresolved tool error that states a requirement (wording such as "needs to" or "must"). The judge picks the tool that meets the requirement, or none. If P ≥ 0.6, the feedback tells the agent to call that tool rather than ask the user §3.5.
- Requested-action gate: checked before every call that changes state. It asks whether the user asked for the call or a tool error demands it. If P(yes) ≤ 0.05, the call is flagged as unrequested §3.5.
# Requested-action gate for BFCL [§3.5]
# Input: call, conversation, tool errors
# Output: (flagged, feedback)
def requested_action_gate(call, conversation, tool_errors):
# Judge: did the user ask for this, or does an error require it? [§3.5]
p_yes = judge_yes(conversation, tool_errors, call)
if p_yes <= 0.05:
# confident no: do only what was asked or required [§3.5]
return True, "unrequested: do only what was asked or required"
return False, ""Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Judge model | Kev-27B, fine-tune of Qwen3.8-27B, no further training | §3.4 §4.3 | Kev-9B and Kev-0.8B also tested §4.3 §5.1 |
| Write-gate "no" threshold | P(yes) ≤ 0.05 | §3.5 §4.3 | set with Kev-27B §4.3 |
| Write-gate revision budget R | 2 | §3.2 §4.3 | |
| Hand-off revision budget R | 1 | §3.2 §4.3 | active on Telecom only §4.3 §4.5 |
| Hand-off open step | P(needed) ≥ 0.5 and P(settled) ≤ 0.5 | §3.5 app. A.1 | |
| End-of-turn threshold | P ≥ 0.6 | §3.5 §4.3 | BFCL |
| Requested-action threshold | P(yes) ≤ 0.05 | §3.5 §4.3 | BFCL |
| Drop rule, write questions | fires on > 10 % of a tool's logged calls | §3.3 app. A.1 | thresholds set once on Airline, reused app. A.1 |
| Drop rule, hand-off steps | flagged on > 5 % of solved hand-offs | §3.3 app. A.1 | 7 Telecom steps dropped app. A.1 |
| Compiler model | Claude Opus 4.6 | §4.3 app. A.1 | single-turn calls, client may retry transport errors app. A.1 |
| Compiler output limit | 16,384 tokens (steps 1, 3, 4, checklist); 8,192 (rule); 2,048 (Telecom prerequisites) | app. A.1 | |
| Agent temperature | 0 | §4.2 | |
| Agent max output tokens | 4,096 | §4.2 | |
| User simulator, τ²-bench | GPT-4.1, temperature 0 | §4.1 §4.2 | gpt-oss-120b used in a separate check §5.1 §6 |
| Runs per task | 3 | §4.4 | pass@3 and pass^3 |
| Airline tasks | 50 | §4.1 | |
| Telecom tasks | 114 (base split) | §4.1 | 40 test-split tasks are mentioned in app. A.1 |
| BFCL multi-turn entries | 800 (200 per category: base, missing function, missing parameter, long context) | §4.1 | up to 20 steps per user turn §4.1 |
| Judge context validation | Kev-0.8B and Kev-9B: 8,192 tokens; Kev-27B: 65,536 tokens | app. A.3 | |
| Judge cost assumption | USD 2 per GPU-hour, full utilisation, A100-80GB bf16, 4 concurrent requests | app. A.6 | estimates only app. A.6 |
| API prices used | GPT-4.1 at USD 2 input / 8 output; Haiku 4.5 at USD 1 input / 5 output, per million tokens | app. A.6 | standard tier, 14 Oct 2026 app. A.6 |
| Compiled items, Airline | 6 write tools, 86 questions (17 dropped) | tab. 1 | |
| Compiled items, Telecom | 6 write tools, 75 questions (5 dropped); 8 problem types; checklist with 7 dropped steps | tab. 1 |
Results
Author-reported. Setup: τ²-bench Airline (50 tasks) and Telecom (114 tasks), BFCL multi-turn (800 entries), three runs per task, each JevAct arm compared with a ReAct arm using the same agent LLM §4. Definitions §4.4:
\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}, \qquad \text{pass}^{k} = \frac{\binom{c}{k}}{\binom{n}{k}}
with c of n runs solved, averaged over tasks §4.4. Setting n = k = 3, pass@3 gives a task credit if one of its three runs succeeds, and pass^3 only if all three do §4.4. Source for the table below: tab. 2.
| Benchmark | Model | pass@3 ReAct | pass@3 JevAct | pass^3 ReAct | pass^3 JevAct |
|---|---|---|---|---|---|
| τ² Airline | GPT-4.1 | 66.0 | 82.0 | 36.0 | 46.0 |
| τ² Airline | Opus 4.6 | 84.0 | 92.0 | 72.0 | 76.0 |
| τ² Airline | gpt-oss-20b | 66.0 | 70.0 | 48.0 | 44.0 |
| τ² Airline | gpt-oss-120b | 68.0 | 66.0 | 42.0 | 50.0 |
| τ² Airline | Qwen3.8-27B | 90.0 | 90.0 | 74.0 | 76.0 |
| τ² Airline | DeepSeek-V4-Pro | 86.0 | 90.0 | 68.0 | 66.0 |
| τ² Telecom | GPT-4.1 | 45.6 | 83.3 | 11.4 | 17.5 |
| τ² Telecom | Haiku 4.5 | 47.4 | 69.3 | 19.3 | 31.6 |
| τ² Telecom | Opus 4.6 | 72.8 | 100.0 | 39.5 | 80.7 |
| τ² Telecom | Opus 5.5 | 99.1 | 100.0 | 74.6 | 82.5 |
| τ² Telecom | gpt-oss-20b | 83.3 | 92.1 | 28.1 | 34.2 |
| τ² Telecom | gpt-oss-120b | 78.9 | 93.9 | 28.9 | 51.8 |
| τ² Telecom | Qwen3.8-27B | 91.2 | 98.2 | 57.9 | 81.6 |
| τ² Telecom | DeepSeek-V4-Pro | 86.8 | 96.5 | 53.5 | 68.4 |
| BFCL | GPT-4.1 | 54.0 | 60.2 | 38.2 | 47.8 |
| BFCL | Opus 4.6 | 63.2 | 66.5 | 55.9 | 59.1 |
| BFCL | gpt-oss-20b | 54.2 | 53.6 | 33.6 | 33.8 |
| BFCL | gpt-oss-120b | 57.8 | 54.2 | 39.9 | 47.6 |
| BFCL | Qwen3.8-27B | 65.9 | 64.9 | 48.1 | 49.9 |
| BFCL | DeepSeek-V4-Pro | 57.1 | 54.9 | 41.4 | 45.1 |
| BFCL | Llama 3.3 70B | 12.2 | 12.0 | 10.5 | 11.5 |
| BFCL | Mistral Small 3.2 24B | 26.0 | 27.9 | 15.5 | 17.1 |
| BFCL | Gemma 4 31B | 62.5 | 63.2 | 61.5 | 61.6 |
Bold marks the higher value of each pair, as in the source table tab. 2. Haiku 4.5 and Opus 5.5 were run on Telecom only; Llama 3.3 70B, Mistral Small 3.2 24B and Gemma 4 31B on BFCL only §4.2.
Headline numbers, as the paper summarises them §5:
- Telecom: every one of the eight agents improves on both pass@3 and pass^3. The average pass@3 rise is 16.0 points and the average pass^3 rise is 16.9 points.
- Airline: four of six agents gain on pass@3; one is flat (Qwen3.8-27B) and one drops (gpt-oss-120b). On pass^3, four gain and two drop.
- BFCL: every one of nine agents gains on pass^3 (0.1 to 9.6 points). On pass@3, four gain and five lose (0.2 to 3.6 points).
Write-gate ablation §5.1 tab. 3
Single trial, Airline, GPT-4.1 as agent (seed and trial details: §5.1). Source: tab. 3. The "flags per task" column counts gate flags per task, not per write call.
| Configuration | pass^1 (%) | Flags per task |
|---|---|---|
| ReAct | 50.0 | -- |
| JevAct (same configuration as Table 2) | 72.0 | 0.38 |
| Code checks only | 68.0 | 0.28 |
| Kev-27B questions only | 57.1 | 0.29 |
| Kev-9B instead of Kev-27B | 58.0 | 0.22 |
| One revision instead of two | 68.0 | 0.32 |
| Placebo: every write call flagged once | 50.0 | 1.00 |
Judge choice §5.1 tab. 4
The same write gate with different judges. Every LLM judge is paired with a dropped-question list derived without labels. ReAct is run separately per block; the GPT-4.1 ReAct baseline is 53.3 pass^1 and the Haiku 4.5 baseline is 55.3 pass^1 tab. 4. Cost is estimated per task app. A.6.
| Judge (params) | Type (size) | GPT-4.1 pass^1 | GPT-4.1 pass^3 | GPT-4.1 USD/task | Haiku pass^1 | Haiku pass^3 | Haiku USD/task |
|---|---|---|---|---|---|---|---|
| none (ReAct) | -- | 53.3 | 42.0 | -- | 55.3 | 36.0 | -- |
| Qwen3.8-27B (Kev-27B's base) | open LLM (27B) | 68.0 | 50.0 | 0.0027 | 64.7 | 48.0 | 0.0020 |
| Gemma-4-31B | open LLM (31B) | 72.0 | 56.0 | 0.0040 | 64.7 | 52.0 | 0.0037 |
| Claude Haiku 4.5 | API LLM | 72.0 | 56.0 | 0.0156 | 65.3 | 50.0 | 0.0127 |
| GPT-4.1 | API LLM | 67.3 | 52.0 | 0.0231 | 66.0 | 56.0 | 0.0195 |
| Kev-0.8B | decision model (0.8B) | 66.0 | 50.0 | 0.0003 | 68.7 | 58.0 | 0.0003 |
| Kev-9B | decision model (9B) | 63.3 | 56.0 | 0.0008 | 68.0 | 58.0 | 0.0006 |
| Kev-27B (JevAct) | decision model (27B) | 72.0 | 56.0 | 0.0017 | 68.0 | 58.0 | 0.0015 |
Judge size §5.1 tab. 5
Same compiled checks and thresholds in every arm (set with Kev-27B) tab. 5. Airline and Telecom; pass^1 and pass^3 in percent.
| Benchmark | Model | pass^1 baseline | 0.8B judge | 9B judge | 27B judge | pass^3 baseline | 0.8B judge | 9B judge | 27B judge |
|---|---|---|---|---|---|---|---|---|---|
| τ² Airline | GPT-4.1 | 52.0 | 58.0 | 59.3 | 65.3 | 36.0 | 40.0 | 40.0 | 46.0 |
| τ² Airline | Opus 4.6 | 79.3 | 86.0 | 83.3 | 84.0 | 72.0 | 82.0 | 76.0 | 76.0 |
| τ² Airline | gpt-oss-120b | 56.7 | 62.0 | 58.7 | 58.0 | 42.0 | 44.0 | 46.0 | 50.0 |
| τ² Airline | DeepSeek-V4-Pro | 77.3 | 79.3 | 80.7 | 80.7 | 68.0 | 70.0 | 70.0 | 66.0 |
| τ² Telecom | GPT-4.1 | 25.7 | 54.4 | 52.3 | 49.7 | 11.4 | 27.2 | 23.7 | 17.5 |
| τ² Telecom | Opus 4.6 | 59.4 | 96.5 | 99.1 | 93.6 | 39.5 | 90.4 | 97.4 | 80.7 |
| τ² Telecom | gpt-oss-120b | 54.4 | 77.8 | 77.8 | 75.1 | 28.9 | 50.9 | 53.5 | 51.8 |
| τ² Telecom | DeepSeek-V4-Pro | 70.5 | 95.0 | 98.2 | 85.7 | 53.5 | 86.0 | 94.7 | 68.4 |
Token use and proposals sent back app. A.5 tab. 9
Mean per run, in thousands of tokens, from the runs in Table 2 tab. 9. Drafts sent back are counts per run. "Not rebuilt" counts Kev requests that could not be rebuilt from logs (BFCL only) app. A.5.
| Benchmark | Model | ReAct in | ReAct out | JevAct in | JevAct out | Kev in | Drafts sent back | Not rebuilt |
|---|---|---|---|---|---|---|---|---|
| τ² airline (50 × 3) | GPT-4.1 | 58.4 | 0.9 | 60.1 | 0.9 | 9.0 | 0.39 | 0 |
| τ² airline | Opus 4.6 | 64.6 | 1.5 | 64.1 | 1.5 | 6.7 | 0.08 | 0 |
| τ² airline | gpt-oss-20b | 50.4 | 3.0 | 56.8 | 3.4 | 5.2 | 0.19 | 0 |
| τ² airline | gpt-oss-120b | 44.1 | 2.5 | 43.8 | 2.6 | 6.5 | 0.36 | 0 |
| τ² airline | Qwen3.8-27B | 79.8 | 8.2 | 80.3 | 8.5 | 6.4 | 0.03 | 0 |
| τ² airline | DeepSeek-V4-Pro | 64.0 | 1.8 | 63.2 | 1.8 | 9.5 | 0.11 | 0 |
| τ² telecom (114 × 3) | GPT-4.1 | 130.6 | 1.2 | 167.3 | 1.4 | 16.8 | 1.44 | 0 |
| τ² telecom | Haiku 4.5 | 138.8 | 2.1 | 184.9 | 2.7 | 21.0 | 1.91 | 0 |
| τ² telecom | Opus 4.6 | 171.7 | 1.7 | 201.4 | 1.8 | 12.7 | 0.61 | 0 |
| τ² telecom | Opus 5.5 | 178.1 | 2.2 | 183.0 | 2.2 | 9.6 | 0.10 | 0 |
| τ² telecom | gpt-oss-20b | 157.3 | 9.1 | 167.9 | 9.4 | 22.1 | 1.30 | 0 |
| τ² telecom | gpt-oss-120b | 117.7 | 4.2 | 139.0 | 5.1 | 24.1 | 1.38 | 0 |
| τ² telecom | Qwen3.8-27B | 143.6 | 8.0 | 143.6 | 8.0 | 12.7 | 0.23 | 0 |
| τ² telecom | DeepSeek-V4-Pro | 199.6 | 1.8 | 207.3 | 1.8 | 15.8 | 0.63 | 0 |
| BFCL (800 × 3) | GPT-4.1 | 44.1 | 0.5 | 46.5 | 0.5 | 3.2 | 0.02 | 0.02 |
| BFCL | Opus 4.6 | 74.0 | 1.4 | 74.9 | 1.4 | 4.4 | 0.11 | 0.19 |
| BFCL | gpt-oss-20b | 49.7 | 2.2 | 49.9 | 2.2 | 3.5 | 0.03 | 0.04 |
| BFCL | gpt-oss-120b | 49.5 | 1.6 | 50.6 | 1.7 | 3.7 | 0.04 | 0.08 |
| BFCL | Qwen3.8-27B | 88.8 | 3.3 | 87.8 | 3.3 | 5.5 | 0.11 | 0.19 |
| BFCL | DeepSeek-V4-Pro | 81.9 | 1.4 | 81.9 | 1.4 | 5.9 | 0.13 | 0.22 |
| BFCL | Llama 3.3 70B | 111.3 | 0.6 | 116.5 | 0.7 | 5.2 | 0.05 | 0.09 |
| BFCL | Mistral Small 3.2 24B | 51.7 | 0.7 | 54.5 | 0.7 | 3.0 | 0.02 | 0.03 |
| BFCL | Gemma 4 31B | 62.4 | 0.5 | 62.3 | 0.5 | 3.6 | 0.05 | 0.09 |
Offline judge replays app. A.2 app. A.4 tab. 7 tab. 8
Replaying 609 Telecom hand-off states logged from the Kev-27B runs, Kev-9B matches Kev-27B's flag-or-pass decision 97.0 % of the time and Kev-0.8B 90.6 % app. A.2 tab. 7. On 260 labelled Airline write calls (101 bad, 159 good), Kev-27B has AUROC 0.85 on long inputs versus 0.61 for Kev-9B; on short inputs 0.75 versus 0.70 for Kev-9B and 0.63 for Kev-0.8B app. A.4 tab. 8. Each judge is compared at the deployed threshold and at its own best threshold, which uses the labels, so it gives the smaller judges their most favourable setting app. A.4.
Paper vs code
No official code.
What I searched:
- The arXiv abstract page lists no comments field and no code or project link.
- The paper's text and bibliography link no repository for JevAct itself. The
only repository it links for the judge model is
jaredpalmer/kev[kev]. That is a separate project by a different author, so I did not use it as official code. - GitHub repository search for
jevactreturned one third-party project that mentions the paper (MIT). It is not by the authors, so I did not use it. - GitHub searches for
JevActanddecision gate LLM agentsinside theIBMorganisation returned zero results.
The arXiv source includes scripts/make_tables.py, which regenerates the
result tables from run logs. It calls analysis scripts (analyze_seeds.py,
analyze_round2.py, analyze_kevsize.py) and runner scripts (run_tau2.py,
run_bfcl.py) that are not in the arXiv source. The harness, compiler and
checks cannot be reproduced from this source alone.
Prompts are in the source directory prompts/ and are shown in app. A.1.
Pitfalls
- Judge thresholds and polarity. The text gives a rule for questions whose
"violation" answer is no (a no counts when P(yes) ≤ 0.05). It does not
say how questions whose violation answer is yes are scored, though the
compile prompt includes that field (
bad_if) app. A.1. Confirm the polarity rule before implementing. - Revision-budget scope. The paper says each check has a budget of R revisions, but not whether the budget is per call, per task or per episode §3.2.
- Keyword filter. The hand-off check uses a keyword filter to find possible give-up messages, but the keyword list is not given §3.5.
- Silent failures in generated code. Generated code runs in a restricted environment, and a raised exception is read as "no violation" app. A.1. A bug in generated code therefore fails open. One repair round on training logs, which fixed bugs such as reading an error string as a user record, reduced wrongful flags on correct writes from 11.3 % to 1.4 % on held-out Airline tasks app. A.1.
- Telecom checklist drops used development runs. The seven dropped steps were chosen from Kev-27B's answers on 74 train-split tasks, from development runs of GPT-4.1 and Opus 4.6 app. A.1. The paper also says the checklist was revised after a results review. The 40 test-split tasks were "held out only for the filter", but the revision means the test split may not be fully independent app. A.1.
- Ablation numbers come from different setups. The write-gate ablation in tab. 3 is one trial with a GPT-4.1 agent. The prose in §5.1 also reports three-run figures for GPT-4.1 and gpt-oss-120b agents with a gpt-oss-120b user simulator (26.0 → 28.0 unconditional re-check, 40.0 full gate; 32.0 → 42.0 for gpt-oss-120b). Those figures are not in any table. Do not mix them with the Table 3 numbers.
- Prose and table disagree. §5.1 says Kev-27B reaches 70.0 % pass^1 with GPT-4.1, but tab. 4 shows 72.0 for the same cell. The Haiku figures match.
- Judge-choice comparison. Each LLM judge uses its own dropped-question list §4.3. The paper does not say whether thresholds were re-tuned for the other judges. The judge-size comparison keeps Kev-27B's thresholds and dropped steps for every size tab. 5. Both comparisons were set up around Kev-27B, so read them as relative, not neutral.
- User simulator sensitivity. The Telecom gain does not appear with a gpt-oss-120b user simulator: pass^3 changes by −1.8 points (95 % CI −7.9 to +4.4) §6. Main results use GPT-4.1 as the user simulator §4.1.
- Hand-off check off on Airline. The hand-off check is off on Airline: in offline tests it raised flags on 26 % of hand-offs in episodes that were otherwise solved §4.5.
- Long inputs. 41 % of Airline write-check inputs are longer than 8,192 tokens. Kev-0.8B and Kev-9B have been validated for inputs up to 8,192 tokens only, and on long inputs Kev-9B catches fewer bad calls than Kev-27B §5.1 app. A.3 app. A.4.
- Passing the gate is not the same as correct. Among 260 labelled dev write calls, the 101 incorrect ones include 42 that break no compiled question. The questions model the policy, not what the user actually asked for §6.
- Unconditional re-checks sometimes help as much. With gpt-oss-120b as agent, re-checking every write unconditionally, or flagging at random at the gate's rate, lifts pass^3 about as much as the full gate does. The paper says the judge's verdicts matter only for some agents §5.1.
- Flag rate. According to §5.1, the gate acts on under half of write calls. Table 3's "flags per task" column does not give the denominator, so the two statements cannot be checked against each other.
- Cost estimates exclude the agent. Judge costs exclude the agent, the user simulator and one-time compilation app. A.6. The GPU cost assumes $2 per GPU-hour and full utilisation app. A.6. Compilation cost is not reported.
- Compiled checks depend on a stronger compiler model. Claude Opus 4.6 compiled the checks §4.3. The paper does not say how the checks change if the compiler is weaker or if the policy is edited.
- pass@3 is not success. pass@3 gives a task credit if one of its three runs succeeds. A BFCL pass@3 drop can coexist with a pass^3 gain, so read both columns §5.
Lineage
- Builds on: ReAct, which the harness wraps unchanged apart from the check step §3. Reflection methods Self-Refine and Reflexion add test-time feedback but do not check actions against a policy §2.1.
- Related: JevLoop assigns action decisions to a decision model, whereas JevAct keeps action choice with the agent and uses the decision model only as a verifier §2.2. Jev introduced the decision-model idea §2.3. Two preprints evaluate Jev as a judge: arXiv:2609.26550 and arXiv:2609.29769 §2.3.
- Compared against: same-model ReAct baselines on τ²-bench and BFCL §4. The paper does not compare against the runtime policy systems in §2.2 (AgentSpec, ShieldAgent, ToolSafe, PolicyGuard and others) on the same benchmarks.
- Superseded by: none known.
Source and license
Adapted from “Think Before You Act: Decision Gate for LLM Agents” by Iftach Shoham, Yuval David, Offer Akrabi, Erez Bilgory, Ido Levy, Sami Marreed, Sergey Zeltyn, Segev Shlomov (arXiv:2610.38620), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0.