Agent-as-a-Router: Agentic Model Routing for Coding Tasks

In plain words

Teams that have access to several AI models know that no single one is best at everything. One model may write new functions well, another may be better at fixing bugs, and a third may be cheaper. Choosing the right model for each job by hand does not scale, so this paper looks at automatic choice, called routing.

The authors first found that a simple router, which picks a model from a description of the task, does worse than it could because it lacks information about how each model has actually performed on similar work. Giving it per-category performance statistics closes most of that gap. Their method goes further: after each task the chosen model's answer is run and checked, and the result (score, cost and what was tried) is stored in a memory. Later tasks look up similar past tasks in that memory and use what worked, so the router keeps learning while it is in use. The paper tests this on a benchmark of about 10,000 coding tasks and on a small set of real repository-level software tasks.

TL;DR

• Routing each coding task to the best of eight LLMs is framed as a loop of context, action, feedback and back to context, so the router improves as the task stream runs §3.2.
• ACRouter, the instance, combines an orchestrator, a sandboxed verifier and an embedding memory §3.3.
• Author-reported: 49.98% AvgPerf and cumulative regret 205.5 on 2,919 in-distribution tasks, the best of the listed routers tab. 3.
• On 176 agentic programming tasks ACRouter reaches 62.50%, but a standalone GPT-5.4 run in an appendix reaches 75.00% on the same split app. D.7.
• No official code could be found; see Paper vs code.

Problem

Users with several model providers want, per task, the model that gives the best quality for its cost. The paper argues that static or one-off routers fail for an information reason, not a reasoning one: a vanilla LLM router that sees only the task prompt scores 41.41 AvgPerf, while the same router given per-dimension performance statistics from a probing set scores 47.74 (+15.3% relative), above a heuristic baseline at 47.50 §3.1 tab. 1. Static routers also cannot use new feedback after training, and the authors report that trained classifiers generalise poorly to an agentic-programming split §5.2. The C-A-F framing is stated as a contextual bandit with cumulative regret as the streaming metric §3.2.

Method

Problem setup §3.2. The router has a pool of M models. Tasks arrive as a stream t_1..t_N. At each step it observes a context c_i, chooses an action a_i (a model index), receives feedback f_i, and writes f_i into the context for the next step eq. 1. Context is the prompt p_i, optional metadata d_i (dimension, difficulty, language) and the memory state. Feedback is a verifier score in [0, 1] plus the monetary cost of the call §3.2.

Reward and regret §3.2 app. A.2. The per-task reward is a weighted sum of score and cost with ε1 > 0 and ε2 < 0 eq. 2. The per-task oracle takes the maximum reward over models for each task eq. 5, and cumulative regret is the sum over tasks of the oracle reward minus the reward of the chosen model eq. 7. The evaluation setting used in the tables is (ε1, ε2) = (1, −0.1) with cost in USD §5.1 app. A.2. Raw AvgPerf is reported without the cost term app. A.2.

ACRouter modules §3.3 app. A.1.

• Orchestrator. Chooses the model. Its inputs are the DimensionBest prior, the top-10 historical neighbours retrieved from memory by cosine kNN, and the task metadata. The core policy is a Qwen3.5-0.8B model fine-tuned on the probing set, combined with heuristic rules by weighted voting. The weights of the vote are not stated.
• Verifier. Evaluates the chosen model's output without ground-truth tests and aggregates tool signals into one score u_i in [0, 1] eq. 8. Tools named are AST parsing, sandbox execution, self-consistency, LLM-as-judge and extraction of in-prompt tests app. A.1. The per-type weights w are not stated in the paper body (the appendix text deferring exact weights is commented out in the source).
• Memory. An online vector store keyed by task-prompt embeddings (voyage-code-3, or BGE-large as an alternative). Each entry logs the chosen model, performance, cost and the verifier trace. Retrieval is cosine kNN with k = 10 and a similarity threshold of 0.5. The store is FIFO-bounded at 20K entries and updated after each attempt §3.3 app. A.1.

Pseudocode of the loop. The steps below follow the C-A-F description in the paper. They are our summary, not code from the release.

# ACRouter over a task stream  [eq. 1] [§3.3] [app. A.1]
# Input:  tasks (N prompts), models (M), memory (kNN store)
# Output: the chosen model index per task

def route_stream(tasks, models, memory):
    # Context: prompt, metadata and the nearest past tasks  [§3.3]
    for task in tasks:
        neighbours = memory.knn(embed(task.prompt), k=10)
        context = (task.prompt, task.metadata, neighbours)
        # Action: orchestrator picks one model  [§3.3]
        action = orchestrator(context, models)
        output = models[action].run(task)
        # Feedback: verifier score in [0, 1] and token cost  [eq. 8] [§3.3]
        score = verifier(task, output)
        cost = token_cost(output)
        # Memorize: store the outcome, FIFO-bounded at E entries  [§3.3]
        memory.add(embed(task.prompt), action, score, cost)
    return

Reward used by the bandit baselines app. A.2 §3.4.

# Cost-aware reward used for CumReg and the bandit updates  [eq. 2] [app. A.2]
# Inputs: score in [0, 1] (ground truth), cost in USD for the call

def reward(score, cost_usd):
    # (eps1, eps2) = (1, -0.1)  [app. A.2]
    return 1.0 * score + (-0.1) * cost_usd

Baseline policies §3.4 app. A.3. Single-model (Always-m); heuristic (DimensionBest, kNN over a frozen probing memory); trained policies (LogReg with TF-IDF, TF-IDF with an MLP, RouteLLM matrix-factorisation and BERT variants, and LoRA-finetuned Qwen3.5 classifiers); and online bandits (LinUCB, LinTS) over categorical or 64-dimensional Johnson–Lindenstrauss-projected task features §3.4.

Benchmark, CodeRouterBench §4. Tasks are drawn from existing coding benchmarks and grouped into nine single-turn dimensions plus an agentic-programming dimension held out as out-of-distribution (OOD) app. B.1 tab. 2. Seven dimensions are scored by execution (pass@1 in sandboxes) and three by proxy metrics with LLM-as-judge §4.1. Eight models are scored on each task: Claude Opus 4.6 and Sonnet 4.6, GPT-5.4, Qwen3-Max, Qwen3.5-Plus, GLM-5, Kimi-K2.5 and MiniMax-M2.7 §4.2. The split of the nine single-turn dimensions is 70/30 by a deterministic MD5 hash with seed coding-router-v1, giving a probing set of 7,080 tasks and an in-distribution test of 2,919 tasks app. B.2 tab. 2. The OOD set has 176 tasks from SWE-bench Verified, LongCLI-Bench, FeatureBench and SWE-CI, run with mini-swe-agent in a Docker harness and capped at 40 steps (the standard limit is 250) §4.1 app. B.2 §5.3.

Hyperparameters

Results

Author-reported. Setup: CodeRouterBench, in-distribution (ID) test n = 2,919 across nine single-turn dimensions and OOD test n = 176 agentic-programming tasks. AvgPerf is the mean score (OOD: resolved rate). CumReg uses (ε1, ε2) = (1, −0.1) §5.1. Perf per USD is AvgPerf per USD §5.1 tab. 3. Dollar costs and token totals are in app. D.6 and app. D.7.

Other reported numbers:

• Preliminary LLM-as-router ablation, Claude Sonnet 4.6 on 2,919 tasks: Vanilla 41.41; +Dimension 41.18; +Perf stats 47.74; DimensionBest 47.50; Oracle 57.00 tab. 1.
• Updated standalone GPT-5.4 run on the OOD split: 75.00% resolved (132/176). Opus 4.6 57.14%, Sonnet 4.6 49.11%, others 2.68% to 28.57% app. D.7.
• Zero-shot and 3-shot LLM-as-router across all eight models (ID): best 0-shot AvgPerf 46.87 (Qwen3.5-Plus), best 3-shot 46.03 (GLM-5); Claude Opus 4.6 is last in both modes at 39.27 and 41.17 app. C.1.
• Agent online (200-task warm-up) with eight router LLMs: 40.95 to 45.27 ID AvgPerf app. C.1.
• Qwen scaling sweep (ID, FT v4): 0.8B 46.41, 2B 46.69, 4B 46.21, 9B 46.56, 27B 46.74 AvgPerf; CumReg reported only for 0.8B (309.1) app. C.2.
• Measured local throughput 35,094 tokens/s; derived USD 0.054 per 1M self-hosted tokens app. B.3.2.

Paper vs code

No official code. The paper's abstract, contributions and project-page link all point to https://github.com/LanceZPF/agent-as-a-router. A shallow clone of that URL failed with "Repository not found". The GitHub user list for LanceZPF is empty, and a repository search for "agent-as-a-router" and for "ACRouter coderouterbench" returned no results. The paper also mentions a released artifact (configs/model_pricing.json, local_pricing_benchmark/, a benchmark script and JSON logs) that we could not access. Because the code is not verifiable, the card has no code field.

Points the paper leaves open, which an implementer would need from code:

• Orchestrator fine-tuning recipe, the heuristic vote weights, and the verifier tool weights §3.3 eq. 8.
• Which embedding model (voyage-code-3 or BGE-large) produced each reported ACRouter number app. A.1.
• Whether the appendix's stated pricing and local-serving numbers come from the same run as the ID table app. B.3.

Pitfalls

• Best-router claim depends on the pool. Table 3 and the text call ACRouter the best router on OOD at 62.50%. That comparison uses the original eight-backend bundle, and the authors also report that a standalone GPT-5.4 run resolves 75.00% of the same split app. D.7 §5.2. The oracle row (75.89%) comes from the same original bundle and so does not include the GPT-5.4 run either.
• Small OOD sample. With 176 tasks each task is about 0.57 AvgPerf points, so the gap between ACRouter (62.50) and Always-Opus (57.14) is about 9 tasks. No variance is reported for ACRouter, which appears to be a single run (seed 42 is mentioned for online agents), while bandits average five seeds tab. 3 app. D.6.
• Verifier scores are used for memory, not ground truth. Memory and the Orchestrator see the verifier's estimated score u_i eq. 8. The paper gives no measurement of how well u_i matches the ground-truth score s_i, which is what CumReg uses. Errors in u_i feed directly into later routing decisions.
• Cost accounting. Self-hosted router tokens are priced at a GPU-rental rate of USD 0.054 per 1M tokens app. B.3.2. The paper does not state whether the embedding calls (voyage-code-3 is an API model) are priced app. A.1. The authors also state that provider-side cache hits are unobservable and that monetary cost is a secondary metric §5.3.
• Reduced OOD step cap. The OOD agent is capped at 40 steps instead of 250, to save budget. The authors say the relative comparison is unaffected §5.3 app. B.2; the absolute resolved rates are not comparable to published SWE-bench numbers.
• Task count mismatch. The text says 10,111 tasks are divided into probing and two test sets §4.1. Table 2 and the split arithmetic give 7,080 + 2,919 + 176 = 10,175. The nine single-turn dimensions each have 1,111 tasks, 9,999 in total app. B.1. Use the component counts when reproducing.
• Dimension count wording. The paper says CodeRouterBench has 10 dimensions §4 and also nine single-turn dimensions plus one OOD dimension app. B.1. Both describe the same set, but the count is easy to misread.
• Cold start. ACRouter's memory is warm-started with 200 validation tasks from the probing set app. B.2. Results with an empty memory are reported only for the online agents in the LLM-router table app. C.1.
• Few-shot fallback. When a router's JSON output cannot be parsed, the default model is claude-sonnet-4-6 app. B.4. This fallback biases the LLM-router baselines toward Sonnet.
• Scaling claim. The authors state that larger Qwen sizes gave similar AvgPerf, but they did not archive per-task decisions for those sizes, so no CumReg is reported for them app. C.2.

Lineage

• Builds on: RouteLLM [ong2024routellm] (its BERT and matrix-factorisation variants are baselines), LinUCB [li2010contextual] and LinTS [agrawal2013thompson] (bandit baselines), and the contextual-bandit framing [li2010contextual; lattimore2020bandit] §3.2.
• Compared against: DimensionBest, kNN retrieval, LogReg, TF-IDF with MLP, RouteLLM, online bandits, LLM-as-a-router (zero-shot and 3-shot) and the always-one-model baselines §3.4 tab. 3 app. C.1.
• Related benchmark named by the authors: LLMRouterBench [duwal2026llmrouterbench], which covers general NLU tasks. The paper gives no arXiv id for it.
• Superseded by: not known from this wiki.