---
kind: concept
slug: model-routing
title: Model routing
summary: >-
  Sends each query, or each step of a task, to one of several models: a router picks it, or a trained planner writes the plan. A survey covers per-query routing as an efficient-inference family; a 2025 paper trains a 7B planner over a worker pool; a GPT-5 system card describes a deployed router with no features given. Results are author-reported.
tags: [inference, efficiency, small-models, orchestration]
papers: ["2409.06857", "2512.04388", "2601.03267"]
related: [knowledge-distillation, speculative-decoding, retrieval-augmented-generation, multi-agent-systems, reinforcement-learning-verifiable-rewards, mixture-of-experts]
status: draft
updated_at: 2026-10-31
---

## Overview

Routing is a dispatch step in front of a set of models. A router looks at a
query and picks a model to answer it [arxiv:2409.06857 §3.1.3]. The survey
places it under efficient inference, next to cascading (a small model answers
first and escalates when a check fails) and speculative decoding (a small draft
model proposes tokens that a large model verifies) [arxiv:2409.06857 §3.1.3]
[fig. 5]. The shared goal is to spend large-model compute only where it is
needed.

The second source uses the word for something wider. A 7B "Conductor" model
writes a plan for each question: which worker model runs each of up to five
steps, what it is asked, and which earlier steps it can read
[arxiv:2512.04388 §3.1] [app. E]. Here the routing decision is made per step,
and one question may use several models. The survey's cost warning about
running several models per query therefore applies to this variant
[arxiv:2409.06857 §3.1.3]. The two sources define the term differently, so
check which sense a card uses before comparing numbers.

A third source, the GPT-5 system card, describes a deployed router. It sends
each query to a fast model or a deeper thinking model [arxiv:2601.03267 §1].
The system is closed, so the card gives no router design.

## Variants

### Per-query router

A router reads the query and names one model. Only that model runs. The
survey's description allows any signal the router can read, but it lists only
the types of signal and gives no thresholds or training details
[arxiv:2409.06857 §3.1.3]. Card: [Chen and Varoquaux's 2024 survey of small models next to LLMs](/papers/2409.06857).

### Training-free router

Routers that use lightweight signals need no training. The survey says these are
cheaper to run than trained routers but rely on heuristics
[arxiv:2409.06857 §3.1.3]. No specific signal is named on the card, so check
the paper's §3.1.3 before choosing one.

### Deployed router with fallback

A product can route between its own models. The GPT-5 card says a real-time
router picks a fast or a thinking model for each query, based on the query's
type, complexity, tool needs and any explicit request to think harder
[arxiv:2601.03267 §1]. The router is trained on signals such as users switching
models, preference rates and measured correctness [§1]. Its features, weights
and thresholds are not stated [§1]. After a usage cap, smaller variants serve
the remaining requests [§1]. Card: [OpenAI GPT-5 System Card](/papers/2601.03267).

### Learned planner over a model pool

The planner is an LLM trained with reinforcement learning to write the whole
plan. The base model is Qwen2.5-7B [arxiv:2512.04388 app. A]. Workers are
shown to the planner only as ordinal ids, with no model names [app. E]. Reward
is 1 for a correct final answer, 0.5 for a wrong but parsable plan and 0 for an
unparsable one [arxiv:2512.04388 §3.1]. Training used 960 problems, and the
authors report 200 iterations on two H100 80GB GPUs [§4.1] [app. A].

For adaptive worker selection, the authors fine-tune on a random subset of k
workers drawn from a pool of n for each question. The value of k is not given
in the main text [arxiv:2512.04388 §3.2]. The paper also tests recursion, in
which the planner may call itself. Recursion adds about one point on three
out-of-distribution tasks, 63.00 against 61.93 [arxiv:2512.04388 tab. 2]. Its
training questions were already seen in training [§4.4], so the gain does not
measure generalisation. Card: [Learning to Orchestrate Agents in Natural Language with the Conductor](/papers/2512.04388).

### Cascade (neighbouring pattern)

A cascade is often confused with routing. It runs a small model first and
sends the query on to a larger model only when a check fails. The survey lists
it as a separate family [arxiv:2409.06857 §3.1.3]. The card's pseudocode gives the
shape, with the confidence threshold left open [arxiv:2409.06857 §3.1.3]. Where
the cascade runs several models on the same query, it adds inference cost
[arxiv:2409.06857 §3.1.3].

## When to use which

The two sources measure different things, so the table states only what each
supports. Results from the Conductor are author-reported.

| Situation | Use | Why |
|---|---|---|
| Each query should hit exactly one model | Per-query router | Routing is defined as one model per query [arxiv:2409.06857 §3.1.3] |
| Running two models on one query is too expensive | Per-query router, not a cascade | The survey notes that methods running several models per query add cost [arxiv:2409.06857 §3.1.3] |
| No training budget, and the router can use simple signals | Training-free router | Cheaper, but relies on heuristics [arxiv:2409.06857 §3.1.3] |
| Task splits into steps that suit different models, and RL training of a planner is affordable | Learned planner | Beats every listed worker on seven unconstrained benchmarks [arxiv:2512.04388 tab. 1]; averages 72.35 against 64.14 for the best single model without self-reflection, in the constrained setting [tab. 7] |
| Only one strong model is available | Single model, or a planner with the pool you have | With GPT 5 as the only worker, the authors' average falls from 72.89 to 69.81 [arxiv:2512.04388 tab. 10] |
| Cost per question matters | Per-query router or cascade, and measure cost yourself | The survey says several-model methods add cost [arxiv:2409.06857 §3.1.3]. The Conductor averages about three steps [arxiv:2512.04388 §4.3] and uses fewer tokens than Mixture-of-Agents [tab. 6] |
| Edge devices or low-latency services that need a small model | Small model as default, large model as fallback | The survey places small models in these settings [arxiv:2409.06857 §4.1] |
| Need the router's features or thresholds to copy a deployed router | Not stated on these cards | The GPT-5 card names the signal types only [arxiv:2601.03267 §1] |
| Latency of sequential planner steps | Not stated on these cards | Neither paper reports wall-clock latency for routed calls |
| Need to pick the router's signals, threshold or step limit | Not stated on these cards | Survey lists signal types only [arxiv:2409.06857 §3.1.3]; recursion limit not given [arxiv:2512.04388 §3.2] |

## Implementation sketch

The survey gives no router algorithm. The Conductor's planner is the only
learned method on these cards. The two sketches below show the dispatch shapes.
Router design, thresholds and planner training must come from the cited works.

```pseudocode
# Per-query routing  [§3.1.3]
# Input:  query; router; models (name -> model)
# Output: answer from exactly one model

def route(query, router, models):
    # The router inspects the query and names one model  [§3.1.3]
    name = router(query)  # router design: not stated
    # Only the chosen model runs for this query  [§3.1.3]
    return models[name](query)
```

```pseudocode
# Learned plan over a model pool  [arxiv:2512.04388 §3.1] [app. E]
# Input:  question; pool (ordinal id -> worker); planner (trained LLM)
# Output: answer from the last step

def plan_and_run(question, pool, planner):
    # The planner writes three equal-length lists: model ids, subtasks,
    # access lists  [§3.1]. Handling of a bad plan: not stated
    plan = parse_three_lists(planner.sample(question, pool))
    history = []
    for model_id, subtask, access in plan:
        # Each worker sees only the earlier steps its access list selects  [app. E]
        context = select_context(history, access)
        history.append((subtask, pool[model_id].respond(context, subtask)))
    # The last step's answer is the response returned  [app. E]
    return history[-1].answer
```

## Contradictions

The survey and the Conductor paper do not measure the same thing, and they
point in different directions on cost. Both sides are stated; neither is
resolved here.

- The survey says methods that run several models per query add inference
  cost, and that training-free routers are cheaper but heuristic
  [arxiv:2409.06857 §3.1.3]. It reports no measurement of its own on this.
- The Conductor runs up to five steps across several models, yet it reports
  1820 tokens and cost 0.02384 per sample, against 11203 tokens and 0.04855 for
  Mixture-of-Agents [arxiv:2512.04388 tab. 6]. It also states fewer than twice
  the original number of agent calls [§4.4]. Its cost definitions are not
  stated [tab. 5] [tab. 6], and the comparison is with Mixture-of-Agents, not
  with a single model.

Within the Conductor paper, the same model has two scores for one benchmark,
depending on the setting. Its LiveCodeBench v6 score is 83.93 with unconstrained
budgets [arxiv:2512.04388 tab. 1] and 64.29 in the constrained multi-agent
table [tab. 7]. Both are reported, but a score should be quoted with its setting.

## Papers

- [Chen and Varoquaux's 2024 survey of small models next to LLMs](/papers/2409.06857): maps routing, cascading and speculative decoding as efficient-inference families, and the trade-offs of running several models per query.
- [Learning to Orchestrate Agents in Natural Language with the Conductor](/papers/2512.04388): a 7B planner trained with GRPO picks worker models, subtasks and access lists for up to five steps; author-reported gains over the listed workers.
- [OpenAI GPT-5 System Card](/papers/2601.03267): a deployed router picks a fast or a thinking model per query, with fallback after usage caps; the card gives no router design or routing numbers.