Model routing

Overview

Routing is a dispatch step in front of a set of models. A router looks at a query and picks a model to answer it 2409.06857 §3.1.3. The survey places it under efficient inference, next to cascading (a small model answers first and escalates when a check fails) and speculative decoding (a small draft model proposes tokens that a large model verifies) 2409.06857 §3.1.3 fig. 5. The shared goal is to spend large-model compute only where it is needed.

The second source uses the word for something wider. A 7B "Conductor" model writes a plan for each question: which worker model runs each of up to five steps, what it is asked, and which earlier steps it can read 2512.04388 §3.1 app. E. Here the routing decision is made per step, and one question may use several models. The survey's cost warning about running several models per query therefore applies to this variant 2409.06857 §3.1.3. The two sources define the term differently, so check which sense a card uses before comparing numbers.

A third source, the GPT-5 system card, describes a deployed router. It sends each query to a fast model or a deeper thinking model 2601.03267 §1. The system is closed, so the card gives no router design.

Variants

Per-query router

A router reads the query and names one model. Only that model runs. The survey's description allows any signal the router can read, but it lists only the types of signal and gives no thresholds or training details 2409.06857 §3.1.3. Card: Chen and Varoquaux's 2024 survey of small models next to LLMs.

Training-free router

Routers that use lightweight signals need no training. The survey says these are cheaper to run than trained routers but rely on heuristics 2409.06857 §3.1.3. No specific signal is named on the card, so check the paper's §3.1.3 before choosing one.

Deployed router with fallback

A product can route between its own models. The GPT-5 card says a real-time router picks a fast or a thinking model for each query, based on the query's type, complexity, tool needs and any explicit request to think harder 2601.03267 §1. The router is trained on signals such as users switching models, preference rates and measured correctness §1. Its features, weights and thresholds are not stated §1. After a usage cap, smaller variants serve the remaining requests §1. Card: OpenAI GPT-5 System Card.

Learned planner over a model pool

The planner is an LLM trained with reinforcement learning to write the whole plan. The base model is Qwen2.5-7B 2512.04388 app. A. Workers are shown to the planner only as ordinal ids, with no model names app. E. Reward is 1 for a correct final answer, 0.5 for a wrong but parsable plan and 0 for an unparsable one 2512.04388 §3.1. Training used 960 problems, and the authors report 200 iterations on two H100 80GB GPUs §4.1 app. A.

For adaptive worker selection, the authors fine-tune on a random subset of k workers drawn from a pool of n for each question. The value of k is not given in the main text 2512.04388 §3.2. The paper also tests recursion, in which the planner may call itself. Recursion adds about one point on three out-of-distribution tasks, 63.00 against 61.93 2512.04388 tab. 2. Its training questions were already seen in training §4.4, so the gain does not measure generalisation. Card: Learning to Orchestrate Agents in Natural Language with the Conductor.

Cascade (neighbouring pattern)

A cascade is often confused with routing. It runs a small model first and sends the query on to a larger model only when a check fails. The survey lists it as a separate family 2409.06857 §3.1.3. The card's pseudocode gives the shape, with the confidence threshold left open 2409.06857 §3.1.3. Where the cascade runs several models on the same query, it adds inference cost 2409.06857 §3.1.3.

When to use which

The two sources measure different things, so the table states only what each supports. Results from the Conductor are author-reported.

Implementation sketch

The survey gives no router algorithm. The Conductor's planner is the only learned method on these cards. The two sketches below show the dispatch shapes. Router design, thresholds and planner training must come from the cited works.

# Per-query routing  [§3.1.3]
# Input:  query; router; models (name -> model)
# Output: answer from exactly one model

def route(query, router, models):
    # The router inspects the query and names one model  [§3.1.3]
    name = router(query)  # router design: not stated
    # Only the chosen model runs for this query  [§3.1.3]
    return models[name](query)

# Learned plan over a model pool  [arxiv:2512.04388 §3.1] [app. E]
# Input:  question; pool (ordinal id -> worker); planner (trained LLM)
# Output: answer from the last step

def plan_and_run(question, pool, planner):
    # The planner writes three equal-length lists: model ids, subtasks,
    # access lists  [§3.1]. Handling of a bad plan: not stated
    plan = parse_three_lists(planner.sample(question, pool))
    history = []
    for model_id, subtask, access in plan:
        # Each worker sees only the earlier steps its access list selects  [app. E]
        context = select_context(history, access)
        history.append((subtask, pool[model_id].respond(context, subtask)))
    # The last step's answer is the response returned  [app. E]
    return history[-1].answer

Contradictions

The survey and the Conductor paper do not measure the same thing, and they point in different directions on cost. Both sides are stated; neither is resolved here.

• The survey says methods that run several models per query add inference cost, and that training-free routers are cheaper but heuristic 2409.06857 §3.1.3. It reports no measurement of its own on this.
• The Conductor runs up to five steps across several models, yet it reports 1820 tokens and cost 0.02384 per sample, against 11203 tokens and 0.04855 for Mixture-of-Agents 2512.04388 tab. 6. It also states fewer than twice the original number of agent calls §4.4. Its cost definitions are not stated tab. 5 tab. 6, and the comparison is with Mixture-of-Agents, not with a single model.

Within the Conductor paper, the same model has two scores for one benchmark, depending on the setting. Its LiveCodeBench v6 score is 83.93 with unconstrained budgets 2512.04388 tab. 1 and 64.29 in the constrained multi-agent table tab. 7. Both are reported, but a score should be quoted with its setting.

Papers

• Chen and Varoquaux's 2024 survey of small models next to LLMs: maps routing, cascading and speculative decoding as efficient-inference families, and the trade-offs of running several models per query.
• Learning to Orchestrate Agents in Natural Language with the Conductor: a 7B planner trained with GRPO picks worker models, subtasks and access lists for up to five steps; author-reported gains over the listed workers.
• OpenAI GPT-5 System Card: a deployed router picks a fast or a thinking model per query, with fallback after usage caps; the card gives no router design or routing numbers.