Small Language Models are the Future of Agentic AI

In plain words

Most AI agents call a large general-purpose language model many times, but most of those calls are small, repetitive jobs such as intent recognition, data extraction or templated messages. The paper argues that a much smaller model, trained for one such job, can do most of these calls well, and at far lower cost and latency. The large model is still the right choice when open-ended conversation is essential, so the paper expects agents to mix both.

It also gives a plan for moving an existing agent over: record its model calls, remove private data, group the calls into tasks, and train a small model for each task. The authors present this as a position and invite critique. They do not claim to have tested it.

TL;DR

• Position paper. It makes a value claim that small language models (SLMs) should replace large ones for most calls inside agents §2.2.
• The three claims are that SLMs are powerful enough, better suited to agent interfaces, and cheaper per call §2.2.
• It proposes a five-step conversion algorithm, from logging to iterative retraining §6.
• It has no experiments of its own. Evidence is cited from other works and three case-study estimates made by the authors §3.1, app. B.

Problem

Agents usually reach language intelligence through API calls to one generalist LLM hosted in the cloud §1. The paper says this model is the cornerstone of a large infrastructure investment, which it estimates at USD 57bn §1. It also says more than half of large IT enterprises already use AI agents §1.

The authors observe that agents use only a narrow part of the model's abilities, through prompts that restrict it to one routine §3.4. Each call is often repetitive and non-conversational §2.3. The paper argues that paying for a generalist on every such call is a misallocation of resources §2.3.

The problem is therefore not a missing capability. It is that the default architecture pays large-model costs for small-model work, and the barriers to change (investment inertia, generalist benchmarks, low awareness) slow the switch §5.

Method

The paper is a position, not a training method. It makes three views and supports each with arguments §2.2, §3.

Definitions §2.1, app. A:

• WD1: an SLM is a language model that fits on a common consumer device and performs inference with latency low enough to serve one user's agentic requests §2.1.
• WD2: an LLM is any language model that is not an SLM §2.1.
• The authors say most models below about 10bn parameters would be SLMs as of 2025, but the definition does not depend on size §2.1, app. A.

Views §2.2:

• V1: SLMs are sufficiently powerful for the language tasks of agentic applications §2.2.
• V2: SLMs are more operationally suitable for agentic systems §2.2.
• V3: SLMs are more economical for most LM uses in agents, because of their size §2.2.
• Conclusion: on these grounds SLMs are the future of agentic AI §2.2.

Arguments §3:

• A1 (power) §3.1: SLMs now reach task performance that earlier LLMs had. The paper says capability, not parameter count, is now the binding constraint. It names reasoning add-ons: fine-tuning, self-consistency, verifier feedback and tool augmentation §3.1.
• A2 (economics) §3.2: cheaper inference, fine-tuning that takes GPU-hours, on-device deployment, and modular systems that combine several models §3.2.
• A3 (flexibility) §3.3: several specialised models are affordable to train and adapt. The paper adds a democratisation argument §3.3.
• A4 (narrow exposure) §3.4: agents expose a limited part of the model, so a tuned SLM would suffice §3.4.
• A5 (alignment) §3.5: agent calls need one strict output format, which an SLM can be trained to produce §3.5.
• A6 (heterogeneity) §3.6: an LLM may act as the root model and an SLM as a subordinate; or every model may be a specialist §3.6, fig. 1.
• A7 (data) §3.7: agent calls produce data that can later train specialist SLMs §3.7.
• A8 to A11 answer the LLM-generalist view §4.1. A12 and A13 answer the centralisation view §4.2.

Alternative views §4:

• AV1: an LLM always has the edge in general language understanding. Counter- arguments are scaling laws (CA1) and a "semantic hub" inside LLMs (CA2) §4.1.
• AV2: at scale, centralised LLM serving is cheaper per token than SLMs. The counter-arguments are harder load balancing for many endpoints (CA3) and setup costs (CA4). The authors call this view valid but case-specific §4.2.
• AV3: SLM and LLM worlds are both possible, but the LLM world is ahead in deployment practice §4.3.

Barriers §5: B1 large upfront investment in centralised inference; B2 generalist benchmarks used for SLMs; B3 low public awareness. The authors regard these as practical hurdles and give no timeline §5.

Conversion algorithm §6. The paper names five steps. The pseudocode below restates them in our words; the paper gives no code for it.

# LLM-to-SLM agent conversion  [§6]
# Input:  call_logs of non-human-facing LLM calls; candidate base SLMs
# Output: one specialist SLM per task, plus the routing to use them

def convert_agent(call_logs, base_slms):
    # Log prompts, responses, tool calls and latency; encrypt, restrict
    # access and anonymise before storage  [§6, S1]
    records = anonymise(encrypt(call_logs))

    # Remove PII, PHI and other sensitive data so no user's data leaks
    # into the specialist. Rule of thumb: 10k-100k examples  [§6, S2]
    records = remove_sensitive(records)

    # Group prompts and agent actions into recurring tasks, such as
    # intent recognition or data extraction  [§6, S3]
    tasks = cluster_by_task(records)

    # One or more candidate SLMs per task, judged on capability,
    # benchmarks, licence and footprint  [§6, S4]
    candidates = select_slms(tasks, base_slms)

    specialists = {}
    for task in tasks:
        for slm in candidates[task]:
            # Fine-tune on that task's data: LoRA or QLoRA, or full
            # tuning; optionally distil from the LLM  [§6, S4]
            specialists[(task, slm)] = fine_tune(slm, records[task])

    # Retrain periodically on new data; loop back to curation or
    # selection as needed  [§6, S5]
    return specialists

Figure fig. 1: one figure with two panels. Left: language-model agency, where the model is both the interface to the human and the orchestrator of tool calls. Right: code agency, where a controller program orchestrates the calls and the model is an optional interface. The figure also labels a "logger" on the calls, which the text ties to the data collection in step S1 §3.6, §6.

Call for discussion §7: the authors invite critique and contributions, and say they will publish the correspondence they receive.

Hyperparameters

The paper gives no training or inference settings. The table lists every number it does state, plus the values it leaves open.

Results

No experiments in this paper. The numbers below are the paper's own summary of other works, or the authors' estimates. Nothing was reproduced. Setups and benchmarks are named only loosely in the text.

The case-study method is not described beyond "we estimate". "Reliably handled" is not defined. The licences given for the three agents are the authors' statements; this card did not verify them.

Paper vs code

No official code.

The paper names no repository. The arXiv metadata has no comment with a link. The project page the paper gives, research.nvidia.com/labs/lpr/slm-agents §1, §7, has no code: its only GitHub link is a website template. A GitHub repository search for "slm-agents" returned nothing, and a search for "small language models agentic" within the NVlabs organisation also returned nothing. Because there is no code, there are no discrepancies to check. The paper has no training procedure to compare against.

Pitfalls

• Position, not method. The paper's central claim is normative: SLMs "should be" the default for agent calls. The authors give no timeline and no test of their claim §2.2, §5. Do not implement it as a proven result.
• No experiments, and the cited numbers come from other setups. The Results table is other papers' numbers, not ours. Several rows name metrics too loosely to compare (for example "commonsense reasoning") §3.1.
• The Phi-3 sentence is ambiguous. It is not clear which metrics match the 70bn models §3.1. Check the Phi-3 report before relying on it.
• Hymba's comparison is unspecified. The paper says it beats "larger 13bn models" on instruction following but does not name them §3.1.
• Case-study numbers have no stated method. The 40%, 60% and 70% are estimates from the authors app. B.
• Size is not part of the definition. WD1 is about device and latency. An implementer who wants a parameter cut-off must choose one; the paper only suggests about 10bn §2.1, app. A.
• Latency and cost numbers are not one measure. §3.1 gives "about 15× faster" for Phi-2. §3.2 gives "10–30× cheaper" for serving, counting latency, energy and FLOPs together. Do not treat them as the same quantity §3.1, §3.2.
• Conflict of interest. Most authors are at NVIDIA Research, one also at Georgia Tech [author block]. The paper cites NVIDIA tools as support (Dynamo, ChatRTX) §3.2, §5. A footnote says the views are the authors' own [abstract, fn.].
• Data collection is the risky step. Logging every model call can capture sensitive content. The paper recommends encryption, access control and anonymisation §6, S1 but gives no method. Paraphrasing for privacy is suggested without detail §6, S2.
• The data-size figure is a rule of thumb. The 10k–100k examples figure is cited to two outside works §6, S2. Check them before sizing a project.
• Licence check is named but not specified. Selection criteria include licence, but the paper gives no rules §6, S4.
• arXiv version. This card was written from the v3 LaTeX source fetched by the fetch command (arXiv updated 2026-09-22). The licence is the one recorded at fetch time.

Lineage

• Builds on: no earlier method. The paper's evidence comes from cited works, named in the text: Phi-2, Phi-3, Nemotron-H, SmolLM2, Hymba, DeepSeek-R1- Distill, RETRO, xLAM-2, Toolformer, LoRA, DoRA, QLoRA and NVIDIA Dynamo §3.1, §3.2, §6.
• Compared against: the alternative views in §4 (LLM generalist advantage, centralised serving cost, equal possibilities).
• Superseded by: none recorded.