What is the Role of Small Models in the LLM Era: A Survey

In plain words

Large language models are powerful, but they are expensive to run and to train, so many researchers and small companies cannot use them freely. This survey asks what small models are still good for. It treats "small" as a relative label: a model is small compared with a bigger one.

It looks at two ways the two sizes work together. Small models can help large ones: they filter training data, check and repair answers, and speed up generation. Large models can teach small ones: they generate training data, write out step-by-step reasoning, and pass on their internal signals.

It also covers where small models simply win: on phones and other edge devices, in fast search systems, on narrow tasks with little data, and where a decision must be explained. The main message is that the two are best used together rather than one replacing the other.

TL;DR

• This is a survey, so there is no single method to reimplement.
• It groups small-model/LLM relationships into collaboration (both directions) and competition or complementarity (edge, low latency, narrow tasks, interpretability) §1.
• The authors' own evidence is one MTEB comparison, shown only in figures §4.1 fig. 9.
• Use it as a map to the cited papers, not as a source of numbers.

Problem

Scaling up LLMs raises computing cost and energy use, which makes them impractical for academic researchers, businesses without a strong revenue stream and edge devices §1. The survey asks what role small models (SMs) keep in this era and what ecological niche they occupy §1.

Small models are defined relatively. A model counts as small when a larger reference model exists, so BERT (110M parameters) is small next to LLaMA-8B, which is small next to GPT-4 §1. Most SMs discussed have fewer than 1B parameters §1, footnote. The survey also includes encoder-only and decoder-only Transformers and, more broadly, other architectures and statistical models §1 §4.

The survey argues that small-model use is underestimated in practice. A HuggingFace download analysis supports this, grouping open-source NLP models into five size bins fig. 1 §1. Table 1 compares the two groups on four dimensions tab. 1:

Method

This survey has no training procedure. Its structure is a classification of how the two model sizes relate. Numbers below are section numbers counted from the LaTeX source (see Pitfalls).

Two perspectives §1. Collaboration §3 asks how the two sizes help each other. Competition and complementarity §4 asks where small models are the better choice. The survey's position is that the best ecosystem is hybrid, with small models in specialised or cost-sensitive roles and large models supporting them §4.

Collaboration, small models enhance LLMs §3.1. The survey follows the LLM life cycle: data preparation, inference and evaluation §3.1.

• Data curation §3.1.1 fig. 3. Pre-training data: a small model or classifier selects high-quality or language-matched text, and small proxy models set domain or instance weights. Fine-tuning data: small models select task-specific samples by embedding similarity or by internal signals, and select alignment data. Weak-to-strong: weaker models supervise stronger ones by labelling data or by steering outputs at test time.
• Augmented reasoning §3.1.2 fig. 4. Four strategies add external signal to a fixed LLM: retrieval-augmented generation (RAG), with textual, structured or other sources; domain adaptation, white-box (a small tuned model shifts the LLM's token distribution) or black-box (a small expert supplies text the LLM reads); prompt engineering (small models retrieve, design or verify prompts and demonstrations); and deficiency repair (contrastive decoding against a small model, or small plug-ins that detect hallucinations or calibrate confidence).
• Efficient inference §3.1.3 fig. 5. Three families allocate work between sizes: cascading (a small model answers first and the query escalates when a check fails), routing (a router picks one model per query) and speculative decoding (a small draft model proposes tokens that the large model verifies).
• Evaluating LLMs §3.1.4 fig. 6. Reference-based metrics compare outputs with references using learned semantic similarity. Reference-free methods use small NLI models, small proxy evaluators or a small judge model.

Collaboration, LLMs enhance small models §3.2. LLMs transfer knowledge by knowledge distillation §3.2. Four forms are described. Three depend only on the teacher's outputs and are black-box; representation distillation needs the teacher's internal states and is white-box §3.2.

• Full data synthesis §3.2.1 fig. 7. The LLM generates inputs and labels for a task (training data generation), or generates instruction-response pairs for alignment.
• Rationale distillation §3.2.2 fig. 7. The student is trained on the teacher's step-by-step rationales (chain-of-thought distillation), or on structured reasoning such as programs or reasoning scaffolds.
• Data augmentation §3.2.3 fig. 7. The LLM labels unlabelled inputs (pseudo-labels), or paraphrases and rewrites existing examples to add diversity.
• Representation distillation §3.2.4 fig. 7. Logit-based: the student matches the teacher's output distribution, for example with soft targets or a reverse KL objective. Feature-based: the student matches hidden states, attention maps or activations, including a fully quantised student.

Competition and complementarity §4 fig. 8. Three settings are discussed.

• Computation-constrained §4.1. Edge devices (phones, IoT) have little memory, power and connectivity. Low-latency services (search, retrieval, recommendation) need fast responses. Other surveys have measured small models such as Phi-3-mini (3.8B), MiniCPM and Gemma-2B on running time for edge settings; this survey cites them rather than re-measuring §4.1.
• Task-specific §4.2. Small models can win where data is scarce or the task is narrow: biomedical and legal text, low-resource languages, tabular data (where tree-based models can be competitive), short text, and niche tasks such as machine-generated text detection, spreadsheet representation and information extraction §4.2.
• Interpretability-required §4.3. Healthcare, finance and law often prefer smaller, more transparent models because non-experts must understand decisions §4.3.

Open challenges §5. Four open problems: reliable small models (capability gap, robustness to shift and forgetting); open-world collaboration beyond a fixed pool of models; trustworthy small evaluators; and small agents that do dedicated functions and cooperate with tools and other models §5.

Conclusion §6. The survey expects the relationship to move from competition towards cooperation, with models of different sizes filling different niches §6.

Pseudocode: confidence-based cascade

The survey describes this pattern but gives no threshold or confidence measure §3.1.3. Thresholds must come from the cited works.

# Confidence-based model cascade  [§3.1.3]
# Input:  query; small_model; large_model; threshold (not stated)
# Output: an answer string

def cascade(query, small_model, large_model, threshold):
    # The small model answers and reports a confidence  [§3.1.3]
    answer, confidence = small_model(query)
    # Keep the cheap answer when confidence clears the threshold  [§3.1.3]
    if confidence >= threshold:
        return answer
    # Below the threshold the query goes to the larger model  [§3.1.3]
    return large_model(query)

Pseudocode: draft-then-verify decoding

The survey describes the two phases and the parallel verification pass, but not the acceptance and resampling rule §3.1.3. Take that rule from the original speculative decoding papers it cites.

# Draft-then-verify decoding  [§3.1.3]
# Input:  prefix tokens; draft_model (small); target_model (large)
# Output: newly accepted tokens

def speculative_step(prefix, draft_model, target_model, num_draft):
    # The small draft model proposes several tokens in sequence  [§3.1.3]
    drafts = draft_model.generate(prefix, steps=num_draft)
    # The large model scores all drafted positions in one parallel pass  [§3.1.3]
    target_probs = target_model.score(prefix, drafts)
    # Accept drafted tokens the target agrees with  [§3.1.3]
    return accept_or_resample(drafts, target_probs)

Hyperparameters

The survey trains nothing. The table lists every numeric setting it mentions.

Results

Author-reported. The survey runs one experiment of its own and quotes others' numbers without re-measuring them.

Survey's own experiment §4.1 fig. 9. The authors evaluate text-embedding models of different sizes, encoder-only and decoder-only, on MTEB §4.1. Four task categories are shown, each averaged over five datasets §4.1 fig. 9. The text gives trends only; values are in the figure.

Download analysis fig. 1 §1. Open-source NLP models on HuggingFace are grouped into five parameter bins: [200M, 500M, 1B, 6B] as printed in the caption. The full bar is total monthly downloads per bin, and the highlighted segment is the most-downloaded model. Data were collected on 6 September 2026 fig. 1. Exact values are not stated in the text.

Cited headline numbers (not measured by this survey):

• DoReMi: a 280M proxy chooses domain weights for an 8B model, with a 6.5-point downstream accuracy gain and baseline accuracy reached 2.6× faster §3.1.1.
• LIMA: fine-tuning on 1,000 curated examples gives a well-aligned model §3.1.1.

Paper vs code

No official code. The abstract links a GitHub repository, tigerchen52/awesome_role_of_small_models, which the abstract calls "code and additional resources" [abstract]. It is a curated awesome-list of resources, not code for this survey's experiment, so it is not cloned. The paper's MTEB comparison has no linked code §4.1. The source contains no other code links.

Pitfalls

• Nothing to reimplement as one method. For cascading and draft-then-verify, the pseudocode above gives the shape only. Thresholds, router training, and the speculative acceptance rule come from the cited works §3.1.3.
• "Small" has no fixed cut-off. It is relative to a reference model, and the scope includes shallow networks and statistical models §1 §4. Do not use this card to set a parameter limit.
• Speculative decoding has a vocabulary constraint. Some methods need the draft and target models to share a vocabulary, which usually limits them to one model family §3.1.3, future directions.
• Routing overhead. Methods that run several models per query add inference cost. Training-free routers that use lightweight signals are cheaper but rely on heuristics §3.1.3.
• Small judges are fragile. Small evaluators can be sensitive to answer order and can carry bias §3.1.4 §5.
• Overlapping categories. Black-box domain adaptation and RAG are both listed under augmented reasoning §3.1.2, so one technique can appear in more than one place.
• Cited numbers are not re-measured. DoReMi and LIMA figures come from the cited papers. The survey gives no detail on how the cited experiments were set up.
• Own experiment is figure-only. The MTEB values and the model list are not in the text, so they cannot be reproduced from this card §4.1 fig. 9.
• Download analysis is thin. The text gives no method beyond the caption, so how models were selected and binned is not stated fig. 1.
• Version. Compiled from arXiv v9 (updated 2026-09-21), the version recorded by wikillm fetch.
• Numbering. Section and figure numbers are counted from the LaTeX source, not from the compiled PDF. Check them against the PDF before quoting.

Lineage

• Builds on: none; this is a survey.
• Compared against: the related surveys in §2. Lu et al. (2025) review 70 open-source small models of 100M–5B parameters; Wang et al. (2024) review small models broadly; a 2024 survey covers architectures, training and compression; Subramanian et al. (2025) examine about 160 papers on 1–8B models. The survey argues these treat small models in isolation, not in relation to LLMs §2.
• Superseded by: not stated.