---
kind: concept
slug: synthetic-training-data
title: Synthetic training data
summary: >-
  Trains a model on examples a generator makes instead of experts: LLM labels, rendered text lines, prior-drawn tables, or tool-use and finance tasks checked by a verifier. Author-reported: IAM CER 7.62 to 4.67, OfficeQA Pro 39.60 to 62.66. One domain per paper; the mixing theory covers Gaussians only.
tags: [data-generation, synthetic-data, few-shot, output-ensembling, pretraining, handwriting-recognition, tabular, tool-use, model-collapse]
papers: ["2110.07356", "2005.13044", "2510.06162", "2610.19560", "2610.26887", "2611.00259"]
related: [text-summarization, in-context-learning, few-shot-learning, synthetic-query-generation, data-augmentation, dataset-curation, knowledge-distillation, weakly-supervised-learning, optical-character-recognition, tabular-foundation-model, continual-learning, tool-use, user-simulation, model-collapse, reinforcement-learning-verifiable-rewards]
status: draft
updated_at: 2026-11-22
---

## Overview

Supervised training needs many labelled examples, and expert labels are slow and
costly to make. Synthetic training data covers part of that cost: a generator
produces examples, and a model is trained on them, usually with some real or
expert data as an anchor [arxiv:2110.07356 §1] [arxiv:2005.13044 §4.3]
[arxiv:2510.06162 § Methods] (A Prior for Synthetic HDLSS Data Generation).

The six papers on this page use four kinds of generator. A general language
model writes labels for text it is shown [arxiv:2110.07356 §4.1]. A renderer draws
text lines from a corpus and a list of fonts [arxiv:2005.13044 §4.3]. A procedural
prior makes tables, which are then widened and used to continue training a tabular
foundation model [arxiv:2510.06162 § Methods] (Continued Pre-Training). Two
pipelines build agent tasks: one generates tool-calling tasks and runs an agent on
them to get trajectories [arxiv:2610.19560 §3.2] [arxiv:2610.19560 §7.3], the other
builds finance questions from facts pulled out of reports [arxiv:2610.26887 §2.3].
A theory paper studies a Gaussian model whose refits mix real and generated samples
[arxiv:2611.00259 §3.1].

A recurring problem is quality. A general model's labels can miss facts or flip
meaning, so the pipeline has to check or select its output [arxiv:2110.07356 §4.1].
Three papers filter their generated data. The LLM-labelling paper samples several
labels per input and keeps the best, scored with a domain concept extractor
[arxiv:2110.07356 §4.2]. The tool-use paper keeps only rollouts whose final database
matches the reference and that the simulated user did not abandon [arxiv:2610.19560
§7.3]. The finance paper keeps a question only if an independent model answers it
correctly from the question and its supporting facts, and programmatic checks pass
[arxiv:2610.26887 §2.5]. The rendering and table-widening papers do not filter.
Real data enters later: the handwriting model is fine-tuned on real lines
[arxiv:2005.13044 §4.6], and the table model is evaluated on real omics and
benchmark datasets [arxiv:2510.06162 § Experiments] (Results on Real-World Wide
Datasets).

## Variants

### Few-shot LLM labeller with ensemble selection

The generator is primed with a fixed number of labelled pairs drawn at random from
a pool. It runs several trials with fresh prompts, and a selector keeps the output
whose domain concepts overlap the input's most [arxiv:2110.07356 §4.2]
[arxiv:2110.07356 alg. 1]. In the paper the prompt holds 21 pairs, and ten trials
use disjoint blocks of a 210-pair pool, so each trial sees different examples
[arxiv:2110.07356 §5] [arxiv:2110.07356 §7]. The resulting 6400 labels train
PEGASUS, which beats human-only PEGASUS on Negation F1 and Concept F1 but not on
ROUGE-L [arxiv:2110.07356 tab. 2]. Card: [GPT-3-ENS](/papers/2110.07356).

### Mixing synthetic and human labels

The same paper also trains with a mix of synthetic and human-written pairs. The
synthetic count is a multiple α of the human count, with α in {0.5, 1, 2, 3}
[arxiv:2110.07356 §7.2]. For PEGASUS every mix beat the human-only baseline on the
three reported metrics. For DR-SUM, α = 0.5 and α = 2 beat it on all three, α = 1
was lower on ROUGE-L, and α = 3 was lower on all three [arxiv:2110.07356 tab. 4].
Card: [GPT-3-ENS](/papers/2110.07356).

### Rendered text-line images for pre-training

Synthetic line images are drawn from over 130,000 corpus lines in 387 cursive
fonts, giving a pre-training set of 138,000 lines [arxiv:2005.13044 §4.3]. The
recogniser is trained on these first and then fine-tuned on real IAM lines
[arxiv:2005.13044 §4.6]. On IAM line level, the same model goes from 7.62 CER and
24.54 WER without synthetic data to 4.67 CER and 15.45 WER with it
[arxiv:2005.13044 tab. 5]. The gain is largest when real data is scarce: with 20%
of the real training set, the model reaches 73.81 CER without synthetic data and
6.51 with it [arxiv:2005.13044 tab. 3]. The authors' recurrent Seq2Seq baseline also
improves, from 11.91 to 10.64 CER [arxiv:2005.13044 tab. 5]. The ablations are
trained on IAM only, so their 4.86 CER is not the 4.67 headline [arxiv:2005.13044
§4.4]. The compiled results have no synthetic-only setting. Card:
[Pay Attention to What You Read](/papers/2005.13044).

### Widened synthetic tables for continued pre-training

A prior generates a table, and the table is widened to a target feature count.
Continuous columns become sparse random projections of the originals with noise
added. Categorical columns copy values from a few donor columns and are merged down
to a target number of categories [arxiv:2510.06162 alg. 1] [arxiv:2510.06162 alg. 2].
The TabPFNv2 classifier is then trained further, with all weights updated, on tables
whose target width is drawn from 200 up to 1,500, 5,000 or 8,000 features
[arxiv:2510.06162 § Methods] (Continued Pre-Training). The authors use the public
TabICL prior as a stand-in, because the TabPFNv2 prior is not public
[arxiv:2510.06162 § Methods] (A Prior for Synthetic HDLSS Data Generation).

On five TCGA omics datasets with 14,229 to 18,614 features, the 5k model's AUROC is
0.971, 0.954, 0.962, 0.988 and 0.981. TabPFNv2 gets 0.733, 0.710, 0.794, 0.973 and
0.836 [arxiv:2510.06162 tab. 3]. On 21 TabArena tasks, the per-task AUROCs of the 5k
model and TabPFNv2 have a Spearman correlation of 0.9935, which the authors read as
no sign of catastrophic forgetting [arxiv:2510.06162 § Experiments] (Results on
Standard Benchmarks and Widened Adaptations) [arxiv:2510.06162 fig. 4].

The same models expose attention from the label token to each feature, which the
authors use as a feature-importance score [arxiv:2510.06162 § Methods] (Feature-wise
Interpretability via Attention Maps). The paper's own limitations section says that
attention maps may mislead on real data and that the synthetic ground-truth test is
more reliable [arxiv:2510.06162 § Conclusion] (Limitations and Outlook). Card:
[TabPFN-Wide](/papers/2510.06162).

### Verified rollouts for tool-calling agents

The Tracer pipeline builds a tool graph for each of ten domains, composes random
walks of tool calls into traces, and grounds each trace against an executable
database to get a reference trajectory [arxiv:2610.19560 §3.2]. A candidate task
is kept only if its reference replays deterministically and it has a concrete
objective and no policy conflict [arxiv:2610.19560 §3.4]. Training rollouts use a
thinking-mode Qwen3.8-27B agent and a Qwen3-235B-A22B-Instruct simulated user, capped
at 50 steps. Of 7,048 kept rollouts, 2,723 succeed (38.6%), and a success needs the
end database to equal the reference and no unsatisfied token from the user
[arxiv:2610.19560 §7.3]. The Tracer-High set holds 1,958 of these
[arxiv:2610.19560 §7.3].

Qwen3-1.7B, 4B and 8B are fine-tuned on that set by supervised learning, for three
epochs at learning rate 5 × 10⁻⁵ [arxiv:2610.19560 §8.1]. The authors report Avg@3
gains of 13.8 to 19.7 points over the base models with the Qwen3-235B user
[arxiv:2610.19560 §4.2]. Out of domain, Tracer-Low at 8B reaches Pass^3 8.3%
against 5.0% for the base and 7.0% for EnvScaler, and its Pass^1 is 0.1 points below
EnvScaler's [arxiv:2610.19560 tab. 3]. In-domain Pass^3 at 8B is 24.7% against
10.5% for the base [arxiv:2610.19560 tab. 4], but the training and test tasks share
a database in those domains, so this does not test unseen data
[arxiv:2610.19560 §7.3]. Card: [τ-Emotion-Bench](/papers/2610.19560).

### Verified multi-step tasks from financial reports

Navalia builds questions from a fact database extracted from each report. It
samples an operator, such as an aggregation over periods or a regression, then
anchors such as companies, periods and metrics [arxiv:2610.26887 §2.2]. A reasoning
model selects the supporting facts from the candidate documents, and the question is
written only after the computation chain is evaluated [arxiv:2610.26887 §2.3]
[arxiv:2610.26887 §2.4]. Complex operators are computed by a fixed numerical routine,
not by the language model [arxiv:2610.26887 §2.4].

The verification is two-stage. An independent review model answers from the question
and the supporting facts alone, and its answer must match the reference. Then
programmatic checks require every non-final step to feed a later step and the periods
in the question to match the facts [arxiv:2610.26887 §2.5]. The verified tasks train
Qwen3.6-35B-A3B by supervised learning on ReAct trajectories with FIFO context, and
then by reinforcement learning, where all segments of a trajectory share the final
task reward [arxiv:2610.26887 §3.2] [arxiv:2610.26887 §3.3]. The authors report
OfficeQA Pro accuracy rising from 39.6 to 62.66 [arxiv:2610.26887 tab. 1]. The
teacher model and the review model are not named [arxiv:2610.26887 §3.2]
[arxiv:2610.26887 §2.5]. The reported training statistics are per task, not a total
task count [arxiv:2610.26887 §4.3]. Card: [Navalia](/papers/2610.26887).

### Iterated refits on pooled real and synthetic samples (theory)

This is not a recipe for generating a training set. It models a loop in which each
round draws new samples from a mix of the real distribution and the current fit,
pools them with all earlier samples, and refits without labels [arxiv:2611.00259
§3.1]. The data are a Gaussian, so the results are not claims about neural networks
[arxiv:2611.00259 §1].

For the mean and covariance, the estimate converges to the truth if and only if the
series of (1 − α_t)(1 − ρ_t) diverges, where α_t is the synthetic share and ρ_t the
pool ratio [arxiv:2611.00259 §3.2.2] [arxiv:2611.00259 eq. 12]. A fixed synthetic
share below one is enough [arxiv:2611.00259 eq. 16]. The schedules 1 − 1/t and
1 − 1/√t leave the error at a positive limit, while 1 − 1/ln(t+2) and
1 − 1/ln ln(t+16) keep it falling [arxiv:2611.00259 §3.2.2] [arxiv:2611.00259 fig. 1].
With all-synthetic batches, the error stays between Tr(Σ)/n_r and 2Tr(Σ)/n_r and
never decreases [arxiv:2611.00259 §3.2.2]. With a constant α, the mean error is of
order N^-1 for α below 1/2 and N^-2(1-α) above it [arxiv:2611.00259 §3.2.3]
[arxiv:2611.00259 eq. 17].

A CIFAR-10 diffusion check follows the same ordering, but it is a single training seed
[arxiv:2611.00259 app. D]. At round 9, the paper reports test FID of about 13.5 for
α = 0.25 and above 27 for α = 0.75, against 9.7 for a real-data reference
[arxiv:2611.00259 app. D]. Card: [Model Collapse Is Avoidable](/papers/2611.00259).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Expert labels are scarce, and the labelling step may send text to an outside model | Few-shot LLM labeller with a selector ([arxiv:2110.07356 §4.2]) | The 6400 synthetic labels came from 210 expert examples. Whether the privacy condition holds depends on which inputs are sent, which the paper does not settle [arxiv:2110.07356 §2] [arxiv:2110.07356 §4.3] |
| You already have some expert labels and want more | Mix synthetic and human labels ([arxiv:2110.07356 §7.2]) | Mixes beat human-only data in most configurations reported, but not at every ratio, so tune α on your own held-out set [arxiv:2110.07356 tab. 4] |
| You have no domain concept extractor | Do not copy the selector as written | The selector depends on an in-house UMLS-based recogniser that is not released. The paper says another recogniser with similar coverage can be used but tests none [arxiv:2110.07356 §4.2] |
| A small real labelled set, and you can render training images from a corpus and fonts | Synthetic pre-training, then fine-tuning on real data ([arxiv:2005.13044 §4.3]) | The gain was largest with 20% of the real data [arxiv:2005.13044 tab. 3]. The corpus and font licences are not stated, so check them before use [arxiv:2005.13044 §4.3] |
| Tables wider than the base model accepts, and you can afford continued training of a pretrained checkpoint | Continued pre-training on widened prior tables ([arxiv:2510.06162 § Methods] (Continued Pre-Training)) | A Wide variant beats TabPFNv2 on all five TCGA sets and has the best AUROC on four of them; RealMLP is best on the fifth [arxiv:2510.06162 tab. 3]. The paper's run used four H100 GPUs with 320 GB combined [arxiv:2510.06162 § Methods] (Continued Pre-Training) |
| You need per-feature importance from attention | Treat attention as a hypothesis | The authors say attention maps may mislead on real data [arxiv:2510.06162 § Conclusion] (Limitations and Outlook) |
| An agent must learn multi-turn tool use, and you can run the tools and check the final state | Verified rollouts, keeping only successes ([arxiv:2610.19560 §7.3]) | The success filter needs a checkable end state. The authors' in-domain gains share a database with training, so measure on held-out databases [arxiv:2610.19560 tab. 3] [arxiv:2610.19560 §7.3] |
| Your tasks need multi-step computation over documents, and you can run a second model as an independent checker | Two-stage verification of generated questions ([arxiv:2610.26887 §2.5]) | The checks catch unused steps and period mismatches. The teacher and reviewer are unnamed, and no code is released, so the pipeline cannot be reproduced exactly [arxiv:2610.26887 §3.2] [arxiv:2610.26887 §2.5] |
| You will keep refitting a model on its own output without labels | Keep a real share, and do not let it shrink fast ([arxiv:2611.00259 §3.2.2]) | The theory says a fixed synthetic share below one converges, and a shrinking real share converges only if it shrinks slowly. It covers Gaussians only, so test on your own data [arxiv:2611.00259 §1] |

The evidence spans medical dialogue [arxiv:2110.07356 §5], handwritten English lines
[arxiv:2005.13044 §4.1], TCGA omics with TabArena [arxiv:2510.06162 § Experiments]
(Datasets and Evaluation Protocol), ten tool-calling domains [arxiv:2610.19560 §4.1],
finance documents [arxiv:2610.26887 §4.1], and a Gaussian model with a CIFAR-10 check
[arxiv:2611.00259 app. D]. The LLM-labelling paper reports single runs without
variance [arxiv:2110.07356 §7], and the finance and tool-use papers also report
single values or three trials per task [arxiv:2610.26887 tab. 1] [arxiv:2610.19560
§4.1]. Treat the trade-offs above as hypotheses to test on your own data.

## Implementation sketch

Language-agnostic pseudocode of the common core of the LLM-labelling variant:
sample candidates from a primed generator, score each against the input, keep the
best. Names are ours; the paper's version is [arxiv:2110.07356 alg. 1].

```pseudocode
# Select the best synthetic label for one input  [arxiv:2110.07356 alg. 1]
# Inputs: x (unlabelled input), pool (expert pairs), K trials, N shots per prompt
def synth_label(x, pool, K, N, generate, concepts):
    # Reference concepts from the input itself  [arxiv:2110.07356 §4.2]
    ref = concepts(x)
    best, best_score = None, -1.0
    for _ in range(K):
        # Fresh shots per trial, no reuse  [arxiv:2110.07356 §5]
        shots = sample_without_replacement(pool, N)
        # Candidate label from the primed generator  [arxiv:2110.07356 §4.1]
        cand = generate(shots, x)
        # Share of reference concepts covered  [arxiv:2110.07356 §4.2]
        score = len(concepts(cand) & ref) / len(ref) if ref else 0.0
        if score > best_score:
            best, best_score = cand, score
    return best
```

Notes where variants differ: the paper's score divides by the reference concept
count, and it does not say how empty references or ties are handled
[arxiv:2110.07356 alg. 1]. The code above returns 0 for an empty reference, which
is our choice, not the paper's. The rendering variant has no selection step; it is
generate once, pre-train, then fine-tune. The table-widening variant's recipes are
in [arxiv:2510.06162 alg. 1] and [arxiv:2510.06162 alg. 2].

The two verifier variants share a shape: generate a candidate, run one or more
checks, and keep it only if every check passes. The tool-use check replays the
reference and compares the end state [arxiv:2610.19560 §3.4]. The finance check
asks a second model to answer blind, then runs structural checks
[arxiv:2610.26887 §2.5]. Neither paper's acceptance rule is a score threshold, so do
not copy a threshold from one into the other.

## Contradictions

No direct contradictions. Four points differ, and this page does not resolve them.

- **Synthetic alone versus synthetic then real.** The LLM-labelling paper reports
  synthetic labels alone, which matched human labels on two of three PEGASUS
  metrics [arxiv:2110.07356 tab. 2]. The rendering paper reports synthetic
  pre-training followed by real fine-tuning and no synthetic-only setting
  [arxiv:2005.13044 tab. 5]. The tasks differ, so the numbers cannot be compared.
- **Synthetic share in theory versus practice.** The Gaussian theory says any fixed
  synthetic share below one keeps the estimate consistent [arxiv:2611.00259 §3.2.2].
  The LLM-labelling paper finds that α = 3 synthetic-heavy mixing is below
  human-only on all three DR-SUM metrics [arxiv:2110.07356 tab. 4]. The settings
  differ: one is an infinite-round limit on a Gaussian, the other a single finite
  fine-tuning run on summarisation. The page does not reconcile them.
- **Paper versus released code, table widening.** The paper gives 10,000 continued
  pre-training steps [arxiv:2510.06162 § Methods] (Continued Pre-Training). The
  released training script sets 30,000, and it differs on other settings too. See
  the card's Paper vs code section: [TabPFN-Wide](/papers/2510.06162).
- **Feature-count convention.** The paper's target dimension d is a total width,
  while the code samples a count of added features [arxiv:2510.06162 § Methods]
  (A Prior for Synthetic HDLSS Data Generation). Check which one a reimplementation
  uses.

## Papers

- [GPT-3-ENS](/papers/2110.07356): primes GPT-3 with expert pairs, selects labels by medical-concept coverage, and trains PEGASUS and DR-SUM on the result, alone or mixed with human labels.
- [Pay Attention to What You Read](/papers/2005.13044): pre-trains a non-recurrent handwritten-line recogniser on rendered synthetic lines, then fine-tunes it on IAM.
- [TabPFN-Wide](/papers/2510.06162): continues TabPFNv2 training on prior tables widened to thousands of features, and reads attention as feature scores.
- [τ-Emotion-Bench](/papers/2610.19560): generates tool-calling tasks and agent rollouts, keeps the successful ones, and fine-tunes Qwen3 1.7B to 8B on them.
- [Navalia](/papers/2610.26887): builds finance questions from extracted facts, verifies them with a blind second answer and programmatic checks, then trains an agent by SFT and RL.
- [Model Collapse Is Avoidable](/papers/2611.00259): proves when iterated refits on pooled real and synthetic Gaussian samples converge, with a CIFAR-10 diffusion check.
