Concept · 6 papers · updated 2026-11-22
Synthetic training data
Trains a model on examples a generator makes instead of experts: LLM labels, rendered text lines, prior-drawn tables, or tool-use and finance tasks checked by a verifier. Author-reported: IAM CER 7.62 to 4.67, OfficeQA Pro 39.60 to 62.66. One domain per paper; the mixing theory covers Gaussians only.
data-generationsynthetic-datafew-shotoutput-ensemblingpretraininghandwriting-recognitiontabulartool-usemodel-collapse
Markdown12 min read
AI-compiled overview, updated 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
Supervised training needs many labelled examples, and expert labels are slow and costly to make. Synthetic training data covers part of that cost: a generator produces examples, and a model is trained on them, usually with some real or expert data as an anchor 2110.07356 §1 2005.13044 §4.3 2510.06162 § Methods (A Prior for Synthetic HDLSS Data Generation).
The six papers on this page use four kinds of generator. A general language model writes labels for text it is shown 2110.07356 §4.1. A renderer draws text lines from a corpus and a list of fonts 2005.13044 §4.3. A procedural prior makes tables, which are then widened and used to continue training a tabular foundation model 2510.06162 § Methods (Continued Pre-Training). Two pipelines build agent tasks: one generates tool-calling tasks and runs an agent on them to get trajectories 2610.19560 §3.2 2610.19560 §7.3, the other builds finance questions from facts pulled out of reports 2610.26887 §2.3. A theory paper studies a Gaussian model whose refits mix real and generated samples 2611.00259 §3.1.
A recurring problem is quality. A general model's labels can miss facts or flip meaning, so the pipeline has to check or select its output 2110.07356 §4.1. Three papers filter their generated data. The LLM-labelling paper samples several labels per input and keeps the best, scored with a domain concept extractor 2110.07356 §4.2. The tool-use paper keeps only rollouts whose final database matches the reference and that the simulated user did not abandon [arxiv:2610.19560 §7.3]. The finance paper keeps a question only if an independent model answers it correctly from the question and its supporting facts, and programmatic checks pass 2610.26887 §2.5. The rendering and table-widening papers do not filter. Real data enters later: the handwriting model is fine-tuned on real lines 2005.13044 §4.6, and the table model is evaluated on real omics and benchmark datasets 2510.06162 § Experiments (Results on Real-World Wide Datasets).
Variants
Few-shot LLM labeller with ensemble selection
The generator is primed with a fixed number of labelled pairs drawn at random from a pool. It runs several trials with fresh prompts, and a selector keeps the output whose domain concepts overlap the input's most 2110.07356 §4.2 2110.07356 alg. 1. In the paper the prompt holds 21 pairs, and ten trials use disjoint blocks of a 210-pair pool, so each trial sees different examples 2110.07356 §5 2110.07356 §7. The resulting 6400 labels train PEGASUS, which beats human-only PEGASUS on Negation F1 and Concept F1 but not on ROUGE-L 2110.07356 tab. 2. Card: GPT-3-ENS.
Mixing synthetic and human labels
The same paper also trains with a mix of synthetic and human-written pairs. The synthetic count is a multiple α of the human count, with α in {0.5, 1, 2, 3} 2110.07356 §7.2. For PEGASUS every mix beat the human-only baseline on the three reported metrics. For DR-SUM, α = 0.5 and α = 2 beat it on all three, α = 1 was lower on ROUGE-L, and α = 3 was lower on all three 2110.07356 tab. 4. Card: GPT-3-ENS.
Rendered text-line images for pre-training
Synthetic line images are drawn from over 130,000 corpus lines in 387 cursive fonts, giving a pre-training set of 138,000 lines 2005.13044 §4.3. The recogniser is trained on these first and then fine-tuned on real IAM lines 2005.13044 §4.6. On IAM line level, the same model goes from 7.62 CER and 24.54 WER without synthetic data to 4.67 CER and 15.45 WER with it 2005.13044 tab. 5. The gain is largest when real data is scarce: with 20% of the real training set, the model reaches 73.81 CER without synthetic data and 6.51 with it 2005.13044 tab. 3. The authors' recurrent Seq2Seq baseline also improves, from 11.91 to 10.64 CER 2005.13044 tab. 5. The ablations are trained on IAM only, so their 4.86 CER is not the 4.67 headline [arxiv:2005.13044 §4.4]. The compiled results have no synthetic-only setting. Card: Pay Attention to What You Read.
Widened synthetic tables for continued pre-training
A prior generates a table, and the table is widened to a target feature count. Continuous columns become sparse random projections of the originals with noise added. Categorical columns copy values from a few donor columns and are merged down to a target number of categories 2510.06162 alg. 1 2510.06162 alg. 2. The TabPFNv2 classifier is then trained further, with all weights updated, on tables whose target width is drawn from 200 up to 1,500, 5,000 or 8,000 features 2510.06162 § Methods (Continued Pre-Training). The authors use the public TabICL prior as a stand-in, because the TabPFNv2 prior is not public 2510.06162 § Methods (A Prior for Synthetic HDLSS Data Generation).
On five TCGA omics datasets with 14,229 to 18,614 features, the 5k model's AUROC is 0.971, 0.954, 0.962, 0.988 and 0.981. TabPFNv2 gets 0.733, 0.710, 0.794, 0.973 and 0.836 2510.06162 tab. 3. On 21 TabArena tasks, the per-task AUROCs of the 5k model and TabPFNv2 have a Spearman correlation of 0.9935, which the authors read as no sign of catastrophic forgetting 2510.06162 § Experiments (Results on Standard Benchmarks and Widened Adaptations) 2510.06162 fig. 4.
The same models expose attention from the label token to each feature, which the authors use as a feature-importance score 2510.06162 § Methods (Feature-wise Interpretability via Attention Maps). The paper's own limitations section says that attention maps may mislead on real data and that the synthetic ground-truth test is more reliable 2510.06162 § Conclusion (Limitations and Outlook). Card: TabPFN-Wide.
Verified rollouts for tool-calling agents
The Tracer pipeline builds a tool graph for each of ten domains, composes random walks of tool calls into traces, and grounds each trace against an executable database to get a reference trajectory 2610.19560 §3.2. A candidate task is kept only if its reference replays deterministically and it has a concrete objective and no policy conflict 2610.19560 §3.4. Training rollouts use a thinking-mode Qwen3.8-27B agent and a Qwen3-235B-A22B-Instruct simulated user, capped at 50 steps. Of 7,048 kept rollouts, 2,723 succeed (38.6%), and a success needs the end database to equal the reference and no unsatisfied token from the user 2610.19560 §7.3. The Tracer-High set holds 1,958 of these 2610.19560 §7.3.
Qwen3-1.7B, 4B and 8B are fine-tuned on that set by supervised learning, for three epochs at learning rate 5 × 10⁻⁵ 2610.19560 §8.1. The authors report Avg@3 gains of 13.8 to 19.7 points over the base models with the Qwen3-235B user 2610.19560 §4.2. Out of domain, Tracer-Low at 8B reaches Pass^3 8.3% against 5.0% for the base and 7.0% for EnvScaler, and its Pass^1 is 0.1 points below EnvScaler's 2610.19560 tab. 3. In-domain Pass^3 at 8B is 24.7% against 10.5% for the base 2610.19560 tab. 4, but the training and test tasks share a database in those domains, so this does not test unseen data 2610.19560 §7.3. Card: τ-Emotion-Bench.
Verified multi-step tasks from financial reports
Navalia builds questions from a fact database extracted from each report. It samples an operator, such as an aggregation over periods or a regression, then anchors such as companies, periods and metrics 2610.26887 §2.2. A reasoning model selects the supporting facts from the candidate documents, and the question is written only after the computation chain is evaluated 2610.26887 §2.3 2610.26887 §2.4. Complex operators are computed by a fixed numerical routine, not by the language model 2610.26887 §2.4.
The verification is two-stage. An independent review model answers from the question and the supporting facts alone, and its answer must match the reference. Then programmatic checks require every non-final step to feed a later step and the periods in the question to match the facts 2610.26887 §2.5. The verified tasks train Qwen3.6-35B-A3B by supervised learning on ReAct trajectories with FIFO context, and then by reinforcement learning, where all segments of a trajectory share the final task reward 2610.26887 §3.2 2610.26887 §3.3. The authors report OfficeQA Pro accuracy rising from 39.6 to 62.66 2610.26887 tab. 1. The teacher model and the review model are not named 2610.26887 §3.2 2610.26887 §2.5. The reported training statistics are per task, not a total task count 2610.26887 §4.3. Card: Navalia.
Iterated refits on pooled real and synthetic samples (theory)
This is not a recipe for generating a training set. It models a loop in which each round draws new samples from a mix of the real distribution and the current fit, pools them with all earlier samples, and refits without labels [arxiv:2611.00259 §3.1]. The data are a Gaussian, so the results are not claims about neural networks 2611.00259 §1.
For the mean and covariance, the estimate converges to the truth if and only if the series of (1 − α_t)(1 − ρ_t) diverges, where α_t is the synthetic share and ρ_t the pool ratio 2611.00259 §3.2.2 2611.00259 eq. 12. A fixed synthetic share below one is enough 2611.00259 eq. 16. The schedules 1 − 1/t and 1 − 1/√t leave the error at a positive limit, while 1 − 1/ln(t+2) and 1 − 1/ln ln(t+16) keep it falling 2611.00259 §3.2.2 2611.00259 fig. 1. With all-synthetic batches, the error stays between Tr(Σ)/n_r and 2Tr(Σ)/n_r and never decreases 2611.00259 §3.2.2. With a constant α, the mean error is of order N^-1 for α below 1/2 and N^-2(1-α) above it 2611.00259 §3.2.3 2611.00259 eq. 17.
A CIFAR-10 diffusion check follows the same ordering, but it is a single training seed 2611.00259 app. D. At round 9, the paper reports test FID of about 13.5 for α = 0.25 and above 27 for α = 0.75, against 9.7 for a real-data reference 2611.00259 app. D. Card: Model Collapse Is Avoidable.
When to use which
| Situation | Use | Why |
|---|---|---|
| Expert labels are scarce, and the labelling step may send text to an outside model | Few-shot LLM labeller with a selector (2110.07356 §4.2) | The 6400 synthetic labels came from 210 expert examples. Whether the privacy condition holds depends on which inputs are sent, which the paper does not settle 2110.07356 §2 2110.07356 §4.3 |
| You already have some expert labels and want more | Mix synthetic and human labels (2110.07356 §7.2) | Mixes beat human-only data in most configurations reported, but not at every ratio, so tune α on your own held-out set 2110.07356 tab. 4 |
| You have no domain concept extractor | Do not copy the selector as written | The selector depends on an in-house UMLS-based recogniser that is not released. The paper says another recogniser with similar coverage can be used but tests none 2110.07356 §4.2 |
| A small real labelled set, and you can render training images from a corpus and fonts | Synthetic pre-training, then fine-tuning on real data (2005.13044 §4.3) | The gain was largest with 20% of the real data 2005.13044 tab. 3. The corpus and font licences are not stated, so check them before use 2005.13044 §4.3 |
| Tables wider than the base model accepts, and you can afford continued training of a pretrained checkpoint | Continued pre-training on widened prior tables (2510.06162 § Methods (Continued Pre-Training)) | A Wide variant beats TabPFNv2 on all five TCGA sets and has the best AUROC on four of them; RealMLP is best on the fifth 2510.06162 tab. 3. The paper's run used four H100 GPUs with 320 GB combined 2510.06162 § Methods (Continued Pre-Training) |
| You need per-feature importance from attention | Treat attention as a hypothesis | The authors say attention maps may mislead on real data 2510.06162 § Conclusion (Limitations and Outlook) |
| An agent must learn multi-turn tool use, and you can run the tools and check the final state | Verified rollouts, keeping only successes (2610.19560 §7.3) | The success filter needs a checkable end state. The authors' in-domain gains share a database with training, so measure on held-out databases 2610.19560 tab. 3 2610.19560 §7.3 |
| Your tasks need multi-step computation over documents, and you can run a second model as an independent checker | Two-stage verification of generated questions (2610.26887 §2.5) | The checks catch unused steps and period mismatches. The teacher and reviewer are unnamed, and no code is released, so the pipeline cannot be reproduced exactly 2610.26887 §3.2 2610.26887 §2.5 |
| You will keep refitting a model on its own output without labels | Keep a real share, and do not let it shrink fast (2611.00259 §3.2.2) | The theory says a fixed synthetic share below one converges, and a shrinking real share converges only if it shrinks slowly. It covers Gaussians only, so test on your own data 2611.00259 §1 |
The evidence spans medical dialogue 2110.07356 §5, handwritten English lines 2005.13044 §4.1, TCGA omics with TabArena 2510.06162 § Experiments (Datasets and Evaluation Protocol), ten tool-calling domains 2610.19560 §4.1, finance documents 2610.26887 §4.1, and a Gaussian model with a CIFAR-10 check 2611.00259 app. D. The LLM-labelling paper reports single runs without variance 2110.07356 §7, and the finance and tool-use papers also report single values or three trials per task 2610.26887 tab. 1 [arxiv:2610.19560 §4.1]. Treat the trade-offs above as hypotheses to test on your own data.
Implementation sketch
Language-agnostic pseudocode of the common core of the LLM-labelling variant: sample candidates from a primed generator, score each against the input, keep the best. Names are ours; the paper's version is 2110.07356 alg. 1.
# Select the best synthetic label for one input [arxiv:2110.07356 alg. 1]
# Inputs: x (unlabelled input), pool (expert pairs), K trials, N shots per prompt
def synth_label(x, pool, K, N, generate, concepts):
# Reference concepts from the input itself [arxiv:2110.07356 §4.2]
ref = concepts(x)
best, best_score = None, -1.0
for _ in range(K):
# Fresh shots per trial, no reuse [arxiv:2110.07356 §5]
shots = sample_without_replacement(pool, N)
# Candidate label from the primed generator [arxiv:2110.07356 §4.1]
cand = generate(shots, x)
# Share of reference concepts covered [arxiv:2110.07356 §4.2]
score = len(concepts(cand) & ref) / len(ref) if ref else 0.0
if score > best_score:
best, best_score = cand, score
return bestNotes where variants differ: the paper's score divides by the reference concept count, and it does not say how empty references or ties are handled 2110.07356 alg. 1. The code above returns 0 for an empty reference, which is our choice, not the paper's. The rendering variant has no selection step; it is generate once, pre-train, then fine-tune. The table-widening variant's recipes are in 2510.06162 alg. 1 and 2510.06162 alg. 2.
The two verifier variants share a shape: generate a candidate, run one or more checks, and keep it only if every check passes. The tool-use check replays the reference and compares the end state 2610.19560 §3.4. The finance check asks a second model to answer blind, then runs structural checks 2610.26887 §2.5. Neither paper's acceptance rule is a score threshold, so do not copy a threshold from one into the other.
Contradictions
No direct contradictions. Four points differ, and this page does not resolve them.
- Synthetic alone versus synthetic then real. The LLM-labelling paper reports synthetic labels alone, which matched human labels on two of three PEGASUS metrics 2110.07356 tab. 2. The rendering paper reports synthetic pre-training followed by real fine-tuning and no synthetic-only setting 2005.13044 tab. 5. The tasks differ, so the numbers cannot be compared.
- Synthetic share in theory versus practice. The Gaussian theory says any fixed synthetic share below one keeps the estimate consistent 2611.00259 §3.2.2. The LLM-labelling paper finds that α = 3 synthetic-heavy mixing is below human-only on all three DR-SUM metrics 2110.07356 tab. 4. The settings differ: one is an infinite-round limit on a Gaussian, the other a single finite fine-tuning run on summarisation. The page does not reconcile them.
- Paper versus released code, table widening. The paper gives 10,000 continued pre-training steps 2510.06162 § Methods (Continued Pre-Training). The released training script sets 30,000, and it differs on other settings too. See the card's Paper vs code section: TabPFN-Wide.
- Feature-count convention. The paper's target dimension d is a total width, while the code samples a count of added features 2510.06162 § Methods (A Prior for Synthetic HDLSS Data Generation). Check which one a reimplementation uses.
Papers
- GPT-3-ENS: primes GPT-3 with expert pairs, selects labels by medical-concept coverage, and trains PEGASUS and DR-SUM on the result, alone or mixed with human labels.
- Pay Attention to What You Read: pre-trains a non-recurrent handwritten-line recogniser on rendered synthetic lines, then fine-tunes it on IAM.
- TabPFN-Wide: continues TabPFNv2 training on prior tables widened to thousands of features, and reads attention as feature scores.
- τ-Emotion-Bench: generates tool-calling tasks and agent rollouts, keeps the successful ones, and fine-tunes Qwen3 1.7B to 8B on them.
- Navalia: builds finance questions from extracted facts, verifies them with a blind second answer and programmatic checks, then trains an agent by SFT and RL.
- Model Collapse Is Avoidable: proves when iterated refits on pooled real and synthetic Gaussian samples converge, with a CIFAR-10 diffusion check.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.